{"benchmarks":[{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":39,"github_stars_30d":19,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"claire_record_id","repository":"gh:sierra-research/hyper-tau-bench","source_record":"bm_tau-tau-bench-an-environment-for-end-to-en_3d823aef"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark where a developer agent builds a complete customer-service agent from business records, client requirements, a production API, inherited codebase, and cost/model limits, scored by deployment against held-out simulated users across 53 tasks in four domains.","descriptions":{"en":"A benchmark where a developer agent builds a complete customer-service agent from business records, client requirements, a production API, inherited codebase, and cost/model limits, scored by deployment against held-out simulated users across 53 tasks in four domains."},"editorial_guide":{"id":"hyper-tau-bench","record_keys":["claire-radar:2609.04611"],"reviewed_at":"2026-10-25","scope":{"en":"Original September 2026 paper and the authors’ construction environment. This is different from τ-bench and τ²-bench.","zh":"依据 2026 年 9 月原论文与作者项目；本评测不同于 τ-bench 和 τ²-bench。"},"sections":[{"id":"tasks","source_ids":["paper","code"],"text":{"en":"Business records, a simulated client, inherited code and a production API form the construction kit. Requirements must be recovered from evidence and conversation rather than handed over as a complete specification.","zh":"开发者面对业务资料、模拟客户、已有代码与生产 API；需求需要通过阅读证据和访谈恢复，而非直接获得完整说明。"},"title":{"en":"What the developer receives","zh":"开发者拿到什么"}},{"id":"grading","source_ids":["paper"],"text":{"en":"The deliverable is an executable customer-service agent. Hidden simulated customer conversations evaluate its behavior after construction. Build time, build cost and serving cost are separate measurements.","zh":"交付物是可执行的客服 Agent，构建后以隐藏的模拟客户对话评估行为。构建耗时、构建费用与服务费用属于不同测量项。"},"title":{"en":"How submissions are evaluated","zh":"怎样评估交付物"}},{"id":"scope","source_ids":["paper"],"text":{"en":"The release has 53 construction tasks: six airline, six retail, six telecom and 35 banking tasks. Overall scores average tasks, so banking has the greatest weight.","zh":"发布题集共 53 道构建任务：航空、零售、通信各 6 道，银行 35 道。总成绩按任务平均，因此银行任务占比最大。"},"title":{"en":"Task coverage","zh":"任务覆盖"}},{"id":"findings","source_ids":["paper"],"text":{"en":"Overall pass rates: GPT-5.6-sol / Codex 22.0%; GPT-5.6-terra / Codex 18.0%; Claude Opus 5 / Claude Code 23.9%; Claude Sonnet 5 / Claude Code 14.9%; Kimi K3 / Kimi Code 16.1%; Kimi K3 / OpenCode 17.9%. Expert-authored reference agents reach 82.2%. These are paper results, not independent reruns.","zh":"总体通过率：GPT-5.6-sol / Codex 22.0%；GPT-5.6-terra / Codex 18.0%；Claude Opus 5 / Claude Code 23.9%；Claude Sonnet 5 / Claude Code 14.9%；Kimi K3 / Kimi Code 16.1%；Kimi K3 / OpenCode 17.9%。专家编写的参考 Agent 为 82.2%。以上来自论文，不是本站复测。"},"title":{"en":"Published results — original paper Table 2","zh":"论文报告结果：表 2"}},{"id":"caveats","source_ids":["paper"],"text":{"en":"A score belongs to a developer model plus its harness and budget. The expert reference is a ceiling, not another developer model. Simulated clients and domain imbalance limit generalization to live deployments.","zh":"成绩属于开发模型、运行框架与预算的组合。专家参考实现代表参考上限，不是另一个开发模型。模拟客户与领域占比差异限制了结论对真实部署的外推。"},"title":{"en":"How to interpret the comparison","zh":"怎样理解比较"}},{"id":"reproduce","source_ids":["code"],"text":{"en":"Use the authors’ repository for construction kits and the evaluation runtime. Construction runs require Docker and configured model access. Inspect a task and its serving constraints before comparing a new run with the paper.","zh":"可从作者仓库查看构建资料包与评估运行环境。构建需要 Docker 及模型访问配置；比较新实验与论文前，应先核对具体任务和服务限制。"},"title":{"en":"Inspect or reproduce the task","zh":"查看或复现任务"}},{"id":"published-table","source_ids":["paper"],"table":{"caption":{"en":"Original paper Table 2 · 53 construction tasks","zh":"原论文表 2 · 53 道构建任务"},"columns":[{"en":"Harness","zh":"运行框架"},{"en":"Developer / reference","zh":"开发模型／参考实现"},{"en":"Overall %","zh":"总体 %"},{"en":"Airline % (6)","zh":"航空 %（6）"},{"en":"Retail % (6)","zh":"零售 %（6）"},{"en":"Telecom % (6)","zh":"通信 %（6）"},{"en":"Banking % (35)","zh":"银行 %（35）"},{"en":"Build minutes","zh":"构建分钟数"},{"en":"Build USD","zh":"构建费用（美元）"},{"en":"Serving / budget","zh":"服务积分／预算"}],"note":{"en":"Overall averages the 53 tasks equally, rather than the four domains equally. The expert reference uses ground-truth access. Serving spend is a credit-budget ratio, not USD; missing reference build values stay unreported.","zh":"总体成绩对 53 道任务等权平均，并非四个领域等权。专家参考实现可访问真值。服务开销为积分预算比例，不是美元费用；未报告的参考构建数据保留为空。"},"rows":[["Codex","GPT-5.6-sol (xhigh)","22.0","49.8","59.2","32.9","9.0","47.9","18.2","0.45"],["Codex","GPT-5.6-terra (xhigh)","18.0","44.5","46.6","27.2","6.9","30.0","7.0","0.38"],["Claude Code","Claude Opus 5 (max)","23.9","55.9","72.8","48.2","5.9","216.3","42.0","0.58"],["Claude Code","Claude Sonnet 5 (max)","14.9","41.9","50.2","21.1","3.2","235.5","30.0","0.76"],["Kimi Code","Kimi K3 (max)","16.1","42.3","49.1","24.9","4.4","205.7","13.5","0.72"],["OpenCode","Kimi K3 (max)","17.9","43.8","52.1","27.6","6.0","360.3","15.1","0.61"],["—","Expert-authored reference","82.2","82.3","84.0","93.8","79.8","—","—","0.96"]]},"text":{"en":"Each row describes a developer model and its harness. The expert reference is a separately authored agent, not another developer model.","zh":"每行对应开发模型及其运行框架；专家参考实现为单独编写的 Agent，不是另一个开发模型。"},"title":{"en":"Published construction and deployment results","zh":"论文中的构建与部署成绩"}}],"sources":[{"id":"paper","title":"Original paper: methods and Table 2","url":"https://arxiv.org/html/2609.04611v1"},{"id":"code","title":"Authors’ executable environment and task definitions","url":"https://github.com/sierra-research/hyper-tau-bench"}],"summary":{"en":"Can a coding agent build a customer-service agent that works in deployment? Hyper-τ tests requirement discovery, implementation and operation under serving constraints.","zh":"编程 Agent 能否交付真正可运行的客服 Agent？Hyper-τ 同时考察需求发现、工程实现与受约束的实际服务表现。"},"title":"Hyper-τ-Bench (τ^τ-Bench)"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.04611","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","tool-use"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_tau-tau-bench-an-environment-for-end-to-en_3d823aef"},"modality":null,"name":"$\\tau^\\tau$-Bench: An Environment for End-To-End, Realistic Agent Construction","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04611","source_url":"https://arxiv.org/abs/2609.04611"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04611","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04611","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Field-effect transistor (FET) layouts on exfoliated two-dimensional flakes are typically drawn by hand for each flake, placing contacts and gates to match its position and outline in optical micrographs.","descriptions":{"en":"Field-effect transistor (FET) layouts on exfoliated two-dimensional flakes are typically drawn by hand for each flake, placing contacts and gates to match its position and outline in optical micrographs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.07423","languages":[],"library_direction":{"basis":"model-review-daily","categories":["engineering-design","visual-reasoning"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_2d-fet-bench_ad7f9e5f"},"modality":null,"name":"2d-fet-bench","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.07423","source_url":"https://arxiv.org/abs/2610.07423"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-07423","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07423","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":30,"github_stars_30d":5,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:360mm-team/360cityarena","source_record":"bm_360cityarena_2a62b93d"},"categories":["Robotics & Embodied AI","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"360CityArena evaluates embodied agents in a photorealistic virtual urban environment built from 360-degree video of Tokyo's Akihabara district. It includes 175 tasks across environment understanding, path reasoning, and spatial reasoning, testing localization, landmark search, path planning, and relational spatial reasoning. Scoring uses mean evaluation score with varied success criteria.","descriptions":{"en":"360CityArena evaluates embodied agents in a photorealistic virtual urban environment built from 360-degree video of Tokyo's Akihabara district. It includes 175 tasks across environment understanding, path reasoning, and spatial reasoning, testing localization, landmark search, path planning, and relational spatial reasoning. Scoring uses mean evaluation score with varied success criteria."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.08814","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","spatial-3d"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_360cityarena_2a62b93d"},"modality":null,"name":"360CityArena","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08814","source_url":"https://arxiv.org/abs/2608.08814"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08814","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08814","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":104,"github_stars_30d":7,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:gaoypeng/3dcodebench","source_record":"bm_3dcodebench_14ee1d5d"},"categories":["General AI","Geometric reasoning","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"3DCodeBench evaluates vision-language model agents on procedural 3D modeling by converting text and image references into Blender Python code. It includes 212 object categories with ground-truth scripts, and scores outputs on executability, image similarity (SigLIP-2/DINOv3), 3D shape distance (Chamfer/Uni3D), and LLM-as-judge, plus a human-preference ranking platform.","descriptions":{"en":"3DCodeBench evaluates vision-language model agents on procedural 3D modeling by converting text and image references into Blender Python code. It includes 212 object categories with ground-truth scripts, and scores outputs on executability, image similarity (SigLIP-2/DINOv3), 3D shape distance (Chamfer/Uni3D), and LLM-as-judge, plus a human-preference ranking platform."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01057","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","code-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_3dcodebench_14ee1d5d"},"modality":null,"name":"3DCodeBench","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01057","source_url":"https://arxiv.org/abs/2606.01057"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01057","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01057","topics":["coding","multimodal"],"unit":null},{"aliases":["4DSynth: Controllable Procedural World Synthesis for Dynamic Embodied Simulation"],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates embodied agents on interactive navigation tasks in procedurally generated 4D environments with independently tunable difficulty axes.","descriptions":{"en":"Evaluates embodied agents on interactive navigation tasks in procedurally generated 4D environments with independently tunable difficulty axes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.26947","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_4dsynth-controllable-procedural-world-synt_fa98bfb4"},"modality":null,"name":"4DSynth-Nav","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26947","source_url":"https://arxiv.org/abs/2608.26947"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26947","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26947","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluation object not clearly defined; abstract mentions alignment of Romanized transliteration with dialectal text, Standard Bangla, English, and subjectivity labels across five Bangla regional varieties, but no explicit task or scoring setup is provided.","descriptions":{"en":"Evaluation object not clearly defined; abstract mentions alignment of Romanized transliteration with dialectal text, Standard Bangla, English, and subjectivity labels across five Bangla regional varieties, but no explicit task or scoring setup is provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09964","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_5-dialects-bn_6e119279"},"modality":null,"name":"5-Dialects-BN","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09964","source_url":"https://arxiv.org/abs/2609.09964"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09964","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09964","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_a-large-scale-dataset-and-benchmark_80e45544"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"InteractBind provides a large-scale dataset of ~100k protein-ligand pairs with fine-grained binding-site localization tasks and interaction maps for six non-covalent interaction types, plus affinity and similarity-controlled splits.","descriptions":{"en":"InteractBind provides a large-scale dataset of ~100k protein-ligand pairs with fine-grained binding-site localization tasks and interaction maps for six non-covalent interaction types, plus affinity and similarity-controlled splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24045","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_a-large-scale-dataset-and-benchmark_80e45544"},"modality":null,"name":"A Large-Scale Dataset and Benchmark","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24045","source_url":"https://arxiv.org/abs/2605.24045"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24045","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24045","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["CAD","General AI","Geometric reasoning","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"CAD segmentation datasets are predominantly synthetic or automatically labelled, leaving it unclear how well existing methods transfer to real-world engineering tasks.","descriptions":{"en":"CAD segmentation datasets are predominantly synthetic or automatically labelled, leaving it unclear how well existing methods transfer to real-world engineering tasks."},"editorial_guide":{"id":"real-world-cad-segmentation","record_keys":["claire-radar:2610.13466"],"reviewed_at":"2026-10-25","scope":{"en":"Introducing paper, arXiv v1; engineering-data and MFCAD++ experiments are distinct.","zh":"依据首发论文 arXiv v1；工程数据实验与 MFCAD++ 迁移实验分别呈现。"},"sections":[{"id":"dataset","source_ids":["paper"],"text":{"en":"Human labels distinguish hole from non-hole B-Rep faces. The dataset contains 2,202 parts and 6,864 holes; 14,471 of 197,003 faces are hole faces.","zh":"人工将 B-Rep 面标为孔或非孔。数据包含 2,202 个零件、6,864 个孔；197,003 个面中有 14,471 个孔面。"},"title":{"en":"What is annotated?","zh":"标注了什么？"}},{"id":"methods","source_ids":["paper"],"text":{"en":"GWS searches boundary loops; MVR combines rendered views and topology. UV-Net and AAGNet learn face labels from B-Rep geometry.","zh":"GWS 搜索边界闭环；MVR 结合多视图渲染与拓扑。UV-Net、AAGNet 从 B-Rep 几何学习面标签。"},"title":{"en":"Rules and learned geometry","zh":"规则方法与几何学习"}},{"id":"scoring","source_ids":["paper"],"text":{"en":"Connected predicted faces form hole candidates. Hole matching uses surface-area-weighted IoU ≥ 0.99; face-level metrics measure a different unit.","zh":"相邻预测面聚为候选孔，孔匹配要求表面积加权 IoU ≥ 0.99；面级指标使用不同的评测单位。"},"title":{"en":"Detecting a complete hole","zh":"如何算完整识别一个孔？"}},{"id":"transfer","source_ids":["paper"],"text":{"en":"The study separately tests 200 selected MFCAD++ parts. Strong in-domain segmentation does not establish robustness to different CAD modelling conventions.","zh":"研究另测了筛选后的 200 个 MFCAD++ 零件。同域分割表现好，并不说明能适应不同 CAD 建模习惯。"},"title":{"en":"Read both test settings","zh":"两种测试要一起看"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Multi-solid parts were excluded. Functional labels can omit geometrically valid holes, so some detections count as false positives despite finding real cavities.","zh":"数据排除了多实体零件。功能性标注可能遗漏几何上成立的孔，因此找到真实孔洞也可能被计为误报。"},"title":{"en":"Annotation and coverage limits","zh":"标注与覆盖边界"}},{"id":"results-in-domain","source_ids":["paper"],"table":{"caption":{"en":"Paper Table 1 · training and testing on the authors’ dataset","zh":"论文表 1 · 在作者数据集上训练与测试"},"columns":[{"en":"Method","zh":"方法"},{"en":"Hole recall","zh":"孔级召回率"},{"en":"Hole precision","zh":"孔级精确率"},{"en":"Hole F1","zh":"孔级 F1"},{"en":"Face recall","zh":"面级召回率"},{"en":"Face precision","zh":"面级精确率"},{"en":"Face F1","zh":"面级 F1"}],"note":{"en":"Values use the original 0–1 scale. Section 5.5 specifies area-weighted hole IoU ≥ 0.99. GWS and MVR are rules, not trained neural models. F1 values are reproduced as published, not recomputed.","zh":"保留原文 0–1 量纲。第 5.5 节规定孔级面积加权 IoU ≥ 0.99。GWS、MVR 为规则方法，并非训练式神经网络；F1 保留论文报告值，不自行重算。"},"rows":[["AAGNet","0.944","0.810","0.871","0.912","0.932","0.922"],["UV-Net","0.934","0.896","0.914","0.883","0.933","0.920"],["GWS","0.927","0.578","0.712","0.977","0.644","0.733"],["MVR","0.728","0.870","0.793","0.768","0.898","0.871"]]},"text":{"en":"The paper reports these values for its own dataset. Keep hole detection and face segmentation separate.","zh":"以下为论文在自建数据集上报告的原值，孔检测与面分割分别列出。"},"title":{"en":"Published results: engineering CAD","zh":"论文成绩：工程 CAD 数据"}},{"id":"results-transfer","source_ids":["paper"],"table":{"caption":{"en":"Paper Table 2 · models trained on the authors’ data, tested on MFCAD++","zh":"论文表 2 · 学习模型在作者数据上训练，在 MFCAD++ 上测试"},"columns":[{"en":"Method","zh":"方法"},{"en":"Hole recall","zh":"孔级召回率"},{"en":"Hole precision","zh":"孔级精确率"},{"en":"Hole F1","zh":"孔级 F1"},{"en":"Face recall","zh":"面级召回率"},{"en":"Face precision","zh":"面级精确率"},{"en":"Face F1","zh":"面级 F1"}],"note":{"en":"Original 0–1 values; hole IoU threshold 0.99. Scores belong to this selected transfer set, not the full MFCAD++ collection.","zh":"保留原文 0–1 数值，孔级 IoU 阈值为 0.99。成绩对应筛选后的迁移测试集，不能代表完整 MFCAD++ 数据集。"},"rows":[["AAGNet","0.013","0.357","0.026","0.023","0.433","0.044"],["UV-Net","0.468","0.933","0.623","0.381","0.957","0.545"],["GWS","0.889","0.805","0.845","0.983","0.768","0.871"],["MVR","0.617","0.987","0.760","0.577","0.987","0.729"]]},"text":{"en":"The transfer evaluation uses 200 selected MFCAD++ parts. It is a different test setting from the preceding table.","zh":"迁移实验使用筛选后的 200 个 MFCAD++ 零件，与上一张表的测试数据不同。"},"title":{"en":"Published results: transfer to MFCAD++","zh":"论文成绩：迁移至 MFCAD++"}}],"sources":[{"id":"paper","title":"Schütte, Heinert & Rottmann · Methods, Tables 1–3 and limitations (arXiv v1)","url":"https://arxiv.org/html/2610.13466v1"}],"summary":{"en":"Detect fastener-related holes in CAD parts and test whether learned segmentation transfers between datasets.","zh":"识别 CAD 零件中的紧固件孔，并检验学习式分割能否迁移到其他数据集。"},"title":"Real-World CAD Segmentation"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13466","languages":[],"library_direction":{"basis":"model-review-daily","categories":["engineering-design","visual-reasoning"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_a-real-world-cad-segmentation-benchmark_0ca461d2"},"modality":null,"name":"A Real-World CAD Segmentation Benchmark","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13466","source_url":"https://arxiv.org/abs/2610.13466"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13466","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13466","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","视觉生成","Visual Generation","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A-Bench is a benchmark designed to diagnose whether LMMs are masters at evaluating AIGIs. 2,864 AIGIs from 16 text-to-image models are sampled, each paired with question-answers annotated by human experts, and tested across 18 leading LMMs. A-Bench是一个旨在诊断 LMMs 是否擅长评估 AIGIs 的基准，从 16 个文本到图像模型中采样了 2,864 个 AIGIs，每个都与由人类专家标注的问题-答案配对。","descriptions":{"en":"A-Bench is a benchmark designed to diagnose whether LMMs are masters at evaluating AIGIs. 2,864 AIGIs from 16 text-to-image models are sampled, each paired with question-answers annotated by human experts, and tested across 18 leading LMMs.","zh":"A-Bench是一个旨在诊断 LMMs 是否擅长评估 AIGIs 的基准，从 16 个文本到图像模型中采样了 2,864 个 AIGIs，每个都与由人类专家标注的问题-答案配对。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1580","languages":[],"library_direction":null,"modality":"multimodal","name":"A-Bench","openness":"unknown","publisher":"SJTU, NTU.","released":"2024-06-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1580-a-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/A-Bench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":120,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2206.01718","repository":null,"source_record":"catalog_50049cb1dc8dc3fd"},"categories":["多模态","Multimodal","推理","Reasoning","知识","Knowledge","VQA","多模态模型","VLM","逻辑推理","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A-OKVQA assesses commonsense reasoning abilities. It is a crowdsourced dataset composed of a diverse set of about 25K questions requiring a broad base of commonsense and world knowledge to answer. A-OKVQA用于评估多模态大模型的常识及推理能力，由25K个不同的问题组成，需要对图像中描述的场景进行某种形式的常识性推理来回答。","descriptions":{"en":"A-OKVQA assesses commonsense reasoning abilities. It is a crowdsourced dataset composed of a diverse set of about 25K questions requiring a broad base of commonsense and world knowledge to answer.","zh":"A-OKVQA用于评估多模态大模型的常识及推理能力，由25K个不同的问题组成，需要对图像中描述的场景进行某种形式的常识性推理来回答。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1367","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2206.01718","primary":"visual-reasoning","source_record":"catalog_50049cb1dc8dc3fd"},"modality":"multimodal","name":"A-OKVQA","openness":"unknown","publisher":"Allen Institute for AI","released":"2022-06-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1367-a-okvqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/A-OKVQA","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A3S-Bench is an executable security test suite for evaluating autonomous agents across multi-turn scenarios and agent-level risks using an action-grounded oracle.","descriptions":{"en":"A3S-Bench is an executable security test suite for evaluating autonomous agents across multi-turn scenarios and agent-level risks using an action-grounded oracle."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.22321","languages":[],"library_direction":null,"modality":null,"name":"A3S-Bench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22321","source_url":"https://arxiv.org/abs/2605.22321"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22321","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22321","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["agentic","business","reasoning"],"collected_at":"2026-08-25T10:41:06Z","description":"Quantitative analysis on spreadsheets & documents","descriptions":{"en":"Quantitative analysis on spreadsheets & documents"},"evidence_summary":{"document_count":1,"model_count":30,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:aa-analystagent:a6340098-d7ae-462d-b372-0a0a67fc44b4","reported_at":"2025-10-15","source_url":"https://artificialanalysis.ai/evaluations/aa-analyst-agent"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:aa-analystagent","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"AA-AnalystAgent","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":30,"score_direction":"higher_is_better","score_summary":{"display_max":60.0,"display_multiplier":100,"model_count":30,"model_count_basis":"source_model_id","numeric_count":30,"raw_max":0.6,"raw_min":0.0125,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:aa-analystagent:b2331108-72ed-415a-82d1-188633875bbc","reported_date":"2026-08-13","source_url":"https://artificialanalysis.ai/evaluations/aa-analyst-agent"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"}],"slug":"artificial-analysis-aa-analystagent","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/aa-analyst-agent","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["agentic","business"],"collected_at":"2026-08-25T10:41:06Z","description":"Agentic knowledge work, Elo","descriptions":{"en":"Agentic knowledge work, Elo"},"evidence_summary":{"document_count":1,"model_count":65,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:aa-briefcase:922c69c7-9037-43c6-8bcf-a1c555e7f3eb","reported_at":"2025-04-05","source_url":"https://artificialanalysis.ai/evaluations/aa-briefcase"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:aa-briefcase","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"AA-Briefcase","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"release_announcement","note":"The publisher launches AA-Briefcase and its four held-out knowledge-work scenarios on June 18.","source_key":"artificial-analysis:aa-briefcase","source_url":"https://artificialanalysis.ai/articles/aa-briefcase"},"repo_kind":null,"repo_resolution_status":null,"score_count":65,"score_direction":"higher_is_better","score_summary":{"display_max":1710.26,"display_multiplier":1,"model_count":65,"model_count_basis":"source_model_id","numeric_count":65,"raw_max":1710.26,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:aa-briefcase:b8fc61f7-5e9a-49e6-8547-6ac56db24627","reported_date":"2026-07-24","source_url":"https://artificialanalysis.ai/evaluations/aa-briefcase"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"}],"slug":"artificial-analysis-aa-briefcase","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/aa-briefcase","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["productivity","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AA-Briefcase is an Artificial Analysis evaluation of AI systems on professional knowledge-work tasks, reported as an Elo score.","descriptions":{"en":"AA-Briefcase is an Artificial Analysis evaluation of AI systems on professional knowledge-work tasks, reported as an Elo score."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aa-briefcase:kimi-k3","reported_at":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-briefcase?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aa-briefcase","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_197d4185398ce127"},"modality":"text","name":"AA-Briefcase","openness":"unknown","publication":{"evidence":[{"note":"Artificial Analysis labels this its original 2026-06-18 launch article, describing its own professional knowledge-work benchmark and Elo-based combined evaluation. This matches both the publisher and task named by the source record. The official page links the live evaluation separately.","url":"https://artificialanalysis.ai/articles/aa-briefcase"},{"note":"The publisher identifies AA-Briefcase as its own private long-horizon knowledge-work evaluation. The live page currently says v1.1; the source record's version remains unspecified. AA-Briefcase-Lite is only a separate public demonstration and is not substituted for the scored benchmark.","url":"https://artificialanalysis.ai/evaluations/aa-briefcase"}],"id":"aa-briefcase-publication","links":[{"kind":"website","title":"AA-Briefcase original launch announcement","url":"https://artificialanalysis.ai/articles/aa-briefcase"},{"kind":"website","title":"AA-Briefcase official evaluation and methodology","url":"https://artificialanalysis.ai/evaluations/aa-briefcase"}],"record_keys":["llm-stats:aa-briefcase"],"reviewed_at":"2026-10-26","scope":{"en":"Artificial Analysis's original AA-Briefcase announcement and official evaluation page. The source record does not identify a version; the current page's v1.1 label is not assigned to the collected scores.","zh":"Artificial Analysis 的 AA-Briefcase 原始发布及官方评测页。收录来源未注明版本，不能把官方当前页面的 v1.1 标签直接套用到已收录成绩。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":1577.0,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":1577.0,"raw_min":917.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aa-briefcase:grok-4.6","reported_date":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-briefcase?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"}],"slug":"llm-stats-aa-briefcase","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-briefcase?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":43,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":217,"join":"llm_stats_id","repository":"gh:zai-org/glm-simple-evals","source_record":"catalog_c6adbc64ef2ca87b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"No official academic documentation found for this benchmark. Extensive research through ArXiv, IEEE/ACL/NeurIPS papers, and university research sites yielded no peer-reviewed sources for an 'aa-index' benchmark. This entry requires verification from official academic sources.","descriptions":{"en":"No official academic documentation found for this benchmark. Extensive research through ArXiv, IEEE/ACL/NeurIPS papers, and university research sites yielded no peer-reviewed sources for an 'aa-index' benchmark. This entry requires verification from official academic sources."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aa-index:glm-4.5","reported_at":"2025-07-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-index?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aa-index","languages":[],"library_direction":{"basis":"model-review","categories":["composite-index"],"join":"llm_stats_id","primary":"composite-index","source_record":"catalog_c6adbc64ef2ca87b"},"modality":"text","name":"AA-Index","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":67.7,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.677,"raw_min":0.61,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aa-index:glm-4.5","reported_date":"2025-07-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-index?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"}],"slug":"llm-stats-aa-index","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-index?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","long-context","reasoning"],"collected_at":"2026-08-25T10:41:06Z","description":"Long context reasoning","descriptions":{"en":"Long context reasoning"},"evidence_summary":{"document_count":1,"model_count":510,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:aa-lcr:217b34ec-5920-4fc1-8886-6a70a324837d","reported_at":"2023-09-27","source_url":"https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:aa-lcr","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["long-context"],"join":"review:14H034160212:2026-10-15","primary":"long-context","source_record":null},"modality":null,"name":"AA-LCR","openness":"unknown","publisher":null,"released":"2025-08-05","released_reference":{"basis":"release_announcement","note":"Artificial Analysis announces the release of this long-context reasoning benchmark, before its later inclusion in Index v2.2.","source_key":"artificial-analysis:aa-lcr","source_url":"https://artificialanalysis.ai/articles/announcing-aa-lcr"},"repo_kind":null,"repo_resolution_status":null,"score_count":510,"score_direction":"higher_is_better","score_summary":{"display_max":83.3333333333333,"display_multiplier":100,"model_count":510,"model_count_basis":"source_model_id","numeric_count":510,"raw_max":0.833333333333333,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:aa-lcr:04ee6719-0327-463b-a1a1-70a6a78254f9","reported_date":"2026-08-05","source_url":"https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2025-02-15","date_precision":"model_announcement","name":"GPT-4o (ChatGPT)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (ChatGPT)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2025-10-30","date_precision":"model_announcement","name":"Kimi Linear 48B A3B Instruct"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 4B v1.1 (Reasoning)"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 3B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 40k"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2 (Jul '24)"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"OLMo 2 32B"},{"date":"2024-11-26","date_precision":"model_announcement","name":"OLMo 2 7B"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2024-04-23","date_precision":"model_announcement","name":"Phi-3 Mini Instruct 3.8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ 32B"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-aa-lcr","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":15977,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_c847be0a0dc9793c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Agent Arena Long Context Reasoning benchmark","descriptions":{"en":"Agent Arena Long Context Reasoning benchmark"},"evidence_summary":{"document_count":1,"model_count":18,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aa-lcr:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-lcr?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aa-lcr","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_c847be0a0dc9793c"},"modality":"text","name":"AA-LCR","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":18,"score_direction":"higher_is_better","score_summary":{"display_max":80.0,"display_multiplier":100,"model_count":18,"model_count_basis":"source_model_id","numeric_count":18,"raw_max":0.8,"raw_min":0.047,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aa-lcr:muse-glimmer-30b","reported_date":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-lcr?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"}],"slug":"llm-stats-aa-lcr","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-lcr?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":["AA-LCR"],"attention_signals":null,"categories":["long_context"],"collected_at":null,"description":"Run by Artificial Analysis rather than the vendor. Third-party execution is the point, but it also means the vendor did not control the setup.","descriptions":{"en":"Run by Artificial Analysis rather than the vendor. Third-party execution is the point, but it also means the vendor did not control the setup."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000aa_lcr\u0000qwen3_5_model_card\u0000aa_lcr\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000aa_lcr\u0000qwen3_5_model_card\u0000aa_lcr\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:aa_lcr","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["long-context"],"join":"review:14H034160212:2026-10-15","primary":"long-context","source_record":null},"modality":null,"name":"AA-LCR","openness":"unknown","publisher":null,"released":"2025-09-16","released_reference":{"basis":"benchmark_release","source_key":"model-reports:aa_lcr","source_url":"https://artificialanalysis.ai/evaluations/aa-lcr"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":74.7,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":74.7,"raw_min":68.7,"source_reference":{"obs_id":"curated\u0000aa_lcr\u0000moonshot_kimi_k3_model_card\u0000aa_lcr\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000aa_lcr\u0000moonshot_kimi_k3_model_card\u0000aa_lcr\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"aa_lcr","source":"model_reports","source_url":"https://artificialanalysis.ai/evaluations/aa-lcr","topics":["long-context"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","knowledge"],"collected_at":"2026-08-25T10:41:06Z","description":"Knowledge","descriptions":{"en":"Knowledge"},"evidence_summary":{"document_count":1,"model_count":489,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:aa-omniscience-accuracy:6a7c0e25-1dcb-4b15-8495-a8536a9da051","reported_at":"2023-03-14","source_url":"https://artificialanalysis.ai/evaluations/omniscience"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:aa-omniscience-accuracy","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["knowledge-expert-qa"],"join":"review:14H034160212:2026-10-15","primary":"knowledge-expert-qa","source_record":null},"modality":null,"name":"AA-Omniscience Accuracy","openness":"unknown","publisher":null,"released":"2025-11-16","released_reference":{"basis":"release_announcement","note":"The Omniscience launch introduces the factual-recall and hallucination evaluation; this source record retains its accuracy metric.","source_key":"artificial-analysis:aa-omniscience-accuracy","source_url":"https://artificialanalysis.ai/articles/aa-omniscience-knowledge-hallucination-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":489,"score_direction":"higher_is_better","score_summary":{"display_max":65.35,"display_multiplier":100,"model_count":489,"model_count_basis":"source_model_id","numeric_count":489,"raw_max":0.6535,"raw_min":0.001,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:aa-omniscience-accuracy:cd55210d-358e-4df1-ba9c-9acb5f186cc9","reported_date":"2026-06-09","source_url":"https://artificialanalysis.ai/evaluations/omniscience"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2023-03-14","date_precision":"model_announcement","name":"GPT-4"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-aa-omniscience-accuracy","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/omniscience","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","knowledge"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AA-Omniscience Index is Artificial Analysis's knowledge-reliability metric. It rewards correct answers, penalizes hallucinations, and does not penalize abstention. Scores range from -100 to 100, where 0 means as many correct as incorrect answers.","descriptions":{"en":"AA-Omniscience Index is Artificial Analysis's knowledge-reliability metric. It rewards correct answers, penalizes hallucinations, and does not penalize abstention. Scores range from -100 to 100, where 0 means as many correct as incorrect answers."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aa-omniscience-index:grok-4.5","reported_at":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-omniscience-index?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aa-omniscience-index","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_4bfe2ff9ab36de99"},"modality":"text","name":"AA-Omniscience Index","openness":"unknown","publisher":null,"released":"2025-11-16","released_reference":{"basis":"release_announcement","note":"The source explicitly describes Artificial Analysis's knowledge-reliability index, introduced in this launch. Its signed index scale remains distinct from accuracy and non-hallucination rate.","source_key":"llm-stats:aa-omniscience-index","source_url":"https://artificialanalysis.ai/articles/aa-omniscience-knowledge-hallucination-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":126.0,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":126.0,"raw_min":-29.5,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aa-omniscience-index:grok-4.5","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-omniscience-index?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"}],"slug":"llm-stats-aa-omniscience-index","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-omniscience-index?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","knowledge","faithfulness"],"collected_at":"2026-08-25T10:41:06Z","description":"1 - hallucination rate","descriptions":{"en":"1 - hallucination rate"},"evidence_summary":{"document_count":1,"model_count":489,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:aa-omniscience-non-hallucination:6a7c0e25-1dcb-4b15-8495-a8536a9da051","reported_at":"2023-03-14","source_url":"https://artificialanalysis.ai/evaluations/omniscience"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:aa-omniscience-non-hallucination","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["factuality"],"join":"review:14H034160212:2026-10-15","primary":"factuality","source_record":null},"modality":null,"name":"AA-Omniscience Non-Hallucination Rate","openness":"unknown","publisher":null,"released":"2025-11-16","released_reference":{"basis":"release_announcement","note":"The Omniscience launch introduces the factual-recall and hallucination evaluation; this source record retains its non-hallucination metric.","source_key":"artificial-analysis:aa-omniscience-non-hallucination","source_url":"https://artificialanalysis.ai/articles/aa-omniscience-knowledge-hallucination-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":489,"score_direction":"higher_is_better","score_summary":{"display_max":99.09909909909909,"display_multiplier":100,"model_count":489,"model_count_basis":"source_model_id","numeric_count":489,"raw_max":0.990990990990991,"raw_min":0.015260480617435568,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:aa-omniscience-non-hallucination:d306cccd-0085-4b2f-8aa0-ffcdbb434695","reported_date":"2026-05-25","source_url":"https://artificialanalysis.ai/evaluations/omniscience"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2023-03-14","date_precision":"model_announcement","name":"GPT-4"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-aa-omniscience-non-hallucination","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/omniscience","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":15,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aarr-bench/aarri-bench","source_record":"bm_aarri-bench_d4d9e662"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AARRI-Bench evaluates LLM agents on entry-level research intern tasks, measuring success rate in containerized environments with fixed tasks and scoring.","descriptions":{"en":"AARRI-Bench evaluates LLM agents on entry-level research intern tasks, measuring success rate in containerized environments with fixed tasks and scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07462","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_aarri-bench_d4d9e662"},"modality":null,"name":"AARRI-Bench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07462","source_url":"https://arxiv.org/abs/2606.07462"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07462","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07462","topics":[],"unit":null},{"aliases":["ABACUS_Bench"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":89,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_abacus-bench_709c4a6b"},"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates instruction-based image editing models on add, remove, and change tasks, measuring exact object count correctness and scene preservation without VLM judges.","descriptions":{"en":"Evaluates instruction-based image editing models on add, remove, and change tasks, measuring exact object count correctness and scene preservation without VLM judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:aymen-lassoued/abacus_bench","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_abacus-bench_709c4a6b"},"modality":null,"name":"ABACUS","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:aymen-lassoued/abacus_bench","source_url":"https://huggingface.co/datasets/Aymen-Lassoued/ABACUS_Bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-aymen-lassoued-abacus-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/Aymen-Lassoued/ABACUS_Bench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ABC-Bench evaluates LLM agents on biosecurity-relevant tasks including liquid handling robot code generation, DNA fragment design, and DNA synthesis screening evasion, with wet-lab validation.","descriptions":{"en":"ABC-Bench evaluates LLM agents on biosecurity-relevant tasks including liquid handling robot code generation, DNA fragment design, and DNA synthesis screening evasion, with wet-lab validation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.11150","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_abc-bench_fecd3eb4"},"modality":null,"name":"ABC-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11150","source_url":"https://arxiv.org/abs/2606.11150"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11150","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11150","topics":["safety","science"],"unit":null},{"aliases":["AbductionRules","Abduction Rules"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Four synthetic abductive-reasoning datasets (Animal and Person domains, each Simple and Confounded): the model must supply the missing fact that explains an observation. The paper itself reports that fine-tuned models partly learned to exploit the structure of the data, so a high fine-tuned score need not mean general abduction, and a number without the subset named is not comparable across reports.","descriptions":{"en":"Four synthetic abductive-reasoning datasets (Animal and Person domains, each Simple and Confounded): the model must supply the missing fact that explains an observation. The paper itself reports that fine-tuned models partly learned to exploit the structure of the data, so a high fine-tuned score need not mean general abduction, and a number without the subset named is not comparable across reports."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:abductionrules","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"AbductionRules","openness":"unknown","publisher":null,"released":"2022-03-23","released_reference":{"basis":"benchmark_release","source_key":"model-reports:abductionrules","source_url":"https://aclanthology.org/2022.findings-acl.19/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"abductionrules","source":"model_reports","source_url":"https://aclanthology.org/2022.findings-acl.19/","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","NAACL 2024","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ABSPYRAMID is a unified entailment graph of 221K textual descriptions of abstraction knowledge. ABSPYRAMID collects abstract knowledge for three components\nof diverse events to comprehensively evaluate the abstraction ability of language\nmodels in the open domain. ABSPYRAMID 是包含 221,000 条文本描述的抽象知识,收集了多种事件的三个组成部分的抽象知识，以全面评估语言模型在开放域中的抽象能力。","descriptions":{"en":"ABSPYRAMID is a unified entailment graph of 221K textual descriptions of abstraction knowledge. ABSPYRAMID collects abstract knowledge for three components\nof diverse events to comprehensively evaluate the abstraction ability of language\nmodels in the open domain.","zh":"ABSPYRAMID 是包含 221,000 条文本描述的抽象知识,收集了多种事件的三个组成部分的抽象知识，以全面评估语言模型在开放域中的抽象能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1148","languages":[],"library_direction":null,"modality":null,"name":"AbsPyramid","openness":"unknown","publisher":"Tencent AI Lab","released":"2024-06-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1148-abspyramid","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AbsPyramid","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_0a7b5bc3b63e66b3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","finance","general","healthcare","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ACEBench is a comprehensive benchmark for evaluating Large Language Models' tool usage capabilities across three primary evaluation types: Normal (basic tool usage scenarios), Special (tool usage with ambiguous or incomplete instructions), and Agent (multi-agent interactions simulating real-world dialogues). The benchmark covers 4,538 APIs across 8 major domains and 68 sub-domains including technology, finance, entertainment, society, health, culture, and environment, supporting both English and Chinese languages.","descriptions":{"en":"ACEBench is a comprehensive benchmark for evaluating Large Language Models' tool usage capabilities across three primary evaluation types: Normal (basic tool usage scenarios), Special (tool usage with ambiguous or incomplete instructions), and Agent (multi-agent interactions simulating real-world dialogues). The benchmark covers 4,538 APIs across 8 major domains and 68 sub-domains including technology, finance, entertainment, society, health, culture, and environment, supporting both English and Chinese languages."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:acebench:kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/acebench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:acebench","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_0a7b5bc3b63e66b3"},"modality":"text","name":"ACEBench","openness":"unknown","publisher":null,"released":"2025-01-22","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: ACEBench: Who Wins the Match Point in Tool Usage?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:acebench","source_url":"https://arxiv.org/abs/2501.12851"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":76.5,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.765,"raw_min":0.765,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:acebench:kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/acebench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"}],"slug":"llm-stats-acebench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/acebench?top_n=500","topics":["agents","reasoning-math","finance","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":null,"source_record":"bm_acg-bench_b4128eb4"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Generalization in multi-arm collaboration can be studied as composing familiar atomic skills in new ways across arms.","descriptions":{"en":"Generalization in multi-arm collaboration can be studied as composing familiar atomic skills in new ways across arms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.06184","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics","multi-agent-games"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_acg-bench_b4128eb4"},"modality":null,"name":"ACG-Bench","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.06184","source_url":"https://arxiv.org/abs/2610.06184"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-06184","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06184","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:zjuicsr/actbench","source_record":"bm_actbench_33a6cb88"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Self-evolving benchmark of behavioral safety in cowork agents, evaluating risk from execution trajectories across 600 cases, 213 scenarios, 15 risk behaviors, and 48 web-service APIs.","descriptions":{"en":"Self-evolving benchmark of behavioral safety in cowork agents, evaluating risk from execution trajectories across 600 cases, 213 scenarios, 15 risk behaviors, and 48 web-service APIs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.09476","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","safety-alignment"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_actbench_33a6cb88"},"modality":null,"name":"ActBench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09476","source_url":"https://arxiv.org/abs/2608.09476"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09476","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09476","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","NeurIPS 2024","多模态模型","VLM","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ActionAtlas is a multiple-choice video question answering benchmark, including 934 videos showcasing 580 unique actions across 56 sports, with a total of 1896 actions within choices. ActionAtlas是一个多项选择视频问答基准测试，包括934个视频，展示了56项运动中的580个独特动作，选项共包含1896个动作。","descriptions":{"en":"ActionAtlas is a multiple-choice video question answering benchmark, including 934 videos showcasing 580 unique actions across 56 sports, with a total of 1896 actions within choices.","zh":"ActionAtlas是一个多项选择视频问答基准测试，包括934个视频，展示了56项运动中的580个独特动作，选项共包含1896个动作。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1319","languages":[],"library_direction":null,"modality":"multimodal","name":"ActionAtlas","openness":"unknown","publisher":"University of Washington","released":"2024-10-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1319-actionatlas","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ActionAtlas","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Video-capable vision-language models score above 80\\% on popular benchmarks yet struggle with spatial-temporal binding: associating the right action with the right person at the right moment.","descriptions":{"en":"Video-capable vision-language models score above 80\\% on popular benchmarks yet struggle with spatial-temporal binding: associating the right action with the right person at the right moment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34547","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_actionlens_0f67b868"},"modality":null,"name":"ActionLens","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34547","source_url":"https://arxiv.org/abs/2609.34547"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34547","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34547","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":102,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:xlearning-scu/active-swe","source_record":"bm_active-swe_8c4ac0e6"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Active-SWE evaluates coding agents on proactive bug fixing: detecting and fixing multiple bugs without issue reports. It includes 1,663 tasks across six bug categories and eight languages, with stages for recorded bugs, potential bugs, and judge validation.","descriptions":{"en":"Active-SWE evaluates coding agents on proactive bug fixing: detecting and fixing multiple bugs without issue reports. It includes 1,663 tasks across six bug categories and eight languages, with stages for recorded bugs, potential bugs, and judge validation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04682","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_active-swe_8c4ac0e6"},"modality":null,"name":"Active-SWE","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04682","source_url":"https://arxiv.org/abs/2608.04682"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04682","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04682","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_activearena-bench_7722594b"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ActiveArena-Bench evaluates active perception in robotic manipulation through 35 tasks across five categories covering visual exploration and interactive information acquisition.","descriptions":{"en":"ActiveArena-Bench evaluates active perception in robotic manipulation through 35 tasks across five categories covering visual exploration and interactive information acquisition."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24124","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_activearena-bench_7722594b"},"modality":null,"name":"ActiveArena-Bench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24124","source_url":"https://arxiv.org/abs/2609.24124"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24124","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24124","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ActiveFly-Bench is a benchmark for UAV embodied perception, decomposing active perception into three tasks: Aerial Embodied Question Answering (Air-EQA), Observation Behavior Planning (OBP), and Fine-grained Language-guided UAV Control (FLUC). It includes datasets from real-world and simulated outdoor environments.","descriptions":{"en":"ActiveFly-Bench is a benchmark for UAV embodied perception, decomposing active perception into three tasks: Aerial Embodied Question Answering (Air-EQA), Observation Behavior Planning (OBP), and Fine-grained Language-guided UAV Control (FLUC). It includes datasets from real-world and simulated outdoor environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.10180","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_activefly-bench_205c2488"},"modality":null,"name":"ActiveFly-Bench","openness":"unknown","publisher":null,"released":"2026-07-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10180","source_url":"https://arxiv.org/abs/2607.10180"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10180","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10180","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["AI Scientist","General AI","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"ActiveSciBench evaluates budget-constrained, closed-loop scientific discovery across enzyme-kinetics and gene-regulatory-network tasks.","descriptions":{"en":"ActiveSciBench evaluates budget-constrained, closed-loop scientific discovery across enzyme-kinetics and gene-regulatory-network tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.24043","languages":[],"library_direction":null,"modality":null,"name":"ActiveSciBench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24043","source_url":"https://arxiv.org/abs/2605.24043"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24043","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24043","topics":[],"unit":null},{"aliases":["An Exam for Active Observers"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"ActiveVision evaluates whether multimodal large language models can perform repeated, active visual perception across 17 tasks in three categories.","descriptions":{"en":"ActiveVision evaluates whether multimodal large language models can perform repeated, active visual perception across 17 tasks in three categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.16165","languages":[],"library_direction":null,"modality":null,"name":"ActiveVision","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.16165","source_url":"https://arxiv.org/abs/2607.16165"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-16165","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16165","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":978,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:activitynet/activitynet","source_record":"catalog_b152800e6ac82bd3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A large-scale video benchmark for human activity understanding. Provides samples from 203 activity classes with an average of 137 untrimmed videos per class and 1.41 activity instances per video, for a total of 849 video hours. The benchmark covers a wide range of complex human activities that are of interest to people in their daily living and can be used to compare algorithms for three scenarios: untrimmed video classification, trimmed activity classification, and activity detection.","descriptions":{"en":"A large-scale video benchmark for human activity understanding. Provides samples from 203 activity classes with an average of 137 untrimmed videos per class and 1.41 activity instances per video, for a total of 849 video hours. The benchmark covers a wide range of complex human activities that are of interest to people in their daily living and can be used to compare algorithms for three scenarios: untrimmed video classification, trimmed activity classification, and activity detection."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:activitynet:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/activitynet?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:activitynet","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_b152800e6ac82bd3"},"modality":"video","name":"ActivityNet","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":61.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.619,"raw_min":0.619,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:activitynet:gpt-4o-2024-08-06","reported_date":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/activitynet?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"}],"slug":"llm-stats-activitynet","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/activitynet?top_n=500","topics":[],"unit":null},{"aliases":["Understanding Autonomous Driving Datasets by Describing Differences between Image Subsets in Natural Language"],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kit-mrt/ad-diff","source_record":"bm_understanding-autonomous-driving-datasets-_149f7829"},"categories":["Autonomous Driving","General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates set-difference captioning on autonomous driving image subsets, with three splits based on annotation filtering, CLIP filtering, and web scraping, yielding natural-language descriptions of differences between target and reference sets.","descriptions":{"en":"Evaluates set-difference captioning on autonomous driving image subsets, with three splits based on annotation filtering, CLIP filtering, and web scraping, yielding natural-language descriptions of differences between target and reference sets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.03677","languages":[],"library_direction":{"basis":"model-review-2","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_understanding-autonomous-driving-datasets-_149f7829"},"modality":null,"name":"AD-Diff Bench","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03677","source_url":"https://arxiv.org/abs/2609.03677"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03677","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03677","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["长文本","Long-Context","NAACL 2024","大语言模型","LLM","长上下文","Long Context","官方自建","Official"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Ada-LEval is a length-adaptable benchmark for evaluating the long-context understanding\nof LLMs. Ada-LEval includes two challenging subsets, TSort and BestAnswer, which enable\na more reliable evaluation of LLMs’ long context capabilities. Ada-LEval 用于评估大型语言模型（LLMs）对长上下文的理解能力。Ada-LEval 包含两个具有挑战性的子集，TSort 和 BestAnswer，能够更可靠地评估 LLMs 的长上下文能力。","descriptions":{"en":"Ada-LEval is a length-adaptable benchmark for evaluating the long-context understanding\nof LLMs. Ada-LEval includes two challenging subsets, TSort and BestAnswer, which enable\na more reliable evaluation of LLMs’ long context capabilities.","zh":"Ada-LEval 用于评估大型语言模型（LLMs）对长上下文的理解能力。Ada-LEval 包含两个具有挑战性的子集，TSort 和 BestAnswer，能够更可靠地评估 LLMs 的长上下文能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1155","languages":[],"library_direction":null,"modality":null,"name":"Ada-LEval","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-06-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1155-ada-leval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Ada-LEval","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:galaxychen/adamem","source_record":"bm_adamem-bench_de779711"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"AdaMem-Bench simulates weeks of interaction with week-by-week question answering to evaluate memory policies for personalized long-horizon LLM agents. It measures QA accuracy and memory volume across different models.","descriptions":{"en":"AdaMem-Bench simulates weeks of interaction with week-by-week question answering to evaluate memory policies for personalized long-horizon LLM agents. It measures QA accuracy and memory volume across different models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.21144","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_adamem-bench_de779711"},"modality":null,"name":"AdaMem-Bench","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21144","source_url":"https://arxiv.org/abs/2606.21144"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21144","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21144","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":28,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":45,"join":"claire_record_id","repository":"gh:jiayujeff/adaplanbench","source_record":"bm_adaplanbench_b501c6c7"},"categories":["Agents & Tool Use","General AI","Planning","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates adaptive planning of LLM agents under progressively disclosed world and user constraints across 307 household tasks.","descriptions":{"en":"Evaluates adaptive planning of LLM agents under progressively disclosed world and user constraints across 307 household tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.05622","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_adaplanbench_b501c6c7"},"modality":null,"name":"AdaPlanBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05622","source_url":"https://arxiv.org/abs/2606.05622"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05622","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05622","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous.","descriptions":{"en":"Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.36488","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","agent-memory"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_adaptarena_91c3197a"},"modality":null,"name":"AdaptArena","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36488","source_url":"https://arxiv.org/abs/2609.36488"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36488","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36488","topics":["computer-use","agents"],"unit":null},{"aliases":["Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security"],"attention_signals":null,"categories":["Agents","Agents & Tool Use","Robotics & Embodied AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A 21-scenario benchmark evaluating memoryless LLM defenders against adaptive attackers that observe earlier responses and change tactics across multiple rounds.","descriptions":{"en":"A 21-scenario benchmark evaluating memoryless LLM defenders against adaptive attackers that observe earlier responses and change tactics across multiple rounds."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.18063","languages":[],"library_direction":null,"modality":null,"name":"Adaptive Adversaries","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18063","source_url":"https://arxiv.org/abs/2607.18063"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18063","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18063","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Manufacturing & Process Control","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Computer Use Agents (CUAs) are increasingly deployed to navigate mobile and desktop applications on behalf of users, yet no benchmark comprehensively evaluates whether they can safely interact with visual interfaces while handling ambiguou…","descriptions":{"en":"Computer Use Agents (CUAs) are increasingly deployed to navigate mobile and desktop applications on behalf of users, yet no benchmark comprehensively evaluates whether they can safely interact with visual interfaces while handling ambiguou…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.26204","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","agent-security"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_adepts-bench_6cc5b43a"},"modality":null,"name":"ADeptS-Bench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26204","source_url":"https://arxiv.org/abs/2608.26204"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26204","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26204","topics":["computer-use","agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"claire_record_id","repository":null,"source_record":"bm_adspark_17415a9b"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Product-centric advertisement video generation aims to create promotional videos that preserve fine-grained product identity while presenting selling points through coherent multi-shot narratives.","descriptions":{"en":"Product-centric advertisement video generation aims to create promotional videos that preserve fine-grained product identity while presenting selling points through coherent multi-shot narratives."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10047","languages":[],"library_direction":{"basis":"model-review-daily","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_adspark_17415a9b"},"modality":null,"name":"AdSpark","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10047","source_url":"https://arxiv.org/abs/2610.10047"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10047","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10047","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"llm_stats_id","repository":null,"source_record":"catalog_53b789eb0f0ba6dc"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","instruction_following","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AdvancedIF is a rubric-based benchmark measuring complex, multi-turn, and system-prompted instruction following ability, scored with a calibrated LLM judge against per-instruction rubrics.","descriptions":{"en":"AdvancedIF is a rubric-based benchmark measuring complex, multi-turn, and system-prompted instruction following ability, scored with a calibrated LLM judge against per-instruction rubrics."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:advancedif:mai-code-1-flash","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/advancedif?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:advancedif","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_53b789eb0f0ba6dc"},"modality":"text","name":"AdvancedIF","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":85.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.85,"raw_min":0.714,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:advancedif:mai-thinking-1","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/advancedif?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"}],"slug":"llm-stats-advancedif","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/advancedif?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":29,"join":"claire_record_id","repository":"gh:internlm/advancedmathbench","source_record":"bm_advancedmathbench_22d95486"},"categories":["Language & Knowledge","Mathematics & Formal Science","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"AdvancedMathBench is a benchmark suite for advanced mathematical reasoning, containing ProverBench (296 proof problems) and VerifierBench (888 proof trajectories with expert labels), with an automatic verification pipeline.","descriptions":{"en":"AdvancedMathBench is a benchmark suite for advanced mathematical reasoning, containing ProverBench (296 proof problems) and VerifierBench (888 proof trajectories with expert labels), with an automatic verification pipeline."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.11849","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_advancedmathbench_22d95486"},"modality":null,"name":"AdvancedMathBench","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11849","source_url":"https://arxiv.org/abs/2607.11849"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11849","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11849","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"AdversaBench is an automated LLM red-teaming evaluation that mutates seed prompts and confirms resulting failures with multiple judges and a tiebreaker.","descriptions":{"en":"AdversaBench is an automated LLM red-teaming evaluation that mutates seed prompts and confirms resulting failures with multiple judges and a tiebreaker."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.24589","languages":[],"library_direction":null,"modality":null,"name":"AdversaBench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24589","source_url":"https://arxiv.org/abs/2606.24589"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24589","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24589","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"AdvPlan-Bench is an offline benchmark for adversarial evaluation of structured plan-generation agents. It uses typed action chains, adversarial response sets, selector diagnostics, and metrics like BLUE-vs-RED advantage and Nash-gap. Includes 150 synthetic scenarios across five planning templates.","descriptions":{"en":"AdvPlan-Bench is an offline benchmark for adversarial evaluation of structured plan-generation agents. It uses typed action chains, adversarial response sets, selector diagnostics, and metrics like BLUE-vs-RED advantage and Nash-gap. Includes 150 synthetic scenarios across five planning templates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00832","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","safety-alignment"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_advplan-bench_7a0b9910"},"modality":null,"name":"AdvPlan-Bench","openness":"unknown","publisher":null,"released":"2026-08-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00832","source_url":"https://arxiv.org/abs/2608.00832"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00832","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00832","topics":["reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":14,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:msp-xen/ae-uav","source_record":"bm_ae-uav_5141a634"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"AE-UAV is an air-to-air event-based UAV tracking benchmark with 178 flight sequences and continuous-time cubic B-spline annotations, supporting evaluation at arbitrary temporal resolutions. It includes multimodal auxiliary data and predefined train/validation/test splits.","descriptions":{"en":"AE-UAV is an air-to-air event-based UAV tracking benchmark with 178 flight sequences and continuous-time cubic B-spline annotations, supporting evaluation at arbitrary temporal resolutions. It includes multimodal auxiliary data and predefined train/validation/test splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.14726","languages":[],"library_direction":null,"modality":null,"name":"AE-UAV","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14726","source_url":"https://arxiv.org/abs/2607.14726"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14726","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14726","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["科学","Science","学科","Examination","知识","Knowledge","科学智能","AI for Science","科学推理","Scientific Reasoning","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AECBench is an open-source benchmark for evaluating LLMs in architecture, engineering, and construction (AEC), covering 23 tasks and about 4,800 samples across five cognitive levels. AECBench 是面向建筑、工程与施工（AEC）领域的大语言模型评测基准，覆盖 5 个认知层级、23 类任务和约 4,800 个样本，用于评估模型在知识记忆、理解、推理、计算与应用方面的能力。","descriptions":{"en":"AECBench is an open-source benchmark for evaluating LLMs in architecture, engineering, and construction (AEC), covering 23 tasks and about 4,800 samples across five cognitive levels.","zh":"AECBench 是面向建筑、工程与施工（AEC）领域的大语言模型评测基准，覆盖 5 个认知层级、23 类任务和约 4,800 个样本，用于评估模型在知识记忆、理解、推理、计算与应用方面的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2452","languages":["Chinese"],"library_direction":null,"modality":null,"name":"AECBench","openness":"restricted","publisher":"华东建筑设计研究院有限公司、同济大学","released":"2026-04-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2452-aecbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AECBench","topics":["reasoning-math","science"],"unit":null},{"aliases":["Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding"],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-09-25T04:40:22Z","description":"AEGBench is a difficulty-stratified evaluation for open-vocabulary audio event grounding, where systems localize queried sound events in time.","descriptions":{"en":"AEGBench is a difficulty-stratified evaluation for open-vocabulary audio event grounding, where systems localize queried sound events in time."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.04383","languages":[],"library_direction":null,"modality":null,"name":"AEGBench","openness":"unknown","publisher":null,"released":"2026-07-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04383","source_url":"https://arxiv.org/abs/2607.04383"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04383","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04383","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":[],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"","descriptions":{},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:community:5f95f778-c521-43fa-b80e-6a55465601e3","languages":[],"library_direction":null,"modality":null,"name":"ael_gate_benchmark_cases_template","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":"higher_is_better","score_summary":{"display_max":null,"display_multiplier":1,"model_count":null,"model_count_basis":null,"numeric_count":0,"raw_max":null,"raw_min":null,"source_reference":null,"unit":null},"scored_models":[],"slug":"llm-stats-community-5f95f778-c521-43fa-b80e-6a55465601e3","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A5f95f778-c521-43fa-b80e-6a55465601e3?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AeroCopilotBench is a two-tier benchmark for evaluating LLM agents as aviation copilots. Tier-1 uses 1,200 multiple-choice questions for knowledge assessment, while Tier-2 includes 73 procedural tasks in an interactive virtual cockpit environment, with safety-gated evaluation.","descriptions":{"en":"AeroCopilotBench is a two-tier benchmark for evaluating LLM agents as aviation copilots. Tier-1 uses 1,200 multiple-choice questions for knowledge assessment, while Tier-2 includes 73 procedural tasks in an interactive virtual cockpit environment, with safety-gated evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.16349","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","knowledge-expert-qa"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_aerocopilotbench_7dd468f7"},"modality":null,"name":"AeroCopilotBench","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16349","source_url":"https://arxiv.org/abs/2608.16349"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16349","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16349","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Reasoning","Robotics & Embodied AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"AeroGround evaluates vision-language models on aerial-ground collaborative reasoning using a simulated dataset of ~29,000 multimodal observation groups and 2,250 QA instances covering cross-view correspondence, spatial understanding, and reasoning.","descriptions":{"en":"AeroGround evaluates vision-language models on aerial-ground collaborative reasoning using a simulated dataset of ~29,000 multimodal observation groups and 2,250 QA instances covering cross-view correspondence, spatial understanding, and reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14721","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_aeroground_3ddedf6a"},"modality":null,"name":"AeroGround","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14721","source_url":"https://arxiv.org/abs/2608.14721"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14721","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14721","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Aerial manipulators extend robotic manipulation into 3D workspaces that are difficult for ground-based robots to access, creating new opportunities for general-purpose manipulation.","descriptions":{"en":"Aerial manipulators extend robotic manipulation into 3D workspaces that are difficult for ground-based robots to access, creating new opportunities for general-purpose manipulation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.36915","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_aeromanip-vla_1bc693da"},"modality":null,"name":"AeroManip-VLA","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36915","source_url":"https://arxiv.org/abs/2609.36915"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36915","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36915","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":14,"join":"llm_stats_id","repository":null,"source_record":"catalog_5355c4d25a985352"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","coding"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AetherCode is a competitive-programming benchmark of olympiad-level algorithmic coding problems.","descriptions":{"en":"AetherCode is a competitive-programming benchmark of olympiad-level algorithmic coding problems."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aethercode:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aethercode?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aethercode","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_5355c4d25a985352"},"modality":"text","name":"AetherCode","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":67.9,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.679,"raw_min":0.658,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aethercode:seed-2.1-turbo","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aethercode?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-aethercode","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aethercode?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["AI Scientist","General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates generative meteorological reasoning through 7,732 expert-written forecast discussions paired with AI weather inputs, using metrics for numerical accuracy, style, and grounding.","descriptions":{"en":"Evaluates generative meteorological reasoning through 7,732 expert-written forecast discussions paired with AI weather inputs, using metrics for numerical accuracy, style, and grounding."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24954","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_afdbench_6e3f5983"},"modality":null,"name":"AFDBench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24954","source_url":"https://arxiv.org/abs/2608.24954"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24954","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24954","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:1171-jpg/affordance20q","source_record":"bm_affordance20q_4a1ecb35"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Affordance20Q is a benchmark for evaluating affordance reasoning in LLMs using a 20-questions game. It comprises 1,009 games over 454 objects and 59 affordances, where models identify a hidden object's affordance by asking yes/no questions about physical properties.","descriptions":{"en":"Affordance20Q is a benchmark for evaluating affordance reasoning in LLMs using a 20-questions game. It comprises 1,009 games over 454 objects and 59 affordances, where models identify a hidden object's affordance by asking yes/no questions about physical properties."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.14240","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_affordance20q_4a1ecb35"},"modality":null,"name":"AFFORDANCE20Q","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14240","source_url":"https://arxiv.org/abs/2606.14240"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14240","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14240","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AFQMC is an Ant Financial chinese semantic similarity task, which requires to judge whether two sentences have the same meaning or not. AFQMC一个蚂蚁金服中文语义相似度任务，要求判断两个句子是否具有相同的语义。","descriptions":{"en":"AFQMC is an Ant Financial chinese semantic similarity task, which requires to judge whether two sentences have the same meaning or not.","zh":"AFQMC一个蚂蚁金服中文语义相似度任务，要求判断两个句子是否具有相同的语义。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:506","languages":["Chinese"],"library_direction":null,"modality":null,"name":"AFQMC","openness":"unknown","publisher":null,"released":"2020-04-13","released_reference":{"basis":"paper_first_version","note":"CLUE's first paper version introduces the Ant Financial semantic-similarity task in section 4.2 and reports its evaluation.","source_key":"opencompass:506","source_url":"https://arxiv.org/abs/2004.05986v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-506-afqmc","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AFQMC","topics":["language"],"unit":null},{"aliases":["Callability Is Not Operability: Controlled Interface Interventions for LLM Agents"],"attention_signals":null,"categories":["Agents","Code & Software","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Holds task, backend, initial state, injected failure, agent, and language model fixed while varying the tool interface to measure callability versus operability.","descriptions":{"en":"Holds task, backend, initial state, injected failure, agent, and language model fixed while varying the tool interface to measure callability versus operability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23628","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_callability-is-not-operability-controlled-_13caf3a3"},"modality":null,"name":"AFT-Bench","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23628","source_url":"https://arxiv.org/abs/2608.23628"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23628","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23628","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"AGC-Bench evaluates artificial general creativity across 78 datasets covering brainstorming, problem solving, STEM, narrative, figurative language, and humor. It uses an agentic harness and a public leaderboard with an open-weight judge model.","descriptions":{"en":"AGC-Bench evaluates artificial general creativity across 78 datasets covering brainstorming, problem solving, STEM, narrative, figurative language, and humor. It uses an agentic harness and a public leaderboard with an open-weight judge model."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01152","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_agc-bench_28cf2135"},"modality":null,"name":"AGC-Bench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01152","source_url":"https://arxiv.org/abs/2607.01152"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01152","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01152","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kyashrathore/agent-app-benchmark","source_record":"bm_agent-app-benchmark_0bc5ce52"},"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates GUI application performance for coding agents using deterministic historical session workloads, measuring app start, session switching, memory, and CPU metrics.","descriptions":{"en":"Evaluates GUI application performance for coding agents using deterministic historical session workloads, measuring app start, session switching, memory, and CPU metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:kyashrathore/agent-app-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding","computer-use","agent-memory"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_agent-app-benchmark_0bc5ce52"},"modality":null,"name":"Agent App Benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:kyashrathore/agent-app-benchmark","source_url":"https://github.com/kyashrathore/agent-app-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-kyashrathore-agent-app-benchmark","source":"claire_radar","source_url":"https://github.com/kyashrathore/agent-app-benchmark","topics":["agents"],"unit":null},{"aliases":["agent-failure-lab"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mostre/agent-failure-lab","source_record":"bm_agent-failure-lab_efea0d2e"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agent failures across 12 categories by converting failed sessions into isolated benchmark tasks with verification commands.","descriptions":{"en":"Evaluates coding agent failures across 12 categories by converting failed sessions into isolated benchmark tasks with verification commands."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:mostre/agent-failure-lab","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_agent-failure-lab_efea0d2e"},"modality":null,"name":"Agent Failure Lab","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:mostre/agent-failure-lab","source_url":"https://github.com/MOSTRE/agent-failure-lab"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-mostre-agent-failure-lab","source":"claire_radar","source_url":"https://github.com/MOSTRE/agent-failure-lab","topics":["coding","agents"],"unit":null},{"aliases":["agent-memory-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:datapace-ai/agent-memory-benchmark","source_record":"bm_agent-memory-benchmark_426c2d6c"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates agent memory systems on a 10-question pilot using LongMemEval, measuring answer accuracy under three judge rules plus token usage, latency, ingestion cost, forgetting curve, and superseded-value handling.","descriptions":{"en":"Evaluates agent memory systems on a 10-question pilot using LongMemEval, measuring answer accuracy under three judge rules plus token usage, latency, ingestion cost, forgetting curve, and superseded-value handling."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:datapace-ai/agent-memory-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_agent-memory-benchmark_426c2d6c"},"modality":null,"name":"Agent Memory Benchmark","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:datapace-ai/agent-memory-benchmark","source_url":"https://github.com/datapace-ai/agent-memory-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-datapace-ai-agent-memory-benchmark","source":"claire_radar","source_url":"https://github.com/datapace-ai/agent-memory-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":6,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:mikivishy/agentplanningbenchmark","source_record":"bm_agent-planning-benchmark_ba86aa47"},"categories":["Agents","General AI","Multimodal","Planning","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Agent Planning Benchmark (APB) is a diagnostic benchmark with 4,209 multimodal cases across 22 domains, evaluating planning capabilities in five settings including tool noise and unsolvable tasks.","descriptions":{"en":"Agent Planning Benchmark (APB) is a diagnostic benchmark with 4,209 multimodal cases across 22 domains, evaluating planning capabilities in five settings including tool noise and unsolvable tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.04874","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","tool-use"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_agent-planning-benchmark_ba86aa47"},"modality":null,"name":"Agent Planning Benchmark","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04874","source_url":"https://arxiv.org/abs/2606.04874"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04874","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04874","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":10,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:eyuansu62/agent-retrieval-bench","source_record":"bm_agent-retrieval-bench_1105bc27"},"categories":["Agents","Information retrieval","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"File-level retrieval benchmark for coding agents, covering four positive tasks (code2test, comment2context, trace2code, edit2ripple) and a selective-retrieval subset with natural no-gold and counterfactual controls across 25 repositories, 427 samples, with frozen base-commit corpora.","descriptions":{"en":"File-level retrieval benchmark for coding agents, covering four positive tasks (code2test, comment2context, trace2code, edit2ripple) and a selective-retrieval subset with natural no-gold and counterfactual controls across 25 repositories, 427 samples, with frozen base-commit corpora."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.24882","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","deep-research"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_agent-retrieval-bench_1105bc27"},"modality":null,"name":"Agent Retrieval Bench","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24882","source_url":"https://arxiv.org/abs/2607.24882"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24882","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24882","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Agent Startup Bench measures AI agents on high-economic-value, startup-style tasks that require autonomous planning and execution to deliver practical, verifiable results.","descriptions":{"en":"Agent Startup Bench measures AI agents on high-economic-value, startup-style tasks that require autonomous planning and execution to deliver practical, verifiable results."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:agent-startup-bench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agent-startup-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:agent-startup-bench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_06a52f8a0bca38b1"},"modality":"text","name":"Agent Startup Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":68.8,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.688,"raw_min":0.54,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:agent-startup-bench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agent-startup-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-agent-startup-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agent-startup-bench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["agent-studio-latency-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jadenguitarman/agent-studio-latency-benchmark","source_record":"bm_agent-studio-latency-benchmark_d27c8fc1"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates latency of a direct Agent Studio integration by running a fixed query set under controlled conditions and reporting timings for Search, Agent Studio, and request-selected index comparisons.","descriptions":{"en":"Evaluates latency of a direct Agent Studio integration by running a fixed query set under controlled conditions and reporting timings for Search, Agent Studio, and request-selected index comparisons."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:jadenguitarman/agent-studio-latency-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Agent Studio latency benchmark","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:jadenguitarman/agent-studio-latency-benchmark","source_url":"https://github.com/jadenguitarman/agent-studio-latency-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-jadenguitarman-agent-studio-latency-benchmark","source":"claire_radar","source_url":"https://github.com/jadenguitarman/agent-studio-latency-benchmark","topics":["agents"],"unit":null},{"aliases":["agent-systems-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:martin-beck/agent-systems-benchmark","source_record":"bm_agent-systems-benchmark_6fb06572"},"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A Linux terminal framework for measuring how AI coding agents scale under concurrent sessions while maintaining task quality, latency, and resource consumption within declared bounds.","descriptions":{"en":"A Linux terminal framework for measuring how AI coding agents scale under concurrent sessions while maintaining task quality, latency, and resource consumption within declared bounds."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:martin-beck/agent-systems-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_agent-systems-benchmark_6fb06572"},"modality":null,"name":"Agent Systems Benchmark (ASB)","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:martin-beck/agent-systems-benchmark","source_url":"https://github.com/martin-beck/agent-systems-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-martin-beck-agent-systems-benchmark","source":"claire_radar","source_url":"https://github.com/martin-beck/agent-systems-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:titusblair/agent-arena","source_record":"bm_agent-arena_0df4c133"},"categories":["Agents","Finance","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"A harness for evaluating multi-agent strategies on financial data tasks, comparing single, fanout, and plan-execute-review approaches with measured cost and confidence intervals.","descriptions":{"en":"A harness for evaluating multi-agent strategies on financial data tasks, comparing single, fanout, and plan-execute-review approaches with measured cost and confidence intervals."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:titusblair/agent-arena","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-skills-evolution","multi-agent-games","finance-legal"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_agent-arena_0df4c133"},"modality":null,"name":"agent-arena","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:titusblair/agent-arena","source_url":"https://github.com/titusblair/agent-arena"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-titusblair-agent-arena","source":"claire_radar","source_url":"https://github.com/titusblair/agent-arena","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:dkautomation23/agent-hijack-bench","source_record":"bm_agent-hijack-bench_b237d694"},"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Measures how often a local LLM agent obeys an instruction hidden in a website's llms.txt instead of the user's request, using a canary token and substring scoring.","descriptions":{"en":"Measures how often a local LLM agent obeys an instruction hidden in a website's llms.txt instead of the user's request, using a canary token and substring scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:dkautomation23/agent-hijack-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_agent-hijack-bench_b237d694"},"modality":null,"name":"agent-hijack-bench","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:dkautomation23/agent-hijack-bench","source_url":"https://github.com/dkautomation23/agent-hijack-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-dkautomation23-agent-hijack-bench","source":"claire_radar","source_url":"https://github.com/dkautomation23/agent-hijack-bench","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sentinelden/agent-injection-bench","source_record":"bm_agent-injection-bench_e2b88690"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates prompt-injection robustness of tool-calling agent stacks using a 24-scenario corpus (21 adversarial, 3 control) delivered through channels such as tool results, documents, web pages, filenames and multi-turn dialogue, with deterministic canary, tool-call, exfiltration and refusal success conditions; results report compromise and over-refusal rates per defense.","descriptions":{"en":"Evaluates prompt-injection robustness of tool-calling agent stacks using a 24-scenario corpus (21 adversarial, 3 control) delivered through channels such as tool results, documents, web pages, filenames and multi-turn dialogue, with deterministic canary, tool-call, exfiltration and refusal success conditions; results report compromise and over-refusal rates per defense."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:sentinelden/agent-injection-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_agent-injection-bench_e2b88690"},"modality":null,"name":"agent-injection-bench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:sentinelden/agent-injection-bench","source_url":"https://github.com/sentinelden/agent-injection-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-sentinelden-agent-injection-bench","source":"claire_radar","source_url":"https://github.com/sentinelden/agent-injection-bench","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ajbermudezh22/agent-model-bench","source_record":"bm_agent-model-bench_4106c088"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates language models on tool calling and strict JSON adherence using 32 cases, scoring exact tool and argument matches and JSON parsing with strict and content-based criteria.","descriptions":{"en":"Evaluates language models on tool calling and strict JSON adherence using 32 cases, scoring exact tool and argument matches and JSON parsing with strict and content-based criteria."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:ajbermudezh22/agent-model-bench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following","tool-use"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_agent-model-bench_4106c088"},"modality":null,"name":"agent-model-bench","openness":"unknown","publisher":null,"released":"2026-08-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:ajbermudezh22/agent-model-bench","source_url":"https://github.com/ajbermudezh22/agent-model-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-ajbermudezh22-agent-model-bench","source":"claire_radar","source_url":"https://github.com/ajbermudezh22/agent-model-bench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":null,"source_record":"bm_agentabstain_d94f5610"},"categories":["Agents","General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentAbstain is a paired-task benchmark for evaluating LLM agents' ability to abstain from acting in scenarios such as ambiguity, conflicting constraints, or tool failures. It includes 263 paired tasks across 42 sandbox environments, with a proposed pipeline for generating fresh task instances.","descriptions":{"en":"AgentAbstain is a paired-task benchmark for evaluating LLM agents' ability to abstain from acting in scenarios such as ambiguity, conflicting constraints, or tool failures. It includes 263 paired tasks across 42 sandbox environments, with a proposed pipeline for generating fresh task instances."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.10059","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","safety-alignment"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_agentabstain_d94f5610"},"modality":null,"name":"AgentAbstain","openness":"unknown","publisher":null,"released":"2026-07-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10059","source_url":"https://arxiv.org/abs/2607.10059"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10059","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10059","topics":["agents","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","NeurIPS 2024","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AgentBoard is tailored to analytical evaluation of LLM agents. It offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit that features easy assessment of agents for multi-faceted analysis through interactive visualization. AgentBoard专用于LLM Agent的分析评估，它提供了一个精细指标用于捕获增量进步，以及一个全面的评估工具包，能基于交互式可视化评估进行多方面分析。","descriptions":{"en":"AgentBoard is tailored to analytical evaluation of LLM agents. It offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit that features easy assessment of agents for multi-faceted analysis through interactive visualization.","zh":"AgentBoard专用于LLM Agent的分析评估，它提供了一个精细指标用于捕获增量进步，以及一个全面的评估工具包，能基于交互式可视化评估进行多方面分析。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1242","languages":[],"library_direction":null,"modality":null,"name":"AgentBoard","openness":"restricted","publisher":"The University of Hong Kong","released":"2024-06-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1242-agentboard","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgentBoard","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AGENTCHAOSBENCH is a dataset of sanitized execution traces from five agentic applications with injected runtime faults, used to evaluate fault detection and localization from telemetry.","descriptions":{"en":"AGENTCHAOSBENCH is a dataset of sanitized execution traces from five agentic applications with injected runtime faults, used to evaluate fault detection and localization from telemetry."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14680","languages":[],"library_direction":null,"modality":null,"name":"AGENTCHAOSBENCH","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14680","source_url":"https://arxiv.org/abs/2608.14680"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14680","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14680","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"AgentCIBench is an executable evaluation harness for testing whether computer-use agents improperly disclose information across application contexts.","descriptions":{"en":"AgentCIBench is an executable evaluation harness for testing whether computer-use agents improperly disclose information across application contexts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.23189","languages":[],"library_direction":null,"modality":null,"name":"AgentCIBench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23189","source_url":"https://arxiv.org/abs/2606.23189"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23189","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23189","topics":[],"unit":null},{"aliases":["Agent Cyber Range","ACR"],"attention_signals":null,"categories":["security"],"collected_at":null,"description":"Verified success depends on the WEB or POST track, information level and retry policy. WEB vulnerabilities and POST ranges have different scoring denominators; Pass@3 average and Pass@3 maximum are distinct statistics. Read the track and configuration alongside any percentage.","descriptions":{"en":"Verified success depends on the WEB or POST track, information level and retry policy. WEB vulnerabilities and POST ranges have different scoring denominators; Pass@3 average and Pass@3 maximum are distinct statistics. Read the track and configuration alongside any percentage."},"editorial_guide":{"id":"agentcyberrange","record_keys":["model-reports:agentcyberrange"],"reviewed_at":"2026-10-20","scope":{"en":"WebExploitBench and PostExploitBench as described in paper version 2 and the official project materials.","zh":"论文第二版及官方项目材料中的 WebExploitBench 与 PostExploitBench。"},"sections":[{"id":"overview","source_ids":["home","web_repo","post_repo"],"text":{"en":"AgentCyberRange separates web application testing from activity inside enterprise-like networks. WebExploitBench covers 110 vulnerabilities across 15 applications, combining known and synthetic cases. PostExploitBench provides eight ranges containing 156 internal hosts. These are two task families with different environments and scoring units, not one interchangeable leaderboard.","zh":"AgentCyberRange 将 Web 应用测试与企业式网络中的任务分开。WebExploitBench 在 15 个应用中设置 110 个漏洞，包含已知漏洞与合成案例；PostExploitBench 提供八个靶场，共有 156 台内部主机。两条轨道的环境与统计单位不同，不能视为可互换的单一榜单。"},"title":{"en":"Two complementary tracks","zh":"测什么"}},{"id":"tasks","source_ids":["paper","cage"],"text":{"en":"Three information levels vary how much help the agent receives, from an entry point alone to structural information and explicit weakness hints. The CAGE framework supplies the execution environment, recording and evaluation infrastructure. Both the chosen track and information level should accompany a reported result.","zh":"三个信息等级控制智能体获得的帮助，从只有入口信息，到增加结构信息，再到明确的弱点提示。CAGE 提供执行环境、过程记录和评测基础设施。展示结果时，应同时标注所选轨道与信息等级。"},"title":{"en":"Information levels and environment","zh":"任务如何设置"}},{"id":"evaluation","source_ids":["paper"],"text":{"en":"Verifiers check observable web outcomes or validated host access, including privilege distinctions. Pass@1 reports single-attempt success; Pass@3 (Avg.) averages three attempts; Pass@3 (Max) credits success in any attempt. Web results use vulnerability-level success, while the paper averages PostExploitBench metrics across eight ranges at each information level.","zh":"验证器检查可观察的 Web 结果或经验证的主机访问，并区分权限层级。Pass@1 表示单次尝试成功率；Pass@3（Avg.）取三次尝试的平均；Pass@3（Max）只要任意一次成功即计成功。Web 结果按漏洞统计，论文中的 PostExploitBench 指标则在每个信息等级内跨八个靶场取平均。"},"title":{"en":"Verified outcomes and repeated attempts","zh":"如何判分"}},{"id":"use_cases","source_ids":["cage","post_repo"],"text":{"en":"Use it when evaluating agents that must maintain context across services, hosts, and longer workflows. Its range setup and recorded trajectories support analysis of planning and tool use over time. Keep web-target findings separate from conclusions about progress through a network.","zh":"适合评估需要跨服务、跨主机维持上下文并完成较长流程的智能体。靶场设置和过程记录便于分析规划及工具使用的持续表现。对单个 Web 目标的结论，应与网络范围内的任务进展分开理解。"},"title":{"en":"When it is useful","zh":"适合用来判断什么"}},{"id":"limitations","source_ids":["web_repo","post_repo","paper"],"text":{"en":"The public GitHub repositories contain subsets, so a repository checkout does not by itself reproduce the complete evaluation. Preserve dataset versions, step limits, agent configuration, and the treatment of failed or excluded runs. A higher score under more informative hints does not demonstrate equal performance with less information.","zh":"公开 GitHub 仓库包含的是部分数据，仅获取仓库并不等于复现完整评测。比较时需保留数据版本、步数限制、智能体配置，以及失败或排除运行的处理方式。更多提示下获得较高分数，不代表在更少信息下具有同样表现。"},"title":{"en":"Reproduction and comparison limits","zh":"读分时的限制"}}],"sources":[{"id":"home","title":"AgentCyberRange official website","url":"https://agentcyberrange.io/"},{"id":"paper","title":"AgentCyberRange paper, version 2","url":"https://arxiv.org/html/2606.14295v2"},{"id":"web_repo","title":"WebExploitBench official repository","url":"https://github.com/AgentCyberRange/WebExploitBench"},{"id":"post_repo","title":"PostExploitBench official repository","url":"https://github.com/AgentCyberRange/PostExploitBench"},{"id":"cage","title":"CAGE benchmark integration documentation","url":"https://github.com/AgentCyberRange/CAGE/blob/main/examples/agent_pentest_bench/README.md"}],"summary":{"en":"AgentCyberRange evaluates agents in web applications and multi-host cyber ranges, with several levels of task information.","zh":"AgentCyberRange 在 Web 应用和多主机网络靶场中评估智能体，并设置不同程度的任务线索。"},"title":"AgentCyberRange"},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:agentcyberrange","languages":[],"library_direction":null,"modality":null,"name":"AgentCyberRange","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"agentcyberrange","source":"model_reports","source_url":"https://agentcyberrange.io/","topics":["agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"As Large Language Models (LLMs) gain tool use capabilities such as web search, they can retrieve and cross-reference public information, creating privacy risks beyond memorization.","descriptions":{"en":"As Large Language Models (LLMs) gain tool use capabilities such as web search, they can retrieve and cross-reference public information, creating privacy risks beyond memorization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.05586","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_agentdoxx_8b183406"},"modality":null,"name":"AgentDoxx","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05586","source_url":"https://arxiv.org/abs/2610.05586"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05586","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05586","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:asif-0209/agentdrift","source_record":"bm_agentdrift_563d3f60"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Step-labeled corpus of 12,536 synthetic LLM agent tool-call trajectories in five domains, labeled per step for benign, injection point, hijacked, or failed injection, supporting detection, localization, and attempt-vs-success evaluation.","descriptions":{"en":"Step-labeled corpus of 12,536 synthetic LLM agent tool-call trajectories in five domains, labeled per step for benign, injection point, hijacked, or failed injection, supporting detection, localization, and attempt-vs-success evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.06972","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_agentdrift_563d3f60"},"modality":null,"name":"AgentDrift","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06972","source_url":"https://arxiv.org/abs/2609.06972"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06972","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06972","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:rohithreddybc/agentfairbench","source_record":"bm_agentfairbench_1a8330ea"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentFairBench evaluates demographic disparity in the actions of LLM agents across hiring, lending, and medical triage. It uses synthetic, demographic-neutral profiles in counterfactual matched sets varying name-coded race/gender. Metrics include counterfactual flip rate, mean absolute score difference, action-rate disparity, and tool-invocation disparity, with bootstrap confidence intervals and FDR control.","descriptions":{"en":"AgentFairBench evaluates demographic disparity in the actions of LLM agents across hiring, lending, and medical triage. It uses synthetic, demographic-neutral profiles in counterfactual matched sets varying name-coded race/gender. Metrics include counterfactual flip rate, mean absolute score difference, action-rate disparity, and tool-invocation disparity, with bootstrap confidence intervals and FDR control."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.16723","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_agentfairbench_1a8330ea"},"modality":null,"name":"AgentFairBench","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16723","source_url":"https://arxiv.org/abs/2606.16723"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16723","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16723","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways.","descriptions":{"en":"Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32795","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_agenthabit_ad141d87"},"modality":null,"name":"AgentHabit","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32795","source_url":"https://arxiv.org/abs/2609.32795"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32795","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32795","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","智能体","Agent","任务执行","Task Execution","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AgentHarm tests the robustness of LLMs to jailbreak attacks. It includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment. AgentHarm用于评估LLM智能体对越狱攻击的鲁棒性，包括110套恶意智能体任务（其中有440个强化任务），涵盖欺诈、网络犯罪和骚扰等11个危害类别。","descriptions":{"en":"AgentHarm tests the robustness of LLMs to jailbreak attacks. It includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment.","zh":"AgentHarm用于评估LLM智能体对越狱攻击的鲁棒性，包括110套恶意智能体任务（其中有440个强化任务），涵盖欺诈、网络犯罪和骚扰等11个危害类别。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1351","languages":[],"library_direction":null,"modality":null,"name":"AgentHarm","openness":"restricted","publisher":"Gray Swan AI","released":"2024-10-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1351-agentharm","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgentHarm","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","安全","Safety","智能体","Agent","任务执行","Task Execution","安全对齐","Safety Alignment","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"移动端 GUI Agent 通过与设备环境的交互来完成任务，在完成任务的过程中会遇到一些未知或不可信的信息来源，这些信息可能含有攻击性的内容，致使 Agent 无法正常完成任务，甚至对用户的隐私和财产带来危害。本评测集兼具动态执行环境和静态评测数据集，旨在为移动端 GUI Agent 提供一个仿真度高的模拟环境，以评估其在真实场景下执行的行为和安全性。 移动端 GUI Agent 通过与设备环境的交互来完成任务，在完成任务的过程中会遇到一些未知或不可信的信息来源，这些信息可能含有攻击性的内容，致使 Agent 无法正常完成任务，甚至对用户的隐私和财产带来危害。本评测集兼具动态执行环境和静态评测数据集，旨在为移动端 GUI Agent 提供一个仿真度高的模拟环境，以评估其在真实场景下执行的行为和安全性。","descriptions":{"en":"移动端 GUI Agent 通过与设备环境的交互来完成任务，在完成任务的过程中会遇到一些未知或不可信的信息来源，这些信息可能含有攻击性的内容，致使 Agent 无法正常完成任务，甚至对用户的隐私和财产带来危害。本评测集兼具动态执行环境和静态评测数据集，旨在为移动端 GUI Agent 提供一个仿真度高的模拟环境，以评估其在真实场景下执行的行为和安全性。","zh":"移动端 GUI Agent 通过与设备环境的交互来完成任务，在完成任务的过程中会遇到一些未知或不可信的信息来源，这些信息可能含有攻击性的内容，致使 Agent 无法正常完成任务，甚至对用户的隐私和财产带来危害。本评测集兼具动态执行环境和静态评测数据集，旨在为移动端 GUI Agent 提供一个仿真度高的模拟环境，以评估其在真实场景下执行的行为和安全性。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2061","languages":[],"library_direction":null,"modality":"multimodal","name":"AgentHazard","openness":"unknown","publisher":"Institute for AI Industry Research, Tsinghua University","released":"2025-07-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2061-agenthazard","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgentHazard","topics":["agents","multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":7,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:tmlr-group/agenthijack","source_record":"bm_agenthijack_15cbe4d0"},"categories":["Agents","Agents & Tool Use","General AI","Robustness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates the robustness of computer use agents under common environment corruptions such as pop-ups, resolution changes, and competing applications. The benchmark introduces 9 configurable corruptions and evaluates agent performance on desktop tasks using multimodal LLM-based agents, measuring task completion rates.","descriptions":{"en":"Evaluates the robustness of computer use agents under common environment corruptions such as pop-ups, resolution changes, and competing applications. The benchmark introduces 9 configurable corruptions and evaluates agent performance on desktop tasks using multimodal LLM-based agents, measuring task completion rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.25707","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_agenthijack_15cbe4d0"},"modality":null,"name":"AgentHijack","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25707","source_url":"https://arxiv.org/abs/2605.25707"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25707","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25707","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_agenthop_3f8b559c"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources.","descriptions":{"en":"Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34428","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_agenthop_3f8b559c"},"modality":null,"name":"AgentHop","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34428","source_url":"https://arxiv.org/abs/2609.34428"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34428","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34428","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_agenthorizon_0a2586b7"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Computer-use agents are capable of completing complex tasks, increasing the use of automatic judges to determine success, either for training or for evaluation without human involvement.","descriptions":{"en":"Computer-use agents are capable of completing complex tasks, increasing the use of automatic judges to determine success, either for training or for evaluation without human involvement."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11050","languages":[],"library_direction":{"basis":"model-review-daily","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_agenthorizon_0a2586b7"},"modality":null,"name":"AgentHorizon","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11050","source_url":"https://arxiv.org/abs/2610.11050"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11050","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11050","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentHPOBench evaluates LLM agents as sequential hyperparameter optimizers across 30 executable ML tasks. Agents observe accumulated configurations, metrics, and logs, then propose the next configuration. Scoring compares agents and conventional HPO baselines under a unified protocol.","descriptions":{"en":"AgentHPOBench evaluates LLM agents as sequential hyperparameter optimizers across 30 executable ML tasks. Agents observe accumulated configurations, metrics, and logs, then propose the next configuration. Scoring compares agents and conventional HPO baselines under a unified protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.29626","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_agenthpobench_61915136"},"modality":null,"name":"AgentHPOBench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29626","source_url":"https://arxiv.org/abs/2607.29626"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29626","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29626","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bluebunnyanon/agent-benchmark-2d-maze","source_record":"bm_agent-benchmark-2d-maze_4d511c00"},"categories":["Agents","Agents & Tool Use","Multimodal","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates interactive multimodal agents in structured 2D maze environments requiring long-horizon planning, mechanism interaction, and error recovery, scored by mechanism-aware progress.","descriptions":{"en":"Evaluates interactive multimodal agents in structured 2D maze environments requiring long-horizon planning, mechanism interaction, and error recovery, scored by mechanism-aware progress."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:bluebunnyanon/agent-benchmark-2d-maze","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_agent-benchmark-2d-maze_4d511c00"},"modality":null,"name":"Agentic Evaluation in 2D Mazes","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:bluebunnyanon/agent-benchmark-2d-maze","source_url":"https://github.com/bluebunnyanon/agent-benchmark-2d-maze"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-bluebunnyanon-agent-benchmark-2d-maze","source":"claire_radar","source_url":"https://github.com/bluebunnyanon/agent-benchmark-2d-maze","topics":["agents","multimodal"],"unit":null},{"aliases":["agentic-infrastructure-change-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aah20/agentic-infrastructure-change-benchmark","source_record":"bm_agentic-infrastructure-change-benchmark_124c16f4"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agent proposals for cloud, Kubernetes, and network infrastructure changes across correctness, safety, reliability, economics, efficiency, and evidence using deterministic scenario-based checks.","descriptions":{"en":"Evaluates AI agent proposals for cloud, Kubernetes, and network infrastructure changes across correctness, safety, reliability, economics, efficiency, and evidence using deterministic scenario-based checks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:aah20/agentic-infrastructure-change-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_agentic-infrastructure-change-benchmark_124c16f4"},"modality":null,"name":"Agentic Infrastructure ChangeBench","openness":"unknown","publisher":null,"released":"2026-09-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:aah20/agentic-infrastructure-change-benchmark","source_url":"https://github.com/AAH20/agentic-infrastructure-change-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-aah20-agentic-infrastructure-change-benchmark","source":"claire_radar","source_url":"https://github.com/AAH20/agentic-infrastructure-change-benchmark","topics":["finance"],"unit":null},{"aliases":["agentic-netdevops-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:andersballegaard/agentic-netdevops-benchmark","source_record":"bm_agentic-netdevops-benchmark_23670ccd"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A multi-stage environment with a Linux management host and five VyOS routers, requiring agents to complete netdevops tasks from information gathering to evaluation.","descriptions":{"en":"A multi-stage environment with a Linux management host and five VyOS routers, requiring agents to complete netdevops tasks from information gathering to evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:andersballegaard/agentic-netdevops-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_agentic-netdevops-benchmark_23670ccd"},"modality":null,"name":"Agentic NetDevOps Benchmark","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:andersballegaard/agentic-netdevops-benchmark","source_url":"https://github.com/AndersBallegaard/agentic-netdevops-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-andersballegaard-agentic-netdevops-benchmark","source":"claire_radar","source_url":"https://github.com/AndersBallegaard/agentic-netdevops-benchmark","topics":[],"unit":null},{"aliases":["agentic-security-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:gitayg/agentic-security-benchmark","source_record":"bm_agentic-security-benchmark_9165db66"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Measures detection and prevention of agentic-AI security products on 286 attack and 875 benign samples across five AMTSO attack vectors via a scoring adapter harness.","descriptions":{"en":"Measures detection and prevention of agentic-AI security products on 286 attack and 875 benign samples across five AMTSO attack vectors via a scoring adapter harness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:gitayg/agentic-security-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_agentic-security-benchmark_9165db66"},"modality":null,"name":"Agentic Security Benchmark","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:gitayg/agentic-security-benchmark","source_url":"https://github.com/gitayg/agentic-security-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-gitayg-agentic-security-benchmark","source":"claire_radar","source_url":"https://github.com/gitayg/agentic-security-benchmark","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":"gh:lamda-bbo/agentic-bbo","source_record":"bm_agenticbbo-bench_7edbb260"},"categories":["Chip Design & EDA","General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensive and limited.","descriptions":{"en":"Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensive and limited."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.12183","languages":[],"library_direction":{"basis":"model-review-daily","categories":["ai-rnd","science"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_agenticbbo-bench_7edbb260"},"modality":null,"name":"AgenticBBO-Bench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12183","source_url":"https://arxiv.org/abs/2610.12183"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12183","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12183","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":43,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":35,"join":"claire_record_id","repository":"gh:agenticdatabench/agenticdatabench","source_record":"bm_agenticdatabench_3427dc68"},"categories":["General AI","Language & Knowledge","cs.DB"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM-based data agents on realistic data science workflows across 15 domains, with fine-grained ground-truth labels and skill-level scoring.","descriptions":{"en":"Evaluates LLM-based data agents on realistic data science workflows across 15 domains, with fine-grained ground-truth labels and skill-level scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.01647","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_agenticdatabench_3427dc68"},"modality":null,"name":"AgenticDataBench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01647","source_url":"https://arxiv.org/abs/2607.01647"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01647","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01647","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_agenticinterpbench_4e5baf78"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates language model agents on explaining components of transformer circuits, with 84 semi-synthetic circuits and 163 component-level annotations.","descriptions":{"en":"Evaluates language model agents on explaining components of transformer circuits, with 84 semi-synthetic circuits and 163 component-level annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.24026","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","science"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_agenticinterpbench_4e5baf78"},"modality":null,"name":"AgenticInterpBench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24026","source_url":"https://arxiv.org/abs/2606.24026"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24026","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24026","topics":["science"],"unit":null},{"aliases":["AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents"],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"AgenticSTS is a reproducible long-horizon agent testbed that evaluates bounded, typed-retrieval memory configurations through game outcomes and controlled ablations.","descriptions":{"en":"AgenticSTS is a reproducible long-horizon agent testbed that evaluates bounded, typed-retrieval memory configurations through game outcomes and controlled ablations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02255","languages":[],"library_direction":null,"modality":null,"name":"AgenticSTS","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02255","source_url":"https://arxiv.org/abs/2607.02255"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02255","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02255","topics":["agents"],"unit":null},{"aliases":["ai-agent-infrastructure-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aah20/ai-agent-infrastructure-benchmark","source_record":"bm_ai-agent-infrastructure-benchmark_aa5659e0"},"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agent infrastructure across isolation, identity, network, authority, durability, observability, and unit economics using weighted scores with hard safety gates.","descriptions":{"en":"Evaluates AI agent infrastructure across isolation, identity, network, authority, durability, observability, and unit economics using weighted scores with hard safety gates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:aah20/ai-agent-infrastructure-benchmark","languages":[],"library_direction":{"basis":"model-review-2","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_ai-agent-infrastructure-benchmark_aa5659e0"},"modality":null,"name":"AgentInfraBench","openness":"unknown","publisher":null,"released":"2026-09-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:aah20/ai-agent-infrastructure-benchmark","source_url":"https://github.com/AAH20/ai-agent-infrastructure-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-aah20-ai-agent-infrastructure-benchmark","source":"claire_radar","source_url":"https://github.com/AAH20/ai-agent-infrastructure-benchmark","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"claire_record_id","repository":null,"source_record":"bm_agentjudgebench_8ae8da3e"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined.","descriptions":{"en":"LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.26623","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_agentjudgebench_8ae8da3e"},"modality":null,"name":"AgentJudgeBench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26623","source_url":"https://arxiv.org/abs/2608.26623"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26623","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26623","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":10,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"claire_record_id","repository":"gh:agent-lens/agent-lens-bench","source_record":"bm_agentlens_07b9cd48"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentLens evaluates interactive coding agents across entire trajectories, combining formal verification with LLM-written trajectory reviews and side-by-side comparisons to score dimensions such as instruction compliance, tool use, and interaction style.","descriptions":{"en":"AgentLens evaluates interactive coding agents across entire trajectories, combining formal verification with LLM-written trajectory reviews and side-by-side comparisons to score dimensions such as instruction compliance, tool use, and interaction style."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.06624","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_agentlens_07b9cd48"},"modality":null,"name":"AgentLens","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06624","source_url":"https://arxiv.org/abs/2607.06624"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06624","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06624","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentMemBench evaluates five long-term memory management strategies for conversational AI agents across three public datasets (LoCoMo, MultiDoc2Dial, MSC), covering multi-session dialogue, document grounding, and persona-grounded chat. Scoring uses retrieval metrics (Recall@k, MRR, nDCG@k), Answer F1, LLM-judge faithfulness, memory footprint, and latency over 491 annotated question turns.","descriptions":{"en":"AgentMemBench evaluates five long-term memory management strategies for conversational AI agents across three public datasets (LoCoMo, MultiDoc2Dial, MSC), covering multi-session dialogue, document grounding, and persona-grounded chat. Scoring uses retrieval metrics (Recall@k, MRR, nDCG@k), Answer F1, LLM-judge faithfulness, memory footprint, and latency over 491 annotated question turns."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00009","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_agentmembench_9938dbed"},"modality":null,"name":"AgentMemBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00009","source_url":"https://arxiv.org/abs/2608.00009"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00009","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00009","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":20,"join":"claire_record_id","repository":null,"source_record":"bm_agentmonbench_29d03d58"},"categories":["Agents","Agents & Tool Use","General AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"As agents take on long-horizon tasks, users shift from making individual decisions to overseeing autonomous execution.","descriptions":{"en":"As agents take on long-horizon tasks, users shift from making individual decisions to overseeing autonomous execution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.06406","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_agentmonbench_29d03d58"},"modality":null,"name":"AgentMonBench","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.06406","source_url":"https://arxiv.org/abs/2610.06406"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-06406","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06406","topics":["agents","safety"],"unit":null},{"aliases":["aa-agentperf-local"],"attention_signals":{"as_of":"2026-10-11","github_stars":91,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:artificialanalysis/aa-agentperf-local","source_record":"bm_aa-agentperf-local_b8ee53bf"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Replays recorded agent conversations against an OpenAI-compatible model server and reports throughput and latency, with a default comparable replay of eight recorded agent tasks and 168 model turns.","descriptions":{"en":"Replays recorded agent conversations against an OpenAI-compatible model server and reports throughput and latency, with a default comparable replay of eight recorded agent tasks and 168 model turns."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:artificialanalysis/aa-agentperf-local","languages":[],"library_direction":null,"modality":null,"name":"agentperf-local","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:artificialanalysis/aa-agentperf-local","source_url":"https://github.com/ArtificialAnalysis/aa-agentperf-local"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-artificialanalysis-aa-agentperf-local","source":"claire_radar","source_url":"https://github.com/ArtificialAnalysis/aa-agentperf-local","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior.","descriptions":{"en":"We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.04379","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_agentpersonabench_cb57cfe4"},"modality":null,"name":"AgentPersonaBench","openness":"unknown","publisher":null,"released":"2026-10-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.04379","source_url":"https://arxiv.org/abs/2610.04379"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-04379","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04379","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentRedBench evaluates LLM agents against indirect prompt injection and underspecified-authorization attacks across 24 enterprise SaaS integrations. It defines 215 attack scenarios with immutable versioning, and tracks attack success rate (ASR) for models and defenses. Open-source codebase and schemas enable replay.","descriptions":{"en":"AgentRedBench evaluates LLM agents against indirect prompt injection and underspecified-authorization attacks across 24 enterprise SaaS integrations. It defines 215 attack scenarios with immutable versioning, and tracks attack success rate (ASR) for models and defenses. Open-source codebase and schemas enable replay."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02240","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_agentredbench_2025428a"},"modality":null,"name":"AgentRedBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02240","source_url":"https://arxiv.org/abs/2606.02240"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02240","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02240","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:shivenkk/agentrelbench","source_record":"bm_agentrelbench_829efc3a"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentRelBench measures repeated agent runs in a fixed task suite, computing severity-weighted damage from database state diffs with no LLM judging.","descriptions":{"en":"AgentRelBench measures repeated agent runs in a fixed task suite, computing severity-weighted damage from database state diffs with no LLM judging."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.15286","languages":[],"library_direction":{"basis":"model-review-2","categories":["agent-security","tool-use"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_agentrelbench_829efc3a"},"modality":null,"name":"AgentRelBench","openness":"unknown","publisher":null,"released":"2026-08-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15286","source_url":"https://arxiv.org/abs/2608.15286"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15286","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15286","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents. AgentRewardBench contains 1302 trajectories across 5 benchmarks and 4 LLMs. AgentRewardBench 是首个用于评估大型语言模型（LLM）评判者评估网络代理有效性的基准测试。AgentRewardBench 包含来自 5 个基准测试和 4 个大型语言模型的 1302 条轨迹。","descriptions":{"en":"AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents. AgentRewardBench contains 1302 trajectories across 5 benchmarks and 4 LLMs.","zh":"AgentRewardBench 是首个用于评估大型语言模型（LLM）评判者评估网络代理有效性的基准测试。AgentRewardBench 包含来自 5 个基准测试和 4 个大型语言模型的 1302 条轨迹。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1778","languages":[],"library_direction":null,"modality":null,"name":"AgentRewardBench","openness":"restricted","publisher":"McGill University，Mila Quebec AI Institute，etc.","released":"2025-04-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1778-agentrewardbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgentRewardBench","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1094,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":393,"join":"llm_stats_id","repository":null,"source_record":"catalog_d0b03ebbcb611ba0"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Agents' Last Exam is a challenging benchmark for AI agents on hard, long-horizon tasks that test sustained reasoning, planning, and tool use, reported with and without tool access.","descriptions":{"en":"Agents' Last Exam is a challenging benchmark for AI agents on hard, long-horizon tasks that test sustained reasoning, planning, and tool use, reported with and without tool access."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:agents-last-exam:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agents-last-exam?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:agents-last-exam","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_d0b03ebbcb611ba0"},"modality":"text","name":"Agents' Last Exam","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":52.7,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.527,"raw_min":0.252,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:agents-last-exam:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agents-last-exam?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0731"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"}],"slug":"llm-stats-agents-last-exam","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agents-last-exam?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["Agents' Last Exam","ALE","Agent's Last Exam","ALE-CLI"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Multi-step agentic benchmark using Claude Code harness. Tool Search disabled. Scores depend on harness, reasoning effort, context length, and timeout settings.","descriptions":{"en":"Multi-step agentic benchmark using Claude Code harness. Tool Search disabled. Scores depend on harness, reasoning effort, context length, and timeout settings."},"evidence_summary":{"document_count":3,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000agents_last_exam\u0000zai_glm_5_3_flash_model_card\u0000agents_last_exam\u0000official evaluation protocol, Claude Code harness, reasoning effort=max, 1M context, 64K max output, Tool Search disabled, scored by official ALE evaluators\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000agents_last_exam\u0000zai_glm_5_3_flash_model_card\u0000agents_last_exam\u0000official evaluation protocol, Claude Code harness, reasoning effort=max, 1M context, 64K max output, Tool Search disabled, scored by official ALE evaluators\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:agents_last_exam","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"Agents' Last Exam","openness":"unknown","publisher":null,"released":"2025-01-23","released_reference":{"basis":"benchmark_release","source_key":"model-reports:agents_last_exam","source_url":"https://lastexam.ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":31.8,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":31.8,"raw_min":22.8,"source_reference":{"obs_id":"curated\u0000agents_last_exam\u0000deepseek_v4_1_flash_technical_report\u0000agents_last_exam\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000agents_last_exam\u0000deepseek_v4_1_flash_technical_report\u0000agents_last_exam\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"agents_last_exam","source":"model_reports","source_url":"https://lastexam.ai/","topics":["coding","agents"],"unit":"percent"},{"aliases":["Agents' Last Exam"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A living benchmark of 1K+ long-horizon professional tasks with verifiable outcomes, covering 55 sub-fields in 13 industry clusters based on O*NET / SOC 2018.","descriptions":{"en":"A living benchmark of 1K+ long-horizon professional tasks with verifiable outcomes, covering 55 sub-fields in 13 industry clusters based on O*NET / SOC 2018."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.05405","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"artifact:arxiv:2606.05405","primary":"professional-work","source_record":"catalog_24fd993b1ae5c226"},"modality":null,"name":"Agents' Last Exam (ALE)","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05405","source_url":"https://arxiv.org/abs/2606.05405"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05405","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05405","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentS4D evaluates runtime safety of LLM-based workspace agents across a four-dimensional framework, with 328 risk-injected cases and seven lifecycle checkpoints, measuring unsafe behavior and evidence across six risk-entry sources and nine harms.","descriptions":{"en":"AgentS4D evaluates runtime safety of LLM-based workspace agents across a four-dimensional framework, with 328 risk-injected cases and seven lifecycle checkpoints, measuring unsafe behavior and evidence across six risk-entry sources and nine harms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27294","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","safety-alignment"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_agents4d_33599565"},"modality":null,"name":"AgentS4D","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27294","source_url":"https://arxiv.org/abs/2607.27294"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27294","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27294","topics":["agents","safety"],"unit":null},{"aliases":["AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints"],"attention_signals":null,"categories":["Agents & Tool Use","Software & AI Compute","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A resource-aware agent benchmark with 16 Dockerized task environments that profiles correctness alongside latency, cost, compute, memory, and network use under declared budgets.","descriptions":{"en":"A resource-aware agent benchmark with 16 Dockerized task environments that profiles correctness alongside latency, cost, compute, memory, and network use under declared budgets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.00805","languages":[],"library_direction":null,"modality":null,"name":"AgentSLABench","openness":"unknown","publisher":null,"released":"2026-08-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00805","source_url":"https://arxiv.org/abs/2608.00805"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00805","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00805","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":35,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:vectorspacelab/agentswe","source_record":"bm_agentswe_fa551309"},"categories":["Agents","Code","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows.","descriptions":{"en":"As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.13361","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_agentswe_fa551309"},"modality":null,"name":"AgentSWE","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13361","source_url":"https://arxiv.org/abs/2610.13361"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13361","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13361","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.OS"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentSysBench is a benchmark suite and measurement toolkit for characterizing agentic workloads on LLM serving systems. It includes ten representative agentic applications and unified instrumentation, identifying six properties that distinguish agentic workloads from conventional LLM inference.","descriptions":{"en":"AgentSysBench is a benchmark suite and measurement toolkit for characterizing agentic workloads on LLM serving systems. It includes ten representative agentic applications and unified instrumentation, identifying six properties that distinguish agentic workloads from conventional LLM inference."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.15127","languages":[],"library_direction":null,"modality":null,"name":"AgentSysBench","openness":"unknown","publisher":null,"released":"2026-08-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15127","source_url":"https://arxiv.org/abs/2608.15127"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15127","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15127","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:kagnlp/agenttell","source_record":"bm_agenttell_e5c724ea"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Browser-use agents often carry information in their context as they move between websites.","descriptions":{"en":"Browser-use agents often carry information in their context as they move between websites."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.32915","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","computer-use"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_agenttell_e5c724ea"},"modality":null,"name":"AgentTell","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32915","source_url":"https://arxiv.org/abs/2609.32915"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32915","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32915","topics":["computer-use","agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":6,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:michaelofengenden/agenttimebench","source_record":"bm_agenttime_d0a06ec2"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"An essential control of AI agents is their ability to manage runtime.","descriptions":{"en":"An essential control of AI agents is their ability to manage runtime."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.09944","languages":[],"library_direction":{"basis":"model-review-daily","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_agenttime_d0a06ec2"},"modality":null,"name":"AgentTime","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.09944","source_url":"https://arxiv.org/abs/2610.09944"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-09944","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09944","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":582,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:krafton-ai/agentvidbench","source_record":"bm_agentvidbench_4fc908af"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates MLLM agents on 100 multi-hop video question answering tasks with 26 answer choices per question, requiring spatial, temporal, and causal reasoning over video content.","descriptions":{"en":"Evaluates MLLM agents on 100 multi-hop video question answering tasks with 26 answer choices per question, requiring spatial, temporal, and causal reasoning over video content."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.21386","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_agentvidbench_4fc908af"},"modality":null,"name":"AgentVidBench","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21386","source_url":"https://arxiv.org/abs/2609.21386"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21386","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21386","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1028,"github_stars_30d":27,"hf_dataset_downloads":976,"hf_paper_upvotes":167,"join":"claire_record_id","repository":"gh:qwenlm/qwen-agentworld","source_record":"bm_agentworldbench_775e3273"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentWorldBench evaluates language world models on simulation fidelity across 7 domains (MCP, Search, Terminal, SWE, Android, Web, OS) using real-world trajectories and rubric-based scoring.","descriptions":{"en":"AgentWorldBench evaluates language world models on simulation fidelity across 7 domains (MCP, Search, Terminal, SWE, Android, Web, OS) using real-world trajectories and rubric-based scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.24597","languages":[],"library_direction":{"basis":"model-review","categories":["world-models"],"join":"claire_record_id","primary":"world-models","source_record":"bm_agentworldbench_775e3273"},"modality":null,"name":"AgentWorldBench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24597","source_url":"https://arxiv.org/abs/2606.24597"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24597","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24597","topics":[],"unit":null},{"aliases":["AGI Maze Prediction Datasets: A Compact Benchmark for Learning World Dynamics with Transformers"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates predictive models on maze prediction tasks to study world dynamics learning.","descriptions":{"en":"Evaluates predictive models on maze prediction tasks to study world dynamics learning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02339","languages":[],"library_direction":null,"modality":null,"name":"AGI Maze Prediction Datasets and Benchmark","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02339","source_url":"https://arxiv.org/abs/2609.02339"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02339","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02339","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":776,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:ruixiangcui/agieval","source_record":"catalog_c100bd50f9e69a95"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A human-centric benchmark for evaluating foundation models on standardized exams including college entrance exams (Gaokao, SAT), law school admission tests (LSAT), math competitions, lawyer qualification tests, and civil service exams. Contains 20 tasks (18 multiple-choice, 2 cloze) designed to assess understanding, knowledge, reasoning, and calculation abilities in real-world academic and professional contexts.","descriptions":{"en":"A human-centric benchmark for evaluating foundation models on standardized exams including college entrance exams (Gaokao, SAT), law school admission tests (LSAT), math competitions, lawyer qualification tests, and civil service exams. Contains 20 tasks (18 multiple-choice, 2 cloze) designed to assess understanding, knowledge, reasoning, and calculation abilities in real-world academic and professional contexts."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:agieval:gemma-2-27b-it","reported_at":"2024-06-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agieval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:agieval","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_c100bd50f9e69a95"},"modality":"text","name":"AGIEval","openness":"unknown","publisher":null,"released":"2023-04-13","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:agieval","source_url":"https://arxiv.org/abs/2304.06364"},"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":65.8,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.658,"raw_min":0.285,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:agieval:mistral-small-24b-base-2501","reported_date":"2025-01-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agieval?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Base 2512)"},{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Base"}],"slug":"llm-stats-agieval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agieval?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["AGIEval"],"attention_signals":{"as_of":"2026-10-13","github_stars":776,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"artifact:gh:ruixiangcui/agieval","repository":"gh:ruixiangcui/agieval","source_record":"catalog_05a3b0ef55120ee4"},"categories":["knowledge"],"collected_at":null,"description":"Human-exam derived; overlaps heavily with MMLU-style coverage.","descriptions":{"en":"Human-exam derived; overlaps heavily with MMLU-style coverage."},"evidence_summary":{"document_count":4,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:agieval","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["knowledge-expert-qa"],"join":"review:14H034160212:2026-10-15","primary":"knowledge-expert-qa","source_record":null},"modality":null,"name":"AGIEval","openness":"unknown","publisher":null,"released":"2023-04-13","released_reference":{"basis":"benchmark_release","source_key":"model-reports:agieval","source_url":"https://github.com/ruixiangcui/AGIEval"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"agieval","source":"model_reports","source_url":"https://github.com/ruixiangcui/AGIEval","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":776,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"artifact:arxiv:2304.06364","repository":"gh:ruixiangcui/agieval","source_record":"catalog_05a3b0ef55120ee4"},"categories":["学科","Examination","大语言模型","LLM","知识储备","Knowledge","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving. This benchmark is derived from 20 official, public, and high-standard admission and qualification exams intended for general human test-takers, such as general college admission tests (e.g., Chinese College Entrance Exam (Gaokao) and American SAT), law school admission tests, math competitions, lawyer qualification tests, and national civil service exams. AGIEval是一个以人为中心的基准测试，专门设计用于评估基础模型在涉及人类认知和问题解决的任务中的一般能力。该基准测试源自20个官方、公开和高标准的入学和资格考试，例如普通大学入学考试（例如中国高考和美国SAT）、法学院入学考试、数学竞赛、律师资格考试以及国家公务员考试","descriptions":{"en":"AGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving. This benchmark is derived from 20 official, public, and high-standard admission and qualification exams intended for general human test-takers, such as general college admission tests (e.g., Chinese College Entrance Exam (Gaokao) and American SAT), law school admission tests, math competitions, lawyer qualification tests, and national civil service exams.","zh":"AGIEval是一个以人为中心的基准测试，专门设计用于评估基础模型在涉及人类认知和问题解决的任务中的一般能力。该基准测试源自20个官方、公开和高标准的入学和资格考试，例如普通大学入学考试（例如中国高考和美国SAT）、法学院入学考试、数学竞赛、律师资格考试以及国家公务员考试"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:497","languages":["Chinese"],"library_direction":null,"modality":null,"name":"AGIEval","openness":"unknown","publisher":null,"released":"2023-09-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-497-agieval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AGIEval","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":16,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":28,"join":"claire_record_id","repository":"gh:vita-group/agingbench","source_record":"bm_agingbench_6fa9d4b4"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates the reliability of deployed AI agents over extended operational lifetimes. The benchmark organizes agent aging into four mechanisms—compression, interference, revision, and maintenance—and uses temporal dependency graphs and paired counterfactual probes to produce diagnostic profiles of the memory pipeline's write, retrieval, and utilization stages. It includes multiple scenarios and memory policies, with scoring based on task performance over sessions.","descriptions":{"en":"Evaluates the reliability of deployed AI agents over extended operational lifetimes. The benchmark organizes agent aging into four mechanisms—compression, interference, revision, and maintenance—and uses temporal dependency graphs and paired counterfactual probes to produce diagnostic profiles of the memory pipeline's write, retrieval, and utilization stages. It includes multiple scenarios and memory policies, with scoring based on task performance over sessions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26302","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_agingbench_6fa9d4b4"},"modality":null,"name":"AgingBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26302","source_url":"https://arxiv.org/abs/2605.26302"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26302","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26302","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","农业","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A Comprehensive Agricultural Multimodal Understanding and Reasoning Benchmark 农业综合多模态理解和推理基准。","descriptions":{"en":"A Comprehensive Agricultural Multimodal Understanding and Reasoning Benchmark","zh":"农业综合多模态理解和推理基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1753","languages":[],"library_direction":null,"modality":"multimodal","name":"AgMMU","openness":"unknown","publisher":"Rice University，etc.","released":"2025-04-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1753-agmmu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgMMU","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":null,"source_record":"bm_agora_fee07ca2"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Agora evaluates agentic document reasoning across eight domain collections of 9,664 authentic workplace documents. It includes 362 questions requiring location of sparse evidence and reconciliation of terminology, units, and time conventions. The benchmark is designed to exceed model context windows, necessitating deliberate exploration.","descriptions":{"en":"Agora evaluates agentic document reasoning across eight domain collections of 9,664 authentic workplace documents. It includes 362 questions requiring location of sparse evidence and reconciliation of terminology, units, and time conventions. The benchmark is designed to exceed model context windows, necessitating deliberate exploration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.24526","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","document-ocr"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_agora_fee07ca2"},"modality":null,"name":"AGORA","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24526","source_url":"https://arxiv.org/abs/2606.24526"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24526","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24526","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AGOS-Bench evaluates VLMs on the urban Air-Ground Object Search task where a UAV and UGV coordinate to locate a target vehicle from multi-view visual references, reporting success rate and success weighted by path length.","descriptions":{"en":"AGOS-Bench evaluates VLMs on the urban Air-Ground Object Search task where a UAV and UGV coordinate to locate a target vehicle from multi-view visual references, reporting success rate and success weighted by path length."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08402","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_agos-bench_29c4a49d"},"modality":null,"name":"AGOS-Bench","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08402","source_url":"https://arxiv.org/abs/2609.08402"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08402","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08402","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":140,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_agrotools_ca69be78"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"AgroTools evaluates tool-augmented multimodal agents in agriculture, with 539 QA instances, 1,097 images, 14 executable tools, and structured tool-use traces for process and outcome evaluation.","descriptions":{"en":"AgroTools evaluates tool-augmented multimodal agents in agriculture, with 539 QA instances, 1,097 images, 14 executable tools, and structured tool-use traces for process and outcome evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22366","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning","tool-use"],"join":"claire_record_id","primary":"science","source_record":"bm_agrotools_ca69be78"},"modality":null,"name":"AgroTools","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22366","source_url":"https://arxiv.org/abs/2605.22366"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22366","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22366","topics":["agents","multimodal","science"],"unit":null},{"aliases":["AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"AgroVG is an agricultural visual grounding benchmark of 10,071 image-query pairs from ten source datasets. It covers box and instance-mask grounding, including queries with no target present.","descriptions":{"en":"AgroVG is an agricultural visual grounding benchmark of 10,071 image-query pairs from ten source datasets. It covers box and instance-mask grounding, including queries with no target present."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22034","languages":[],"library_direction":null,"modality":null,"name":"AgroVG","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22034","source_url":"https://arxiv.org/abs/2605.22034"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22034","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22034","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"AGVBench compares vein-recognition augmentation strategies across datasets and model families using accuracy, calibration, and adversarial-robustness considerations.","descriptions":{"en":"AGVBench compares vein-recognition augmentation strategies across datasets and model families using accuracy, calibration, and adversarial-robustness considerations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.02271","languages":[],"library_direction":null,"modality":null,"name":"AGVBench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02271","source_url":"https://arxiv.org/abs/2607.02271"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02271","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02271","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aiondadotcom/ai-sim-benchmark","source_record":"bm_ai-sim-benchmark_0f263e79"},"categories":["Agents","Agents & Tool Use","Geometric reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates autonomous coding agents on a reproducible 3D water-simulation challenge, measuring maximum capability through correctness of the generated simulation, with interventions counted against each run.","descriptions":{"en":"Evaluates autonomous coding agents on a reproducible 3D water-simulation challenge, measuring maximum capability through correctness of the generated simulation, with interventions counted against each run."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:aiondadotcom/ai-sim-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding","spatial-3d"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_ai-sim-benchmark_0f263e79"},"modality":null,"name":"AI Coding Agent Water Simulation Benchmark","openness":"unknown","publisher":null,"released":"2026-08-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:aiondadotcom/ai-sim-benchmark","source_url":"https://github.com/AiondaDotCom/ai-sim-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-aiondadotcom-ai-sim-benchmark","source":"claire_radar","source_url":"https://github.com/AiondaDotCom/ai-sim-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":7,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:enterpilot/ai-gateway-reproducible-benchmark","source_record":"bm_ai-gateway-reproducible-benchmark_99ff14f3"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Compares four open-source AI gateway proxies (GoModel, LiteLLM, Portkey, Bifrost) for routing overhead on a fixed AWS c7i.large instance with a mock backend. Measures latency percentiles, throughput, memory, cold start, and image size across streaming and non-streaming workloads.","descriptions":{"en":"Compares four open-source AI gateway proxies (GoModel, LiteLLM, Portkey, Bifrost) for routing overhead on a fixed AWS c7i.large instance with a mock backend. Measures latency percentiles, throughput, memory, cold start, and image size across streaming and non-streaming workloads."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:enterpilot/ai-gateway-reproducible-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-memory","visual-reasoning"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_ai-gateway-reproducible-benchmark_99ff14f3"},"modality":null,"name":"AI gateway reproducible benchmark","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:enterpilot/ai-gateway-reproducible-benchmark","source_url":"https://github.com/ENTERPILOT/ai-gateway-reproducible-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-enterpilot-ai-gateway-reproducible-benchmark","source":"claire_radar","source_url":"https://github.com/ENTERPILOT/ai-gateway-reproducible-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"The AI Security Leaderboard evaluates frontier AI model safeguards against the FAR.AI Minimal Standard for Safeguards across severe misuse requests (CBRNE threats and offensive cybersecurity), testing for universal jailbreaks.","descriptions":{"en":"The AI Security Leaderboard evaluates frontier AI model safeguards against the FAR.AI Minimal Standard for Safeguards across severe misuse requests (CBRNE threats and offensive cybersecurity), testing for universal jailbreaks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03070","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","cybersecurity"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_ai-security-leaderboard_0982a1ca"},"modality":null,"name":"AI Security Leaderboard","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03070","source_url":"https://arxiv.org/abs/2608.03070"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03070","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03070","topics":["safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_19ec567faa5ba4ac"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A dataset of 7,787 genuine grade-school level, multiple-choice science questions assembled to encourage research in advanced question-answering. The dataset is partitioned into a Challenge Set and Easy Set, where the Challenge Set contains only questions answered incorrectly by both retrieval-based and word co-occurrence algorithms. Covers multiple scientific domains including biology, physics, earth science, and chemistry, requiring scientific reasoning, causal understanding, and conceptual knowledge beyond simple fact retrieval. Includes a supporting corpus of over 14 million science sentences.","descriptions":{"en":"A dataset of 7,787 genuine grade-school level, multiple-choice science questions assembled to encourage research in advanced question-answering. The dataset is partitioned into a Challenge Set and Easy Set, where the Challenge Set contains only questions answered incorrectly by both retrieval-based and word co-occurrence algorithms. Covers multiple scientific domains including biology, physics, earth science, and chemistry, requiring scientific reasoning, causal understanding, and conceptual knowledge beyond simple fact retrieval. Includes a supporting corpus of over 14 million science sentences."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ai2-reasoning-challenge-(arc):gpt-4-0613","reported_at":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2-reasoning-challenge-%28arc%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ai2-reasoning-challenge-(arc)","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_19ec567faa5ba4ac"},"modality":"text","name":"AI2 Reasoning Challenge (ARC)","openness":"unknown","publisher":null,"released":"2018-03-14","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:ai2-reasoning-challenge-(arc)","source_url":"https://arxiv.org/abs/1803.05457"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":96.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.963,"raw_min":0.963,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ai2-reasoning-challenge-(arc):gpt-4-0613","reported_date":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2-reasoning-challenge-%28arc%29?top_n=500"},"unit":null},"scored_models":[{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"}],"slug":"llm-stats-ai2-reasoning-challenge-arc","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2-reasoning-challenge-%28arc%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":39,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_17891a74c7877e1e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AI2D is a dataset of 4,903 illustrative diagrams from grade school natural sciences (such as food webs, human physiology, and life cycles) with over 15,000 multiple choice questions and answers. The benchmark evaluates diagram understanding and visual reasoning capabilities, requiring models to interpret diagrammatic elements, relationships, and structure to answer questions about scientific concepts represented in visual form.","descriptions":{"en":"AI2D is a dataset of 4,903 illustrative diagrams from grade school natural sciences (such as food webs, human physiology, and life cycles) with over 15,000 multiple choice questions and answers. The benchmark evaluates diagram understanding and visual reasoning capabilities, requiring models to interpret diagrammatic elements, relationships, and structure to answer questions about scientific concepts represented in visual form."},"evidence_summary":{"document_count":1,"model_count":33,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ai2d:grok-1.5v","reported_at":"2024-04-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2d?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ai2d","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_17891a74c7877e1e"},"modality":"multimodal","name":"AI2D","openness":"unknown","publication":{"evidence":[{"note":"The authors' ECCV paper introduces the diagram corpus and question-answering task. Its science-diagram task matches this record's description; no score protocol is inferred.","url":"https://ai2-website.s3.amazonaws.com/publications/Diagrams_ECCV2016.pdf"}],"id":"ai2d-original","links":[{"kind":"paper","title":"A Diagram Is Worth A Dozen Images","url":"https://arxiv.org/abs/1603.07396"}],"record_keys":["llm-stats:ai2d"],"reviewed_at":"2026-10-26","scope":{"en":"Original AI2D diagram-understanding publication. The source record does not specify which image-label treatment or evaluation split its scores use.","zh":"AI2D 图解理解数据集的原始论文；收录来源未明确每项成绩采用的图中文字标注处理方式或评测划分。"}},"publisher":null,"released":"2016-03-24","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: A Diagram Is Worth A Dozen Images. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:ai2d","source_url":"https://arxiv.org/abs/1603.07396"},"repo_kind":"not_found","repo_resolution_status":"not_found","score_count":33,"score_direction":"higher_is_better","score_summary":{"display_max":94.69999999999999,"display_multiplier":100,"model_count":33,"model_count_basis":"source_model_id","numeric_count":33,"raw_max":0.947,"raw_min":0.716,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ai2d:claude-3-5-sonnet-20241022","reported_date":"2024-10-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2d?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-04-12","date_precision":"model_announcement","name":"Grok-1.5V"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-vision-instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-ai2d","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2d?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_ai4ai-bench_cbcf9954"},"categories":["Language & Knowledge","Self-Evolution","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"AI4AI-Bench evaluates LLM agents on algorithmic design across 10 frozen research repositories. Agents rewrite training algorithms within 4 hours, then scored by fixed evaluators against baseline algorithms, with submissions released for repeatable measurement.","descriptions":{"en":"AI4AI-Bench evaluates LLM agents on algorithmic design across 10 frozen research repositories. Agents rewrite training algorithms within 4 hours, then scored by fixed evaluators against baseline algorithms, with submissions released for repeatable measurement."},"editorial_guide":{"id":"ai4ai-bench","record_keys":["model-reports:ai4ai_bench","claire-radar:2608.20318"],"reviewed_at":"2026-10-27","scope":{"en":"The ten-task AI4AI-Bench suite, with paper v1 results and the separately identified v1.5 official chart data. Model, harness and reasoning effort define each evaluated configuration.","zh":"对应 AI4AI-Bench 的十任务套件。论文 v1 成绩与官网标注 v1.5 的图表数据分别保留；模型、Agent 框架和推理强度共同定义一次配置。"},"sections":[{"id":"tasks","source_ids":["paper","repository"],"text":{"en":"The ten frozen projects cover supervised fine-tuning, agentic reinforcement learning, on-policy distillation, reward modeling, preference optimization, diffusion reinforcement learning, unlearning, graph diffusion, model merging and pruning. The model and assets are fixed; the agent may change the learning method and training code. Model Soup and OWL do not train: they combine existing checkpoints or prune a model.","zh":"十个固定项目覆盖监督微调、Agent 强化学习、同策略蒸馏、奖励建模、偏好优化、扩散强化学习、遗忘学习、图扩散、模型融合和剪枝。起始模型与资产固定，Agent 可以修改学习方法和训练代码。Model Soup 与 OWL 不进行训练，分别组合已有权重和执行剪枝。"},"title":{"en":"Ten different algorithm families","zh":"十种算法家族"}},{"id":"protocol","source_ids":["paper"],"text":{"en":"Each configuration gets four hours on one NVIDIA B300 to inspect code and run experiments using a proxy metric. Only source code crosses into a fresh formal environment. Eight training tasks then run from initialization for up to twelve hours, with the best result among at most the three latest checkpoints. The two non-training tasks execute once; twelve hours is their verification ceiling.","zh":"每个配置有一张 NVIDIA B300 和四小时探索时间，可以读代码并用代理指标试验。只有源码进入全新的正式环境。八个训练任务从初始化状态运行，最多十二小时，并从最近至多三个检查点中取最佳结果；两个非训练任务执行一次，十二小时只是验收上限。"},"title":{"en":"Exploration and final execution are separate","zh":"探索与正式执行分开"}},{"id":"score","source_ids":["paper","evaluation"],"text":{"en":"Different raw task metrics are mapped to a common 0–1 scale before their mean is taken. Zero is an uninformative model, 0.1 is the unchanged repository recipe under the same conditions, and 1 is the metric-defined optimum. Perplexity uses a logarithmic coordinate. A score of 0.288 is not 28.8% of tasks solved. The paper assigns zero to submissions that return no model; raw execution receipts with missing or invalid results must not be converted into a task-metric zero.","zh":"各任务的原始指标先映射到 0–1，再取平均：0 对应无信息模型，0.1 对应同条件下未经修改的原仓库方法，1 对应指标定义的理想点。困惑度使用对数坐标。0.288 不等于完成了 28.8% 的任务。论文对未交付模型的提交计归一化零分；原始执行凭据中的缺失或无效结果，不能强行改成任务原始指标的零。"},"title":{"en":"Read the normalized score correctly","zh":"怎样读归一化总分"}},{"id":"comparison","source_ids":["paper","chart-data"],"text":{"en":"The study contains 29 configurations of six systems, each attempting all ten tasks. The paper reports Claude Opus 5 at medium effort at 0.288, while 0.250 averages its five effort settings. These answer different questions. The official JSON labels itself v1.5 and preserves more decimals, such as 0.2881 for medium effort; retain the cited source and printed precision instead of replacing one with the other.","zh":"研究包含六个系统的 29 个配置，每个配置都尝试十道任务。论文中 Claude Opus 5 的 medium 配置为 0.288，而 0.250 是它五档推理强度的平均，不能混为同一成绩。官网 JSON 自标 v1.5，并保留更多小数，例如 medium 为 0.2881；应保留各自来源及原始精度，不互相覆盖。"},"title":{"en":"Compare configurations, not pooled model names","zh":"比较配置，不混淆汇总口径"}},{"id":"limits","source_ids":["paper","evaluation"],"text":{"en":"This measures algorithm changes in ten selected projects under one GPU and fixed time budgets, not a demonstrated recursive self-improvement cycle. The baseline is the original repository recipe, not a human expert. Some cheap proxies share a corpus with final evaluation, so separation does not guarantee disjoint samples. The public release supports self-hosted evaluation; its documentation says an official blind service is not yet available and local warning-mode runs are non-official.","zh":"它测量单卡和固定时间内对十个特定项目的改进，并未演示完整的递归自我改进循环。基线是仓库原有方法，不是人类专家。部分代理指标与最终评测来自同一语料，因此分开执行不代表样本完全隔离。公开版本支持自托管评测，文档明确尚无官方盲测服务；使用默认警告模式的本地结果不算官方结果。"},"title":{"en":"What this can and cannot establish","zh":"结果的适用边界"}},{"id":"official-configurations","source_ids":["chart-data","homepage","paper"],"table":{"caption":{"en":"Official v1.5 chart data · all ten tasks · observed 27 October 2026","zh":"官网 v1.5 图表数据 · 全部十道任务 · 2026 年 10 月 27 日核对"},"columns":[{"en":"Model","zh":"模型"},{"en":"Organization","zh":"模型机构"},{"en":"Harness","zh":"运行框架"},{"en":"Reasoning effort","zh":"推理强度"},{"en":"Mean normalized score","zh":"平均归一化分数"}],"note":{"en":"Original JSON precision retained. The scale is 0–1 with the unchanged recipe at 0.1; these are not percentages. Scores evaluate model, harness and effort together. Paper Table 3 reports separate, lower-precision values; this table does not overwrite them.","zh":"保留 JSON 原始精度。量表为 0–1，原仓库方法基线为 0.1；这些数字不是百分比。成绩对应模型、框架与推理强度的组合。论文表 3 保留其独立的较低精度读数，本表不覆盖论文成绩。"},"rows":[["Claude Opus 5","Anthropic","Claude Code","low","0.2722"],["Claude Opus 5","Anthropic","Claude Code","medium","0.2881"],["Claude Opus 5","Anthropic","Claude Code","high","0.2776"],["Claude Opus 5","Anthropic","Claude Code","xhigh","0.1898"],["Claude Opus 5","Anthropic","Claude Code","max","0.2245"],["Claude Sonnet 5","Anthropic","Claude Code","low","0.0991"],["Claude Sonnet 5","Anthropic","Claude Code","medium","0.1092"],["Claude Sonnet 5","Anthropic","Claude Code","high","0.216"],["Claude Sonnet 5","Anthropic","Claude Code","xhigh","0.137"],["Claude Sonnet 5","Anthropic","Claude Code","max","0.1635"],["Kimi K3","Moonshot AI","Claude Code","max","0.1738"],["GPT-5.6 Sol","OpenAI","Codex","none","0.1261"],["GPT-5.6 Sol","OpenAI","Codex","low","0.1976"],["GPT-5.6 Sol","OpenAI","Codex","medium","0.144"],["GPT-5.6 Sol","OpenAI","Codex","high","0.235"],["GPT-5.6 Sol","OpenAI","Codex","xhigh","0.1952"],["GPT-5.6 Sol","OpenAI","Codex","max","0.2455"],["GPT-5.6 Terra","OpenAI","Codex","none","0.065"],["GPT-5.6 Terra","OpenAI","Codex","low","0.1085"],["GPT-5.6 Terra","OpenAI","Codex","medium","0.1349"],["GPT-5.6 Terra","OpenAI","Codex","high","0.1293"],["GPT-5.6 Terra","OpenAI","Codex","xhigh","0.1438"],["GPT-5.6 Terra","OpenAI","Codex","max","0.2282"],["GPT-5.6 Luna","OpenAI","Codex","none","0.0914"],["GPT-5.6 Luna","OpenAI","Codex","low","0.0786"],["GPT-5.6 Luna","OpenAI","Codex","medium","0.1242"],["GPT-5.6 Luna","OpenAI","Codex","high","0.1191"],["GPT-5.6 Luna","OpenAI","Codex","xhigh","0.1468"],["GPT-5.6 Luna","OpenAI","Codex","max","0.1393"]]},"text":{"en":"These are the exact mean values from the official points.json labeled “AI4AI-Bench v1.5 mapped scores”, observed on 27 October 2026. Each configuration attempts all ten tasks. The source provides no exact publication date for this data file, so the observation date is not presented as a new release or score-report date.","zh":"下表逐项保留官网 points.json 的平均值，源文件标注“AI4AI-Bench v1.5 mapped scores”，于 2026 年 10 月 27 日核对。每个配置都尝试十道任务。文件未给出精确发布日期，因此核对日期不作为新版本发布日期或成绩报告日期。"},"title":{"en":"All 29 official chart configurations","zh":"官网图表中的全部 29 个配置"}},{"id":"pooled-systems","source_ids":["chart-data","paper"],"table":{"caption":{"en":"Official pooled means · configuration-task cells, not unique task counts","zh":"官方跨配置平均值 · 单元数是配置×任务，不是独立题目数"},"columns":[{"en":"Model","zh":"模型"},{"en":"Harness","zh":"运行框架"},{"en":"Included effort settings","zh":"纳入的推理强度"},{"en":"Configuration-task cells","zh":"配置×任务单元数"},{"en":"Pooled mean score","zh":"汇总平均分数"}],"note":{"en":"All rows use the same ten-task suite, but pool different numbers of configurations. These author-reported averages are retained exactly; they must not replace the 29 individual configuration scores.","zh":"各行使用同一套十道任务，但合并的配置数量不同。保留作者发布的原始均值，不用它们替换上表 29 个单独配置的分数。"},"rows":[["Claude Opus 5","Claude Code","low, medium, high, xhigh, max","50","0.25"],["GPT-5.6 Sol","Codex","none, low, medium, high, xhigh, max","60","0.191"],["Kimi K3","Claude Code","max","10","0.174"],["Claude Sonnet 5","Claude Code","low, medium, high, xhigh, max","50","0.145"],["GPT-5.6 Terra","Codex","none, low, medium, high, xhigh, max","60","0.135"],["GPT-5.6 Luna","Codex","none, low, medium, high, xhigh, max","60","0.117"]]},"text":{"en":"The official file also publishes these six pooled system means. They combine the configurations listed in each row and therefore have different denominators: Claude has five tested effort settings, GPT-5.6 has six, and Kimi K3 has one. They are not additional single-effort configurations.","zh":"官方文件还给出以下六个系统汇总。每行合并所列的推理强度，因此分母不同：Claude 测五档，GPT-5.6 测六档，Kimi K3 只测一档。它们不是新增的单一推理强度配置。"},"title":{"en":"Six system averages across their tested effort settings","zh":"六个系统在已测推理强度上的汇总"}}],"sources":[{"id":"homepage","title":"AI4AI-Bench official project and results","url":"https://lab.einsia.ai/ai4ai/"},{"id":"paper","title":"AI4AI-Bench paper v1 — task protocol, scoring and Tables 2–3","url":"https://arxiv.org/html/2608.20318v1"},{"id":"publication","title":"arXiv submission history — 20 August 2026","url":"https://arxiv.org/abs/2608.20318v1"},{"id":"repository","title":"Official AI4AI-Bench repository","url":"https://github.com/Einsia/AI4AI-Bench"},{"id":"evaluation","title":"Official evaluation and replay documentation","url":"https://github.com/Einsia/AI4AI-Bench/blob/main/tools/docs/evaluation.md"},{"id":"chart-data","title":"Official chart data — AI4AI-Bench v1.5 mapped scores","url":"https://lab.einsia.ai/ai4ai/data/points.json"}],"summary":{"en":"Can a coding agent improve the algorithm inside an existing AI research project? AI4AI-Bench lets it rewrite a working repository, then reruns only the submitted source under a fixed training and evaluation budget.","zh":"编程 Agent 能否改进现有 AI 研究项目中的算法？AI4AI-Bench 让它修改可运行的研究仓库，再用固定预算从头执行提交的源码，检验改进是否成立。"},"title":"AI4AI-Bench"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.20318","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_ai4ai-bench_cbcf9954"},"modality":null,"name":"AI4AI-Bench","openness":"unknown","publication":{"evidence":[{"note":"The author project names this benchmark and links its introducing paper and available release resources. This exact-record review preserves the documented task version, metrics and evaluation setup.","url":"https://lab.einsia.ai/ai4ai/"},{"note":"Navers Lab / Einsia research index links this named project. Method papers and memory software in the same index are not added as benchmarks.","url":"https://lab.einsia.ai/research/"}],"id":"einsia-ai4ai-bench-original","links":[{"kind":"website","title":"AI4AI-Bench · official project","url":"https://lab.einsia.ai/ai4ai/"},{"kind":"paper","title":"AI4AI-Bench · introducing paper","url":"https://arxiv.org/abs/2608.20318v1"},{"kind":"repo","title":"AI4AI-Bench · official repository","url":"https://github.com/Einsia/AI4AI-Bench"}],"record_keys":["model-reports:ai4ai_bench","claire-radar:2608.20318"],"reviewed_at":"2026-10-27","scope":{"en":"The ten-task AI4AI-Bench suite, with paper v1 results and the separately identified v1.5 official chart data. Model, harness and reasoning effort define each evaluated configuration.","zh":"对应 AI4AI-Bench 的十任务套件。论文 v1 成绩与官网标注 v1.5 的图表数据分别保留；模型、Agent 框架和推理强度共同定义一次配置。"}},"publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.20318","source_url":"https://arxiv.org/abs/2608.20318"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-20318","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20318","topics":["coding","agents"],"unit":null},{"aliases":["AI4AI Bench","AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"artifact:arxiv:2608.20318","repository":null,"source_record":"bm_ai4ai-bench_cbcf9954"},"categories":["agents"],"collected_at":null,"description":"Ten fixed research projects, with model, harness and reasoning effort jointly evaluated under 4 h exploration on one B300 and up to 12 h fresh execution. The mean normalized score uses 0.1 for the original recipe, not percent solved; some proxy/final evaluations share a corpus.","descriptions":{"en":"Ten fixed research projects, with model, harness and reasoning effort jointly evaluated under 4 h exploration on one B300 and up to 12 h fresh execution. The mean normalized score uses 0.1 for the original recipe, not percent solved; some proxy/final evaluations share a corpus."},"editorial_guide":{"id":"ai4ai-bench","record_keys":["model-reports:ai4ai_bench","claire-radar:2608.20318"],"reviewed_at":"2026-10-27","scope":{"en":"The ten-task AI4AI-Bench suite, with paper v1 results and the separately identified v1.5 official chart data. Model, harness and reasoning effort define each evaluated configuration.","zh":"对应 AI4AI-Bench 的十任务套件。论文 v1 成绩与官网标注 v1.5 的图表数据分别保留；模型、Agent 框架和推理强度共同定义一次配置。"},"sections":[{"id":"tasks","source_ids":["paper","repository"],"text":{"en":"The ten frozen projects cover supervised fine-tuning, agentic reinforcement learning, on-policy distillation, reward modeling, preference optimization, diffusion reinforcement learning, unlearning, graph diffusion, model merging and pruning. The model and assets are fixed; the agent may change the learning method and training code. Model Soup and OWL do not train: they combine existing checkpoints or prune a model.","zh":"十个固定项目覆盖监督微调、Agent 强化学习、同策略蒸馏、奖励建模、偏好优化、扩散强化学习、遗忘学习、图扩散、模型融合和剪枝。起始模型与资产固定，Agent 可以修改学习方法和训练代码。Model Soup 与 OWL 不进行训练，分别组合已有权重和执行剪枝。"},"title":{"en":"Ten different algorithm families","zh":"十种算法家族"}},{"id":"protocol","source_ids":["paper"],"text":{"en":"Each configuration gets four hours on one NVIDIA B300 to inspect code and run experiments using a proxy metric. Only source code crosses into a fresh formal environment. Eight training tasks then run from initialization for up to twelve hours, with the best result among at most the three latest checkpoints. The two non-training tasks execute once; twelve hours is their verification ceiling.","zh":"每个配置有一张 NVIDIA B300 和四小时探索时间，可以读代码并用代理指标试验。只有源码进入全新的正式环境。八个训练任务从初始化状态运行，最多十二小时，并从最近至多三个检查点中取最佳结果；两个非训练任务执行一次，十二小时只是验收上限。"},"title":{"en":"Exploration and final execution are separate","zh":"探索与正式执行分开"}},{"id":"score","source_ids":["paper","evaluation"],"text":{"en":"Different raw task metrics are mapped to a common 0–1 scale before their mean is taken. Zero is an uninformative model, 0.1 is the unchanged repository recipe under the same conditions, and 1 is the metric-defined optimum. Perplexity uses a logarithmic coordinate. A score of 0.288 is not 28.8% of tasks solved. The paper assigns zero to submissions that return no model; raw execution receipts with missing or invalid results must not be converted into a task-metric zero.","zh":"各任务的原始指标先映射到 0–1，再取平均：0 对应无信息模型，0.1 对应同条件下未经修改的原仓库方法，1 对应指标定义的理想点。困惑度使用对数坐标。0.288 不等于完成了 28.8% 的任务。论文对未交付模型的提交计归一化零分；原始执行凭据中的缺失或无效结果，不能强行改成任务原始指标的零。"},"title":{"en":"Read the normalized score correctly","zh":"怎样读归一化总分"}},{"id":"comparison","source_ids":["paper","chart-data"],"text":{"en":"The study contains 29 configurations of six systems, each attempting all ten tasks. The paper reports Claude Opus 5 at medium effort at 0.288, while 0.250 averages its five effort settings. These answer different questions. The official JSON labels itself v1.5 and preserves more decimals, such as 0.2881 for medium effort; retain the cited source and printed precision instead of replacing one with the other.","zh":"研究包含六个系统的 29 个配置，每个配置都尝试十道任务。论文中 Claude Opus 5 的 medium 配置为 0.288，而 0.250 是它五档推理强度的平均，不能混为同一成绩。官网 JSON 自标 v1.5，并保留更多小数，例如 medium 为 0.2881；应保留各自来源及原始精度，不互相覆盖。"},"title":{"en":"Compare configurations, not pooled model names","zh":"比较配置，不混淆汇总口径"}},{"id":"limits","source_ids":["paper","evaluation"],"text":{"en":"This measures algorithm changes in ten selected projects under one GPU and fixed time budgets, not a demonstrated recursive self-improvement cycle. The baseline is the original repository recipe, not a human expert. Some cheap proxies share a corpus with final evaluation, so separation does not guarantee disjoint samples. The public release supports self-hosted evaluation; its documentation says an official blind service is not yet available and local warning-mode runs are non-official.","zh":"它测量单卡和固定时间内对十个特定项目的改进，并未演示完整的递归自我改进循环。基线是仓库原有方法，不是人类专家。部分代理指标与最终评测来自同一语料，因此分开执行不代表样本完全隔离。公开版本支持自托管评测，文档明确尚无官方盲测服务；使用默认警告模式的本地结果不算官方结果。"},"title":{"en":"What this can and cannot establish","zh":"结果的适用边界"}},{"id":"official-configurations","source_ids":["chart-data","homepage","paper"],"table":{"caption":{"en":"Official v1.5 chart data · all ten tasks · observed 27 October 2026","zh":"官网 v1.5 图表数据 · 全部十道任务 · 2026 年 10 月 27 日核对"},"columns":[{"en":"Model","zh":"模型"},{"en":"Organization","zh":"模型机构"},{"en":"Harness","zh":"运行框架"},{"en":"Reasoning effort","zh":"推理强度"},{"en":"Mean normalized score","zh":"平均归一化分数"}],"note":{"en":"Original JSON precision retained. The scale is 0–1 with the unchanged recipe at 0.1; these are not percentages. Scores evaluate model, harness and effort together. Paper Table 3 reports separate, lower-precision values; this table does not overwrite them.","zh":"保留 JSON 原始精度。量表为 0–1，原仓库方法基线为 0.1；这些数字不是百分比。成绩对应模型、框架与推理强度的组合。论文表 3 保留其独立的较低精度读数，本表不覆盖论文成绩。"},"rows":[["Claude Opus 5","Anthropic","Claude Code","low","0.2722"],["Claude Opus 5","Anthropic","Claude Code","medium","0.2881"],["Claude Opus 5","Anthropic","Claude Code","high","0.2776"],["Claude Opus 5","Anthropic","Claude Code","xhigh","0.1898"],["Claude Opus 5","Anthropic","Claude Code","max","0.2245"],["Claude Sonnet 5","Anthropic","Claude Code","low","0.0991"],["Claude Sonnet 5","Anthropic","Claude Code","medium","0.1092"],["Claude Sonnet 5","Anthropic","Claude Code","high","0.216"],["Claude Sonnet 5","Anthropic","Claude Code","xhigh","0.137"],["Claude Sonnet 5","Anthropic","Claude Code","max","0.1635"],["Kimi K3","Moonshot AI","Claude Code","max","0.1738"],["GPT-5.6 Sol","OpenAI","Codex","none","0.1261"],["GPT-5.6 Sol","OpenAI","Codex","low","0.1976"],["GPT-5.6 Sol","OpenAI","Codex","medium","0.144"],["GPT-5.6 Sol","OpenAI","Codex","high","0.235"],["GPT-5.6 Sol","OpenAI","Codex","xhigh","0.1952"],["GPT-5.6 Sol","OpenAI","Codex","max","0.2455"],["GPT-5.6 Terra","OpenAI","Codex","none","0.065"],["GPT-5.6 Terra","OpenAI","Codex","low","0.1085"],["GPT-5.6 Terra","OpenAI","Codex","medium","0.1349"],["GPT-5.6 Terra","OpenAI","Codex","high","0.1293"],["GPT-5.6 Terra","OpenAI","Codex","xhigh","0.1438"],["GPT-5.6 Terra","OpenAI","Codex","max","0.2282"],["GPT-5.6 Luna","OpenAI","Codex","none","0.0914"],["GPT-5.6 Luna","OpenAI","Codex","low","0.0786"],["GPT-5.6 Luna","OpenAI","Codex","medium","0.1242"],["GPT-5.6 Luna","OpenAI","Codex","high","0.1191"],["GPT-5.6 Luna","OpenAI","Codex","xhigh","0.1468"],["GPT-5.6 Luna","OpenAI","Codex","max","0.1393"]]},"text":{"en":"These are the exact mean values from the official points.json labeled “AI4AI-Bench v1.5 mapped scores”, observed on 27 October 2026. Each configuration attempts all ten tasks. The source provides no exact publication date for this data file, so the observation date is not presented as a new release or score-report date.","zh":"下表逐项保留官网 points.json 的平均值，源文件标注“AI4AI-Bench v1.5 mapped scores”，于 2026 年 10 月 27 日核对。每个配置都尝试十道任务。文件未给出精确发布日期，因此核对日期不作为新版本发布日期或成绩报告日期。"},"title":{"en":"All 29 official chart configurations","zh":"官网图表中的全部 29 个配置"}},{"id":"pooled-systems","source_ids":["chart-data","paper"],"table":{"caption":{"en":"Official pooled means · configuration-task cells, not unique task counts","zh":"官方跨配置平均值 · 单元数是配置×任务，不是独立题目数"},"columns":[{"en":"Model","zh":"模型"},{"en":"Harness","zh":"运行框架"},{"en":"Included effort settings","zh":"纳入的推理强度"},{"en":"Configuration-task cells","zh":"配置×任务单元数"},{"en":"Pooled mean score","zh":"汇总平均分数"}],"note":{"en":"All rows use the same ten-task suite, but pool different numbers of configurations. These author-reported averages are retained exactly; they must not replace the 29 individual configuration scores.","zh":"各行使用同一套十道任务，但合并的配置数量不同。保留作者发布的原始均值，不用它们替换上表 29 个单独配置的分数。"},"rows":[["Claude Opus 5","Claude Code","low, medium, high, xhigh, max","50","0.25"],["GPT-5.6 Sol","Codex","none, low, medium, high, xhigh, max","60","0.191"],["Kimi K3","Claude Code","max","10","0.174"],["Claude Sonnet 5","Claude Code","low, medium, high, xhigh, max","50","0.145"],["GPT-5.6 Terra","Codex","none, low, medium, high, xhigh, max","60","0.135"],["GPT-5.6 Luna","Codex","none, low, medium, high, xhigh, max","60","0.117"]]},"text":{"en":"The official file also publishes these six pooled system means. They combine the configurations listed in each row and therefore have different denominators: Claude has five tested effort settings, GPT-5.6 has six, and Kimi K3 has one. They are not additional single-effort configurations.","zh":"官方文件还给出以下六个系统汇总。每行合并所列的推理强度，因此分母不同：Claude 测五档，GPT-5.6 测六档，Kimi K3 只测一档。它们不是新增的单一推理强度配置。"},"title":{"en":"Six system averages across their tested effort settings","zh":"六个系统在已测推理强度上的汇总"}}],"sources":[{"id":"homepage","title":"AI4AI-Bench official project and results","url":"https://lab.einsia.ai/ai4ai/"},{"id":"paper","title":"AI4AI-Bench paper v1 — task protocol, scoring and Tables 2–3","url":"https://arxiv.org/html/2608.20318v1"},{"id":"publication","title":"arXiv submission history — 20 August 2026","url":"https://arxiv.org/abs/2608.20318v1"},{"id":"repository","title":"Official AI4AI-Bench repository","url":"https://github.com/Einsia/AI4AI-Bench"},{"id":"evaluation","title":"Official evaluation and replay documentation","url":"https://github.com/Einsia/AI4AI-Bench/blob/main/tools/docs/evaluation.md"},{"id":"chart-data","title":"Official chart data — AI4AI-Bench v1.5 mapped scores","url":"https://lab.einsia.ai/ai4ai/data/points.json"}],"summary":{"en":"Can a coding agent improve the algorithm inside an existing AI research project? AI4AI-Bench lets it rewrite a working repository, then reruns only the submitted source under a fixed training and evaluation budget.","zh":"编程 Agent 能否改进现有 AI 研究项目中的算法？AI4AI-Bench 让它修改可运行的研究仓库，再用固定预算从头执行提交的源码，检验改进是否成立。"},"title":"AI4AI-Bench"},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000ai4ai_bench\u0000ai4ai_bench_paper_v1_table3\u0000ai4ai_bench_arxiv_2608_20318_v1\u0000Claude Code; reasoning effort high; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)\u0000Claude Opus 5","reported_at":"2026-08-20","source_url":"https://arxiv.org/html/2608.20318v1#S3.T3"},"first_score_reported_at":"2026-08-20","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000ai4ai_bench\u0000ai4ai_bench_paper_v1_table3\u0000ai4ai_bench_arxiv_2608_20318_v1\u0000Claude Code; reasoning effort high; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)\u0000Claude Opus 5","reported_at":"2026-08-20","source_url":"https://arxiv.org/html/2608.20318v1#S3.T3"},"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"model-reports:ai4ai_bench","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"artifact:arxiv:2608.20318","primary":"ai-rnd","source_record":"bm_ai4ai-bench_cbcf9954"},"modality":null,"name":"AI4AI-Bench","openness":"unknown","publication":{"evidence":[{"note":"The author project names this benchmark and links its introducing paper and available release resources. This exact-record review preserves the documented task version, metrics and evaluation setup.","url":"https://lab.einsia.ai/ai4ai/"},{"note":"Navers Lab / Einsia research index links this named project. Method papers and memory software in the same index are not added as benchmarks.","url":"https://lab.einsia.ai/research/"}],"id":"einsia-ai4ai-bench-original","links":[{"kind":"website","title":"AI4AI-Bench · official project","url":"https://lab.einsia.ai/ai4ai/"},{"kind":"paper","title":"AI4AI-Bench · introducing paper","url":"https://arxiv.org/abs/2608.20318v1"},{"kind":"repo","title":"AI4AI-Bench · official repository","url":"https://github.com/Einsia/AI4AI-Bench"}],"record_keys":["model-reports:ai4ai_bench","claire-radar:2608.20318"],"reviewed_at":"2026-10-27","scope":{"en":"The ten-task AI4AI-Bench suite, with paper v1 results and the separately identified v1.5 official chart data. Model, harness and reasoning effort define each evaluated configuration.","zh":"对应 AI4AI-Bench 的十任务套件。论文 v1 成绩与官网标注 v1.5 的图表数据分别保留；模型、Agent 框架和推理强度共同定义一次配置。"}},"publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"model-reports:ai4ai_bench","source_url":"https://lab.einsia.ai/ai4ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":29,"score_direction":"higher_is_better","score_summary":{"display_max":0.288,"display_multiplier":1,"model_count":6,"model_count_basis":"source_model_id","numeric_count":29,"raw_max":0.288,"raw_min":0.065,"source_reference":{"obs_id":"curated\u0000ai4ai_bench\u0000ai4ai_bench_paper_v1_table3\u0000ai4ai_bench_arxiv_2608_20318_v1\u0000Claude Code; reasoning effort medium; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)\u0000Claude Opus 5","observation_id":"curated\u0000ai4ai_bench\u0000ai4ai_bench_paper_v1_table3\u0000ai4ai_bench_arxiv_2608_20318_v1\u0000Claude Code; reasoning effort medium; 10 tasks; 4 h exploration on 1 NVIDIA B300; 8 training tasks: clean-start run up to 12 h, best of up to 3 latest checkpoints; 2 non-training tasks: one execution under 12 h verification ceiling; mean mapped score (paper v1 Table 3)\u0000Claude Opus 5","reported_at":"2026-08-20","reported_date":"2026-08-20","source_id":"ai4ai_bench_paper_v1_table3","source_url":"https://arxiv.org/html/2608.20318v1#S3.T3"},"unit":"points"},"scored_models":[{"date":"2026-08-20","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2026-08-20","date_precision":"document_publication","name":"Claude Sonnet 5"},{"date":"2026-08-20","date_precision":"document_publication","name":"GPT-5.6 Luna"},{"date":"2026-08-20","date_precision":"document_publication","name":"GPT-5.6 Sol"},{"date":"2026-08-20","date_precision":"document_publication","name":"GPT-5.6 Terra"},{"date":"2026-08-20","date_precision":"document_publication","name":"Kimi K3"}],"slug":"ai4ai_bench","source":"model_reports","source_url":"https://lab.einsia.ai/ai4ai/","topics":["coding","agents"],"unit":"points"},{"aliases":["Auditory Illusion Benchmark for Large Audio Language Models"],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:gillosae/aib","source_record":"bm_auditory-illusion-benchmark-for-large-audi_9ce2461b"},"categories":["Factuality","General AI","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates large audio language models on 14,829 forced-choice trials across ten auditory illusions, scoring human-likeness and reality alignment.","descriptions":{"en":"Evaluates large audio language models on 14,829 forced-choice trials across ten auditory illusions, scoring human-likeness and reality alignment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.02277","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_auditory-illusion-benchmark-for-large-audi_9ce2461b"},"modality":null,"name":"AIB","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02277","source_url":"https://arxiv.org/abs/2609.02277"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02277","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02277","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_aicompanionbench_4fc0af83"},"categories":["Robotics & Embodied AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"AICompanionBench provides a dataset of 2,123 human-AI companion conversations with safety risk annotations to evaluate LLMs-as-judges for detecting unsafe interactions.","descriptions":{"en":"AICompanionBench provides a dataset of 2,123 human-AI companion conversations with safety risk annotations to evaluate LLMs-as-judges for detecting unsafe interactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.04867","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_aicompanionbench_4fc0af83"},"modality":null,"name":"AICompanionBench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04867","source_url":"https://arxiv.org/abs/2606.04867"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04867","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04867","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","智能体","Agent","AI数据分析","生产力工具","数据可视化","逻辑推理","任务执行","Task Execution","数理能力","Math","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AIDABench provides a professional benchmarking suite for AI office data analytics tools, focusing on the core data-processing needs in everyday office workflows, covering high-frequency and reproducible data-processing scenarios commonly seen in real business contexts. AIDABench致力于为AI办公数据分析工具提供专业评测基准，聚焦日常办公中的核心数据处理需求。评测输入以 Excel 文件为主，辅以少量DOC、 PDF、图片及无文件文本等形态，覆盖真实业务中高频、可复现的数据处理场景。核心考察能力包括：数据结构识别、数据清洗、条件筛选、分组聚合、描述统计、排序与排名。任务交付类型包括：问答（Q/A）、文件生成、数据可视化。","descriptions":{"en":"AIDABench provides a professional benchmarking suite for AI office data analytics tools, focusing on the core data-processing needs in everyday office workflows, covering high-frequency and reproducible data-processing scenarios commonly seen in real business contexts.","zh":"AIDABench致力于为AI办公数据分析工具提供专业评测基准，聚焦日常办公中的核心数据处理需求。评测输入以 Excel 文件为主，辅以少量DOC、 PDF、图片及无文件文本等形态，覆盖真实业务中高频、可复现的数据处理场景。核心考察能力包括：数据结构识别、数据清洗、条件筛选、分组聚合、描述统计、排序与排名。任务交付类型包括：问答（Q/A）、文件生成、数据可视化。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:2396","languages":[],"library_direction":null,"modality":"multimodal","name":"AIDABench","openness":"unknown","publisher":"商汤科技 & 上海人工智能实验室","released":"2026-02-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2396-aidabench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AIDABench","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":49532,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_ebde709e306badca"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Aider is a comprehensive code editing benchmark based on 133 practice exercises from Exercism's Python repository, designed to evaluate AI models' ability to translate natural language coding requests into executable code that passes unit tests. The benchmark measures end-to-end code editing capabilities, including GPT's ability to edit existing code and format code changes for automated saving to local files. The Aider Polyglot variant extends this evaluation across 225 challenging exercises spanning C++, Go, Java, JavaScript, Python, and Rust, making it a standard benchmark for assessing multilingual code editing performance in AI research.","descriptions":{"en":"Aider is a comprehensive code editing benchmark based on 133 practice exercises from Exercism's Python repository, designed to evaluate AI models' ability to translate natural language coding requests into executable code that passes unit tests. The benchmark measures end-to-end code editing capabilities, including GPT's ability to edit existing code and format code changes for automated saving to local files. The Aider Polyglot variant extends this evaluation across 225 challenging exercises spanning C++, Go, Java, JavaScript, Python, and Rust, making it a standard benchmark for assessing multilingual code editing performance in AI research."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aider:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aider","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_ebde709e306badca"},"modality":"text","name":"Aider","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":72.2,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.722,"raw_min":0.502,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aider:deepseek-v2.5","reported_date":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 32B"}],"slug":"llm-stats-aider","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":["Aider Polyglot","Aider"],"attention_signals":null,"categories":["coding"],"collected_at":null,"description":"Edit-format sensitive; whole-file and diff modes differ substantially.","descriptions":{"en":"Edit-format sensitive; whole-file and diff modes differ substantially."},"evidence_summary":{"document_count":4,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000aider_polyglot\u0000deepseek_r1_report\u0000aider_polyglot\u0000Accuracy\u0000DeepSeek-R1","reported_at":"2025-01-22","source_url":"https://arxiv.org/abs/2501.12948"},"first_score_reported_at":"2025-01-22","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000aider_polyglot\u0000deepseek_r1_report\u0000aider_polyglot\u0000Accuracy\u0000DeepSeek-R1","reported_at":"2025-01-22","source_url":"https://arxiv.org/abs/2501.12948"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:aider_polyglot","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["code-generation"],"join":"review:14H034160212:2026-10-15","primary":"code-generation","source_record":null},"modality":null,"name":"Aider Polyglot","openness":"unknown","publisher":null,"released":"2024-12-21","released_reference":{"basis":"benchmark_release","source_key":"model-reports:aider_polyglot","source_url":"https://aider.chat/docs/leaderboards/"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":82.2,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":82.2,"raw_min":53.3,"source_reference":{"obs_id":"curated\u0000aider_polyglot\u0000google_gemini_2_5_report\u0000aider_polyglot\u0000pass rate, average of 3 trials\u0000Gemini 2.5 Pro","observation_id":"curated\u0000aider_polyglot\u0000google_gemini_2_5_report\u0000aider_polyglot\u0000pass rate, average of 3 trials\u0000Gemini 2.5 Pro","reported_at":"2025-06-17","reported_date":"2025-06-17","source_id":"google_gemini_2_5_report","source_url":"https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf"},"unit":"percent"},"scored_models":[{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2025-06-17","date_precision":"document_publication","name":"Gemini 2.5 Pro"}],"slug":"aider_polyglot","source":"model_reports","source_url":"https://aider.chat/docs/leaderboards/","topics":["coding"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":49532,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:aider-ai/aider","source_record":"lib_aider_polyglot"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.","descriptions":{"en":"A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages."},"evidence_summary":{"document_count":1,"model_count":22,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aider-polyglot:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:aider-polyglot","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"lib_aider_polyglot"},"modality":"text","name":"Aider-Polyglot","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":22,"score_direction":"higher_is_better","score_summary":{"display_max":88.0,"display_multiplier":100,"model_count":22,"model_count_basis":"source_model_id","numeric_count":22,"raw_max":0.88,"raw_min":0.098,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aider-polyglot:gpt-5-2025-08-07","reported_date":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"}],"slug":"llm-stats-aider-polyglot","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":49532,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:aider-ai/aider","source_record":"catalog_a7c2ffe234cb2fe7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.","descriptions":{"en":"A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aider-polyglot-edit:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot-edit?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aider-polyglot-edit","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_a7c2ffe234cb2fe7"},"modality":"text","name":"Aider-Polyglot Edit","openness":"unknown","publication":{"evidence":[{"note":"Aider's December 2024 announcement describes the 225 problems selected from 697 exercises in C++, Go, Java, JavaScript, Python and Rust after testing seven models. It links the official problem repository and separates completion correctness from correct edit format, matching the exact record without choosing a per-run format.","url":"https://aider.chat/2024/12/21/polyglot.html"},{"note":"The accessible Aider-AI repository links back to the original announcement and credits the same six Exercism tracks. It is the benchmark-problem release, not a claim that all recorded runs use a single harness configuration.","url":"https://github.com/Aider-AI/polyglot-benchmark/blob/main/README.md"}],"id":"aider-polyglot-edit-original","links":[{"kind":"website","title":"Aider polyglot benchmark original announcement","url":"https://aider.chat/2024/12/21/polyglot.html"},{"kind":"repo","title":"Aider polyglot benchmark exercises","url":"https://github.com/Aider-AI/polyglot-benchmark"}],"record_keys":["llm-stats:aider-polyglot-edit"],"reviewed_at":"2026-10-26","scope":{"en":"Aider's polyglot code-editing benchmark uses 225 selected Exercism problems in six languages. Edit format, model configuration and scoring columns remain specific to each reported run.","zh":"Aider 多语言代码编辑基准使用从六种语言中筛选的 225 道 Exercism 题目。编辑格式、模型配置与计分列仍以每次已报告运行的设置为准。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":79.7,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.797,"raw_min":0.062,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aider-polyglot-edit:deepseek-v3","reported_date":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot-edit?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"}],"slug":"llm-stats-aider-polyglot-edit","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot-edit?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":84,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"lib_aime"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"American Invitational Mathematics Examination (AIME) benchmark for evaluating mathematical reasoning capabilities of large language models. Contains 30 challenging mathematical problems from AIME 2024 competition that require multi-step reasoning and advanced mathematical insight. Each problem has an integer answer between 000-999.","descriptions":{"en":"American Invitational Mathematics Examination (AIME) benchmark for evaluating mathematical reasoning capabilities of large language models. Contains 30 challenging mathematical problems from AIME 2024 competition that require multi-step reasoning and advanced mathematical insight. Each problem has an integer answer between 000-999."},"editorial_guide":{"id":"aime-years","record_keys":["model-reports:aime","llm-stats:aime","llm-stats:aime-2024","llm-stats:aime-2025","artificial-analysis:aime-2025"],"reviewed_at":"2026-10-25","scope":{"en":"This guide covers the 2024 and 2025 AIME I/II papers. Some linked records also contain 2026 or unspecified-year results; the 30-question descriptions below do not establish the task count or protocol for those rows. Check each result’s year and protocol before comparing.","zh":"本导读介绍 2024 与 2025 年 AIME I／II 试题。部分关联记录还含 2026 年或未注明年份的成绩；下文的 30 题说明不代表这些成绩的题量或协议，应逐条核对年份与评测条件后再比较。"},"sections":[{"id":"task","source_ids":["maa","aa"],"text":{"en":"AIME is a mathematics competition whose answers are integers from 0 to 999. In the AI evaluation, a model solves a problem and provides that integer; eloquent reasoning alone does not substitute for a correct final answer.","zh":"AIME 是数学竞赛，答案为 0 至 999 的整数。在 AI 评测中，模型需要解题并给出该整数；推理文字再流畅，也不能替代正确的最终答案。"},"title":{"en":"What the model must produce","zh":"模型需要交付什么"}},{"id":"years","source_ids":["data2024","data2025"],"text":{"en":"The commonly used 2024 collection contains 30 problems from AIME I and II. The 2025 collection also contains 30, but different, problems. Equal sizes do not make scores interchangeable. This guide does not cover AIME 2026.","zh":"常用的 2024 题集包含当年 AIME I 和 II 共 30 题；2025 题集也有 30 题，但题目不同。题量一致不意味着成绩可互换。本导读不覆盖 AIME 2026。"},"title":{"en":"Keep 2024 and 2025 separate","zh":"2024 与 2025 要分开看"}},{"id":"grading","source_ids":["methodology"],"text":{"en":"Evaluators must extract the model's final answer before checking it. Artificial Analysis documents its prompt and mathematical answer-checking procedure. Record these details alongside tool access and reasoning limits, because they define the evaluated setup.","zh":"评分前需要从输出中提取最终答案。Artificial Analysis 公布了提示词和数学答案检查流程。应同时记录这些细节、工具权限和推理限制，因为它们共同定义实际评测条件。"},"title":{"en":"Answer extraction is part of the test","zh":"答案提取也是评测的一部分"}},{"id":"samples","source_ids":["matharena"],"text":{"en":"Ask whether a reported percentage averages attempts, accepts any successful attempt, or grades a voted answer. These answer different questions about reliability and compute. MathArena exposes run configuration and saved outputs so the protocol can be inspected.","zh":"阅读百分比时应确认：是对多次尝试取平均、任意一次成功即可，还是对投票选出的答案评分。这些分别衡量不同的可靠性与计算投入；MathArena 提供运行配置和已保存输出供核对。"},"title":{"en":"Distinguish repeated attempts and voting","zh":"区分重复尝试与投票"}},{"id":"interpretation","source_ids":["data2024","data2025"],"text":{"en":"On one 30-question run, one additional correct answer changes accuracy by about 3.33 percentage points. This is arithmetic, not an uncertainty estimate. Inspect per-problem outcomes and repeated runs before treating a small lead as broad mathematical superiority.","zh":"单次完整作答 30 题时，多答对一题就会提高约 3.33 个百分点。这只是算术关系，并不是误差区间。应结合逐题结果和重复实验判断，小幅领先不足以说明全面数学优势。"},"title":{"en":"Read small differences cautiously","zh":"小幅分差怎样理解"}}],"sources":[{"id":"maa","title":"MAA · AIME competition and integer answer format","url":"https://maa.org/maa-invitational-competitions/"},{"id":"data2024","title":"Hugging Face H4 · AIME 2024 dataset","url":"https://huggingface.co/datasets/HuggingFaceH4/aime_2024"},{"id":"data2025","title":"MathArena · AIME 2025 dataset","url":"https://huggingface.co/datasets/MathArena/aime_2025"},{"id":"aa","title":"Artificial Analysis · AIME 2025 evaluation","url":"https://artificialanalysis.ai/evaluations/aime-2025"},{"id":"methodology","title":"Artificial Analysis · Mathematical answer extraction and evaluation","url":"https://artificialanalysis.ai/methodology/intelligence-benchmarking"},{"id":"matharena","title":"MathArena · Evaluation code and run configurations","url":"https://github.com/eth-sri/matharena"}],"summary":{"en":"AIME asks for precise integer answers to difficult mathematics problems. The year, number of attempts and answer selection rule are essential parts of any model comparison.","zh":"AIME 要求模型为较难的数学题给出准确整数答案。比较成绩时，年份、尝试次数和最终答案的选择规则缺一不可。"},"title":"AIME 2024 and 2025"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aime:phi-4-mini-reasoning","reported_at":"2025-04-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aime","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"lib_aime"},"modality":"text","name":"AIME","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":57.49999999999999,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.575,"raw_min":0.373,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aime:phi-4-mini-reasoning","reported_date":"2025-04-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Mini Reasoning"}],"slug":"llm-stats-aime","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["AIME","AIME 2024","AIME 2025","AIME 2026"],"attention_signals":null,"categories":["math"],"collected_at":null,"description":"pass@k, majority vote and Python tool access each shift this by double digits.","descriptions":{"en":"pass@k, majority vote and Python tool access each shift this by double digits."},"editorial_guide":{"id":"aime-years","record_keys":["model-reports:aime","llm-stats:aime","llm-stats:aime-2024","llm-stats:aime-2025","artificial-analysis:aime-2025"],"reviewed_at":"2026-10-25","scope":{"en":"This guide covers the 2024 and 2025 AIME I/II papers. Some linked records also contain 2026 or unspecified-year results; the 30-question descriptions below do not establish the task count or protocol for those rows. Check each result’s year and protocol before comparing.","zh":"本导读介绍 2024 与 2025 年 AIME I／II 试题。部分关联记录还含 2026 年或未注明年份的成绩；下文的 30 题说明不代表这些成绩的题量或协议，应逐条核对年份与评测条件后再比较。"},"sections":[{"id":"task","source_ids":["maa","aa"],"text":{"en":"AIME is a mathematics competition whose answers are integers from 0 to 999. In the AI evaluation, a model solves a problem and provides that integer; eloquent reasoning alone does not substitute for a correct final answer.","zh":"AIME 是数学竞赛，答案为 0 至 999 的整数。在 AI 评测中，模型需要解题并给出该整数；推理文字再流畅，也不能替代正确的最终答案。"},"title":{"en":"What the model must produce","zh":"模型需要交付什么"}},{"id":"years","source_ids":["data2024","data2025"],"text":{"en":"The commonly used 2024 collection contains 30 problems from AIME I and II. The 2025 collection also contains 30, but different, problems. Equal sizes do not make scores interchangeable. This guide does not cover AIME 2026.","zh":"常用的 2024 题集包含当年 AIME I 和 II 共 30 题；2025 题集也有 30 题，但题目不同。题量一致不意味着成绩可互换。本导读不覆盖 AIME 2026。"},"title":{"en":"Keep 2024 and 2025 separate","zh":"2024 与 2025 要分开看"}},{"id":"grading","source_ids":["methodology"],"text":{"en":"Evaluators must extract the model's final answer before checking it. Artificial Analysis documents its prompt and mathematical answer-checking procedure. Record these details alongside tool access and reasoning limits, because they define the evaluated setup.","zh":"评分前需要从输出中提取最终答案。Artificial Analysis 公布了提示词和数学答案检查流程。应同时记录这些细节、工具权限和推理限制，因为它们共同定义实际评测条件。"},"title":{"en":"Answer extraction is part of the test","zh":"答案提取也是评测的一部分"}},{"id":"samples","source_ids":["matharena"],"text":{"en":"Ask whether a reported percentage averages attempts, accepts any successful attempt, or grades a voted answer. These answer different questions about reliability and compute. MathArena exposes run configuration and saved outputs so the protocol can be inspected.","zh":"阅读百分比时应确认：是对多次尝试取平均、任意一次成功即可，还是对投票选出的答案评分。这些分别衡量不同的可靠性与计算投入；MathArena 提供运行配置和已保存输出供核对。"},"title":{"en":"Distinguish repeated attempts and voting","zh":"区分重复尝试与投票"}},{"id":"interpretation","source_ids":["data2024","data2025"],"text":{"en":"On one 30-question run, one additional correct answer changes accuracy by about 3.33 percentage points. This is arithmetic, not an uncertainty estimate. Inspect per-problem outcomes and repeated runs before treating a small lead as broad mathematical superiority.","zh":"单次完整作答 30 题时，多答对一题就会提高约 3.33 个百分点。这只是算术关系，并不是误差区间。应结合逐题结果和重复实验判断，小幅领先不足以说明全面数学优势。"},"title":{"en":"Read small differences cautiously","zh":"小幅分差怎样理解"}}],"sources":[{"id":"maa","title":"MAA · AIME competition and integer answer format","url":"https://maa.org/maa-invitational-competitions/"},{"id":"data2024","title":"Hugging Face H4 · AIME 2024 dataset","url":"https://huggingface.co/datasets/HuggingFaceH4/aime_2024"},{"id":"data2025","title":"MathArena · AIME 2025 dataset","url":"https://huggingface.co/datasets/MathArena/aime_2025"},{"id":"aa","title":"Artificial Analysis · AIME 2025 evaluation","url":"https://artificialanalysis.ai/evaluations/aime-2025"},{"id":"methodology","title":"Artificial Analysis · Mathematical answer extraction and evaluation","url":"https://artificialanalysis.ai/methodology/intelligence-benchmarking"},{"id":"matharena","title":"MathArena · Evaluation code and run configurations","url":"https://github.com/eth-sri/matharena"}],"summary":{"en":"AIME asks for precise integer answers to difficult mathematics problems. The year, number of attempts and answer selection rule are essential parts of any model comparison.","zh":"AIME 要求模型为较难的数学题给出准确整数答案。比较成绩时，年份、尝试次数和最终答案的选择规则缺一不可。"},"title":"AIME 2024 and 2025"},"evidence_summary":{"document_count":19,"model_count":21,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000aime\u0000deepseek_v3_report\u0000aime_2024\u0000Pass@1\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"first_score_reported_at":"2024-12-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000aime\u0000deepseek_v3_report\u0000aime_2024\u0000Pass@1\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:aime","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["math"],"join":"review:14H034160212:2026-10-15","primary":"math","source_record":null},"modality":null,"name":"AIME","openness":"unknown","publisher":null,"released":"2024-02-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:aime","source_url":"https://maa.org/maa-invitational-competitions/"},"repo_kind":null,"repo_resolution_status":null,"score_count":24,"score_direction":"higher_is_better","score_summary":{"display_max":99.2,"display_multiplier":1,"model_count":21,"model_count_basis":"source_model_id","numeric_count":24,"raw_max":99.2,"raw_min":6.7,"source_reference":{"obs_id":"curated\u0000aime\u0000zai_glm_5_2_model_card\u0000aime\u0000AIME 2026, temp 1.0, top_p 0.95, max gen 163840 tokens, GPT-5.5 (medium) judge\u0000GLM-5.2","observation_id":"curated\u0000aime\u0000zai_glm_5_2_model_card\u0000aime\u0000AIME 2026, temp 1.0, top_p 0.95, max gen 163840 tokens, GPT-5.5 (medium) judge\u0000GLM-5.2","reported_at":"2026-06-16","reported_date":"2026-06-16","source_id":"zai_glm_5_2_model_card","source_url":"https://huggingface.co/zai-org/GLM-5.2"},"unit":"percent"},"scored_models":[{"date":"2025-04-30","date_precision":"document_publication","name":"Bespoke-Stratos-7B"},{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Opus 4"},{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Sonnet 4"},{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2025-04-30","date_precision":"document_publication","name":"DeepSeek-R1-Distill-Llama-8B"},{"date":"2025-04-30","date_precision":"document_publication","name":"DeepSeek-R1-Distill-Qwen-7B"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2025-04-30","date_precision":"document_publication","name":"Llama-3.2-3B-Instruct"},{"date":"2025-04-30","date_precision":"document_publication","name":"o1-mini"},{"date":"2025-04-30","date_precision":"document_publication","name":"OpenThinker-7B"},{"date":"2025-04-30","date_precision":"document_publication","name":"Phi-4-Mini (base model, 3.8B)"},{"date":"2025-04-30","date_precision":"document_publication","name":"Phi-4-mini-reasoning (3.8B)"},{"date":"2025-05-14","date_precision":"document_publication","name":"Qwen3-235B-A22B (Thinking)"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"aime","source":"model_reports","source_url":"https://maa.org/maa-invitational-competitions/","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":14209,"github_stars_30d":null,"hf_dataset_downloads":49872,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eleutherai/lm-evaluation-harness","source_record":"catalog_f1b502e7764977ee"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.","descriptions":{"en":"American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty."},"editorial_guide":{"id":"aime-years","record_keys":["model-reports:aime","llm-stats:aime","llm-stats:aime-2024","llm-stats:aime-2025","artificial-analysis:aime-2025"],"reviewed_at":"2026-10-25","scope":{"en":"This guide covers the 2024 and 2025 AIME I/II papers. Some linked records also contain 2026 or unspecified-year results; the 30-question descriptions below do not establish the task count or protocol for those rows. Check each result’s year and protocol before comparing.","zh":"本导读介绍 2024 与 2025 年 AIME I／II 试题。部分关联记录还含 2026 年或未注明年份的成绩；下文的 30 题说明不代表这些成绩的题量或协议，应逐条核对年份与评测条件后再比较。"},"sections":[{"id":"task","source_ids":["maa","aa"],"text":{"en":"AIME is a mathematics competition whose answers are integers from 0 to 999. In the AI evaluation, a model solves a problem and provides that integer; eloquent reasoning alone does not substitute for a correct final answer.","zh":"AIME 是数学竞赛，答案为 0 至 999 的整数。在 AI 评测中，模型需要解题并给出该整数；推理文字再流畅，也不能替代正确的最终答案。"},"title":{"en":"What the model must produce","zh":"模型需要交付什么"}},{"id":"years","source_ids":["data2024","data2025"],"text":{"en":"The commonly used 2024 collection contains 30 problems from AIME I and II. The 2025 collection also contains 30, but different, problems. Equal sizes do not make scores interchangeable. This guide does not cover AIME 2026.","zh":"常用的 2024 题集包含当年 AIME I 和 II 共 30 题；2025 题集也有 30 题，但题目不同。题量一致不意味着成绩可互换。本导读不覆盖 AIME 2026。"},"title":{"en":"Keep 2024 and 2025 separate","zh":"2024 与 2025 要分开看"}},{"id":"grading","source_ids":["methodology"],"text":{"en":"Evaluators must extract the model's final answer before checking it. Artificial Analysis documents its prompt and mathematical answer-checking procedure. Record these details alongside tool access and reasoning limits, because they define the evaluated setup.","zh":"评分前需要从输出中提取最终答案。Artificial Analysis 公布了提示词和数学答案检查流程。应同时记录这些细节、工具权限和推理限制，因为它们共同定义实际评测条件。"},"title":{"en":"Answer extraction is part of the test","zh":"答案提取也是评测的一部分"}},{"id":"samples","source_ids":["matharena"],"text":{"en":"Ask whether a reported percentage averages attempts, accepts any successful attempt, or grades a voted answer. These answer different questions about reliability and compute. MathArena exposes run configuration and saved outputs so the protocol can be inspected.","zh":"阅读百分比时应确认：是对多次尝试取平均、任意一次成功即可，还是对投票选出的答案评分。这些分别衡量不同的可靠性与计算投入；MathArena 提供运行配置和已保存输出供核对。"},"title":{"en":"Distinguish repeated attempts and voting","zh":"区分重复尝试与投票"}},{"id":"interpretation","source_ids":["data2024","data2025"],"text":{"en":"On one 30-question run, one additional correct answer changes accuracy by about 3.33 percentage points. This is arithmetic, not an uncertainty estimate. Inspect per-problem outcomes and repeated runs before treating a small lead as broad mathematical superiority.","zh":"单次完整作答 30 题时，多答对一题就会提高约 3.33 个百分点。这只是算术关系，并不是误差区间。应结合逐题结果和重复实验判断，小幅领先不足以说明全面数学优势。"},"title":{"en":"Read small differences cautiously","zh":"小幅分差怎样理解"}}],"sources":[{"id":"maa","title":"MAA · AIME competition and integer answer format","url":"https://maa.org/maa-invitational-competitions/"},{"id":"data2024","title":"Hugging Face H4 · AIME 2024 dataset","url":"https://huggingface.co/datasets/HuggingFaceH4/aime_2024"},{"id":"data2025","title":"MathArena · AIME 2025 dataset","url":"https://huggingface.co/datasets/MathArena/aime_2025"},{"id":"aa","title":"Artificial Analysis · AIME 2025 evaluation","url":"https://artificialanalysis.ai/evaluations/aime-2025"},{"id":"methodology","title":"Artificial Analysis · Mathematical answer extraction and evaluation","url":"https://artificialanalysis.ai/methodology/intelligence-benchmarking"},{"id":"matharena","title":"MathArena · Evaluation code and run configurations","url":"https://github.com/eth-sri/matharena"}],"summary":{"en":"AIME asks for precise integer answers to difficult mathematics problems. The year, number of attempts and answer selection rule are essential parts of any model comparison.","zh":"AIME 要求模型为较难的数学题给出准确整数答案。比较成绩时，年份、尝试次数和最终答案的选择规则缺一不可。"},"title":"AIME 2024 and 2025"},"evidence_summary":{"document_count":1,"model_count":53,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aime-2024:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2024?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aime-2024","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_f1b502e7764977ee"},"modality":"text","name":"AIME 2024","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"not_found","repo_resolution_status":"not_found","score_count":53,"score_direction":"higher_is_better","score_summary":{"display_max":95.8,"display_multiplier":100,"model_count":53,"model_count_basis":"source_model_id","numeric_count":53,"raw_max":0.958,"raw_min":0.131,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aime-2024:grok-3-mini","reported_date":"2025-02-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2024?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 7B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Zero"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek-V3 0324"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Gemini 2.0 Flash Thinking"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Grok-3"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Grok-3 Mini"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-01-20","date_precision":"model_announcement","name":"Kimi-k1.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 2506"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Min  istral 3 (3B Reasoning 2512)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Reasoning 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Reasoning 2512)"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1-pro"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 30B A3B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 32B"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ-32B"},{"date":"2024-11-28","date_precision":"model_announcement","name":"QwQ-32B-Preview"}],"slug":"llm-stats-aime-2024","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2024?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["reasoning","math"],"collected_at":"2026-08-25T10:41:06Z","description":"Mathematical reasoning","descriptions":{"en":"Mathematical reasoning"},"editorial_guide":{"id":"aime-years","record_keys":["model-reports:aime","llm-stats:aime","llm-stats:aime-2024","llm-stats:aime-2025","artificial-analysis:aime-2025"],"reviewed_at":"2026-10-25","scope":{"en":"This guide covers the 2024 and 2025 AIME I/II papers. Some linked records also contain 2026 or unspecified-year results; the 30-question descriptions below do not establish the task count or protocol for those rows. Check each result’s year and protocol before comparing.","zh":"本导读介绍 2024 与 2025 年 AIME I／II 试题。部分关联记录还含 2026 年或未注明年份的成绩；下文的 30 题说明不代表这些成绩的题量或协议，应逐条核对年份与评测条件后再比较。"},"sections":[{"id":"task","source_ids":["maa","aa"],"text":{"en":"AIME is a mathematics competition whose answers are integers from 0 to 999. In the AI evaluation, a model solves a problem and provides that integer; eloquent reasoning alone does not substitute for a correct final answer.","zh":"AIME 是数学竞赛，答案为 0 至 999 的整数。在 AI 评测中，模型需要解题并给出该整数；推理文字再流畅，也不能替代正确的最终答案。"},"title":{"en":"What the model must produce","zh":"模型需要交付什么"}},{"id":"years","source_ids":["data2024","data2025"],"text":{"en":"The commonly used 2024 collection contains 30 problems from AIME I and II. The 2025 collection also contains 30, but different, problems. Equal sizes do not make scores interchangeable. This guide does not cover AIME 2026.","zh":"常用的 2024 题集包含当年 AIME I 和 II 共 30 题；2025 题集也有 30 题，但题目不同。题量一致不意味着成绩可互换。本导读不覆盖 AIME 2026。"},"title":{"en":"Keep 2024 and 2025 separate","zh":"2024 与 2025 要分开看"}},{"id":"grading","source_ids":["methodology"],"text":{"en":"Evaluators must extract the model's final answer before checking it. Artificial Analysis documents its prompt and mathematical answer-checking procedure. Record these details alongside tool access and reasoning limits, because they define the evaluated setup.","zh":"评分前需要从输出中提取最终答案。Artificial Analysis 公布了提示词和数学答案检查流程。应同时记录这些细节、工具权限和推理限制，因为它们共同定义实际评测条件。"},"title":{"en":"Answer extraction is part of the test","zh":"答案提取也是评测的一部分"}},{"id":"samples","source_ids":["matharena"],"text":{"en":"Ask whether a reported percentage averages attempts, accepts any successful attempt, or grades a voted answer. These answer different questions about reliability and compute. MathArena exposes run configuration and saved outputs so the protocol can be inspected.","zh":"阅读百分比时应确认：是对多次尝试取平均、任意一次成功即可，还是对投票选出的答案评分。这些分别衡量不同的可靠性与计算投入；MathArena 提供运行配置和已保存输出供核对。"},"title":{"en":"Distinguish repeated attempts and voting","zh":"区分重复尝试与投票"}},{"id":"interpretation","source_ids":["data2024","data2025"],"text":{"en":"On one 30-question run, one additional correct answer changes accuracy by about 3.33 percentage points. This is arithmetic, not an uncertainty estimate. Inspect per-problem outcomes and repeated runs before treating a small lead as broad mathematical superiority.","zh":"单次完整作答 30 题时，多答对一题就会提高约 3.33 个百分点。这只是算术关系，并不是误差区间。应结合逐题结果和重复实验判断，小幅领先不足以说明全面数学优势。"},"title":{"en":"Read small differences cautiously","zh":"小幅分差怎样理解"}}],"sources":[{"id":"maa","title":"MAA · AIME competition and integer answer format","url":"https://maa.org/maa-invitational-competitions/"},{"id":"data2024","title":"Hugging Face H4 · AIME 2024 dataset","url":"https://huggingface.co/datasets/HuggingFaceH4/aime_2024"},{"id":"data2025","title":"MathArena · AIME 2025 dataset","url":"https://huggingface.co/datasets/MathArena/aime_2025"},{"id":"aa","title":"Artificial Analysis · AIME 2025 evaluation","url":"https://artificialanalysis.ai/evaluations/aime-2025"},{"id":"methodology","title":"Artificial Analysis · Mathematical answer extraction and evaluation","url":"https://artificialanalysis.ai/methodology/intelligence-benchmarking"},{"id":"matharena","title":"MathArena · Evaluation code and run configurations","url":"https://github.com/eth-sri/matharena"}],"summary":{"en":"AIME asks for precise integer answers to difficult mathematics problems. The year, number of attempts and answer selection rule are essential parts of any model comparison.","zh":"AIME 要求模型为较难的数学题给出准确整数答案。比较成绩时，年份、尝试次数和最终答案的选择规则缺一不可。"},"title":"AIME 2024 and 2025"},"evidence_summary":{"document_count":1,"model_count":270,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:aime-2025:9f873c2f-2c2d-4ccb-9e1b-71bf61b052be","reported_at":"2024-02-26","source_url":"https://artificialanalysis.ai/evaluations/aime-2025"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:aime-2025","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["math"],"join":"review:14H034160212:2026-10-15","primary":"math","source_record":null},"modality":null,"name":"AIME 2025","openness":"unknown","publisher":null,"released":"2025-02-12","released_reference":{"basis":"dataset_published","note":"This source evaluates all 30 AIME I and II problems. MAA dates AIME I to February 6 and AIME II to February 12; the latter completes the 2025 problem set.","source_key":"artificial-analysis:aime-2025","source_url":"https://maa.org/news/aime-thresholds-are-available/"},"repo_kind":null,"repo_resolution_status":null,"score_count":270,"score_direction":"higher_is_better","score_summary":{"display_max":99.0,"display_multiplier":100,"model_count":270,"model_count_basis":"source_model_id","numeric_count":270,"raw_max":0.99,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:aime-2025:498862c3-f9ac-49d2-852f-16a02bb0c38f","reported_date":"2025-12-11","source_url":"https://artificialanalysis.ai/evaluations/aime-2025"},"unit":null},"scored_models":[{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-03-27","date_precision":"model_announcement","name":"GPT-4o (March 2025, chatgpt-4o-latest)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (ChatGPT)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2025-10-30","date_precision":"model_announcement","name":"Kimi Linear 48B A3B Instruct"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 4B v1.1 (Reasoning)"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 3B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 40k"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2 (Jul '24)"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"OLMo 2 32B"},{"date":"2024-11-26","date_precision":"model_announcement","name":"OLMo 2 7B"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2024-04-23","date_precision":"model_announcement","name":"Phi-3 Mini Instruct 3.8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ 32B"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"}],"slug":"artificial-analysis-aime-2025","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/aime-2025","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":34028,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eth-sri/matharena","source_record":"catalog_c99837e299b2d648"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.","descriptions":{"en":"All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning."},"editorial_guide":{"id":"aime-years","record_keys":["model-reports:aime","llm-stats:aime","llm-stats:aime-2024","llm-stats:aime-2025","artificial-analysis:aime-2025"],"reviewed_at":"2026-10-25","scope":{"en":"This guide covers the 2024 and 2025 AIME I/II papers. Some linked records also contain 2026 or unspecified-year results; the 30-question descriptions below do not establish the task count or protocol for those rows. Check each result’s year and protocol before comparing.","zh":"本导读介绍 2024 与 2025 年 AIME I／II 试题。部分关联记录还含 2026 年或未注明年份的成绩；下文的 30 题说明不代表这些成绩的题量或协议，应逐条核对年份与评测条件后再比较。"},"sections":[{"id":"task","source_ids":["maa","aa"],"text":{"en":"AIME is a mathematics competition whose answers are integers from 0 to 999. In the AI evaluation, a model solves a problem and provides that integer; eloquent reasoning alone does not substitute for a correct final answer.","zh":"AIME 是数学竞赛，答案为 0 至 999 的整数。在 AI 评测中，模型需要解题并给出该整数；推理文字再流畅，也不能替代正确的最终答案。"},"title":{"en":"What the model must produce","zh":"模型需要交付什么"}},{"id":"years","source_ids":["data2024","data2025"],"text":{"en":"The commonly used 2024 collection contains 30 problems from AIME I and II. The 2025 collection also contains 30, but different, problems. Equal sizes do not make scores interchangeable. This guide does not cover AIME 2026.","zh":"常用的 2024 题集包含当年 AIME I 和 II 共 30 题；2025 题集也有 30 题，但题目不同。题量一致不意味着成绩可互换。本导读不覆盖 AIME 2026。"},"title":{"en":"Keep 2024 and 2025 separate","zh":"2024 与 2025 要分开看"}},{"id":"grading","source_ids":["methodology"],"text":{"en":"Evaluators must extract the model's final answer before checking it. Artificial Analysis documents its prompt and mathematical answer-checking procedure. Record these details alongside tool access and reasoning limits, because they define the evaluated setup.","zh":"评分前需要从输出中提取最终答案。Artificial Analysis 公布了提示词和数学答案检查流程。应同时记录这些细节、工具权限和推理限制，因为它们共同定义实际评测条件。"},"title":{"en":"Answer extraction is part of the test","zh":"答案提取也是评测的一部分"}},{"id":"samples","source_ids":["matharena"],"text":{"en":"Ask whether a reported percentage averages attempts, accepts any successful attempt, or grades a voted answer. These answer different questions about reliability and compute. MathArena exposes run configuration and saved outputs so the protocol can be inspected.","zh":"阅读百分比时应确认：是对多次尝试取平均、任意一次成功即可，还是对投票选出的答案评分。这些分别衡量不同的可靠性与计算投入；MathArena 提供运行配置和已保存输出供核对。"},"title":{"en":"Distinguish repeated attempts and voting","zh":"区分重复尝试与投票"}},{"id":"interpretation","source_ids":["data2024","data2025"],"text":{"en":"On one 30-question run, one additional correct answer changes accuracy by about 3.33 percentage points. This is arithmetic, not an uncertainty estimate. Inspect per-problem outcomes and repeated runs before treating a small lead as broad mathematical superiority.","zh":"单次完整作答 30 题时，多答对一题就会提高约 3.33 个百分点。这只是算术关系，并不是误差区间。应结合逐题结果和重复实验判断，小幅领先不足以说明全面数学优势。"},"title":{"en":"Read small differences cautiously","zh":"小幅分差怎样理解"}}],"sources":[{"id":"maa","title":"MAA · AIME competition and integer answer format","url":"https://maa.org/maa-invitational-competitions/"},{"id":"data2024","title":"Hugging Face H4 · AIME 2024 dataset","url":"https://huggingface.co/datasets/HuggingFaceH4/aime_2024"},{"id":"data2025","title":"MathArena · AIME 2025 dataset","url":"https://huggingface.co/datasets/MathArena/aime_2025"},{"id":"aa","title":"Artificial Analysis · AIME 2025 evaluation","url":"https://artificialanalysis.ai/evaluations/aime-2025"},{"id":"methodology","title":"Artificial Analysis · Mathematical answer extraction and evaluation","url":"https://artificialanalysis.ai/methodology/intelligence-benchmarking"},{"id":"matharena","title":"MathArena · Evaluation code and run configurations","url":"https://github.com/eth-sri/matharena"}],"summary":{"en":"AIME asks for precise integer answers to difficult mathematics problems. The year, number of attempts and answer selection rule are essential parts of any model comparison.","zh":"AIME 要求模型为较难的数学题给出准确整数答案。比较成绩时，年份、尝试次数和最终答案的选择规则缺一不可。"},"title":"AIME 2024 and 2025"},"evidence_summary":{"document_count":1,"model_count":115,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aime-2025:deepseek-v3.1","reported_at":"2025-01-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2025?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:aime-2025","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_c99837e299b2d648"},"modality":"text","name":"AIME 2025","openness":"unknown","publisher":null,"released":"2025-02-12","released_reference":{"basis":"dataset_published","note":"The source explicitly evaluates all 30 AIME I and II problems. MAA dates the second exam to February 12, completing the 2025 problem set.","source_key":"llm-stats:aime-2025","source_url":"https://maa.org/news/aime-thresholds-are-available/"},"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":115,"score_direction":"higher_is_better","score_summary":{"display_max":100.0,"display_multiplier":100,"model_count":115,"model_count_basis":"source_model_id","numeric_count":115,"raw_max":1.0,"raw_min":0.067,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aime-2025:gemini-3-pro-preview","reported_date":"2025-11-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2025?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Opus 4"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude Opus 4.1"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Sonnet 4"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude Sonnet 4.5"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2-Speciale"},{"date":"2026-01-22","date_precision":"model_announcement","name":"ERNIE 5.0"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B High"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 20B High"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 High"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 Medium"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Codex High"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Codex Mini"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 High"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Instant"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Medium"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Thinking"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Pro"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok 4 Fast"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Grok-3"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Grok-3 Mini"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4 Heavy"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE-236B-A23B"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 8B V1"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama-3.3 Nemotron Super 49B v1"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 2506"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Min  istral 3 (3B Reasoning 2512)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Reasoning 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Reasoning 2512)"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"Nemotron Nano 9B v2"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 30B A3B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 32B"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Lite"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Pro"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step-3.5-Flash"},{"date":"2026-01-15","date_precision":"model_announcement","name":"Step3-VL-10B"}],"slug":"llm-stats-aime-2025","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2025?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eth-sri/matharena","source_record":"catalog_b3ca1ee4206be75c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"All 30 problems from the 2026 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.","descriptions":{"en":"All 30 problems from the 2026 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning."},"evidence_summary":{"document_count":1,"model_count":21,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aime-2026:seed-2.0-lite","reported_at":"2026-02-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2026?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:aime-2026","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_b3ca1ee4206be75c"},"modality":"text","name":"AIME 2026","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":21,"score_direction":"higher_is_better","score_summary":{"display_max":99.2,"display_multiplier":100,"model_count":21,"model_count_basis":"source_model_id","numeric_count":21,"raw_max":0.992,"raw_min":0.375,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aime-2026:glm-5.2","reported_date":"2026-06-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2026?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Sakana Namazu"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Lite"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Pro"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"}],"slug":"llm-stats-aime-2026","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2026?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"AIR-BENCH Live is a self-evolving safety benchmark for foundation models, with an automated pipeline that updates risk taxonomy and prompts based on new regulations. It evaluates model safety across multilingual prompts.","descriptions":{"en":"AIR-BENCH Live is a self-evolving safety benchmark for foundation models, with an automated pipeline that updates risk taxonomy and prompts based on new regulations. It evaluates model safety across multilingual prompts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.22671","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_air-bench_2ac1e887"},"modality":null,"name":"AIR-BENCH","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22671","source_url":"https://arxiv.org/abs/2607.22671"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22671","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22671","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AIR-Bench 2024 is a safety benchmark grounded in risk categories derived from government regulations and company policies. It evaluates policy-grounded refusal across a broad regulatory and policy-derived harm taxonomy, using category-specific LLM-judge prompts that reward safe engagement rather than only penalizing unsafe responses.","descriptions":{"en":"AIR-Bench 2024 is a safety benchmark grounded in risk categories derived from government regulations and company policies. It evaluates policy-grounded refusal across a broad regulatory and policy-derived harm taxonomy, using category-specific LLM-judge prompts that reward safe engagement rather than only penalizing unsafe responses."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:air-bench:mai-thinking-1","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/air-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:air-bench","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"llm_stats_id","primary":"safety-alignment","source_record":"bm_air-bench_2ac1e887"},"modality":"text","name":"AIR-Bench","openness":"unknown","publisher":null,"released":"2024-07-11","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:air-bench","source_url":"https://arxiv.org/abs/2407.17436"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":88.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.88,"raw_min":0.88,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:air-bench:mai-thinking-1","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/air-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"}],"slug":"llm-stats-air-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/air-bench?top_n=500","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":135,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2402.07729","repository":null,"source_record":"review_air_bench_audio"},"categories":["知识","Knowledge","ACL 2024","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AIR-Bench is the first benchmark designed to evaluate the\nability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench 是第一个旨在评估 LALMs 理解各种音频信号（包括人类语言、自然声音和音乐）能力的基准，并进一步评估其以文本形式与人类互动的能力。AIR-Bench 包括两个维度：基础基准和聊天基准。前者由19个任务组成，包含约19,000个单选题，旨在检查LALMs的基本单任务能力。后者包含2,000个开放式问答数据实例，直接评估模型对复杂音频的理解及其遵循指令的能力。","descriptions":{"en":"AIR-Bench is the first benchmark designed to evaluate the\nability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format.","zh":"AIR-Bench 是第一个旨在评估 LALMs 理解各种音频信号（包括人类语言、自然声音和音乐）能力的基准，并进一步评估其以文本形式与人类互动的能力。AIR-Bench 包括两个维度：基础基准和聊天基准。前者由19个任务组成，包含约19,000个单选题，旨在检查LALMs的基本单任务能力。后者包含2,000个开放式问答数据实例，直接评估模型对复杂音频的理解及其遵循指令的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1069","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"artifact:arxiv:2402.07729","primary":"speech-audio","source_record":"review_air_bench_audio"},"modality":null,"name":"AIR-Bench","openness":"unknown","publisher":"Zhejiang University, Alibaba Group","released":"2024-02-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1069-air-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AIR-Bench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AIR-Bench 2024, the first policy-aligned AI safety benchmark, structures 8 government regulations and 16 corporate policies into four security tiers, with 5,694 diverse prompts spanning these categories. AIR-Bench 2024是首个与新兴政府法规和企业政策相一致的 AI 安全基准， 将 8 项政府法规和 16 项企业政策分解为四级安全分类，涵盖了这些类别的 5,694 个多样化的提示。","descriptions":{"en":"AIR-Bench 2024, the first policy-aligned AI safety benchmark, structures 8 government regulations and 16 corporate policies into four security tiers, with 5,694 diverse prompts spanning these categories.","zh":"AIR-Bench 2024是首个与新兴政府法规和企业政策相一致的 AI 安全基准， 将 8 项政府法规和 16 项企业政策分解为四级安全分类，涵盖了这些类别的 5,694 个多样化的提示。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1557","languages":[],"library_direction":null,"modality":null,"name":"AIRBench-2024","openness":"restricted","publisher":"Virtue AI,  Virginia Tech, University of California, Los Angeles, etc.","released":"2024-08-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1557-airbench-2024","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AIRBench-2024","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AirGroundBench evaluates multi-view spatial intelligence in multimodal large language models through UAV-UGV collaborative tasks, including 62,000 dual-view multiple-choice questions and 115 navigation episodes across 11 simulated environments.","descriptions":{"en":"AirGroundBench evaluates multi-view spatial intelligence in multimodal large language models through UAV-UGV collaborative tasks, including 62,000 dual-view multiple-choice questions and 115 navigation episodes across 11 simulated environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.28049","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_airgroundbench_74494075"},"modality":null,"name":"AirGroundBench","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.28049","source_url":"https://arxiv.org/abs/2606.28049"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-28049","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28049","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Goal-oriented Vision-and-Language Navigation (VLN) requires agents to locate and reach targets described in natural language without prescribed routes.","descriptions":{"en":"Goal-oriented Vision-and-Language Navigation (VLN) requires agents to locate and reach targets described in natural language without prescribed routes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10421","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics","spatial-3d"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_airgroundvln_fe5081ef"},"modality":null,"name":"AirGroundVLN","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10421","source_url":"https://arxiv.org/abs/2610.10421"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10421","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10421","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AIRTBench is a benchmark designed to evaluate large language models (LLMs) on their autonomous AI red teaming capabilities. AIRTBench 是一个专为评估大型语言模型（LLM）在“红队”安全任务中的自主攻击能力而设计的评测基准。本基准包含 70 个黑盒 CTF（夺旗赛）挑战，模拟真实 AI/ML 系统漏洞环境，要求模型独立编写 Python 代码进行漏洞发现、利用与夺旗操作，体现其计划、推理与系统操控等综合能力。","descriptions":{"en":"AIRTBench is a benchmark designed to evaluate large language models (LLMs) on their autonomous AI red teaming capabilities.","zh":"AIRTBench 是一个专为评估大型语言模型（LLM）在“红队”安全任务中的自主攻击能力而设计的评测基准。本基准包含 70 个黑盒 CTF（夺旗赛）挑战，模拟真实 AI/ML 系统漏洞环境，要求模型独立编写 Python 代码进行漏洞发现、利用与夺旗操作，体现其计划、推理与系统操控等综合能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1995","languages":[],"library_direction":null,"modality":null,"name":"AIRTBench","openness":"open","publisher":"dreadnode","released":"2025-06-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1995-airtbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AIRTBench","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":6,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:zai-org/aise-bench","source_record":"bm_aise-bench_0ef6ecf5"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AISE-Bench is a benchmark for evaluating multi-step API-using LLM agents in information seeking on academic knowledge graphs. It comprises 1,133 QA pairs with API trajectories, validated parameters, and grounded answers, and evaluates answer quality, reference grounding, API-planning correctness, and execution success.","descriptions":{"en":"AISE-Bench is a benchmark for evaluating multi-step API-using LLM agents in information seeking on academic knowledge graphs. It comprises 1,133 QA pairs with API trajectories, validated parameters, and grounded answers, and evaluates answer quality, reference grounding, API-planning correctness, and execution success."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.20498","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","tool-use"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_aise-bench_0ef6ecf5"},"modality":null,"name":"AISE-Bench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20498","source_url":"https://arxiv.org/abs/2607.20498"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20498","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20498","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":104,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:imnearth/coat","source_record":"catalog_15356ad8448b717b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Android-In-The-Zoo (AitZ) benchmark for evaluating autonomous GUI agents on smartphones. Contains 18,643 screen-action pairs with chain-of-action-thought annotations spanning over 70 Android apps. Designed to connect perception (screen layouts and UI elements) with cognition (action decision-making) for natural language-triggered smartphone task completion.","descriptions":{"en":"Android-In-The-Zoo (AitZ) benchmark for evaluating autonomous GUI agents on smartphones. Contains 18,643 screen-action pairs with chain-of-action-thought annotations spanning over 70 Android apps. Designed to connect perception (screen layouts and UI elements) with cognition (action decision-making) for natural language-triggered smartphone task completion."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:aitz-em:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aitz-em?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:aitz-em","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_15356ad8448b717b"},"modality":"multimodal","name":"AITZ_EM","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":83.2,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.832,"raw_min":0.819,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:aitz-em:qwen2.5-vl-72b","reported_date":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aitz-em?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-aitz-em","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aitz-em?top_n=500","topics":["computer-use","agents","multimodal","reasoning-math"],"unit":null},{"aliases":["A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":null,"source_record":"bm_a-pathway-to-general-purpose-scientific-ai_1bd05759"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"ALD/E-ImageMiner is a benchmark of 1,951 scientific figures from 205 publications, expert-annotated for classification, table extraction, summarization, and visual question answering.","descriptions":{"en":"ALD/E-ImageMiner is a benchmark of 1,951 scientific figures from 205 publications, expert-annotated for classification, table extraction, summarization, and visual question answering."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14075","languages":[],"library_direction":{"basis":"model-review","categories":["science","document-ocr","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_a-pathway-to-general-purpose-scientific-ai_1bd05759"},"modality":null,"name":"ALD/E-ImageMiner","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14075","source_url":"https://arxiv.org/abs/2608.14075"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14075","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14075","topics":["multimodal","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ALE-Bench is a benchmark for evaluating AI systems on score-based algorithmic programming contests. ALE-Bench 是一个用于评估 AI 系统在基于分数的算法编程竞赛中的基准测试。","descriptions":{"en":"ALE-Bench is a benchmark for evaluating AI systems on score-based algorithmic programming contests.","zh":"ALE-Bench 是一个用于评估 AI 系统在基于分数的算法编程竞赛中的基准测试。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1969","languages":["English","Japanese"],"library_direction":null,"modality":null,"name":"ALE-Bench","openness":"restricted","publisher":"SakanaAI, Japan , The University of Tokyo, Japan , AtCoder, etc.","released":"2025-06-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1969-ale-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ALE-Bench","topics":["coding"],"unit":null},{"aliases":["Benchmarking Open-Ended Multi-Agent Coordination in Language Agents"],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"alem is a JAX-based, long-horizon environment for evaluating coordination among language-agent teams through survival, exploration, crafting, trading, and combat.","descriptions":{"en":"alem is a JAX-based, long-horizon environment for evaluating coordination among language-agent teams through survival, exploration, crafting, trading, and combat."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.08340","languages":[],"library_direction":null,"modality":null,"name":"alem","openness":"unknown","publisher":null,"released":"2026-06-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08340","source_url":"https://arxiv.org/abs/2606.08340"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08340","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08340","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:xzx34/algoworlds","source_record":"bm_algoworlds_02d60d03"},"categories":["Agents","Agents & Tool Use","General AI","Tool use"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates language-model agents on 240 algorithmic worlds across ten combinatorial optimization families, assessing structured final decisions for feasibility and exact global optimality using certified optima.","descriptions":{"en":"Evaluates language-model agents on 240 algorithmic worlds across ten combinatorial optimization families, assessing structured final decisions for feasibility and exact global optimality using certified optima."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29397","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","math"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_algoworlds_02d60d03"},"modality":null,"name":"AlgoWorlds","openness":"unknown","publisher":null,"released":"2026-08-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29397","source_url":"https://arxiv.org/abs/2608.29397"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29397","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29397","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":431,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:thudm/alignbench","source_record":"catalog_a4a4b02f3db1d72c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","roleplay","language","general","creativity","writing"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AlignBench is a comprehensive multi-dimensional benchmark for evaluating Chinese alignment of Large Language Models. It contains 8 main categories: Fundamental Language Ability, Advanced Chinese Understanding, Open-ended Questions, Writing Ability, Logical Reasoning, Mathematics, Task-oriented Role Play, and Professional Knowledge. The benchmark includes 683 real-scenario rooted queries with human-verified references and uses a rule-calibrated multi-dimensional LLM-as-Judge approach with Chain-of-Thought for evaluation.","descriptions":{"en":"AlignBench is a comprehensive multi-dimensional benchmark for evaluating Chinese alignment of Large Language Models. It contains 8 main categories: Fundamental Language Ability, Advanced Chinese Understanding, Open-ended Questions, Writing Ability, Logical Reasoning, Mathematics, Task-oriented Role Play, and Professional Knowledge. The benchmark includes 683 real-scenario rooted queries with human-verified references and uses a rule-calibrated multi-dimensional LLM-as-Judge approach with Chain-of-Thought for evaluation."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:alignbench:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/alignbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:alignbench","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","multilingual"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_a4a4b02f3db1d72c"},"modality":"text","name":"AlignBench","openness":"unknown","publisher":null,"released":"2023-11-30","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: AlignBench: Benchmarking Chinese Alignment of Large Language Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:alignbench","source_url":"https://arxiv.org/abs/2311.18743"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":81.6,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.816,"raw_min":0.721,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:alignbench:qwen-2.5-72b-instruct","reported_date":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/alignbench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"}],"slug":"llm-stats-alignbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/alignbench?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":431,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"artifact:gh:thudm/alignbench","repository":"gh:thudm/alignbench","source_record":"catalog_a4a4b02f3db1d72c"},"categories":["理解","Understanding","ACL 2024","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ALIGNBENCH is a comprehensive multidimensional benchmark for evaluating LLMs’ alignment in Chinese. We tailor a humanin-the-loop data curation pipeline, containing 8 main categories, 683 real-scenario rooted queries and corresponding human verified references. AlignBench 是一个用于评估中文大语言模型对齐性能的全面、多维度的评测基准。AlignBench 构建了人类参与的数据构建流程，来保证评测数据的动态更新。AlignBench 采用多维度、规则校准的模型评价方法（LLM-as-Judge），并且结合思维链（Chain-of-Thought）生成对模型回复的多维度分析和最终的综合评分，增强了评测的高可靠性和可解释性。","descriptions":{"en":"ALIGNBENCH is a comprehensive multidimensional benchmark for evaluating LLMs’ alignment in Chinese. We tailor a humanin-the-loop data curation pipeline, containing 8 main categories, 683 real-scenario rooted queries and corresponding human verified references.","zh":"AlignBench 是一个用于评估中文大语言模型对齐性能的全面、多维度的评测基准。AlignBench 构建了人类参与的数据构建流程，来保证评测数据的动态更新。AlignBench 采用多维度、规则校准的模型评价方法（LLM-as-Judge），并且结合思维链（Chain-of-Thought）生成对模型回复的多维度分析和最终的综合评分，增强了评测的高可靠性和可解释性。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1075","languages":["Chinese"],"library_direction":{"basis":"model-review","categories":["chat-writing","multilingual"],"join":"artifact:gh:thudm/alignbench","primary":"chat-writing","source_record":"catalog_a4a4b02f3db1d72c"},"modality":null,"name":"AlignBench","openness":"unknown","publisher":"THUDM","released":"2024-08-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1075-alignbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AlignBench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object is unclear from the provided information.","descriptions":{"en":"The evaluation object is unclear from the provided information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14573","languages":[],"library_direction":null,"modality":null,"name":"Alipay-PIBench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14573","source_url":"https://arxiv.org/abs/2607.14573"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14573","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14573","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","Robotics & Embodied AI","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A paired benchmark tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in an abstract text-based formulation and a code-construction task.","descriptions":{"en":"A paired benchmark tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in an abstract text-based formulation and a code-construction task."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.23332","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_allocbench_5aa496ac"},"modality":null,"name":"AllocBench","openness":"unknown","publisher":null,"released":"2026-07-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23332","source_url":"https://arxiv.org/abs/2607.23332"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23332","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23332","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jbpayton/almanac","source_record":"bm_almanac_f919dcd3"},"categories":["Agents","Factuality","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates agent memory systems on generated multi-month conversation lives, testing time, plans, provenance, absence, tasks, and hygiene via 215 questions.","descriptions":{"en":"Evaluates agent memory systems on generated multi-month conversation lives, testing time, plans, provenance, absence, tasks, and hygiene via 215 questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:jbpayton/almanac","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_almanac_f919dcd3"},"modality":null,"name":"Almanac","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:jbpayton/almanac","source_url":"https://github.com/jbpayton/almanac"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-jbpayton-almanac","source":"claire_radar","source_url":"https://github.com/jbpayton/almanac","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":7,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":"gh:qcri/almieyar-oryx-bloombench","source_record":"bm_almieyar-oryx-bloombench_1b7c3bae"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language models on six cognitive levels (Remember to Create) with bilingual English-Arabic image-question-answer tasks.","descriptions":{"en":"Evaluates vision-language models on six cognitive levels (Remember to Create) with bilingual English-Arabic image-question-answer tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.05531","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","multilingual"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_almieyar-oryx-bloombench_1b7c3bae"},"modality":null,"name":"Almieyar-Oryx-BloomBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05531","source_url":"https://arxiv.org/abs/2606.05531"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05531","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05531","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":2019,"github_stars_30d":null,"hf_dataset_downloads":14206,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:tatsu-lab/alpaca_eval","source_record":"catalog_6e0dd16efc4427f8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","creativity","writing"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AlpacaEval 2.0 is a length-controlled automatic evaluator for instruction-following language models that uses GPT-4 Turbo to assess model responses against a baseline. It evaluates models on 805 diverse instruction-following tasks including creative writing, classification, programming, and general knowledge questions. The benchmark achieves 0.98 Spearman correlation with ChatBot Arena while being fast (< 3 minutes) and affordable (< $10 in OpenAI credits). It addresses length bias in automatic evaluation through length-controlled win-rates and uses weighted scoring based on response quality.","descriptions":{"en":"AlpacaEval 2.0 is a length-controlled automatic evaluator for instruction-following language models that uses GPT-4 Turbo to assess model responses against a baseline. It evaluates models on 805 diverse instruction-following tasks including creative writing, classification, programming, and general knowledge questions. The benchmark achieves 0.98 Spearman correlation with ChatBot Arena while being fast (< 3 minutes) and affordable (< $10 in OpenAI credits). It addresses length bias in automatic evaluation through length-controlled win-rates and uses weighted scoring based on response quality."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:alpacaeval-2.0:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/alpacaeval-2.0?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:alpacaeval-2.0","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_6e0dd16efc4427f8"},"modality":"text","name":"AlpacaEval 2.0","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":62.68,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.6268,"raw_min":0.3516,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:alpacaeval-2.0:granite-3.3-8b-base","reported_date":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/alpacaeval-2.0?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"}],"slug":"llm-stats-alpacaeval-2-0","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/alpacaeval-2.0?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:hervemottaran/alzheimer-classification-benchmark","source_record":"bm_alzheimer-classification-benchmark_abb5f538"},"categories":["Biology & Drug Discovery","Robotics & Embodied AI","Software & AI Compute","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Comparative evaluation of 11 classifiers for Alzheimer's disease detection on a synthetic health dataset, using KNIME workflows and a fixed experimental framework with sensitivity as primary metric.","descriptions":{"en":"Comparative evaluation of 11 classifiers for Alzheimer's disease detection on a synthetic health dataset, using KNIME workflows and a fixed experimental framework with sensitivity as primary metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:hervemottaran/alzheimer-classification-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science","health-medicine"],"join":"claire_record_id","primary":"science","source_record":"bm_alzheimer-classification-benchmark_abb5f538"},"modality":null,"name":"alzheimer-classification-benchmark","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:hervemottaran/alzheimer-classification-benchmark","source_url":"https://github.com/hervemottaran/alzheimer-classification-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-hervemottaran-alzheimer-classification-benchmark","source":"claire_radar","source_url":"https://github.com/hervemottaran/alzheimer-classification-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates aerial manipulation policy learning with a modular simulation suite covering embodiments, 12 tasks, disturbances, controllers, and baseline algorithms.","descriptions":{"en":"Evaluates aerial manipulation policy learning with a modular simulation suite covering embodiments, 12 tasks, disturbances, controllers, and baseline algorithms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.00641","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_am-bench_5de2497d"},"modality":null,"name":"AM-Bench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00641","source_url":"https://arxiv.org/abs/2609.00641"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00641","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00641","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","大语言模型","LLM","代码工程","Code"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AMBROSIA is a new benchmark for recognizing and interpreting ambiguous requests in text-to-SQL. It contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries. AMBROSIA是识别和解释text-to-SQL中歧义请求的新基准，其中包含三种不同类型的歧义（范围歧义、附件歧义和模糊性）问题、它们的解释和相应的SQL查询。","descriptions":{"en":"AMBROSIA is a new benchmark for recognizing and interpreting ambiguous requests in text-to-SQL. It contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries.","zh":"AMBROSIA是识别和解释text-to-SQL中歧义请求的新基准，其中包含三种不同类型的歧义（范围歧义、附件歧义和模糊性）问题、它们的解释和相应的SQL查询。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1280","languages":[],"library_direction":null,"modality":null,"name":"AMBROSIA","openness":"unknown","publisher":"University of Edinburgh","released":"2024-06-27","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1280-ambrosia","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AMBROSIA","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"American Mathematics Competition problems from the 2022-23 academic year, consisting of multiple-choice mathematics competition problems designed for high school students. These problems require advanced mathematical reasoning, problem-solving strategies, and mathematical knowledge covering topics like algebra, geometry, number theory, and combinatorics. The benchmark is derived from the official AMC competitions sponsored by the Mathematical Association of America.","descriptions":{"en":"American Mathematics Competition problems from the 2022-23 academic year, consisting of multiple-choice mathematics competition problems designed for high school students. These problems require advanced mathematical reasoning, problem-solving strategies, and mathematical knowledge covering topics like algebra, geometry, number theory, and combinatorics. The benchmark is derived from the official AMC competitions sponsored by the Mathematical Association of America."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:amc-2022-23:gemini-1.5-flash","reported_at":"2024-05-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/amc-2022-23?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:amc-2022-23","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_b3237d7431b6fe34"},"modality":"text","name":"AMC_2022_23","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":52.0,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.52,"raw_min":0.348,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:amc-2022-23:mistral-large-3-675B-instruct-2512-eagle","reported_date":"2025-12-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/amc-2022-23?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Base)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 Eagle)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 NVFP4)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512)"}],"slug":"llm-stats-amc-2022-23","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/amc-2022-23?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"AmchiBias is a benchmark for measuring socio-cultural stereotypical bias for Goan identity groups, with 313 minimal pairs across eight sociodemographic dimensions in English and Devanagari Konkani.","descriptions":{"en":"AmchiBias is a benchmark for measuring socio-cultural stereotypical bias for Goan identity groups, with 313 minimal pairs across eight sociodemographic dimensions in English and Devanagari Konkani."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15191","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_amchibias_ca7c8175"},"modality":null,"name":"AmchiBias","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15191","source_url":"https://arxiv.org/abs/2606.15191"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15191","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15191","topics":["language","safety"],"unit":null},{"aliases":["Amharic-Tokenization-Benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bedr-ux/amharic-tokenization-benchmark","source_record":"bm_amharic-tokenization-benchmark_645319af"},"categories":["General AI","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates tokenizer fertility and fragmentation for Amharic versus English using 10 parallel sentence pairs, reporting fertility ratio and characters per token for standard, multilingual, and Amharic-specialized LLM tokenizers.","descriptions":{"en":"Evaluates tokenizer fertility and fragmentation for Amharic versus English using 10 parallel sentence pairs, reporting fertility ratio and characters per token for standard, multilingual, and Amharic-specialized LLM tokenizers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:bedr-ux/amharic-tokenization-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_amharic-tokenization-benchmark_645319af"},"modality":null,"name":"Amharic-Tokenization-Benchmark","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:bedr-ux/amharic-tokenization-benchmark","source_url":"https://github.com/bedr-ux/Amharic-Tokenization-Benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-bedr-ux-amharic-tokenization-benchmark","source":"claire_radar","source_url":"https://github.com/bedr-ux/Amharic-Tokenization-Benchmark","topics":[],"unit":null},{"aliases":["AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis"],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"An open-source medical-unlearning benchmark containing 70,560 question-answer pairs from 8,820 patient notes across 11 disease categories, with factual-recall and clinical-reasoning evaluations.","descriptions":{"en":"An open-source medical-unlearning benchmark containing 70,560 question-answer pairs from 8,820 patient notes across 11 disease categories, with factual-recall and clinical-reasoning evaluations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30599","languages":[],"library_direction":null,"modality":null,"name":"AMNESIA","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30599","source_url":"https://arxiv.org/abs/2605.30599"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30599","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30599","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":166,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":36,"join":"llm_stats_id","repository":null,"source_record":"catalog_f084adf51c3b4343"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AMO Bench is an olympiad-level mathematics benchmark that evaluates advanced mathematical problem-solving and multi-step reasoning on competition-style problems.","descriptions":{"en":"AMO Bench is an olympiad-level mathematics benchmark that evaluates advanced mathematical problem-solving and multi-step reasoning on competition-style problems."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:amo-bench:mai-code-1-flash","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/amo-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:amo-bench","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_f084adf51c3b4343"},"modality":"text","name":"AMO Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":40.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.4,"raw_min":0.4,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:amo-bench:mai-code-1-flash","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/amo-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"}],"slug":"llm-stats-amo-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/amo-bench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":166,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:meituan-longcat/amo-bench","source_record":"bm_amo-bench-p_ed92afee"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AMO-Bench-P is a 39-problem subset of AMO-Bench with parser-graded answers (numbers, sets, variables), excluding description-type answers that require an LLM judge.","descriptions":{"en":"AMO-Bench-P is a 39-problem subset of AMO-Bench with parser-graded answers (numbers, sets, variables), excluding description-type answers that require an LLM judge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:djalexj/amo-bench-p","languages":[],"library_direction":null,"modality":null,"name":"amo-bench-p","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:djalexj/amo-bench-p","source_url":"https://huggingface.co/datasets/djalexj/amo-bench-p"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-djalexj-amo-bench-p","source":"claire_radar","source_url":"https://huggingface.co/datasets/djalexj/amo-bench-p","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":370,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_ampbench-mt_ce75d23c"},"categories":["Biology & Drug Discovery","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"AMPBench-MT evaluates antimicrobial peptide prediction across binary recognition, species-conditioned potency regression, and endpoint-specific safety readouts (hemolysis, toxicity, selectivity) under a sequence-homology-controlled protocol. It includes 13 source databases and multiple task configurations.","descriptions":{"en":"AMPBench-MT evaluates antimicrobial peptide prediction across binary recognition, species-conditioned potency regression, and endpoint-specific safety readouts (hemolysis, toxicity, selectivity) under a sequence-homology-controlled protocol. It includes 13 source databases and multiple task configurations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.25518","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_ampbench-mt_ce75d23c"},"modality":null,"name":"AMPBench-MT","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25518","source_url":"https://arxiv.org/abs/2607.25518"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25518","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25518","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","创作","Creation","AMS","Circuit","EDA","科学智能","AI for Science","图像理解","Image Understanding","语言生成","Generation","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AMSbench is a benchmark of ~8000 questions to evaluate multi-modal LLMs on analog/mixed-signal circuit tasks like schematic recognition, analysis, and design. AMSbench 是一个包含约8000道题目的基准测试集，用于评估多模态大语言模型在模拟/混合信号电路任务中的表现，包括识图、分析与设计。","descriptions":{"en":"AMSbench is a benchmark of ~8000 questions to evaluate multi-modal LLMs on analog/mixed-signal circuit tasks like schematic recognition, analysis, and design.","zh":"AMSbench 是一个包含约8000道题目的基准测试集，用于评估多模态大语言模型在模拟/混合信号电路任务中的表现，包括识图、分析与设计。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1966","languages":[],"library_direction":null,"modality":"multimodal","name":"AMSbench","openness":"unknown","publisher":"宁波东方理工大学","released":"2025-06-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1966-amsbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AMSbench","topics":["multimodal","reasoning-math","language","science"],"unit":null},{"aliases":["analog-design-bench"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":235,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_analog-design-bench_8213864b"},"categories":["Chip Design & EDA","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Fifty analog and mixed-signal circuit design tasks in the SkyWater SKY130 process where an agent delivers a SPICE netlist that a hidden ngspice verifier checks against every specification.","descriptions":{"en":"Fifty analog and mixed-signal circuit design tasks in the SkyWater SKY130 process where an agent delivers a SPICE netlist that a hidden ngspice verifier checks against every specification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:arcadia-2026/analog-design-bench","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_analog-design-bench_8213864b"},"modality":null,"name":"Analog Design Bench","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:arcadia-2026/analog-design-bench","source_url":"https://huggingface.co/datasets/Arcadia-2026/analog-design-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-arcadia-2026-analog-design-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/Arcadia-2026/analog-design-bench","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_anchorbench_ca7f11f6"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AnchorBench evaluates anchoring effects in LLMs across multiple pathways and anchor relevance, measuring answer shifts through controlled prompts.","descriptions":{"en":"AnchorBench evaluates anchoring effects in LLMs across multiple pathways and anchor relevance, measuring answer shifts through controlled prompts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14320","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_anchorbench_ca7f11f6"},"modality":null,"name":"AnchorBench","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14320","source_url":"https://arxiv.org/abs/2608.14320"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14320","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14320","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":6,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:scut-dlvclab/ancient_bench","source_record":"bm_ancient-bench_967fba8e"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Ancient Chinese artifact text recognition is fundamental to heritage digitization, and benchmarks for ancient texts are essential for evaluating current model capabilities.","descriptions":{"en":"Ancient Chinese artifact text recognition is fundamental to heritage digitization, and benchmarks for ancient texts are essential for evaluating current model capabilities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.27169","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_ancient-bench_967fba8e"},"modality":null,"name":"Ancient-Bench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27169","source_url":"https://arxiv.org/abs/2608.27169"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27169","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27169","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":38909,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_b20ceafccb29be99"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Android device control benchmark using high exact match evaluation metric for assessing agent performance on mobile interface tasks","descriptions":{"en":"Android device control benchmark using high exact match evaluation metric for assessing agent performance on mobile interface tasks"},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:android-control-high-em:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-high-em?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:android-control-high-em","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_b20ceafccb29be99"},"modality":"multimodal","name":"Android Control High_EM","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":69.6,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.696,"raw_min":0.601,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:android-control-high-em:qwen2.5-vl-32b","reported_date":"2025-02-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-high-em?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-android-control-high-em","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-high-em?top_n=500","topics":["computer-use","agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":38909,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_4574108d375ebc00"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Android control benchmark evaluating autonomous agents on mobile device interaction tasks with low exact match scoring criteria","descriptions":{"en":"Android control benchmark evaluating autonomous agents on mobile device interaction tasks with low exact match scoring criteria"},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:android-control-low-em:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-low-em?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:android-control-low-em","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_4574108d375ebc00"},"modality":"multimodal","name":"Android Control Low_EM","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":93.7,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.937,"raw_min":0.914,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:android-control-low-em:qwen2.5-vl-72b","reported_date":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-low-em?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-android-control-low-em","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-low-em?top_n=500","topics":["computer-use","agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":302,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:android-bench/android-bench","source_record":"catalog_328649396c915a4d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AndroidBench evaluates coding agents on Android application development tasks.","descriptions":{"en":"AndroidBench evaluates coding agents on Android application development tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:androidbench:qwen3.8-max","reported_at":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:androidbench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_328649396c915a4d"},"modality":"multimodal","name":"AndroidBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":75.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.751,"raw_min":0.751,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:androidbench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-androidbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidbench?top_n=500","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Mobile & Personal Computing","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"AndroidDaily evaluates mobile GUI agents on 350 daily-use tasks across 94 closed-source Android applications. Automatic scoring is based on a three-tiered guideline system (operational obligations, output quality, negative constraints), with step-level diagnostic judgments.","descriptions":{"en":"AndroidDaily evaluates mobile GUI agents on 350 daily-use tasks across 94 closed-source Android applications. Automatic scoring is based on a three-tiered guideline system (operational obligations, output quality, negative constraints), with step-level diagnostic judgments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27761","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_androiddaily_b00e0630"},"modality":null,"name":"AndroidDaily","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27761","source_url":"https://arxiv.org/abs/2605.27761"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27761","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27761","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":947,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_b2b65b1f73bb83a1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AndroidWorld evaluates an agent's ability to operate in real Android GUI environments, completing multi-step tasks by perceiving screen content and executing touch/type actions.","descriptions":{"en":"AndroidWorld evaluates an agent's ability to operate in real Android GUI environments, completing multi-step tasks by perceiving screen content and executing touch/type actions."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:androidworld:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:androidworld","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_b2b65b1f73bb83a1"},"modality":"multimodal","name":"AndroidWorld","openness":"unknown","publisher":null,"released":"2024-05-23","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:androidworld","source_url":"https://arxiv.org/abs/2405.14573"},"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":85.3,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.853,"raw_min":0.703,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:androidworld:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-androidworld","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld?top_n=500","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":947,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_c762c77f5f399234"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AndroidWorld Success Rate (SR) benchmark - A dynamic benchmarking environment for autonomous agents operating on Android devices. Evaluates agents on 116 programmatic tasks across 20 real-world Android apps using multimodal inputs (screen screenshots, accessibility trees, and natural language instructions). Measures success rate of agents completing tasks like sending messages, creating calendar events, and navigating mobile interfaces. Published at ICLR 2025. Best current performance: 30.6% success rate (M3A agent) vs 80.0% human performance.","descriptions":{"en":"AndroidWorld Success Rate (SR) benchmark - A dynamic benchmarking environment for autonomous agents operating on Android devices. Evaluates agents on 116 programmatic tasks across 20 real-world Android apps using multimodal inputs (screen screenshots, accessibility trees, and natural language instructions). Measures success rate of agents completing tasks like sending messages, creating calendar events, and navigating mobile interfaces. Published at ICLR 2025. Best current performance: 30.6% success rate (M3A agent) vs 80.0% human performance."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:androidworld-sr:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld-sr?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:androidworld-sr","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_c762c77f5f399234"},"modality":"multimodal","name":"AndroidWorld_SR","openness":"unknown","publisher":null,"released":"2024-05-23","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:androidworld-sr","source_url":"https://arxiv.org/abs/2405.14573"},"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":71.1,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.711,"raw_min":0.22,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:androidworld-sr:qwen3.5-35b-a3b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld-sr?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-androidworld-sr","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld-sr?top_n=500","topics":["computer-use","agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:zjudbxai/anestrace","source_record":"bm_anestrace_2f56493b"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Intraoperative anesthesia requires systems to interpret evolving multimodal evidence, recommend timely management, and revise decisions as patient states change, yet existing benchmarks usually isolate perception or single-point reasoning.","descriptions":{"en":"Intraoperative anesthesia requires systems to interpret evolving multimodal evidence, recommend timely management, and revise decisions as patient states change, yet existing benchmarks usually isolate perception or single-point reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.32740","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_anestrace_2f56493b"},"modality":null,"name":"AnesTRACE","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32740","source_url":"https://arxiv.org/abs/2609.32740"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32740","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32740","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","Medical Imagery","Aneurysm","Computational Fluid Dynamics","科学智能","AI for Science","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Based on 427 real aneurysm geometries, we synthesized 10,660 3D shapes via controlled deformation to simulate aneurysm evolution. CFD computations were performed on each shape under eight steady-state mass flow conditions, generating a total of 85,280 blood flow dynamics data covering key parameters Based on 427 real aneurysm geometries, we synthesized 10,660 3D shapes via controlled deformation to simulate aneurysm evolution. CFD computations were performed on each shape under eight steady-state mass flow conditions, generating a total of 85,280 blood flow dynamics data covering key parameters","descriptions":{"en":"Based on 427 real aneurysm geometries, we synthesized 10,660 3D shapes via controlled deformation to simulate aneurysm evolution. CFD computations were performed on each shape under eight steady-state mass flow conditions, generating a total of 85,280 blood flow dynamics data covering key parameters","zh":"Based on 427 real aneurysm geometries, we synthesized 10,660 3D shapes via controlled deformation to simulate aneurysm evolution. CFD computations were performed on each shape under eight steady-state mass flow conditions, generating a total of 85,280 blood flow dynamics data covering key parameters"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1876","languages":[],"library_direction":null,"modality":"multimodal","name":"Aneumo","openness":"restricted","publisher":"Shanghai Academy of Artificial Intelligence for Science","released":"2025-05-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1876-aneumo","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Aneumo","topics":["multimodal","reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_animation2code_7763a440"},"categories":["Code","Code & Software","Code generation","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Animation2Code evaluates temporal visual reasoning in video-to-code generation. It includes 1,069 web animation videos with corresponding HTML/CSS/JavaScript implementations, and uses appearance and temporal similarity metrics to assess model performance.","descriptions":{"en":"Animation2Code evaluates temporal visual reasoning in video-to-code generation. It includes 1,069 web animation videos with corresponding HTML/CSS/JavaScript implementations, and uses appearance and temporal similarity metrics to assess model performance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.28593","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","video-understanding"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_animation2code_7763a440"},"modality":null,"name":"Animation2Code","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.28593","source_url":"https://arxiv.org/abs/2606.28593"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-28593","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28593","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:yan-liu05/anipro","source_record":"bm_anipro-sfd-bench_812496de"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates provenance detection for anime images across Real, Inpainting, and Text2Image categories using statistical features derived from five semantic dimensions.","descriptions":{"en":"Evaluates provenance detection for anime images across Real, Inpainting, and Text2Image categories using statistical features derived from five semantic dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.23345","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_anipro-sfd-bench_812496de"},"modality":null,"name":"AniPrO-SFD-Bench","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23345","source_url":"https://arxiv.org/abs/2609.23345"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23345","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23345","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.HC"],"collected_at":"2026-10-14T09:44:51Z","description":"AnnoBench evaluates visualization annotation generation across four representation formats, five chart description conditions, and two prompt specification levels. It uses a VLM-as-a-judge protocol aligned with human assessment to score annotation quality.","descriptions":{"en":"AnnoBench evaluates visualization annotation generation across four representation formats, five chart description conditions, and two prompt specification levels. It uses a VLM-as-a-judge protocol aligned with human assessment to score annotation quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.25911","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_annobench_f60658c7"},"modality":null,"name":"AnnoBench","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25911","source_url":"https://arxiv.org/abs/2607.25911"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25911","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25911","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:abdullahuseyinli-dot/annomi-counselling-dialogue-analysis","source_record":"bm_annomi-counselling-dialogue-analysis_beb022e0"},"categories":["Biology & Drug Discovery","Finance","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates therapist-behaviour classification on the AnnoMI counselling-dialogue corpus using transcript-grouped splits, comparing elastic-net and RoBERTa models with metrics including macro-F1 and accuracy.","descriptions":{"en":"Evaluates therapist-behaviour classification on the AnnoMI counselling-dialogue corpus using transcript-grouped splits, comparing elastic-net and RoBERTa models with metrics including macro-F1 and accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:abdullahuseyinli-dot/annomi-counselling-dialogue-analysis","languages":[],"library_direction":null,"modality":null,"name":"AnnoMI Counselling Dialogue Analysis","openness":"unknown","publisher":null,"released":"2026-08-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:abdullahuseyinli-dot/annomi-counselling-dialogue-analysis","source_url":"https://github.com/abdullahuseyinli-dot/annomi-counselling-dialogue-analysis"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-abdullahuseyinli-dot-annomi-counselling-dialogue-analysis","source":"claire_radar","source_url":"https://github.com/abdullahuseyinli-dot/annomi-counselling-dialogue-analysis","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"claire_record_id","repository":"gh:yonglixiang/antiskillbench","source_record":"bm_antiskillbench_e40657d8"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"AntiSkillBench is a benchmark for evaluating privacy leakage and impersonation risks in persona-skills. It includes 7,500 persona-grounded dialogue traces and evaluates across three skill-distillation strategies.","descriptions":{"en":"AntiSkillBench is a benchmark for evaluating privacy leakage and impersonation risks in persona-skills. It includes 7,500 persona-grounded dialogue traces and evaluates across three skill-distillation strategies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.03700","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_antiskillbench_e40657d8"},"modality":null,"name":"AntiSkillBench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03700","source_url":"https://arxiv.org/abs/2608.03700"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03700","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03700","topics":["agents","safety"],"unit":null},{"aliases":["Any2Poster: Any-Source Poster Generation Across Modalities and Domains"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark for poster generation from eight source modalities and five domains, scored with quiz-based fidelity probes and VLM judgments of visual quality.","descriptions":{"en":"A benchmark for poster generation from eight source modalities and five domains, scored with quiz-based fidelity probes and VLM judgments of visual quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.02915","languages":[],"library_direction":null,"modality":null,"name":"Any2Poster Bench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02915","source_url":"https://arxiv.org/abs/2606.02915"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02915","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02915","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Mobile & Personal Computing","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Mobile GUI agents deployed in real settings must work across different applications that support the same functionality.","descriptions":{"en":"Mobile GUI agents deployed in real settings must work across different applications that support the same functionality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34139","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_anyappbench_787c3e24"},"modality":null,"name":"AnyAppBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34139","source_url":"https://arxiv.org/abs/2609.34139"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34139","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34139","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:rinost081/anygroundbench","source_record":"bm_anygroundbench_08e0374e"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"AnyGroundBench evaluates video grounding in vision-language models across five specialized domains (animal, industry, sports, surgery, public security) with spatio-temporal annotations. It provides training and test splits per domain for zero-shot and in-context learning evaluation.","descriptions":{"en":"AnyGroundBench evaluates video grounding in vision-language models across five specialized domains (animal, industry, sports, surgery, public security) with spatio-temporal annotations. It provides training and test splits per domain for zero-shot and in-context learning evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.02269","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_anygroundbench_08e0374e"},"modality":null,"name":"AnyGroundBench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02269","source_url":"https://arxiv.org/abs/2607.02269"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02269","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02269","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"AOR-Bench is a benchmark of 3,000 pseudo-harmful audio samples across six categories, designed to evaluate over-refusal in large audio language models. It assesses whether models incorrectly reject benign audio queries.","descriptions":{"en":"AOR-Bench is a benchmark of 3,000 pseudo-harmful audio samples across six categories, designed to evaluate over-refusal in large audio language models. It assesses whether models incorrectly reject benign audio queries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21147","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","speech-audio"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_aor-bench_86ee63c7"},"modality":null,"name":"AOR-Bench","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21147","source_url":"https://arxiv.org/abs/2606.21147"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21147","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21147","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Apex is a challenging frontier reasoning benchmark testing advanced multi-step problem solving across difficult STEM and logical tasks.","descriptions":{"en":"Apex is a challenging frontier reasoning benchmark testing advanced multi-step problem solving across difficult STEM and logical tasks."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:apex:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:apex","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_846be5fc541c5039"},"modality":"text","name":"Apex","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":84.8,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.848,"raw_min":0.227,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:apex:nemotron-3-ultra-550b-a55b","reported_date":"2026-06-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-apex","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"APEX-Accounting evaluates frontier models on expert-authored accounting and bookkeeping work conducted across simulated accounting systems and supporting files.","descriptions":{"en":"APEX-Accounting evaluates frontier models on expert-authored accounting and bookkeeping work conducted across simulated accounting systems and supporting files."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27189","languages":[],"library_direction":null,"modality":null,"name":"APEX-Accounting","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27189","source_url":"https://arxiv.org/abs/2607.27189"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27189","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27189","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_0c16e811aeab795e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"APEX-Agents is a benchmark evaluating AI agents on long horizon professional tasks that require sustained reasoning, planning, and execution across complex multi-step workflows.","descriptions":{"en":"APEX-Agents is a benchmark evaluating AI agents on long horizon professional tasks that require sustained reasoning, planning, and execution across complex multi-step workflows."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:apex-agents:gemini-3.1-pro-preview","reported_at":"2026-02-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-agents?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:apex-agents","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_0c16e811aeab795e"},"modality":"text","name":"APEX-Agents","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":57.49999999999999,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.575,"raw_min":0.256,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:apex-agents:grok-4.6","reported_date":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-agents?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-apex-agents","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-agents?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["APEX-Agents","APEX","Apex","Apex Shortlist"],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"artifact:arxiv:2601.14242","repository":null,"source_record":"catalog_0c16e811aeab795e"},"categories":["agent"],"collected_at":null,"description":"Expert-authored professional tasks with rubric grading; graders are LLMs.","descriptions":{"en":"Expert-authored professional tasks with rubric grading; graders are LLMs."},"evidence_summary":{"document_count":5,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000apex_agents\u0000google_gemini_3_1_pro_model_card\u0000apex_agents\u0000Thinking (High)\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"first_score_reported_at":"2026-02-19","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000apex_agents\u0000google_gemini_3_1_pro_model_card\u0000apex_agents\u0000Thinking (High)\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:apex_agents","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"artifact:arxiv:2601.14242","primary":"professional-work","source_record":"catalog_0c16e811aeab795e"},"modality":null,"name":"APEX-Agents","openness":"unknown","publisher":null,"released":"2026-01-20","released_reference":{"basis":"benchmark_release","source_key":"model-reports:apex_agents","source_url":"https://arxiv.org/abs/2601.14242"},"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":41.0,"display_multiplier":1,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":41.0,"raw_min":33.0,"source_reference":{"obs_id":"curated\u0000apex_agents\u0000moonshot_kimi_k3_model_card\u0000apex_agents\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000apex_agents\u0000moonshot_kimi_k3_model_card\u0000apex_agents\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"}],"slug":"apex_agents","source":"model_reports","source_url":"https://arxiv.org/abs/2601.14242","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["agentic","reasoning"],"collected_at":"2026-08-25T10:41:06Z","description":"Long-horizon agentic tasks","descriptions":{"en":"Long-horizon agentic tasks"},"evidence_summary":{"document_count":1,"model_count":29,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:apex-agents-aa:36f73aaf-d38a-4b56-a2b3-d04d17186910","reported_at":"2025-08-05","source_url":"https://artificialanalysis.ai/evaluations/apex-agents-aa"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:apex-agents-aa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"APEX-Agents-AA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":29,"score_direction":"higher_is_better","score_summary":{"display_max":47.0501474926254,"display_multiplier":100,"model_count":29,"model_count_basis":"source_model_id","numeric_count":29,"raw_max":0.470501474926254,"raw_min":0.00737463126843658,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:apex-agents-aa:0097ebf5-124f-42f6-9463-33b00e711f03","reported_date":"2026-05-19","source_url":"https://artificialanalysis.ai/evaluations/apex-agents-aa"},"unit":null},"scored_models":[{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"}],"slug":"artificial-analysis-apex-agents-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/apex-agents-aa","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Finance","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Professional agents increasingly work through command-line interfaces and multiple tools over shared workspaces containing files, messages, and business applications.","descriptions":{"en":"Professional agents increasingly work through command-line interfaces and multiple tools over shared workspaces containing files, messages, and business applications."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13623","languages":[],"library_direction":{"basis":"model-review-daily","categories":["safety-alignment","professional-work"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_apex-harm_ffdf9511"},"modality":null,"name":"APEX-Harm","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13623","source_url":"https://arxiv.org/abs/2610.13623"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13623","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13623","topics":["agents","safety","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_019660c276df8920"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"APEX-SWE evaluates AI agents on software engineering tasks requiring multi-step coding, debugging, and verification.","descriptions":{"en":"APEX-SWE evaluates AI agents on software engineering tasks requiring multi-step coding, debugging, and verification."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:apex-swe:grok-4.6","reported_at":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-swe?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:apex-swe","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_019660c276df8920"},"modality":"text","name":"APEX-SWE","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":56.39999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.564,"raw_min":0.564,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:apex-swe:grok-4.6","reported_date":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-swe?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"}],"slug":"llm-stats-apex-swe","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-swe?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_apex-voice_2a66e71e"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Full-duplex voice agents can now listen, speak, use tools, and act during spoken interactions, but fluent dialogue does not guarantee correct completion of delegated professional workflows.","descriptions":{"en":"Full-duplex voice agents can now listen, speak, use tools, and act during spoken interactions, but fluent dialogue does not guarantee correct completion of delegated professional workflows."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34973","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","professional-work","tool-use"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_apex-voice_2a66e71e"},"modality":null,"name":"APEX-Voice","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34973","source_url":"https://arxiv.org/abs/2609.34973"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34973","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34973","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:advancedphotonsource/apexa-aps-beamline-assistant","source_record":"bm_apexa-bench_2d46435d"},"categories":["Agents","Language & Knowledge","Scientific Facilities"],"collected_at":"2026-10-14T09:44:51Z","description":"APEXA-Bench evaluates multi-agent LLM automation on 58 synchrotron data reduction tasks organized by a four-class physical-consequence taxonomy, with grading against NIST-traceable lattice constants in a cross-detector slice.","descriptions":{"en":"APEXA-Bench evaluates multi-agent LLM automation on 58 synchrotron data reduction tasks organized by a four-class physical-consequence taxonomy, with grading against NIST-traceable lattice constants in a cross-detector slice."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.24165","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_apexa-bench_2d46435d"},"modality":null,"name":"APEXA-Bench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24165","source_url":"https://arxiv.org/abs/2609.24165"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24165","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24165","topics":["agents","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_dd09bda504101131"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive benchmark for tool-augmented LLMs that evaluates API planning, retrieval, and calling capabilities. Contains 314 tool-use dialogues with 753 API calls across 73 API tools, designed to assess how effectively LLMs can utilize external tools and overcome obstacles in tool leveraging.","descriptions":{"en":"A comprehensive benchmark for tool-augmented LLMs that evaluates API planning, retrieval, and calling capabilities. Contains 314 tool-use dialogues with 753 API calls across 73 API tools, designed to assess how effectively LLMs can utilize external tools and overcome obstacles in tool leveraging."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:api-bank:llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/api-bank?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:api-bank","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_dd09bda504101131"},"modality":"text","name":"API-Bank","openness":"unknown","publisher":null,"released":"2023-04-14","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:api-bank","source_url":"https://arxiv.org/abs/2304.08244"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":92.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.92,"raw_min":0.826,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:api-bank:llama-3.1-405b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/api-bank?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"}],"slug":"llm-stats-api-bank","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/api-bank?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:postmanlabs/apiflow-bench","source_record":"bm_apiflow-bench_b256ba9f"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates tool-using agents on 467 long-horizon REST API tasks with deterministic provenance-sensitive grading, measuring reliability, efficiency, robustness, and other engineering capabilities.","descriptions":{"en":"Evaluates tool-using agents on 467 long-horizon REST API tasks with deterministic provenance-sensitive grading, measuring reliability, efficiency, robustness, and other engineering capabilities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29128","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_apiflow-bench_b256ba9f"},"modality":null,"name":"APIFlow-Bench","openness":"unknown","publisher":null,"released":"2026-08-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29128","source_url":"https://arxiv.org/abs/2608.29128"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29128","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29128","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:liningbest/apitest","source_record":"bm_apitest_7d8e6901"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Not applicable; excluded.","descriptions":{"en":"Not applicable; excluded."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:liningbest/apitest","languages":[],"library_direction":null,"modality":null,"name":"apitest","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:liningbest/apitest","source_url":"https://github.com/liningbest/apitest"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-liningbest-apitest","source":"claire_radar","source_url":"https://github.com/liningbest/apitest","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":47,"join":"claire_record_id","repository":"gh:jianguo-huang11/apm-bench","source_record":"bm_apm-bench_312d0e12"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"To serve as real-world personal assistants, streaming video models need persistent memory that retains past experiences for later use.","descriptions":{"en":"To serve as real-world personal assistants, streaming video models need persistent memory that retains past experiences for later use."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.37559","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","agent-memory"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_apm-bench_312d0e12"},"modality":null,"name":"APM-Bench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37559","source_url":"https://arxiv.org/abs/2609.37559"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37559","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37559","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"APPS is a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. APPS 是一个代码生成评测基准，该评测基准测量模型根据任意自然语言规范生成令人满意的 Python 代码的能力。","descriptions":{"en":"APPS is a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code.","zh":"APPS 是一个代码生成评测基准，该评测基准测量模型根据任意自然语言规范生成令人满意的 Python 代码的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1093","languages":[],"library_direction":null,"modality":null,"name":"APPS","openness":"open","publisher":"UC Berkeley","released":"2021-11-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1093-apps","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/APPS","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":4,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:acrab-agentic-labs/appsim","source_record":"bm_appsim-bench_0821b282"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Controllable simulated apps for mobile GUI agent evaluation, containing 557 tasks across 17 Chinese and English apps with deterministic backend data and outcome-based verification.","descriptions":{"en":"Controllable simulated apps for mobile GUI agent evaluation, containing 557 tasks across 17 Chinese and English apps with deterministic backend data and outcome-based verification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.07712","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["computer-use","multilingual"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_appsim-bench_0821b282"},"modality":null,"name":"AppSim-Bench","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07712","source_url":"https://arxiv.org/abs/2609.07712"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07712","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07712","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Scientific Facilities","physics.acc-ph"],"collected_at":"2026-10-14T09:44:51Z","description":"APS-Bench is a 50-question QA dataset with auditable gold answers for evaluating retrieval-augmented generation over scientific facility operations knowledge.","descriptions":{"en":"APS-Bench is a 50-question QA dataset with auditable gold answers for evaluating retrieval-augmented generation over scientific facility operations knowledge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24663","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","science"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_aps-bench_6b2027af"},"modality":null,"name":"APS-Bench","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24663","source_url":"https://arxiv.org/abs/2607.24663"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24663","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24663","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Document databases such as MongoDB are core infrastructure for modern applications, and natural-language interfaces to them---text-to-MQL---would let non-experts query complex, semi-structured data without mastering the query language.","descriptions":{"en":"Document databases such as MongoDB are core infrastructure for modern applications, and natural-language interfaces to them---text-to-MQL---would let non-experts query complex, semi-structured data without mastering the query language."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.02770","languages":[],"library_direction":{"basis":"model-review-daily","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_aptmql-bench_60d58b06"},"modality":null,"name":"AptMQL-Bench","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.02770","source_url":"https://arxiv.org/abs/2610.02770"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-02770","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02770","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["数学","Math","大语言模型","LLM","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AQUA-RAT contains the algebraic word problems. The dataset consists of about 100,000 algebraic word problems with natural language rationales. AQUA-RAT 包含代数文字问题。该数据集由约 100,000 道带有自然语言推理的代数文字问题组成。","descriptions":{"en":"AQUA-RAT contains the algebraic word problems. The dataset consists of about 100,000 algebraic word problems with natural language rationales.","zh":"AQUA-RAT 包含代数文字问题。该数据集由约 100,000 道带有自然语言推理的代数文字问题组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1116","languages":[],"library_direction":null,"modality":null,"name":"AQUA-RAT","openness":"open","publisher":"DeepMind","released":"2017-10-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1116-aqua-rat","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AQUA-RAT","topics":["reasoning-math"],"unit":null},{"aliases":["ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"ARAC-Bench evaluates auto-research systems on alignment and completeness of research trajectories across proposal, experiment, and synthesis stages using rubrics derived from academic cognition skills.","descriptions":{"en":"ARAC-Bench evaluates auto-research systems on alignment and completeness of research trajectories across proposal, experiment, and synthesis stages using rubrics derived from academic cognition skills."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12788","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","science"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_arac-benchmarking-auto-research-s-alignmen_57539495"},"modality":null,"name":"ARAC-Bench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12788","source_url":"https://arxiv.org/abs/2608.12788"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12788","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12788","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:pavelgolikov/arbigraph","source_record":"bm_arbigraph_c1f2133e"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"ArbiGraph is a benchmark generator for evaluating tool-assisted language agents' context management via scalable task graphs with exact automatic verification.","descriptions":{"en":"ArbiGraph is a benchmark generator for evaluating tool-assisted language agents' context management via scalable task graphs with exact automatic verification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.20764","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","long-context"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_arbigraph_c1f2133e"},"modality":null,"name":"ArbiGraph","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20764","source_url":"https://arxiv.org/abs/2607.20764"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20764","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20764","topics":["agents","reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_404fcfb394d23199"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The Abstraction and Reasoning Corpus (ARC) is a benchmark designed to measure human-like general fluid intelligence through grid-based reasoning tasks. It consists of 800 tasks (400 training, 400 evaluation) where each task presents input-output grids that require understanding abstract patterns and transformations. Test-takers must produce exactly correct output grids for all test inputs in a task to solve it, with 3 trials allowed per test input. ARC aims to enable fair comparisons of general intelligence between AI systems and humans using priors designed to be as close as possible to innate human priors.","descriptions":{"en":"The Abstraction and Reasoning Corpus (ARC) is a benchmark designed to measure human-like general fluid intelligence through grid-based reasoning tasks. It consists of 800 tasks (400 training, 400 evaluation) where each task presents input-output grids that require understanding abstract patterns and transformations. Test-takers must produce exactly correct output grids for all test inputs in a task to solve it, with 3 trials allowed per test input. ARC aims to enable fair comparisons of general intelligence between AI systems and humans using priors designed to be as close as possible to innate human priors."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arc:gemini-2.5-flash-lite","reported_at":"2025-06-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:arc","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_404fcfb394d23199"},"modality":"multimodal","name":"Arc","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":2.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.025,"raw_min":0.025,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arc:gemini-2.5-flash-lite","reported_date":"2025-06-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"}],"slug":"llm-stats-arc","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4852,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_220634b4ce946da2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","spatial_reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) is a benchmark designed to test general intelligence and abstract reasoning capabilities through visual grid-based transformation tasks. Each task consists of 2-5 demonstration pairs showing input grids transformed into output grids according to underlying rules, with test-takers required to infer these rules and apply them to novel test inputs. The benchmark uses colored grids (up to 30x30) with 10 discrete colors/symbols, designed to measure human-like general fluid intelligence and skill-acquisition efficiency with minimal prior knowledge.","descriptions":{"en":"The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) is a benchmark designed to test general intelligence and abstract reasoning capabilities through visual grid-based transformation tasks. Each task consists of 2-5 demonstration pairs showing input grids transformed into output grids according to underlying rules, with test-takers required to infer these rules and apply them to novel test inputs. The benchmark uses colored grids (up to 30x30) with 10 discrete colors/symbols, designed to measure human-like general fluid intelligence and skill-acquisition efficiency with minimal prior knowledge."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arc-agi:o3-2025-04-16","reported_at":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:arc-agi","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_220634b4ce946da2"},"modality":"image","name":"ARC-AGI","openness":"unknown","publisher":null,"released":"2019-11-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: On the Measure of Intelligence. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:arc-agi","source_url":"https://arxiv.org/abs/1911.01547"},"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":95.0,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.95,"raw_min":0.418,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arc-agi:gpt-5.5","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Pro"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"}],"slug":"llm-stats-arc-agi","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["ARC-AGI","ARC-AGI-1","ARC-AGI-2"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Compute per task is reported alongside score by the maintainers and should not be dropped.","descriptions":{"en":"Compute per task is reported alongside score by the maintainers and should not be dropped."},"evidence_summary":{"document_count":4,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000arc_agi\u0000bytedance_seed2_0_model_card\u0000arc_agi_1\u0000ARC-AGI-1, as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000arc_agi\u0000bytedance_seed2_0_model_card\u0000arc_agi_1\u0000ARC-AGI-1, as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:arc_agi","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"ARC-AGI","openness":"unknown","publisher":null,"released":"2019-11-05","released_reference":{"basis":"benchmark_release","source_key":"model-reports:arc_agi","source_url":"https://arcprize.org/"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":85.4,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":85.4,"raw_min":72.3,"source_reference":{"obs_id":"curated\u0000arc_agi\u0000bytedance_seed2_0_model_card\u0000arc_agi_1\u0000ARC-AGI-1, as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000arc_agi\u0000bytedance_seed2_0_model_card\u0000arc_agi_1\u0000ARC-AGI-1, as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"arc_agi","source":"model_reports","source_url":"https://arcprize.org/","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":744,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_4dae675f25ab2857"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","spatial_reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ARC-AGI-2 is an upgraded benchmark for measuring abstract reasoning and problem-solving abilities in AI systems through visual grid transformation tasks. It evaluates fluid intelligence via input-output grid pairs (1x1 to 30x30) using colored cells (0-9), requiring models to identify underlying transformation rules from demonstration examples and apply them to test cases. Designed to be easy for humans but challenging for AI, focusing on core cognitive abilities like spatial reasoning, pattern recognition, and compositional generalization.","descriptions":{"en":"ARC-AGI-2 is an upgraded benchmark for measuring abstract reasoning and problem-solving abilities in AI systems through visual grid transformation tasks. It evaluates fluid intelligence via input-output grid pairs (1x1 to 30x30) using colored cells (0-9), requiring models to identify underlying transformation rules from demonstration examples and apply them to test cases. Designed to be easy for humans but challenging for AI, focusing on core cognitive abilities like spatial reasoning, pattern recognition, and compositional generalization."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arc-agi-v2:o3-2025-04-16","reported_at":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:arc-agi-v2","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_4dae675f25ab2857"},"modality":"multimodal","name":"ARC-AGI v2","openness":"unknown","publication":{"evidence":[{"note":"The ARC Prize Foundation announcement introduces the second ARC-AGI benchmark, explains visual-grid reasoning and links the official public training and evaluation tasks.","url":"https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025"},{"note":"The official repository explicitly contains ARC-AGI-2 task data and describes input/output grids with dimensions 1x1 to 30x30 and colors 0-9, matching the source record. It distinguishes public, semi-private and private evaluation sets.","url":"https://github.com/arcprize/ARC-AGI-2/blob/main/readme.md"}],"id":"arc-agi-2-publication","links":[{"kind":"website","title":"ARC-AGI-2 official launch","url":"https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025"},{"kind":"repo","title":"ARC-AGI-2 official public task repository","url":"https://github.com/arcprize/ARC-AGI-2"}],"record_keys":["llm-stats:arc-agi-v2"],"reviewed_at":"2026-10-26","scope":{"en":"ARC-AGI-2's original release and public task repository. The repository does not expose the separate private evaluation sets or establish which split produced a collected score.","zh":"ARC-AGI-2 的原始发布与公开任务仓库。仓库不包含独立的私有测试集，也不能据此确认某条已收录成绩使用了哪个划分。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":85.0,"display_multiplier":100,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":0.85,"raw_min":0.049,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arc-agi-v2:gpt-5.5","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Opus 4"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Pro"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"}],"slug":"llm-stats-arc-agi-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-v2?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["ARC-AGI-2","ARC-AGI 2"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Cost per task is part of the official result and is routinely dropped when the score is quoted on its own.","descriptions":{"en":"Cost per task is part of the official result and is routinely dropped when the score is quoted on its own."},"evidence_summary":{"document_count":2,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000arc_agi_2\u0000google_gemini_3_1_pro_model_card\u0000arc_agi_2\u0000Thinking (High), ARC Prize Verified\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"first_score_reported_at":"2026-02-19","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000arc_agi_2\u0000google_gemini_3_1_pro_model_card\u0000arc_agi_2\u0000Thinking (High), ARC Prize Verified\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:arc_agi_2","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"ARC-AGI-2","openness":"unknown","publisher":null,"released":"2025-03-24","released_reference":{"basis":"benchmark_release","source_key":"model-reports:arc_agi_2","source_url":"https://arcprize.org/arc-agi/2/"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":77.1,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":77.1,"raw_min":77.1,"source_reference":{"obs_id":"curated\u0000arc_agi_2\u0000google_gemini_3_1_pro_model_card\u0000arc_agi_2\u0000Thinking (High), ARC Prize Verified\u0000Gemini 3.1 Pro","observation_id":"curated\u0000arc_agi_2\u0000google_gemini_3_1_pro_model_card\u0000arc_agi_2\u0000Thinking (High), ARC Prize Verified\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","reported_date":"2026-02-19","source_id":"google_gemini_3_1_pro_model_card","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"unit":"percent"},"scored_models":[{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"}],"slug":"arc_agi_2","source":"model_reports","source_url":"https://arcprize.org/arc-agi/2/","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_7d3ff108e044ad47"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ARC-AGI-3 is the third-generation Abstraction and Reasoning Corpus benchmark, an interactive-reasoning evaluation designed to measure fluid, novel problem-solving ability that remains far from saturated for frontier models.","descriptions":{"en":"ARC-AGI-3 is the third-generation Abstraction and Reasoning Corpus benchmark, an interactive-reasoning evaluation designed to measure fluid, novel problem-solving ability that remains far from saturated for frontier models."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arc-agi-3:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-3?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:arc-agi-3","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","multi-agent-games"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_7d3ff108e044ad47"},"modality":"multimodal","name":"ARC-AGI-3","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":30.2,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.302,"raw_min":0.0018,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arc-agi-3:claude-opus-5","reported_date":"2026-07-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-3?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-arc-agi-3","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-3?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["ARC-AGI-3","ARC-AGI 3"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Interactive multi-step format, so the agent harness is part of the measurement. Scores are low and spread wide, which makes small absolute gaps look larger than they are.","descriptions":{"en":"Interactive multi-step format, so the agent harness is part of the measurement. Scores are low and spread wide, which makes small absolute gaps look larger than they are."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000arc_agi_3\u0000anthropic_claude_opus_5_system_card\u0000arc_agi_3\u0000high effort, verified score, semi-private evaluation set\u0000Claude Opus 5","reported_at":"2026-07-24","source_url":"https://www.anthropic.com/news/claude-opus-5"},"first_score_reported_at":"2026-07-24","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000arc_agi_3\u0000anthropic_claude_opus_5_system_card\u0000arc_agi_3\u0000high effort, verified score, semi-private evaluation set\u0000Claude Opus 5","reported_at":"2026-07-24","source_url":"https://www.anthropic.com/news/claude-opus-5"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:arc_agi_3","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"ARC-AGI-3","openness":"unknown","publisher":null,"released":"2026-01-15","released_reference":{"basis":"benchmark_release","source_key":"model-reports:arc_agi_3","source_url":"https://arcprize.org/arc-agi/3/"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":30.16,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":30.16,"raw_min":30.16,"source_reference":{"obs_id":"curated\u0000arc_agi_3\u0000anthropic_claude_opus_5_system_card\u0000arc_agi_3\u0000high effort, verified score, semi-private evaluation set\u0000Claude Opus 5","observation_id":"curated\u0000arc_agi_3\u0000anthropic_claude_opus_5_system_card\u0000arc_agi_3\u0000high effort, verified score, semi-private evaluation set\u0000Claude Opus 5","reported_at":"2026-07-24","reported_date":"2026-07-24","source_id":"anthropic_claude_opus_5_system_card","source_url":"https://www.anthropic.com/news/claude-opus-5"},"unit":"percent"},"scored_models":[{"date":"2026-07-24","date_precision":"document_publication","name":"Claude Opus 5"}],"slug":"arc_agi_3","source":"model_reports","source_url":"https://arcprize.org/arc-agi/3/","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_da72f57a8db8bba7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The AI2 Reasoning Challenge (ARC) Challenge Set is a multiple-choice question-answering benchmark containing grade-school level science questions that require advanced reasoning capabilities. ARC-C specifically contains questions that were answered incorrectly by both retrieval-based and word co-occurrence algorithms, making it a particularly challenging subset designed to test commonsense reasoning abilities in AI systems.","descriptions":{"en":"The AI2 Reasoning Challenge (ARC) Challenge Set is a multiple-choice question-answering benchmark containing grade-school level science questions that require advanced reasoning capabilities. ARC-C specifically contains questions that were answered incorrectly by both retrieval-based and word co-occurrence algorithms, making it a particularly challenging subset designed to test commonsense reasoning abilities in AI systems."},"evidence_summary":{"document_count":1,"model_count":34,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arc-c:claude-3-opus-20240229","reported_at":"2024-02-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-c?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"llm-stats:arc-c","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_da72f57a8db8bba7"},"modality":"text","name":"ARC-C","openness":"unknown","publisher":null,"released":"2018-03-14","released_reference":{"basis":"paper_first_version","note":"The ARC introduction releases both the Challenge and Easy partitions.","source_key":"opencompass:502","source_url":"https://arxiv.org/abs/1803.05457"},"repo_kind":"not_found","repo_resolution_status":"not_found","score_count":34,"score_direction":"higher_is_better","score_summary":{"display_max":97.2,"display_multiplier":100,"model_count":34,"model_count_basis":"source_model_id","numeric_count":34,"raw_max":0.972,"raw_min":0.406,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arc-c:mimo-v2.5-pro","reported_date":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-c?top_n=500"},"unit":null},"scored_models":[{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-08-30","date_precision":"model_announcement","name":"Command R+"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Base"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-arc-c","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-c?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"artifact:arxiv:1803.05457","repository":null,"source_record":"catalog_0e86e28ad3f67105"},"categories":["学科","Examination","大语言模型","LLM","知识储备","Knowledge","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The AI2’s Reasoning Challenge (ARC) dataset is a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9. The dataset is split in two partitions: Easy and Challenge, where the latter partition contains the more difficult questions that require reasoning. Most of the questions have 4 answer choices, with <1% of all the questions having either 3 or 5 answer choices. AI2的推理挑战（ARC）数据集是一个多项选择问题回答数据集，包含了从三年级到九年级的科学考试中提取的问题。该数据集分为两个部分：简单和挑战，其中后者包含了需要推理能力的更难的问题。大多数问题有4个答案选项，仅有不到1％的问题有3个或5个答案选项。","descriptions":{"en":"The AI2’s Reasoning Challenge (ARC) dataset is a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9. The dataset is split in two partitions: Easy and Challenge, where the latter partition contains the more difficult questions that require reasoning. Most of the questions have 4 answer choices, with <1% of all the questions having either 3 or 5 answer choices.","zh":"AI2的推理挑战（ARC）数据集是一个多项选择问题回答数据集，包含了从三年级到九年级的科学考试中提取的问题。该数据集分为两个部分：简单和挑战，其中后者包含了需要推理能力的更难的问题。大多数问题有4个答案选项，仅有不到1％的问题有3个或5个答案选项。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:502","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"artifact:arxiv:1803.05457","primary":"knowledge-expert-qa","source_record":"catalog_0e86e28ad3f67105"},"modality":null,"name":"ARC-c","openness":"unknown","publisher":null,"released":"2018-03-14","released_reference":{"basis":"paper_first_version","note":"The ARC introduction releases both the Challenge and Easy partitions.","source_key":"opencompass:502","source_url":"https://arxiv.org/abs/1803.05457"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-502-arc-c","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ARC-c","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_0e86e28ad3f67105"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ARC-E (AI2 Reasoning Challenge - Easy Set) is a subset of grade-school level, multiple-choice science questions that requires knowledge and reasoning capabilities. Part of the AI2 Reasoning Challenge dataset containing 5,197 questions that test scientific reasoning and factual knowledge. The Easy Set contains questions that are answerable by retrieval-based and word co-occurrence algorithms, making them more accessible than the Challenge Set.","descriptions":{"en":"ARC-E (AI2 Reasoning Challenge - Easy Set) is a subset of grade-school level, multiple-choice science questions that requires knowledge and reasoning capabilities. Part of the AI2 Reasoning Challenge dataset containing 5,197 questions that test scientific reasoning and factual knowledge. The Easy Set contains questions that are answerable by retrieval-based and word co-occurrence algorithms, making them more accessible than the Challenge Set."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arc-e:gemma-2-27b-it","reported_at":"2024-06-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-e?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:arc-e","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_0e86e28ad3f67105"},"modality":"text","name":"ARC-E","openness":"unknown","publisher":null,"released":"2018-03-14","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:arc-e","source_url":"https://arxiv.org/abs/1803.05457"},"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":88.6,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.886,"raw_min":0.607,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arc-e:gemma-2-27b-it","reported_date":"2024-06-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-e?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"}],"slug":"llm-stats-arc-e","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-e?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"artifact:arxiv:1803.05457","repository":null,"source_record":"catalog_0e86e28ad3f67105"},"categories":["学科","Examination","大语言模型","LLM","知识储备","Knowledge","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The AI2’s Reasoning Challenge (ARC) dataset is a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9. The dataset is split in two partitions: Easy and Challenge, where the latter partition contains the more difficult questions that require reasoning. Most of the questions have 4 answer choices, with <1% of all the questions having either 3 or 5 answer choices. AI2的推理挑战（ARC）数据集是一个多项选择问题回答数据集，包含了从三年级到九年级的科学考试中提取的问题。该数据集分为两个部分：简单和挑战，其中后者包含了需要推理能力的更难的问题。大多数问题有4个答案选项，仅有不到1％的问题有3个或5个答案选项。","descriptions":{"en":"The AI2’s Reasoning Challenge (ARC) dataset is a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9. The dataset is split in two partitions: Easy and Challenge, where the latter partition contains the more difficult questions that require reasoning. Most of the questions have 4 answer choices, with <1% of all the questions having either 3 or 5 answer choices.","zh":"AI2的推理挑战（ARC）数据集是一个多项选择问题回答数据集，包含了从三年级到九年级的科学考试中提取的问题。该数据集分为两个部分：简单和挑战，其中后者包含了需要推理能力的更难的问题。大多数问题有4个答案选项，仅有不到1％的问题有3个或5个答案选项。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:503","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"artifact:arxiv:1803.05457","primary":"knowledge-expert-qa","source_record":"catalog_0e86e28ad3f67105"},"modality":null,"name":"ARC-e","openness":"unknown","publisher":null,"released":"2018-03-14","released_reference":{"basis":"paper_first_version","note":"The ARC introduction releases both the Easy and Challenge partitions.","source_key":"opencompass:503","source_url":"https://arxiv.org/abs/1803.05457"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-503-arc-e","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ARC-e","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_0138a663d691f494"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ARC-AGI-2 is the second-generation Abstraction and Reasoning Corpus benchmark measuring fluid, general reasoning and abstraction.","descriptions":{"en":"ARC-AGI-2 is the second-generation Abstraction and Reasoning Corpus benchmark measuring fluid, general reasoning and abstraction."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arcagi2:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arcagi2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:arcagi2","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_0138a663d691f494"},"modality":"text","name":"ArcAGI2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":62.5,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.625,"raw_min":0.613,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arcagi2:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arcagi2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-arcagi2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arcagi2?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["ARCGym: Benchmarking Deep Reinforcement Learning in Autonomous Robotic Colonoscopy"],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for deep reinforcement learning in autonomous robotic colonoscopy, focusing on image-based navigation in clinically derived deformable colon anatomies.","descriptions":{"en":"Benchmark for deep reinforcement learning in autonomous robotic colonoscopy, focusing on image-based navigation in clinically derived deformable colon anatomies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.22803","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_arcgym-benchmarking-deep-reinforcement-lea_ebba1e07"},"modality":null,"name":"ARCGym","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.22803","source_url":"https://arxiv.org/abs/2609.22803"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-22803","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22803","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Multimodal models increasingly interpret visual environments, but their ability to recognize the same building across photographs, floor plans, elevations, sections, and renderings remains poorly characterized.","descriptions":{"en":"Multimodal models increasingly interpret visual environments, but their ability to recognize the same building across photographs, floor plans, elevations, sections, and renderings remains poorly characterized."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34047","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_arch-b_54d808d1"},"modality":null,"name":"ARCH-B","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34047","source_url":"https://arxiv.org/abs/2609.34047"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34047","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34047","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_archegraph_38554c04"},"categories":["General AI","Geometric reasoning","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ArchEGraph evaluates geometry-topology-physics aligned building energy modeling through graph reconstruction and topology-informed load prediction tasks, with standardized protocols and generalization experiments.","descriptions":{"en":"ArchEGraph evaluates geometry-topology-physics aligned building energy modeling through graph reconstruction and topology-informed load prediction tasks, with standardized protocols and generalization experiments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06772","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_archegraph_38554c04"},"modality":null,"name":"ArchEGraph","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06772","source_url":"https://arxiv.org/abs/2608.06772"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06772","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06772","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1080,"github_stars_30d":null,"hf_dataset_downloads":3593,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":"gh:lmarena/arena-hard-auto","source_record":"catalog_82bde15e43356e7f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","creativity","writing"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Arena-Hard-Auto is an automatic evaluation benchmark for instruction-tuned LLMs consisting of 500 challenging real-world prompts curated by BenchBuilder. It includes open-ended software engineering problems, mathematical questions, and creative writing tasks. The benchmark uses LLM-as-a-Judge methodology with GPT-4.1 and Gemini-2.5 as automatic judges to approximate human preference. Arena-Hard achieves 98.6% correlation with human preference rankings and provides 3x higher separation of model performances compared to MT-Bench, making it highly effective for distinguishing between models of similar quality.","descriptions":{"en":"Arena-Hard-Auto is an automatic evaluation benchmark for instruction-tuned LLMs consisting of 500 challenging real-world prompts curated by BenchBuilder. It includes open-ended software engineering problems, mathematical questions, and creative writing tasks. The benchmark uses LLM-as-a-Judge methodology with GPT-4.1 and Gemini-2.5 as automatic judges to approximate human preference. Arena-Hard achieves 98.6% correlation with human preference rankings and provides 3x higher separation of model performances compared to MT-Bench, making it highly effective for distinguishing between models of similar quality."},"evidence_summary":{"document_count":1,"model_count":26,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arena-hard:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:arena-hard","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_82bde15e43356e7f"},"modality":"text","name":"Arena Hard","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":26,"score_direction":"higher_is_better","score_summary":{"display_max":95.6,"display_multiplier":100,"model_count":26,"model_count_basis":"source_model_id","numeric_count":26,"raw_max":0.956,"raw_min":0.267,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arena-hard:qwen3-235b-a22b","reported_date":"2025-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama-3.3 Nemotron Super 49B v1"},{"date":"2025-12-04","date_precision":"model_announcement","name":"MiniStral 3 (14B Instruct 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Instruct 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Instruct 2512)"},{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"},{"date":"2025-09-01","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Instruct"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 30B A3B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 32B"}],"slug":"llm-stats-arena-hard","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":["Arena-Hard","ArenaHard"],"attention_signals":null,"categories":["human_preference"],"collected_at":null,"description":"LLM-judge dependent, with known style and length bias.","descriptions":{"en":"LLM-judge dependent, with known style and length bias."},"evidence_summary":{"document_count":4,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000arena_hard\u0000deepseek_v3_report\u0000arena_hard\u0000GPT-4-1106 judge\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"first_score_reported_at":"2024-12-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000arena_hard\u0000deepseek_v3_report\u0000arena_hard\u0000GPT-4-1106 judge\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:arena_hard","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"artifact:gh:lmarena/arena-hard-auto","primary":"chat-writing","source_record":"catalog_0f07233db8031494"},"modality":null,"name":"Arena-Hard","openness":"unknown","publisher":null,"released":"2024-04-19","released_reference":{"basis":"benchmark_release","source_key":"model-reports:arena_hard","source_url":"https://github.com/lmarena/arena-hard-auto"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":95.6,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":95.6,"raw_min":85.5,"source_reference":{"obs_id":"curated\u0000arena_hard\u0000qwen3_technical_report\u0000arena_hard\u0000thinking mode\u0000Qwen3-235B-A22B (Thinking)","observation_id":"curated\u0000arena_hard\u0000qwen3_technical_report\u0000arena_hard\u0000thinking mode\u0000Qwen3-235B-A22B (Thinking)","reported_at":"2025-05-14","reported_date":"2025-05-14","source_id":"qwen3_technical_report","source_url":"https://arxiv.org/abs/2505.09388"},"unit":"percent"},"scored_models":[{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2025-05-14","date_precision":"document_publication","name":"Qwen3-235B-A22B (Thinking)"}],"slug":"arena_hard","source":"model_reports","source_url":"https://github.com/lmarena/arena-hard-auto","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1080,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:lmarena/arena-hard-auto","source_record":"catalog_0f07233db8031494"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","creativity","writing"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Arena-Hard-Auto v2 is a challenging benchmark consisting of 500 carefully curated prompts sourced from Chatbot Arena and WildChat-1M, designed to evaluate large language models on real-world user queries. The benchmark covers diverse domains including open-ended software engineering problems, mathematics, creative writing, and technical problem-solving. It uses LLM-as-a-Judge for automatic evaluation, achieving 98.6% correlation with human preference rankings while providing 3x higher separation of model performances compared to MT-Bench. The benchmark emphasizes prompt specificity, complexity, and domain knowledge to better distinguish between model capabilities.","descriptions":{"en":"Arena-Hard-Auto v2 is a challenging benchmark consisting of 500 carefully curated prompts sourced from Chatbot Arena and WildChat-1M, designed to evaluate large language models on real-world user queries. The benchmark covers diverse domains including open-ended software engineering problems, mathematics, creative writing, and technical problem-solving. It uses LLM-as-a-Judge for automatic evaluation, achieving 98.6% correlation with human preference rankings while providing 3x higher separation of model performances compared to MT-Bench. The benchmark emphasizes prompt specificity, complexity, and domain knowledge to better distinguish between model capabilities."},"evidence_summary":{"document_count":1,"model_count":16,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arena-hard-v2:qwen3-235b-a22b-instruct-2507","reported_at":"2025-07-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:arena-hard-v2","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_0f07233db8031494"},"modality":"text","name":"Arena-Hard v2","openness":"unknown","publication":{"evidence":[{"note":"The authors' repository links the original paper and announces Arena-Hard v2.0 on 2025-04-23. It documents 500 hard queries and an additional 250 creative-writing prompts. The 2024 paper's 98.6% correlation and 3x separation claims are not treated as evidence about the v2 evaluation configuration.","url":"https://github.com/lmarena/arena-hard-auto/blob/main/README.md"},{"note":"The introducing Arena-Hard/BenchBuilder paper is accessible and predates the v2 release; it supports benchmark-family provenance rather than the identity of each recorded v2 score.","url":"https://arxiv.org/abs/2406.11939"}],"id":"arena-hard-v2-original","links":[{"kind":"paper","title":"From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (family paper)","url":"https://arxiv.org/abs/2406.11939"},{"kind":"repo","title":"Arena-Hard-Auto repository and v2 release","url":"https://github.com/lmarena/arena-hard-auto"}],"record_keys":["llm-stats:arena-hard-v2"],"reviewed_at":"2026-10-26","scope":{"en":"Arena-Hard-Auto v2 is documented in the authors' repository; the 2024 paper introduces the benchmark family. The source record does not establish its v2 prompt subset, judge or baseline configuration.","zh":"作者仓库记录 Arena-Hard-Auto v2，2024 年论文介绍其基准系列。当前来源记录未明确 v2 的题目子集、评审模型或对照模型设置。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":16,"score_direction":"higher_is_better","score_summary":{"display_max":86.2,"display_multiplier":100,"model_count":16,"model_count_basis":"source_model_id","numeric_count":16,"raw_max":0.862,"raw_min":0.368,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arena-hard-v2:mimo-v2-flash","reported_date":"2025-12-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"}],"slug":"llm-stats-arena-hard-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard-v2?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","大语言模型","LLM","安全对齐","Safety Alignment","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Arena-Hard-Auto is an automated benchmark for instruction-tuned LLMs, designed to efficiently approximate human preferences. Arena-Hard-Auto 是一个用于评估 LLM 的基准，自动甄选 500 条高难度开放式提示，从模型区分度、人类偏好一致性与提示质量三维度进行严苛评测。依托 BenchBuilder 管道、主题建模与 LLM 裁判，实现众包数据→筛选→评分的全自动闭环。","descriptions":{"en":"Arena-Hard-Auto is an automated benchmark for instruction-tuned LLMs, designed to efficiently approximate human preferences.","zh":"Arena-Hard-Auto 是一个用于评估 LLM 的基准，自动甄选 500 条高难度开放式提示，从模型区分度、人类偏好一致性与提示质量三维度进行严苛评测。依托 BenchBuilder 管道、主题建模与 LLM 裁判，实现众包数据→筛选→评分的全自动闭环。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2075","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"artifact:arxiv:2406.11939","primary":"chat-writing","source_record":"catalog_0f07233db8031494"},"modality":null,"name":"Arena-Hard-Auto","openness":"open","publisher":"University of California, Berkeley","released":"2024-04-19","released_reference":{"basis":"release_announcement","note":"The original LMSYS announcement introduces Arena-Hard-Auto v0.1 and its leaderboard, before the June paper.","source_key":"opencompass:2075","source_url":"https://www.lmsys.org/blog/2024-04-19-arena-hard/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2075-arena-hard-auto","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Arena-Hard-Auto","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable rewards, particularly in mathematics, code, and formal logic.","descriptions":{"en":"Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable rewards, particularly in mathematics, code, and formal logic."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38409","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_arggym_6721ee0f"},"modality":null,"name":"ArgGYM","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38409","source_url":"https://arxiv.org/abs/2609.38409"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38409","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38409","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","视觉问答","Visual-Qa","物理智能","Embodied AI","逻辑推理","图像理解","Image Understanding","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"As Multimodal Large Language Models (MLLMs) continue to evolve, their cognitive and reasoning capabilities have seen remarkable progress. However, challenges in visual fine-grained perception and commonsense causal inference persist. This paper introduces Argus Inspection, a multimodal benchmark wit As Multimodal Large Language Models (MLLMs) continue to evolve, their cognitive and reasoning capabilities have seen remarkable progress. However, challenges in visual fine-grained perception and commonsense causal inference persist. This paper introduces Argus Inspection, a multimodal benchmark wit","descriptions":{"en":"As Multimodal Large Language Models (MLLMs) continue to evolve, their cognitive and reasoning capabilities have seen remarkable progress. However, challenges in visual fine-grained perception and commonsense causal inference persist. This paper introduces Argus Inspection, a multimodal benchmark wit","zh":"As Multimodal Large Language Models (MLLMs) continue to evolve, their cognitive and reasoning capabilities have seen remarkable progress. However, challenges in visual fine-grained perception and commonsense causal inference persist. This paper introduces Argus Inspection, a multimodal benchmark wit"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:2370","languages":[],"library_direction":null,"modality":"multimodal","name":"ArgusInspection","openness":"unknown","publisher":"Shanghai Artificial Intelligence Laboratory","released":"2025-10-27","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2370-argusinspection","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ArgusInspection","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":968,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_7e1829fbc34027db"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["spatial_reasoning","3d","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ARKitScenes evaluates 3D scene understanding and spatial reasoning in AR/VR contexts.","descriptions":{"en":"ARKitScenes evaluates 3D scene understanding and spatial reasoning in AR/VR contexts."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arkitscenes:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arkitscenes?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:arkitscenes","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_7e1829fbc34027db"},"modality":"multimodal","name":"ARKitScenes","openness":"unknown","publisher":null,"released":"2021-11-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:arkitscenes","source_url":"https://arxiv.org/abs/2111.08897"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.537,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.537,"raw_min":0.537,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arkitscenes:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arkitscenes?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-arkitscenes","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arkitscenes?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for robot manipulation policies evaluated on a fleet of low-cost SO-101 cells. It compares 7 policies across 12 tasks in single-arm and bimanual configurations, with 2,518 policy rollouts and 600 reference demonstrations, all labeled successful, suboptimal, or failure. Data is released in LeRobot and RoboMeter formats.","descriptions":{"en":"A benchmark for robot manipulation policies evaluated on a fleet of low-cost SO-101 cells. It compares 7 policies across 12 tasks in single-arm and bimanual configurations, with 2,518 policy rollouts and 600 reference demonstrations, all labeled successful, suboptimal, or failure. Data is released in LeRobot and RoboMeter formats."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24481","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_armnetbench_46ca6487"},"modality":null,"name":"ArmnetBench","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24481","source_url":"https://arxiv.org/abs/2607.24481"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24481","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24481","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Robotics & Embodied AI","cs.CR"],"collected_at":"2026-09-25T04:40:22Z","description":"ART-SafeBench evaluates attacks against multimodal agentic RAG systems across four attack surfaces, using attack success rate and query cost as comparable outcomes.","descriptions":{"en":"ART-SafeBench evaluates attacks against multimodal agentic RAG systems across four attack surfaces, using attack success rate and query cost as comparable outcomes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.26793","languages":[],"library_direction":null,"modality":null,"name":"ART-SafeBench","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26793","source_url":"https://arxiv.org/abs/2606.26793"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26793","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26793","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"ArtECulture is a benchmark for culture-conditioned visual emotion understanding with 6,792 artworks and culture-specific labels across three cultures. It evaluates MLLMs on predicting cultural emotion perception and explaining rationale.","descriptions":{"en":"ArtECulture is a benchmark for culture-conditioned visual emotion understanding with 6,792 artworks and culture-specific labels across three cultures. It evaluates MLLMs on predicting cultural emotion perception and explaining rationale."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03358","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_arteculture_b1d1e3cf"},"modality":null,"name":"ArtECulture","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03358","source_url":"https://arxiv.org/abs/2608.03358"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03358","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03358","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":279,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"llm_stats_id","repository":"gh:tencent-hunyuan/artifactsbenchmark","source_record":"catalog_588d925488858d2e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["frontend_development","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Artifacts Bench evaluates a model's ability to generate visual code artifacts, measuring the quality of generated interactive and visual front-end outputs from natural-language requests.","descriptions":{"en":"Artifacts Bench evaluates a model's ability to generate visual code artifacts, measuring the quality of generated interactive and visual front-end outputs from natural-language requests."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:artifacts-bench:mai-code-1-flash","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/artifacts-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:artifacts-bench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_588d925488858d2e"},"modality":"text","name":"Artifacts Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":51.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.51,"raw_min":0.364,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:artifacts-bench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/artifacts-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-artifacts-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/artifacts-bench?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":279,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"artifact:arxiv:2507.04952","repository":"gh:tencent-hunyuan/artifactsbenchmark","source_record":"catalog_588d925488858d2e"},"categories":["多模态","Multimodal","推理","Reasoning","代码","Code","ArtifactsBench","代码可视化","代码生成","多模态模型","VLM","逻辑推理","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ArtifactsBench is a benchmark with 1,825 tasks for evaluating LLM-generated visual and interactive code. It addresses the gap of traditional benchmarks that focus only on algorithmic correctness by assessing visual fidelity and user interaction. rtifactsBench 是一个专注于弥合传统代码评测中“视觉-交互”鸿沟的新型基准。它旨在全面评估大语言模型（LLM）生成动态可视化与交互式代码的能力，而非仅仅考核算法正确性。\n该基准包含1825个真实世界的任务，并开创了一套自动化多模态评估流程：系统会自动渲染代码、捕捉其视觉与交互行为，再由一个多模态大模型（MLLM）依据详细清单进行评分。该流程与人类专家判断的一致性高达94.4%，证明了其高度可靠性。\nArtifactsBench 已将数据集、评估框架及工具链完全开源，为社区提供了一个可扩展且精准的工具，以推动能创造丰富用户体验的新一代生成模型的发展。","descriptions":{"en":"ArtifactsBench is a benchmark with 1,825 tasks for evaluating LLM-generated visual and interactive code. It addresses the gap of traditional benchmarks that focus only on algorithmic correctness by assessing visual fidelity and user interaction.","zh":"rtifactsBench 是一个专注于弥合传统代码评测中“视觉-交互”鸿沟的新型基准。它旨在全面评估大语言模型（LLM）生成动态可视化与交互式代码的能力，而非仅仅考核算法正确性。\n该基准包含1825个真实世界的任务，并开创了一套自动化多模态评估流程：系统会自动渲染代码、捕捉其视觉与交互行为，再由一个多模态大模型（MLLM）依据详细清单进行评分。该流程与人类专家判断的一致性高达94.4%，证明了其高度可靠性。\nArtifactsBench 已将数据集、评估框架及工具链完全开源，为社区提供了一个可扩展且精准的工具，以推动能创造丰富用户体验的新一代生成模型的发展。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2052","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"artifact:arxiv:2507.04952","primary":"code-generation","source_record":"catalog_588d925488858d2e"},"modality":"multimodal","name":"ArtifactsBench","openness":"unknown","publisher":"Tencent","released":"2025-07-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2052-artifactsbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ArtifactsBench","topics":["coding","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Artificial Analysis benchmark evaluates AI models across quality, speed, and pricing dimensions, providing a composite assessment of model capabilities for real-world usage.","descriptions":{"en":"Artificial Analysis benchmark evaluates AI models across quality, speed, and pricing dimensions, providing a composite assessment of model capabilities for real-world usage."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:artificial-analysis:minimax-m2.7","reported_at":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/artificial-analysis?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:artificial-analysis","languages":[],"library_direction":{"basis":"model-review","categories":["composite-index"],"join":"llm_stats_id","primary":"composite-index","source_record":"catalog_373b3921ce6ac401"},"modality":"text","name":"Artificial Analysis","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":59.0,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.59,"raw_min":0.4,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:artificial-analysis:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/artificial-analysis?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"}],"slug":"llm-stats-artificial-analysis","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/artificial-analysis?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A validation framework for assessing whether synthetic populations support intended analyses.","descriptions":{"en":"A validation framework for assessing whether synthetic populations support intended analyses."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.30030","languages":[],"library_direction":null,"modality":null,"name":"Artificial Societies Benchmark","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30030","source_url":"https://arxiv.org/abs/2609.30030"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30030","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30030","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eth-sri/matharena","source_record":"catalog_3ea1a5e4de61cb92"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ArXivMath is a final-answer benchmark of research-level mathematics maintained by MathArena. Problems are extracted monthly from recent arXiv paper abstracts, then filtered through automated and manual checks to ensure they are self-contained, non-trivial, and verifiable. Because problems are drawn from active research, the benchmark is more realistic and more closely connected to mathematical research than contest or olympiad benchmarks.","descriptions":{"en":"ArXivMath is a final-answer benchmark of research-level mathematics maintained by MathArena. Problems are extracted monthly from recent arXiv paper abstracts, then filtered through automated and manual checks to ensure they are self-contained, non-trivial, and verifiable. Because problems are drawn from active research, the benchmark is more realistic and more closely connected to mathematical research than contest or olympiad benchmarks."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:arxivmath:claude-sonnet-5","reported_at":"2026-06-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arxivmath?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:arxivmath","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_3ea1a5e4de61cb92"},"modality":"text","name":"ArXivMath","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":72.2,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.722,"raw_min":0.522,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:arxivmath:claude-sonnet-5","reported_date":"2026-06-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arxivmath?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"}],"slug":"llm-stats-arxivmath","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arxivmath?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["math"],"collected_at":null,"description":"Competition-style mathematics drawn from arXiv; test-time compute budget is part of the result.","descriptions":{"en":"Competition-style mathematics drawn from arXiv; test-time compute budget is part of the result."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000arxivmath\u0000tencent_hy4_preview\u0000arxivmath\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000arxivmath\u0000tencent_hy4_preview\u0000arxivmath\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:arxivmath","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["math"],"join":"review:14H034160212:2026-10-15","primary":"math","source_record":null},"modality":null,"name":"ArXivMath","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":66.6,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":66.6,"raw_min":66.6,"source_reference":{"obs_id":"curated\u0000arxivmath\u0000tencent_hy4_preview\u0000arxivmath\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000arxivmath\u0000tencent_hy4_preview\u0000arxivmath\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"arxivmath","source":"model_reports","source_url":"https://matharena.ai/arxivmath","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["数学","Math","大语言模型","LLM","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ASDiv is a new MWP corpus that contains diverse lexicon patterns with wide problem type coverage. Each problem provides consistent equations and answers. It is further annotated with the corresponding problem type and grade level. ASDiv 是一个新的数学文字问题（MWP）语料库，包含多样的词汇模式，覆盖广泛的问题类型。每个问题提供对应的方程和答案。它进一步标注了相应的问题类型和年级水平，可用于测试系统的能力，并指明问题的难度等级。","descriptions":{"en":"ASDiv is a new MWP corpus that contains diverse lexicon patterns with wide problem type coverage. Each problem provides consistent equations and answers. It is further annotated with the corresponding problem type and grade level.","zh":"ASDiv 是一个新的数学文字问题（MWP）语料库，包含多样的词汇模式，覆盖广泛的问题类型。每个问题提供对应的方程和答案。它进一步标注了相应的问题类型和年级水平，可用于测试系统的能力，并指明问题的难度等级。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1114","languages":[],"library_direction":null,"modality":null,"name":"ASDiv","openness":"restricted","publisher":"Institute of Information Science, Academia Sinica","released":"2020-07-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1114-asdiv","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ASDiv","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":301,"github_stars_30d":13,"hf_dataset_downloads":1151,"hf_paper_upvotes":68,"join":"claire_record_id","repository":"gh:apexin-ai/asi-bench","source_record":"bm_asi-bench_faf0dd90"},"categories":["AI Scientist","Agents","Agents & Tool Use","Autonomous execution","Reasoning","Research planning","Scientific Research & AI for Science","Scientific discovery"],"collected_at":"2026-10-14T09:44:51Z","description":"ASI-Bench evaluates AI systems on 60 project-level scientific research tasks across 11 domains, with four guidance levels B1-B4 measuring autonomous execution and innovation.","descriptions":{"en":"ASI-Bench evaluates AI systems on 60 project-level scientific research tasks across 11 domains, with four guidance levels B1-B4 measuring autonomous execution and innovation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.17271","languages":[],"library_direction":{"basis":"model-review","categories":["science","ai-rnd"],"join":"claire_record_id","primary":"science","source_record":"bm_asi-bench_faf0dd90"},"modality":null,"name":"ASI-Bench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"first_public","source_key":"claire-radar:2608.17271","source_url":"https://arxiv.org/abs/2608.17271"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17271","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17271","topics":["agents","reasoning-math","science"],"unit":null},{"aliases":["ASI-Bench","ASI Bench"],"attention_signals":{"as_of":"2026-10-11","github_stars":301,"github_stars_30d":13,"hf_dataset_downloads":1151,"hf_paper_upvotes":68,"join":"artifact:arxiv:2608.17271","repository":"gh:apexin-ai/asi-bench","source_record":"bm_asi-bench_faf0dd90"},"categories":["science"],"collected_at":null,"description":"First benchmark that jointly evaluates general intelligence, innovation, and autonomous execution, via 60 project-level scientific research tasks spanning 11 domains, progressively withdrawing human guidance. Very new (arXiv 2608.17271, code at github.com/apexin-ai/ASI-Bench) and intended to probe superintelligence-level behaviour, so high variance across seeds and scaffolds should be expected before any vendor adopts it. Only the seed-31415 instances are public; the seed-42 references stay private for official leaderboard scoring, so a locally computed score and a leaderboard score are not the same measurement. This registry's maintainer is one of the benchmark's authors.","descriptions":{"en":"First benchmark that jointly evaluates general intelligence, innovation, and autonomous execution, via 60 project-level scientific research tasks spanning 11 domains, progressively withdrawing human guidance. Very new (arXiv 2608.17271, code at github.com/apexin-ai/ASI-Bench) and intended to probe superintelligence-level behaviour, so high variance across seeds and scaffolds should be expected before any vendor adopts it. Only the seed-31415 instances are public; the seed-42 references stay private for official leaderboard scoring, so a locally computed score and a leaderboard score are not the same measurement. This registry's maintainer is one of the benchmark's authors."},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000asi_bench\u0000asi_bench_results_table\u0000asi_bench_b3_60_tasks\u0000B3 guidance (only research goal, data, constraints and required outputs); 60 official seed-42 project tasks in 11 domains; no external tool access; macro-average over 3 runs except Claude-Opus-5 (single run); harness: Claude Code\u0000Claude Opus 4.8","reported_at":"2026-08-18","source_url":"https://arxiv.org/html/2608.17271v1"},"first_score_reported_at":"2026-08-18","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000asi_bench\u0000asi_bench_results_table\u0000asi_bench_b3_60_tasks\u0000B3 guidance (only research goal, data, constraints and required outputs); 60 official seed-42 project tasks in 11 domains; no external tool access; macro-average over 3 runs except Claude-Opus-5 (single run); harness: Claude Code\u0000Claude Opus 4.8","reported_at":"2026-08-18","source_url":"https://arxiv.org/html/2608.17271v1"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:asi_bench","languages":[],"library_direction":{"basis":"model-review","categories":["science","ai-rnd"],"join":"artifact:arxiv:2608.17271","primary":"science","source_record":"bm_asi-bench_faf0dd90"},"modality":null,"name":"ASI-Bench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"model-reports:asi_bench","source_url":"https://arxiv.org/abs/2608.17271"},"repo_kind":null,"repo_resolution_status":null,"score_count":18,"score_direction":"higher_is_better","score_summary":{"display_max":51.6,"display_multiplier":1,"model_count":13,"model_count_basis":"source_model_id","numeric_count":18,"raw_max":51.6,"raw_min":11.5,"source_reference":{"obs_id":"curated\u0000asi_bench\u0000asi_bench_results_table\u0000asi_bench_b3_60_tasks\u0000B3 guidance (only research goal, data, constraints and required outputs); 60 official seed-42 project tasks in 11 domains; no external tool access; macro-average over 3 runs; harness: Codex\u0000GPT-5.6 Sol (ultra)","observation_id":"curated\u0000asi_bench\u0000asi_bench_results_table\u0000asi_bench_b3_60_tasks\u0000B3 guidance (only research goal, data, constraints and required outputs); 60 official seed-42 project tasks in 11 domains; no external tool access; macro-average over 3 runs; harness: Codex\u0000GPT-5.6 Sol (ultra)","reported_at":"2026-08-18","reported_date":"2026-08-18","source_id":"asi_bench_results_table","source_url":"https://arxiv.org/html/2608.17271v1"},"unit":"percent"},"scored_models":[{"date":"2026-08-18","date_precision":"document_publication","name":"Claude Opus 4.8"},{"date":"2026-08-18","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2026-08-18","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-08-18","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-08-18","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-08-18","date_precision":"document_publication","name":"GLM-5.3"},{"date":"2026-08-18","date_precision":"document_publication","name":"GPT-5.5 (xhigh)"},{"date":"2026-08-18","date_precision":"document_publication","name":"GPT-5.6 Sol (ultra)"},{"date":"2026-08-18","date_precision":"document_publication","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-08-18","date_precision":"document_publication","name":"Kimi K2.7"},{"date":"2026-08-18","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-08-18","date_precision":"document_publication","name":"MiMo-V2.5-Pro"},{"date":"2026-08-18","date_precision":"document_publication","name":"MiniMax M3"}],"slug":"asi_bench","source":"model_reports","source_url":"https://arxiv.org/abs/2608.17271","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ASOB-Bench evaluates diffusion classifiers along three bias dimensions: attribute binding, size-order bias, and background dependency, using newly constructed datasets and extending existing frameworks.","descriptions":{"en":"ASOB-Bench evaluates diffusion classifiers along three bias dimensions: attribute binding, size-order bias, and background dependency, using newly constructed datasets and extending existing frameworks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.03831","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_asob-bench_b6e17903"},"modality":null,"name":"ASOB-Bench","openness":"unknown","publisher":null,"released":"2026-07-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03831","source_url":"https://arxiv.org/abs/2607.03831"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03831","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03831","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":273,"join":"claire_record_id","repository":null,"source_record":"bm_aspire_c720473f"},"categories":["General AI","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"ASPIRE evaluates LLM agents and models on self-evolution from vague natural-language goals, using a hidden expert-authored set of 520 items across six goals in an interactive environment.","descriptions":{"en":"ASPIRE evaluates LLM agents and models on self-evolution from vague natural-language goals, using a hidden expert-authored set of 520 items across six goals in an interactive environment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.31111","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_aspire_c720473f"},"modality":null,"name":"ASPIRE","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.31111","source_url":"https://arxiv.org/abs/2608.31111"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-31111","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31111","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:anuragkokate09/asr-model-benchmark","source_record":"bm_asr-model-benchmark_50d41822"},"categories":["General AI","Speech & Audio","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Compares Whisper Small, Faster-Whisper Small, and Wav2Vec2 Base on 20 clean and 20 noisy LibriSpeech samples using WER, inference latency, RTF, and memory usage on CPU.","descriptions":{"en":"Compares Whisper Small, Faster-Whisper Small, and Wav2Vec2 Base on 20 clean and 20 noisy LibriSpeech samples using WER, inference latency, RTF, and memory usage on CPU."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:anuragkokate09/asr-model-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio","agent-memory"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_asr-model-benchmark_50d41822"},"modality":null,"name":"ASR Model Benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:anuragkokate09/asr-model-benchmark","source_url":"https://github.com/Anuragkokate09/asr-model-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-anuragkokate09-asr-model-benchmark","source":"claire_radar","source_url":"https://github.com/Anuragkokate09/asr-model-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","q-bio.QM"],"collected_at":"2026-10-14T09:44:51Z","description":"AssayBench-Loop is a large-scale benchmark for adaptive hit discovery using 1,389 CRISPR screens across five phenotype categories.","descriptions":{"en":"AssayBench-Loop is a large-scale benchmark for adaptive hit discovery using 1,389 CRISPR screens across five phenotype categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11877","languages":[],"library_direction":null,"modality":null,"name":"AssayBench-Loop","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11877","source_url":"https://arxiv.org/abs/2609.11877"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11877","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11877","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:fouad-bahrpeyma/assemblygrid_v1","source_record":"bm_assemblygrid-v1_bce798fe"},"categories":["Geometric reasoning","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AssemblyGrid v1 is a benchmark for repeated multi-robot production that combines explicit process progression, decentralized observations, material transfer, temporary multi-robot coalitions, productive concurrency, and geometry-dependent feasibility within one task-level formulation.","descriptions":{"en":"AssemblyGrid v1 is a benchmark for repeated multi-robot production that combines explicit process progression, decentralized observations, material transfer, temporary multi-robot coalitions, productive concurrency, and geometry-dependent feasibility within one task-level formulation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.16075","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","multi-agent-games"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_assemblygrid-v1_bce798fe"},"modality":null,"name":"AssemblyGrid v1","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16075","source_url":"https://arxiv.org/abs/2609.16075"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16075","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16075","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Language & Knowledge","cs.AR"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM generation of SystemVerilog assertions from structured design specifications and RTL, across two tasks: bug-prevention and bug-hunting. The benchmark includes 83 real-world designs with golden and mutated RTL, and assesses syntactic validity, formal provability, coverage, and mutation-based bug detection.","descriptions":{"en":"Evaluates LLM generation of SystemVerilog assertions from structured design specifications and RTL, across two tasks: bug-prevention and bug-hunting. The benchmark includes 83 real-world designs with golden and mutated RTL, and assesses syntactic validity, formal provability, coverage, and mutation-based bug detection."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27472","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_assertllm2_9bd7c97e"},"modality":null,"name":"AssertLLM2","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27472","source_url":"https://arxiv.org/abs/2605.27472"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27472","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27472","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AssetOpsBench is a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) and AI agents in complex asset operation and maintenance tasks. AssetOpsBench 是一个专注于评估大语言模型（LLM）和智能体在资产运维领域复杂任务中实际表现的多维度评测基准。该基准旨在检验模型在工业场景下的任务规划、多步推理、工具调用、安全合规性以及领域知识理解等核心能力，覆盖设备维护、异常诊断、风险评估等典型运维场景。测试集包含 1,000 个高质量样本，涉及 5 大类任务和 20 余种细分领域，数据来源于真实运维手册、工单记录及专家验证案例。","descriptions":{"en":"AssetOpsBench is a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) and AI agents in complex asset operation and maintenance tasks.","zh":"AssetOpsBench 是一个专注于评估大语言模型（LLM）和智能体在资产运维领域复杂任务中实际表现的多维度评测基准。该基准旨在检验模型在工业场景下的任务规划、多步推理、工具调用、安全合规性以及领域知识理解等核心能力，覆盖设备维护、异常诊断、风险评估等典型运维场景。测试集包含 1,000 个高质量样本，涉及 5 大类任务和 20 余种细分领域，数据来源于真实运维手册、工单记录及专家验证案例。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1991","languages":[],"library_direction":null,"modality":null,"name":"AssetOpsBench","openness":"unknown","publisher":"IBMResearch-Yorktown , IBMResearch-Ireland","released":"2025-06-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1991-assetopsbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AssetOpsBench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"AstroMind evaluates LLM reasoning about spacecraft behavior across intent inference, maneuver parameter estimation, and threat assessment, using physics-grounded simulations with realistic sensing noise. Metrics capture semantic correctness and quantitative consistency under physical constraints.","descriptions":{"en":"AstroMind evaluates LLM reasoning about spacecraft behavior across intent inference, maneuver parameter estimation, and threat assessment, using physics-grounded simulations with realistic sensing noise. Metrics capture semantic correctness and quantitative consistency under physical constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24573","languages":[],"library_direction":{"basis":"model-review","categories":["science","abstract-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_astromind_587e5d1f"},"modality":null,"name":"AstroMind","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24573","source_url":"https://arxiv.org/abs/2605.24573"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24573","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24573","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates code agents on 50 web tasks with underspecified intent and multi-round refinement via browser-rendered behavior.","descriptions":{"en":"Evaluates code agents on 50 web tasks with underspecified intent and multi-round refinement via browser-rendered behavior."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05920","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_asuka-bench_f32387bf"},"modality":null,"name":"Asuka-Bench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05920","source_url":"https://arxiv.org/abs/2606.05920"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05920","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05920","topics":["coding","agents"],"unit":null},{"aliases":["Towards Detecting AI-Assisted Responses in Online Surveys"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mike-qz-wang/asurre","source_record":"bm_towards-detecting-ai-assisted-responses-in_ffe1dccf"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Dataset for detecting AI-assisted responses in online surveys.","descriptions":{"en":"Dataset for detecting AI-assisted responses in online surveys."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.17317","languages":[],"library_direction":null,"modality":null,"name":"ASURRE","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17317","source_url":"https://arxiv.org/abs/2609.17317"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17317","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17317","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Code","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem.","descriptions":{"en":"Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32662","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","multi-agent-games"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_asyncodebench_ae4b21be"},"modality":null,"name":"AsynCodeBench","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32662","source_url":"https://arxiv.org/abs/2609.32662"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32662","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32662","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":102,"github_stars_30d":-5,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:stokou/repo-asynctool","source_record":"bm_asynctool_407c214b"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AsyncTool is a benchmark for evaluating asynchronous function calling in multi-task tool-use environments. It presents multiple tasks with simulated tool response latency, assessing step-level tool-call correctness, sub-task completion, and task-level end-to-end success, along with efficiency-oriented metrics.","descriptions":{"en":"AsyncTool is a benchmark for evaluating asynchronous function calling in multi-task tool-use environments. It presents multiple tasks with simulated tool response latency, assessing step-level tool-call correctness, sub-task completion, and task-level end-to-end success, along with efficiency-oriented metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.27995","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_asynctool_407c214b"},"modality":null,"name":"AsyncTool","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27995","source_url":"https://arxiv.org/abs/2605.27995"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27995","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27995","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AtelierEval evaluates prompting proficiency of humans and MLLMs for text-to-image systems via 360 tasks, using a skill-based agentic evaluator (AtelierJudge) that scores prompt-image pairs subjectively and objectively.","descriptions":{"en":"AtelierEval evaluates prompting proficiency of humans and MLLMs for text-to-image systems via 360 tasks, using a skill-based agentic evaluator (AtelierJudge) that scores prompt-image pairs subjectively and objectively."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22645","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_ateliereval_cc1f5b0c"},"modality":null,"name":"AtelierEval","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22645","source_url":"https://arxiv.org/abs/2605.22645"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22645","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22645","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":[],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"","descriptions":{},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:community:ed90e889-4678-4fbd-98ab-0e654f4bf35e","languages":[],"library_direction":null,"modality":null,"name":"atlas","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":"higher_is_better","score_summary":{"display_max":null,"display_multiplier":1,"model_count":null,"model_count_basis":null,"numeric_count":0,"raw_max":null,"raw_min":null,"source_reference":null,"unit":null},"scored_models":[],"slug":"llm-stats-community-ed90e889-4678-4fbd-98ab-0e654f4bf35e","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3Aed90e889-4678-4fbd-98ab-0e654f4bf35e?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:acodercat/atmoscoder-bench","source_record":"bm_atmoscoder-bench_24ddfa7c"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"AtmosCoder-Bench is an execution-grounded benchmark for LLMs on atmospheric science computation, with 436 problems and 3,910 variants, grading by executing code solutions against ground truth.","descriptions":{"en":"AtmosCoder-Bench is an execution-grounded benchmark for LLMs on atmospheric science computation, with 436 problems and 3,910 variants, grading by executing code solutions against ground truth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.18726","languages":[],"library_direction":{"basis":"model-review","categories":["science","code-generation"],"join":"claire_record_id","primary":"science","source_record":"bm_atmoscoder-bench_24ddfa7c"},"modality":null,"name":"AtmosCoder-Bench","openness":"unknown","publisher":null,"released":"2026-08-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.18726","source_url":"https://arxiv.org/abs/2608.18726"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-18726","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18726","topics":["coding","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ATOM-Bench evaluates atomic skills and compositional generalization in manipulation policies across 30 atomic tasks and 24 held-out compositional tasks, using paired single-arm and dual-arm robot tracks, with 3,000 human demonstrations and evaluation rollout data released.","descriptions":{"en":"ATOM-Bench evaluates atomic skills and compositional generalization in manipulation policies across 30 atomic tasks and 24 held-out compositional tasks, using paired single-arm and dual-arm robot tracks, with 3,000 human demonstrations and evaluation rollout data released."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16826","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_atom-bench_82c6474b"},"modality":null,"name":"ATOM-Bench","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16826","source_url":"https://arxiv.org/abs/2606.16826"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16826","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16826","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":36,"github_stars_30d":5,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:alibaba/atrex-bench","source_record":"bm_atrex-bench_ab05aff2"},"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on generating GPU kernels from PyTorch references across 30 operators and 440 shapes derived from production inference traces. Scoring uses a three-stage evaluator measuring compile success, numerical correctness, and speed-of-light (SOL) efficiency against a cached roofline. The benchmark includes 4 DSL backends and supports multi-vendor GPUs.","descriptions":{"en":"Evaluates coding agents on generating GPU kernels from PyTorch references across 30 operators and 440 shapes derived from production inference traces. Scoring uses a three-stage evaluator measuring compile success, numerical correctness, and speed-of-light (SOL) efficiency against a cached roofline. The benchmark includes 4 DSL backends and supports multi-vendor GPUs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.14541","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_atrex-bench_ab05aff2"},"modality":null,"name":"Atrex-Bench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14541","source_url":"https://arxiv.org/abs/2607.14541"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14541","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14541","topics":["agents"],"unit":null},{"aliases":["kaggle-art-benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mziqudhd92/kaggle-art-benchmark","source_record":"bm_kaggle-art-benchmark_61ed421d"},"categories":["Cybersecurity","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ART evaluates whether language models separate attacker-reachable vulnerabilities from patched twins, safe code, and vacuous noise, scoring vulnerability, patched, and filler accuracy via a weighted ART score and a Twin Gap metric.","descriptions":{"en":"ART evaluates whether language models separate attacker-reachable vulnerabilities from patched twins, safe code, and vacuous noise, scoring vulnerability, patched, and filler accuracy via a weighted ART score and a Twin Gap metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:mziqudhd92/kaggle-art-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_kaggle-art-benchmark_61ed421d"},"modality":null,"name":"Attacker-Reachable Sink Triage (ART)","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:mziqudhd92/kaggle-art-benchmark","source_url":"https://github.com/mziqudhd92/kaggle-art-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-mziqudhd92-kaggle-art-benchmark","source":"claire_radar","source_url":"https://github.com/mziqudhd92/kaggle-art-benchmark","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":null,"source_record":"catalog_669c5f1c9d6e93a2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AttaQ is a unique dataset containing adversarial examples in the form of questions designed to provoke harmful or inappropriate responses from large language models. The benchmark evaluates safety vulnerabilities by using specialized clustering techniques that analyze both the semantic similarity of input attacks and the harmfulness of model responses, facilitating targeted improvements to model safety mechanisms.","descriptions":{"en":"AttaQ is a unique dataset containing adversarial examples in the form of questions designed to provoke harmful or inappropriate responses from large language models. The benchmark evaluates safety vulnerabilities by using specialized clustering techniques that analyze both the semantic similarity of input attacks and the harmfulness of model responses, facilitating targeted improvements to model safety mechanisms."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:attaq:granite-3.3-8b-base","reported_at":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/attaq?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:attaq","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"llm_stats_id","primary":"safety-alignment","source_record":"catalog_669c5f1c9d6e93a2"},"modality":"text","name":"AttaQ","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":88.5,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.885,"raw_min":0.861,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:attaq:granite-3.3-8b-base","reported_date":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/attaq?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"}],"slug":"llm-stats-attaq","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/attaq?top_n=500","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_auau_d78e1117"},"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"AuAu combines psychometric instruments, vignettes, and realistic prompts to assess authoritarian tendencies in LLM responses, measuring sub-concepts like aggression, submission, and conventionalism.","descriptions":{"en":"AuAu combines psychometric instruments, vignettes, and realistic prompts to assess authoritarian tendencies in LLM responses, measuring sub-concepts like aggression, submission, and conventionalism."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16127","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_auau_d78e1117"},"modality":null,"name":"AuAu","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16127","source_url":"https://arxiv.org/abs/2606.16127"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16127","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16127","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","语言","Language","安全","Safety","audio","jailbreak","LAM","多模态模型","VLM","语言理解","Comprehension","安全对齐","Safety Alignment","音频理解","Audio Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LAMs face jailbreak risks. AJailBench, our new benchmark, reveals leading LAMs lack robustness. Subtle audio perturbations significantly degrade their safety. We release AJailBench for research. LAM 面临越狱风险。我们新的基准测试 AJailBench 揭示，领先的 LAM 缺乏稳健性。细微的音频干扰会显著降低其安全性。我们发布 AJailBench 进行研究。","descriptions":{"en":"LAMs face jailbreak risks. AJailBench, our new benchmark, reveals leading LAMs lack robustness. Subtle audio perturbations significantly degrade their safety. We release AJailBench for research.","zh":"LAM 面临越狱风险。我们新的基准测试 AJailBench 揭示，领先的 LAM 缺乏稳健性。细微的音频干扰会显著降低其安全性。我们发布 AJailBench 进行研究。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1835","languages":[],"library_direction":null,"modality":"multimodal","name":"AudioJailbreak","openness":"open","publisher":"mbzuai","released":"2025-05-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1835-audiojailbreak","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AudioJailbreak","topics":["multimodal","language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","安全","Safety","多模态模型","VLM","安全对齐","Safety Alignment","音频理解","Audio Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AudioTrust is a comprehensive trust evaluation framework for Audio Large Language Models (ALLMs) that effectively reveals potential risks in six dimensions: fairness, hallucination, security, privacy, robustness, and authentication. It aggregates over 4,420 real-world audio/text data samples, coveri AudioTrust针对Audio Large Language Models（ALLMs）的全方位可信评估框架，有效揭示音频大模型在公平性、幻觉、安全、隐私、鲁棒性和身份验证六大维度的潜在风险。汇集4,420+条真实场景音频/文本数据，覆盖日常对话、紧急呼叫、语音助手等18种实验设置，设计9项音频特定评测指标，构建自动化评估流水线。主要发现：闭源模型在鲁棒性和安全防护上表现更佳，开源模型对隐私和公平性仍存盲区；多数ALLMs对性别、口音、年龄等敏感属性存在系统性偏见。期待研究者基于AudioTrust继续优化音频大模型，共同推动更安全、可信的AI音频生态发展！","descriptions":{"en":"AudioTrust is a comprehensive trust evaluation framework for Audio Large Language Models (ALLMs) that effectively reveals potential risks in six dimensions: fairness, hallucination, security, privacy, robustness, and authentication. It aggregates over 4,420 real-world audio/text data samples, coveri","zh":"AudioTrust针对Audio Large Language Models（ALLMs）的全方位可信评估框架，有效揭示音频大模型在公平性、幻觉、安全、隐私、鲁棒性和身份验证六大维度的潜在风险。汇集4,420+条真实场景音频/文本数据，覆盖日常对话、紧急呼叫、语音助手等18种实验设置，设计9项音频特定评测指标，构建自动化评估流水线。主要发现：闭源模型在鲁棒性和安全防护上表现更佳，开源模型对隐私和公平性仍存盲区；多数ALLMs对性别、口音、年龄等敏感属性存在系统性偏见。期待研究者基于AudioTrust继续优化音频大模型，共同推动更安全、可信的AI音频生态发展！"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1908","languages":["English"],"library_direction":null,"modality":"multimodal","name":"AudioTrust","openness":"open","publisher":"Tsinghua University, Nanyang Technological University","released":"2025-06-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1908-audiotrust","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AudioTrust","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":1,"hf_dataset_downloads":136,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:mao-code/authbench","source_record":"bm_authbench_049afd70"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A multilingual authorship representation benchmark with 428,150 documents across 10 languages, 9 primary genres, and four length buckets, supporting same-author retrieval and verification.","descriptions":{"en":"A multilingual authorship representation benchmark with 428,150 documents across 10 languages, 9 primary genres, and four length buckets, supporting same-author retrieval and verification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.06771","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_authbench_049afd70"},"modality":null,"name":"AuthBench","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06771","source_url":"https://arxiv.org/abs/2609.06771"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06771","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06771","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_authmem-bench_56d03df5"},"categories":["Agents","General AI","Language & Knowledge","Safety","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"AuthMem-Bench evaluates authority collapse in persistent memory for LLM agents. It uses a paired benchmark holding claims and tasks fixed while varying source authority, measuring write-time collapse, authorization errors, and automatic authority preservation.","descriptions":{"en":"AuthMem-Bench evaluates authority collapse in persistent memory for LLM agents. It uses a paired benchmark holding claims and tasks fixed while varying source authority, measuring write-time collapse, authorization errors, and automatic authority preservation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01679","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","agent-security"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_authmem-bench_56d03df5"},"modality":null,"name":"AuthMem-Bench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01679","source_url":"https://arxiv.org/abs/2608.01679"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01679","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01679","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:floating-reeds/authoritybench","source_record":"bm_authoritybench_5d25d571"},"categories":["General AI","Safety & Trustworthiness","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"AuthorityBench evaluates how citation-based authority signals affect epistemic behavior in LLMs using 220,564 prompts across four domains, with a 2x2 factorial design crossing claim veracity and citation veracity.","descriptions":{"en":"AuthorityBench evaluates how citation-based authority signals affect epistemic behavior in LLMs using 220,564 prompts across four domains, with a 2x2 factorial design crossing claim veracity and citation veracity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.13104","languages":[],"library_direction":{"basis":"model-review-2","categories":["factuality","safety-alignment"],"join":"claire_record_id","primary":"factuality","source_record":"bm_authoritybench_5d25d571"},"modality":null,"name":"AuthorityBench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13104","source_url":"https://arxiv.org/abs/2606.13104"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13104","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13104","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"AuthTrace is a diagnostic benchmark for evidence construction in thematically dense single-author corpora, with fan-in annotations and pack-level protocol.","descriptions":{"en":"AuthTrace is a diagnostic benchmark for evidence construction in thematically dense single-author corpora, with fan-in annotations and pack-level protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.25382","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_authtrace_cea4d208"},"modality":null,"name":"AuthTrace","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25382","source_url":"https://arxiv.org/abs/2605.25382"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25382","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25382","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","智能体","Agent","任务执行","Task Execution","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AutoAdvExBench is a benchmark designed to evaluate large language models' (LLMs) ability to autonomously exploit adversarial example defenses, directly measuring LLMs' success on tasks regularly performed by machine learning security experts. AutoAdvExBench 是一个评估大型语言模型（LLMs）自主利用对抗性样本防御能力的基准，直接衡量LLMs在机器学习安全专家任务上的成功率。它主要评估模型理解学术论文、代码实现及生成对抗性攻击的能力。测试集包含75个对抗性样本防御实现。","descriptions":{"en":"AutoAdvExBench is a benchmark designed to evaluate large language models' (LLMs) ability to autonomously exploit adversarial example defenses, directly measuring LLMs' success on tasks regularly performed by machine learning security experts.","zh":"AutoAdvExBench 是一个评估大型语言模型（LLMs）自主利用对抗性样本防御能力的基准，直接衡量LLMs在机器学习安全专家任务上的成功率。它主要评估模型理解学术论文、代码实现及生成对抗性攻击的能力。测试集包含75个对抗性样本防御实现。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2078","languages":[],"library_direction":null,"modality":null,"name":"AutoAdvExBench","openness":"unknown","publisher":"GoogleDeepMind , ETHZurich","released":"2025-03-03","released_reference":{"basis":"paper_first_version","note":"First version introducing AutoAdvExBench and evaluating LLM agents on its defenses.","source_key":"opencompass:2078","source_url":"https://arxiv.org/abs/2503.01811"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2078-autoadvexbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AutoAdvExBench","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":26,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"claire_record_id","repository":"gh:stardewxxx/autodatabench","source_record":"bm_autodatabench_812fc1f6"},"categories":["General AI","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates agents' ability to autonomously author executable training tasks that meet acceptance criteria, scored via gate, difficulty, and quality terms.","descriptions":{"en":"Evaluates agents' ability to autonomously author executable training tasks that meet acceptance criteria, scored via gate, difficulty, and quality terms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.35025","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_autodatabench_812fc1f6"},"modality":null,"name":"AutoDataBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35025","source_url":"https://arxiv.org/abs/2609.35025"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35025","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35025","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":49,"join":"claire_record_id","repository":"gh:autodatabench/autodatabench","source_record":"bm_autodatabench_e428e4b7"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Existing auto-research benchmarks often entangle multiple sources of improvement, including training frameworks, hyperparameters, compute budgets, and data, making it difficult to attribute why one frontier agent outperforms another to spe…","descriptions":{"en":"Existing auto-research benchmarks often entangle multiple sources of improvement, including training frameworks, hyperparameters, compute budgets, and data, making it difficult to attribute why one frontier agent outperforms another to spe…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.40097","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_autodatabench_e428e4b7"},"modality":null,"name":"AutoDataBench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.40097","source_url":"https://arxiv.org/abs/2609.40097"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-40097","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.40097","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":169,"github_stars_30d":5,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"claire_record_id","repository":"gh:autolabhq/autolab","source_record":"bm_autolab_db01499e"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AutoLab evaluates frontier models on long-horizon closed-loop optimization tasks across system optimization, CUDA kernel optimization, model development, and puzzle challenges, with 36 expert-curated tasks and a strict wall-clock budget.","descriptions":{"en":"AutoLab evaluates frontier models on long-horizon closed-loop optimization tasks across system optimization, CUDA kernel optimization, model development, and puzzle challenges, with 36 expert-curated tasks and a strict wall-clock budget."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.05080","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_autolab_db01499e"},"modality":null,"name":"AutoLab","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05080","source_url":"https://arxiv.org/abs/2606.05080"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05080","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05080","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AutoLogi is an automated method for synthesizing open-ended logic puzzles to evaluate reasoning abilities of Large Language Models. The benchmark addresses limitations of existing multiple-choice reasoning evaluations by featuring program-based verification and controllable difficulty levels. It includes 1,575 English and 883 Chinese puzzles, enabling more reliable evaluation that better distinguishes models' reasoning capabilities across languages.","descriptions":{"en":"AutoLogi is an automated method for synthesizing open-ended logic puzzles to evaluate reasoning abilities of Large Language Models. The benchmark addresses limitations of existing multiple-choice reasoning evaluations by featuring program-based verification and controllable difficulty levels. It includes 1,575 English and 883 Chinese puzzles, enabling more reliable evaluation that better distinguishes models' reasoning capabilities across languages."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:autologi:kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/autologi?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:autologi","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_83aeecd3f494da49"},"modality":"text","name":"AutoLogi","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":89.5,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.895,"raw_min":0.895,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:autologi:kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/autologi?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"}],"slug":"llm-stats-autologi","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/autologi?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_e59105b35af353d8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AutomationBench is a tool-use benchmark that evaluates AI agents on automating real-world workflows, testing their ability to orchestrate tools and complete multi-step automation tasks.","descriptions":{"en":"AutomationBench is a tool-use benchmark that evaluates AI agents on automating real-world workflows, testing their ability to orchestrate tools and complete multi-step automation tasks."},"evidence_summary":{"document_count":1,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:automationbench:claude-fable-5","reported_at":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:automationbench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_e59105b35af353d8"},"modality":"text","name":"AutomationBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":12,"score_direction":"higher_is_better","score_summary":{"display_max":48.199999999999996,"display_multiplier":100,"model_count":12,"model_count_basis":"source_model_id","numeric_count":12,"raw_max":0.482,"raw_min":0.135,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:automationbench:glm-5.3","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0731"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-automationbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["AutomationBench","Zapier AutomationBench","HLEAutomationBench"],"attention_signals":null,"categories":["agent"],"collected_at":null,"description":"Published by Zapier over its own automation surface, and cards report different task subsets of it.","descriptions":{"en":"Published by Zapier over its own automation surface, and cards report different task subsets of it."},"evidence_summary":{"document_count":7,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000automationbench\u0000moonshot_kimi_k3_model_card\u0000automationbench\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"first_score_reported_at":"2026-06-13","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000automationbench\u0000moonshot_kimi_k3_model_card\u0000automationbench\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:automationbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"AutomationBench","openness":"unknown","publisher":null,"released":"2026-03-10","released_reference":{"basis":"benchmark_release","source_key":"model-reports:automationbench","source_url":"https://github.com/zapier/automation-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":54.8,"display_multiplier":1,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":54.8,"raw_min":26.0,"source_reference":{"obs_id":"curated\u0000automationbench\u0000deepseek_v4_1_flash_technical_report\u0000automationbench\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000automationbench\u0000deepseek_v4_1_flash_technical_report\u0000automationbench\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-07-24","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2026-08-13","date_precision":"document_publication","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"}],"slug":"automationbench","source":"model_reports","source_url":"https://github.com/zapier/automation-bench","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["agentic","tool-use","business"],"collected_at":"2026-08-25T10:41:06Z","description":"Agentic SaaS workflows","descriptions":{"en":"Agentic SaaS workflows"},"evidence_summary":{"document_count":1,"model_count":39,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:automationbench-aa:a6340098-d7ae-462d-b372-0a0a67fc44b4","reported_at":"2025-10-15","source_url":"https://artificialanalysis.ai/evaluations/automationbench-aa"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:automationbench-aa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"AutomationBench-AA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":39,"score_direction":"higher_is_better","score_summary":{"display_max":62.74556199710793,"display_multiplier":100,"model_count":39,"model_count_basis":"source_model_id","numeric_count":39,"raw_max":0.6274556199710792,"raw_min":0.013565301670673362,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:automationbench-aa:b2331108-72ed-415a-82d1-188633875bbc","reported_date":"2026-08-13","source_url":"https://artificialanalysis.ai/evaluations/automationbench-aa"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"}],"slug":"artificial-analysis-automationbench-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/automationbench-aa","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":336,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:zapier/automationbench","source_record":"catalog_530d0ae739bef10b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"AutomationBench-AA is Artificial Analysis's independently run version of AutomationBench, covering 657 real-world SaaS workflow tasks across 40 simulated applications (e.g. Gmail, Slack, Salesforce, HubSpot). It scores the share of objectives an agent completes without violating business guardrails, using a private held-out task set.","descriptions":{"en":"AutomationBench-AA is Artificial Analysis's independently run version of AutomationBench, covering 657 real-world SaaS workflow tasks across 40 simulated applications (e.g. Gmail, Slack, Salesforce, HubSpot). It scores the share of objectives an agent completes without violating business guardrails, using a private held-out task set."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:automationbench-aa:grok-4.5","reported_at":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench-aa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:automationbench-aa","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_530d0ae739bef10b"},"modality":"text","name":"AutomationBench-AA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":51.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.514,"raw_min":0.514,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:automationbench-aa:grok-4.5","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench-aa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"}],"slug":"llm-stats-automationbench-aa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench-aa?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":62,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":26,"join":"claire_record_id","repository":"gh:automedbench/automedbench","source_record":"bm_automedbench_61e5be52"},"categories":["Agents","Agents & Tool Use","Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"AutoMedBench evaluates autonomous AI agents on end-to-end medical-AI research tasks spanning segmentation, image enhancement, VQA, report generation, and lesion detection. Tasks follow a five-stage workflow (Plan, Setup, Validate, Inference, Submit), with scoring based on both final task performance and stage-level rubric scores.","descriptions":{"en":"AutoMedBench evaluates autonomous AI agents on end-to-end medical-AI research tasks spanning segmentation, image enhancement, VQA, report generation, and lesion detection. Tasks follow a five-stage workflow (Plan, Setup, Validate, Inference, Submit), with scoring based on both final task performance and stage-level rubric scores."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01961","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","ai-rnd"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_automedbench_61e5be52"},"modality":null,"name":"AutoMedBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01961","source_url":"https://arxiv.org/abs/2606.01961"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01961","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01961","topics":["agents","multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CE"],"collected_at":"2026-10-14T09:44:51Z","description":"Partial differential equations (PDEs) are essential for modeling complex physical systems, and neural solvers have recently emerged as powerful data-driven tools for numerically solving them.","descriptions":{"en":"Partial differential equations (PDEs) are essential for modeling complex physical systems, and neural solvers have recently emerged as powerful data-driven tools for numerically solving them."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32245","languages":[],"library_direction":{"basis":"model-review","categories":["science","ai-rnd"],"join":"claire_record_id","primary":"science","source_record":"bm_autopdebench_61c31705"},"modality":null,"name":"AutoPDEBench","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32245","source_url":"https://arxiv.org/abs/2609.32245"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32245","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32245","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:li-ch/autotunebench","source_record":"bm_autotunebench_b07424bd"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark and measurement protocol for agent auto-tuning of LLM serving engines, featuring a proposal-evaluation loop with correctness checks, fair timing, profiling, and anti-cheating gates to ensure trustworthy results.","descriptions":{"en":"A benchmark and measurement protocol for agent auto-tuning of LLM serving engines, featuring a proposal-evaluation loop with correctness checks, fair timing, profiling, and anti-cheating gates to ensure trustworthy results."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.18123","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_autotunebench_b07424bd"},"modality":null,"name":"AutoTuneBench","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18123","source_url":"https://arxiv.org/abs/2609.18123"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18123","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18123","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":14,"join":"claire_record_id","repository":"gh:electronicarts/autoworldmodelbench","source_record":"bm_autoworldmodel-bench_01d33f6e"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AutoWorldModel-Bench evaluates coding agents on autonomously improving a base world model under fixed compute budget across eight game environments. Uses structured-state representation, with held-out test split and closed-loop iteration.","descriptions":{"en":"AutoWorldModel-Bench evaluates coding agents on autonomously improving a base world model under fixed compute budget across eight game environments. Uses structured-state representation, with held-out test split and closed-loop iteration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11216","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","world-models"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_autoworldmodel-bench_01d33f6e"},"modality":null,"name":"AutoWorldModel-Bench","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11216","source_url":"https://arxiv.org/abs/2608.11216"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11216","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11216","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Multimodal foundation models are increasingly used for evaluating and generating user interfaces (UIs), often producing seemingly reasonable aesthetic judgments and visually plausible pages.","descriptions":{"en":"Multimodal foundation models are increasingly used for evaluating and generating user interfaces (UIs), often producing seemingly reasonable aesthetic judgments and visually plausible pages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34854","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_auv-bench_c1df993d"},"modality":null,"name":"AUV-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34854","source_url":"https://arxiv.org/abs/2609.34854"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34854","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34854","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","其他","Other","audio-visual","多模态模型","VLM","音频理解","Audio Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AV-Odyssey Bench. This benchmark encompasses 26 different tasks and 4,555 carefully crafted problems, each incorporating text, visual, and audio components. All data are newly collected and annotated by humans, not from any existing audio-visual dataset. AV-Odyssey Bench. This benchmark encompasses 26 different tasks and 4,555 carefully crafted problems, each incorporating text, visual, and audio components. All data are newly collected and annotated by humans, not from any existing audio-visual dataset.","descriptions":{"en":"AV-Odyssey Bench. This benchmark encompasses 26 different tasks and 4,555 carefully crafted problems, each incorporating text, visual, and audio components. All data are newly collected and annotated by humans, not from any existing audio-visual dataset.","zh":"AV-Odyssey Bench. This benchmark encompasses 26 different tasks and 4,555 carefully crafted problems, each incorporating text, visual, and audio components. All data are newly collected and annotated by humans, not from any existing audio-visual dataset."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1396","languages":[],"library_direction":null,"modality":"multimodal","name":"AV-Odyssey-Bench","openness":"open","publisher":"CUHK MMLab, CUHK (SZ), Stanford University, UC Berkeley, Yale University","released":"2024-12-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1396-av-odyssey-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AV-Odyssey-Bench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","cs.MM"],"collected_at":"2026-10-14T09:44:51Z","description":"Audio-visual object removal benchmark with objective metrics and MLLM-based evaluation protocol.","descriptions":{"en":"Audio-visual object removal benchmark with objective metrics and MLLM-based evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.25864","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_av-remove-bench_79cc44ee"},"modality":null,"name":"AV-Remove-Bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25864","source_url":"https://arxiv.org/abs/2609.25864"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25864","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25864","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Proposed safety benchmark for text-to-audio-video generation, with a four-axis, 13-category taxonomy and 5,200 manually reviewed prompts, evaluated under Full-AV, Video-Only, and Audio-Only views.","descriptions":{"en":"Proposed safety benchmark for text-to-audio-video generation, with a four-axis, 13-category taxonomy and 5,200 manually reviewed prompts, evaluated under Full-AV, Video-Only, and Audio-Only views."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.06991","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio","safety-alignment","image-video-generation"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_av-safetybench_5dfce700"},"modality":null,"name":"AV-SafetyBench","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06991","source_url":"https://arxiv.org/abs/2609.06991"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06991","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06991","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fgt7t6g/av-syncbench","source_record":"bm_av-syncbench_20190a72"},"categories":["General AI","Multimodal","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"AV-SyncBench evaluates audio-visual synchronization by separating temporal and semantic consistency. It contains 3,269 videos and 38,390 samples across 10 scenarios and 5 tasks, with data filtered and verified for on-screen sound sources.","descriptions":{"en":"AV-SyncBench evaluates audio-visual synchronization by separating temporal and semantic consistency. It contains 3,269 videos and 38,390 samples across 10 scenarios and 5 tasks, with data filtered and verified for on-screen sound sources."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.00726","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_av-syncbench_20190a72"},"modality":null,"name":"AV-SyncBench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00726","source_url":"https://arxiv.org/abs/2607.00726"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00726","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00726","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DB"],"collected_at":"2026-10-14T09:44:51Z","description":"AvalancheBench evaluates enterprise data agents on latent world recovery, scoring analytical understanding of segments, drivers, temporal events, and relationships from generated observations.","descriptions":{"en":"AvalancheBench evaluates enterprise data agents on latent world recovery, scoring analytical understanding of segments, drivers, temporal events, and relationships from generated observations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24183","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_avalanchebench_c5aa4b06"},"modality":null,"name":"AvalancheBench","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24183","source_url":"https://arxiv.org/abs/2605.24183"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24183","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24183","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Avalon-ToM-Bench evaluates fine-grained theory of mind in LLMs using a 2x2 taxonomy of epistemic/motivational reasoning crossed with inference/action, via human-crafted perspective-constrained queries from The Resistance: Avalon.","descriptions":{"en":"Avalon-ToM-Bench evaluates fine-grained theory of mind in LLMs using a 2x2 taxonomy of epistemic/motivational reasoning crossed with inference/action, via human-crafted perspective-constrained queries from The Resistance: Avalon."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09638","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_avalon-tom-bench_a3c88250"},"modality":null,"name":"Avalon-ToM-Bench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09638","source_url":"https://arxiv.org/abs/2608.09638"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09638","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09638","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_avbench_ce8c8830"},"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AVBench evaluates audio-video generative models across ten fine-grained dimensions covering visual quality, audio quality, and cross-modal consistency for human-centric scenarios. It uses specialized evaluators trained via preference learning and provides continuous scores from prediction confidence.","descriptions":{"en":"AVBench evaluates audio-video generative models across ten fine-grained dimensions covering visual quality, audio quality, and cross-modal consistency for human-centric scenarios. It uses specialized evaluators trained via preference learning and provides continuous scores from prediction confidence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24652","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","speech-audio"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_avbench_ce8c8830"},"modality":null,"name":"AVBench","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24652","source_url":"https://arxiv.org/abs/2605.24652"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24652","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24652","topics":[],"unit":null},{"aliases":["AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_avcap-reinforcing-audio-video-joint-captio_13492b7b"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates atomic-level details in audiovisual captions using a specialized benchmark and metric, with associated dataset AVCap-100K and model AVCap.","descriptions":{"en":"Evaluates atomic-level details in audiovisual captions using a specialized benchmark and metric, with associated dataset AVCap-100K and model AVCap."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06930","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_avcap-reinforcing-audio-video-joint-captio_13492b7b"},"modality":null,"name":"AVCap-Bench","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06930","source_url":"https://arxiv.org/abs/2608.06930"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06930","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06930","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":10,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"claire_record_id","repository":"gh:nju-link/ave-compass","source_record":"bm_ave-compass_477a3826"},"categories":["General AI","Vision & 3D","cs.MM"],"collected_at":"2026-10-14T09:44:51Z","description":"AVE-Compass evaluates audio-visual editing models on 145 source videos and 196 instructions with 2,688 checklist items, scoring Instruction Following, Fidelity Preserving, Realism, and Editing Intent via MLLM judging and automated metrics.","descriptions":{"en":"AVE-Compass evaluates audio-visual editing models on 145 source videos and 196 instructions with 2,688 checklist items, scoring Instruction Following, Fidelity Preserving, Realism, and Editing Intent via MLLM judging and automated metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.24821","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","speech-audio"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_ave-compass_477a3826"},"modality":null,"name":"AVE-Compass","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24821","source_url":"https://arxiv.org/abs/2607.24821"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24821","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24821","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":18,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_avi-bench_16159c13"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"AVI-Bench evaluates omni-multimodal LLMs on audio-visual tasks across perception, understanding, and reasoning stages, with an extension probing primitive audio-visual sensation using unfamiliar low-semantic stimuli.","descriptions":{"en":"AVI-Bench evaluates omni-multimodal LLMs on audio-visual tasks across perception, understanding, and reasoning stages, with an extension probing primitive audio-visual sensation using unfamiliar low-semantic stimuli."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07643","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_avi-bench_16159c13"},"modality":null,"name":"AVI-Bench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07643","source_url":"https://arxiv.org/abs/2606.07643"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07643","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07643","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["When Writing Style Drifts: Benchmarking Authorship Verification under Distribution Shifts in Genre, Time and the AI-Era"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"AVShift evaluates authorship verification across over 150,000 German text pairs spanning three genres and 21 years, with controlled cross-genre, temporal, and AI-era distribution shifts.","descriptions":{"en":"AVShift evaluates authorship verification across over 150,000 German text pairs spanning three genres and 21 years, with controlled cross-genre, temporal, and AI-era distribution shifts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17979","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_when-writing-style-drifts-benchmarking-aut_3da16bfb"},"modality":null,"name":"AVShift","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17979","source_url":"https://arxiv.org/abs/2608.17979"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17979","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17979","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"As language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential.","descriptions":{"en":"As language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35409","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_awarenessbench_6f4fb8b9"},"modality":null,"name":"AwarenessBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35409","source_url":"https://arxiv.org/abs/2609.35409"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35409","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35409","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"artifact:arxiv:1905.00537","repository":null,"source_record":"catalog_56ba0cb792f6bfce"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AX-b is a broad-coverage diagnostic task, which requires to determine the logical relation between the given sentence pair, with three relations: entailment, contradiction and neutral. This task is selected from a subset of the GLUE broad-coverage diagnostic dataset, mainly used to test the model's understanding ability in grammar, semantics, world knowledge and so on. AX-b是一个广覆盖诊断任务，要求根据给定的句子对，判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。这个任务是从GLUE的广覆盖诊断数据集中选取了一部分数据，主要用来测试模型在语法、语义、世界知识等方面的理解能力。","descriptions":{"en":"AX-b is a broad-coverage diagnostic task, which requires to determine the logical relation between the given sentence pair, with three relations: entailment, contradiction and neutral. This task is selected from a subset of the GLUE broad-coverage diagnostic dataset, mainly used to test the model's understanding ability in grammar, semantics, world knowledge and so on.","zh":"AX-b是一个广覆盖诊断任务，要求根据给定的句子对，判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。这个任务是从GLUE的广覆盖诊断数据集中选取了一部分数据，主要用来测试模型在语法、语义、世界知识等方面的理解能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:526","languages":[],"library_direction":null,"modality":null,"name":"AX-b","openness":"unknown","publisher":null,"released":"2019-05-02","released_reference":{"basis":"paper_first_version","note":"The SuperGLUE first version introduces its collapsed broad-coverage diagnostic in section 3.2, derived from GLUE's original three-way diagnostic.","source_key":"opencompass:526","source_url":"https://arxiv.org/abs/1905.00537v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-526-ax-b","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AX-b","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"artifact:arxiv:1905.00537","repository":null,"source_record":"catalog_56ba0cb792f6bfce"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AX-g is a Winogender diagnostic task, which requires to determine which noun the pronoun refers to according to the given sentence and pronoun. This task is selected from a subset of the Winogender dataset, mainly used to test the model's ability in dealing with gender bias and discrimination. AX-g是一个Winogender诊断任务，要求根据给定的句子和代词，判断代词指代的是哪个名词。这个任务是从Winogender数据集中选取了一部分数据，主要用来测试模型在处理性别偏见和性别歧视方面的能力。","descriptions":{"en":"AX-g is a Winogender diagnostic task, which requires to determine which noun the pronoun refers to according to the given sentence and pronoun. This task is selected from a subset of the Winogender dataset, mainly used to test the model's ability in dealing with gender bias and discrimination.","zh":"AX-g是一个Winogender诊断任务，要求根据给定的句子和代词，判断代词指代的是哪个名词。这个任务是从Winogender数据集中选取了一部分数据，主要用来测试模型在处理性别偏见和性别歧视方面的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:527","languages":[],"library_direction":null,"modality":null,"name":"AX-g","openness":"unknown","publisher":null,"released":"2019-07-12","released_reference":{"basis":"paper_publication","note":"The first paper version containing SuperGLUE's filtered Winogender diagnostic is v2, dated July 12. V1 does not include this task.","source_key":"opencompass:527","source_url":"https://arxiv.org/abs/1905.00537v2"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-527-ax-g","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AX-g","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AXBENCH is a benchmark for large-scale evaluation of Language Model (LLM) control methods using synthetic data, focusing on fine-grained steering for safety and reliability. AXBENCH 是一个旨在评估LLM控制能力的基准。它通过概念检测和模型操控（包含概念、指令、流畅度）评估，旨在实现安全可靠的细粒度操控。基准使用大规模合成数据集，并集成了多种基线方法。","descriptions":{"en":"AXBENCH is a benchmark for large-scale evaluation of Language Model (LLM) control methods using synthetic data, focusing on fine-grained steering for safety and reliability.","zh":"AXBENCH 是一个旨在评估LLM控制能力的基准。它通过概念检测和模型操控（包含概念、指令、流畅度）评估，旨在实现安全可靠的细粒度操控。基准使用大规模合成数据集，并集成了多种基线方法。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2084","languages":[],"library_direction":null,"modality":null,"name":"AXBENCH","openness":"unknown","publisher":"Department of Computer Science,Stanford University , Pr(AI) R Group.","released":"2025-01-28","released_reference":{"basis":"paper_first_version","note":"AxBench's own introduction, not later steering papers referenced by its README.","source_key":"opencompass:2084","source_url":"https://arxiv.org/abs/2501.17148"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2084-axbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AXBENCH","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_axis_d184e18d"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AXIS is a community-driven data engine and benchmark for robot manipulation, providing browser-based teleoperation for data collection, automated task generation and validation, and a dataset of 207 tasks and 50K+ trajectories, with a systematic held-out protocol for policy evaluation.","descriptions":{"en":"AXIS is a community-driven data engine and benchmark for robot manipulation, providing browser-based teleoperation for data collection, automated task generation and validation, and a dataset of 207 tasks and 50K+ trajectories, with a systematic held-out protocol for policy evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.21588","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_axis_d184e18d"},"modality":null,"name":"AXIS","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21588","source_url":"https://arxiv.org/abs/2607.21588"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21588","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21588","topics":[],"unit":null},{"aliases":["BABE"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Biology Arena BEnchmark; measures experimental reasoning and causal inference over peer-reviewed biological research papers.","descriptions":{"en":"Biology Arena BEnchmark; measures experimental reasoning and causal inference over peer-reviewed biological research papers."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000babe\u0000bytedance_seed2_0_model_card\u0000babe\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000babe\u0000bytedance_seed2_0_model_card\u0000babe\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:babe","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["science"],"join":"review:14H034160212:2026-10-15","primary":"science","source_record":null},"modality":null,"name":"BABE","openness":"unknown","publisher":null,"released":"2026-02-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: BABE: Biology Arena BEnchmark. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"model-reports:babe","source_url":"https://arxiv.org/abs/2602.05857"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":50.2,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":50.2,"raw_min":40.4,"source_reference":{"obs_id":"curated\u0000babe\u0000bytedance_seed2_0_model_card\u0000babe\u0000as printed in Table 4 (Efficient Models)\u0000Seed2.0 Lite","observation_id":"curated\u0000babe\u0000bytedance_seed2_0_model_card\u0000babe\u0000as printed in Table 4 (Efficient Models)\u0000Seed2.0 Lite","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"babe","source":"model_reports","source_url":"https://arxiv.org/abs/2602.05857","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating multilingual LLM agents using a general localization workflow.","descriptions":{"en":"A benchmark for evaluating multilingual LLM agents using a general localization workflow."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23490","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_babelarena_f22828e6"},"modality":null,"name":"BabelArena","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23490","source_url":"https://arxiv.org/abs/2609.23490"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23490","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23490","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:shreyaskc/babeljudge","source_record":"bm_babeljudge_6d1b19e2"},"categories":["Agents","General AI","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"BabelJudge audits LLM-as-a-judge reliability across languages and agent trajectories, measuring position bias, verbosity bias, order inconsistency, and cross-lingual degradation without human labels, providing a composite bias-penalised reliability score.","descriptions":{"en":"BabelJudge audits LLM-as-a-judge reliability across languages and agent trajectories, measuring position bias, verbosity bias, order inconsistency, and cross-lingual degradation without human labels, providing a composite bias-penalised reliability score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22329","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","multilingual"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_babeljudge_6d1b19e2"},"modality":null,"name":"BabelJudge","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22329","source_url":"https://arxiv.org/abs/2606.22329"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22329","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22329","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["长文本","Long-Context","NeurIPS 2024","大语言模型","LLM","长上下文","Long Context","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BABILong is designed to test language models' ability to reason across facts distributed in extremely long documents. It contains a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. BABILong旨在测试语言模型对分布在极长文档中的事实进行推理的能力，涵盖事实链接、简单归纳、推导、计数和处理列表/集合等20种各类推理任务。","descriptions":{"en":"BABILong is designed to test language models' ability to reason across facts distributed in extremely long documents. It contains a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets.","zh":"BABILong旨在测试语言模型对分布在极长文档中的事实进行推理的能力，涵盖事实链接、简单归纳、推导、计数和处理列表/集合等20种各类推理任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1266","languages":[],"library_direction":null,"modality":null,"name":"BABILong","openness":"restricted","publisher":"AIRI","released":"2024-06-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1266-babilong","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BABILong","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":258,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":202,"join":"llm_stats_id","repository":null,"source_record":"catalog_f7192a6a6674f631"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A benchmark for early-stage visual reasoning and perception on child-like vision tasks.","descriptions":{"en":"A benchmark for early-stage visual reasoning and perception on child-like vision tasks."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:babyvision:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/babyvision?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:babyvision","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_f7192a6a6674f631"},"modality":"multimodal","name":"BabyVision","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":85.7,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.857,"raw_min":0.384,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:babyvision:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/babyvision?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-babyvision","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/babyvision?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["BabyVision"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Multimodal vision benchmark. Temperature=1.0, top_p=0.95, max context 164K tokens. Images resized to shorter side at least 1.5K pixels.","descriptions":{"en":"Multimodal vision benchmark. Temperature=1.0, top_p=0.95, max context 164K tokens. Images resized to shorter side at least 1.5K pixels."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000babyvision\u0000zai_glm_5_3_flash_model_card\u0000babyvision\u0000temp 1.0, top_p 0.95, max context 164K, images resized so shorter side >= 1.5K pixels\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000babyvision\u0000zai_glm_5_3_flash_model_card\u0000babyvision\u0000temp 1.0, top_p 0.95, max context 164K, images resized so shorter side >= 1.5K pixels\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:babyvision","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["visual-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"visual-reasoning","source_record":null},"modality":null,"name":"BabyVision","openness":"unknown","publisher":null,"released":"2025-12-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:babyvision","source_url":"https://github.com/babyvision/babyvision"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":89.6,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":89.6,"raw_min":53.4,"source_reference":{"obs_id":"curated\u0000babyvision\u0000deepseek_v4_1_flash_technical_report\u0000babyvision\u0000think max, Pass@1, with tools\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000babyvision\u0000deepseek_v4_1_flash_technical_report\u0000babyvision\u0000think max, Pass@1, with tools\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"}],"slug":"babyvision","source":"model_reports","source_url":"https://github.com/babyvision/babyvision","topics":["multimodal"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"BackendForge is a benchmark of 56 contract-defined backend generation tasks from real open-source applications. LLMs must generate Dockerized services evaluated through HTTP tests against an OpenAPI contract.","descriptions":{"en":"BackendForge is a benchmark of 56 contract-defined backend generation tasks from real open-source applications. LLMs must generate Dockerized services evaluated through HTTP tests against an OpenAPI contract."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.11042","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_backendforge_1bb8c499"},"modality":null,"name":"BackendForge","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11042","source_url":"https://arxiv.org/abs/2607.11042"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11042","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11042","topics":["coding","agents"],"unit":null},{"aliases":["Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs"],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"An algorithmic-trading benchmark for LLM coding agents. It uses multiple-choice questions derived from backtests, with an independent checker that re-derives every answer, and scores models by accuracy.","descriptions":{"en":"An algorithmic-trading benchmark for LLM coding agents. It uses multiple-choice questions derived from backtests, with an independent checker that re-derives every answer, and scores models by accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11232","languages":[],"library_direction":null,"modality":null,"name":"Backtrader-Bench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11232","source_url":"https://arxiv.org/abs/2608.11232"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11232","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11232","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A retrospective benchmark where systems recover problem-space and solution-space precursor signals for a mature target topic under a temporal evidence constraint.","descriptions":{"en":"A retrospective benchmark where systems recover problem-space and solution-space precursor signals for a mature target topic under a temporal evidence constraint."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24921","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_backtrend_d357c50d"},"modality":null,"name":"BackTrend","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24921","source_url":"https://arxiv.org/abs/2609.24921"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24921","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24921","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Cybersecurity","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates whether LLM agents exploit optional shortcuts in three synthetic tabular ML tasks to inflate public test scores while failing hidden tests.","descriptions":{"en":"Evaluates whether LLM agents exploit optional shortcuts in three synthetic tabular ML tasks to inflate public test scores while failing hidden tests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30724","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","ai-rnd"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_baitbench_067a4af0"},"modality":null,"name":"BAITBENCH","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30724","source_url":"https://arxiv.org/abs/2608.30724"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30724","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30724","topics":["agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Mental health assessment relies on episodic self-report scales, which convert subjective states such as stress into numerical scores but provide only sparse snapshots of wellbeing.","descriptions":{"en":"Mental health assessment relies on episodic self-report scales, which convert subjective states such as stress into numerical scores but provide only sparse snapshots of wellbeing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27219","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_balms_57758cdd"},"modality":null,"name":"BALMS","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27219","source_url":"https://arxiv.org/abs/2608.27219"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27219","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27219","topics":["medical"],"unit":null},{"aliases":["Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:banglallm/banglasafe","source_record":"bm_register-shifts-break-llm-safety-a-bengali_9b6dd4e8"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM safety on 879 Bengali prompts spanning 17 harm categories and five prompting conditions varying language, writing style, and authority framing.","descriptions":{"en":"Evaluates LLM safety on 879 Bengali prompts spanning 17 harm categories and five prompting conditions varying language, writing style, and authority framing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22335","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_register-shifts-break-llm-safety-a-bengali_9b6dd4e8"},"modality":null,"name":"BanglaSafe","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22335","source_url":"http://arxiv.org/abs/2608.22335v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22335","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22335v1","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates Bangla LLM safety across six language forms using 2,366 prompts and a held-out 354-prompt split spanning unsafe, safe, and safe-sensitive requests, with deterministic response scoring and guardrail screening.","descriptions":{"en":"Evaluates Bangla LLM safety across six language forms using 2,366 prompts and a held-out 354-prompt split spanning unsafe, safe, and safe-sensitive requests, with deterministic response scoring and guardrail screening."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.21880","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_banglaveilguard_8771f924"},"modality":null,"name":"BanglaVeilGuard","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.21880","source_url":"http://arxiv.org/abs/2608.21880v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-21880","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21880v1","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"BanglaWild is a benchmark of 2,535 Bengali scene text images with verbatim gold transcriptions and diagnostic attributes. It evaluates OCR and vision-language models on in-the-wild scene text recognition.","descriptions":{"en":"BanglaWild is a benchmark of 2,535 Bengali scene text images with verbatim gold transcriptions and diagnostic attributes. It evaluates OCR and vision-language models on in-the-wild scene text recognition."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03884","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","multilingual"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_banglawild_bb99b93a"},"modality":null,"name":"BanglaWild","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03884","source_url":"https://arxiv.org/abs/2608.03884"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03884","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03884","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_9dcbfe996a77cf11"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["finance","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BankerToolBench is a public benchmark that evaluates models on banking and finance tool-use tasks. Models are scored against dataset rubrics, measuring their ability to correctly invoke tools and complete multi-step financial workflows.","descriptions":{"en":"BankerToolBench is a public benchmark that evaluates models on banking and finance tool-use tasks. Models are scored against dataset rubrics, measuring their ability to correctly invoke tools and complete multi-step financial workflows."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bankertoolbench:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bankertoolbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bankertoolbench","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","tool-use"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_9dcbfe996a77cf11"},"modality":"text","name":"BankerToolBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":76.12,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.7612,"raw_min":0.7612,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bankertoolbench:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bankertoolbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-bankertoolbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bankertoolbench?top_n=500","topics":["agents","finance"],"unit":null},{"aliases":["BankerToolBench","Banker Tool Bench","BTB"],"attention_signals":null,"categories":["professional"],"collected_at":null,"description":"End-to-end investment-banking tasks produce spreadsheets, presentations, and documents; the agent harness, financial-data tools, and rubric-grader configuration are part of the score.","descriptions":{"en":"End-to-end investment-banking tasks produce spreadsheets, presentations, and documents; the agent harness, financial-data tools, and rubric-grader configuration are part of the score."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000bankertoolbench\u0000tencent_hy4_preview\u0000bankertoolbench\u0000OpenCode scaffold, Gemini 3 Flash judge\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000bankertoolbench\u0000tencent_hy4_preview\u0000bankertoolbench\u0000OpenCode scaffold, Gemini 3 Flash judge\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:bankertoolbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["finance-legal"],"join":"review:14H034160212:2026-10-15","primary":"finance-legal","source_record":null},"modality":null,"name":"BankerToolBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":78.6,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":78.6,"raw_min":78.6,"source_reference":{"obs_id":"curated\u0000bankertoolbench\u0000tencent_hy4_preview\u0000bankertoolbench\u0000OpenCode scaffold, Gemini 3 Flash judge\u0000Hy4 preview","observation_id":"curated\u0000bankertoolbench\u0000tencent_hy4_preview\u0000bankertoolbench\u0000OpenCode scaffold, Gemini 3 Flash judge\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"bankertoolbench","source":"model_reports","source_url":"https://github.com/Handshake-AI-Research/bankertoolbench","topics":["agents","finance"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["General AI","Robustness","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Recent speech LLMs can leverage rich paralinguistic cues, enabling more complex and personalized interactions.","descriptions":{"en":"Recent speech LLMs can leverage rich paralinguistic cues, enabling more complex and personalized interactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13550","languages":[],"library_direction":{"basis":"model-review-daily","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_basis-bench_92c1bf5d"},"modality":null,"name":"BASIS-Bench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13550","source_url":"https://arxiv.org/abs/2610.13550"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13550","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13550","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates basketball understanding through 7,980 questions over text, image, and video, covering ten tasks from event recognition to structured game knowledge.","descriptions":{"en":"Evaluates basketball understanding through 7,980 questions over text, image, and video, covering ten tasks from event recognition to structured game knowledge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23435","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","knowledge-expert-qa"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_basketballbench_30421d6c"},"modality":null,"name":"BasketballBench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23435","source_url":"http://arxiv.org/abs/2608.23435v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23435","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23435v1","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_bavground_960d10f0"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"BavGround evaluates regional cultural grounding and dialect competence in Bavarian across English, German, and Bavarian, with 618 multi-parallel multiple-choice questions across eight cultural domains.","descriptions":{"en":"BavGround evaluates regional cultural grounding and dialect competence in Bavarian across English, German, and Bavarian, with 618 multi-parallel multiple-choice questions across eight cultural domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12894","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_bavground_960d10f0"},"modality":null,"name":"BavGround","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12894","source_url":"https://arxiv.org/abs/2608.12894"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12894","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12894","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Physical active vision allows robots to change their viewpoint when task-relevant observations become unreliable, yet existing manipulation benchmarks provide limited support for studying how policies recover from occlusion during executio…","descriptions":{"en":"Physical active vision allows robots to change their viewpoint when task-relevant observations become unreliable, yet existing manipulation benchmarks provide limited support for studying how policies recover from occlusion during executio…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37292","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_bavo-bench_3a1462dc"},"modality":null,"name":"BAVO-Bench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37292","source_url":"https://arxiv.org/abs/2609.37292"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37292","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37292","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ziyan-wang98/bazaarbench","source_record":"bm_bazaarbench_5b8e7fa1"},"categories":["Cybersecurity","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"In decentralized consumer-to-consumer (C2C) marketplaces, people list goods, negotiate with strangers, and rate one another, so trust rests on reputation.","descriptions":{"en":"In decentralized consumer-to-consumer (C2C) marketplaces, people list goods, negotiate with strangers, and rate one another, so trust rests on reputation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.06748","languages":[],"library_direction":{"basis":"model-review-daily","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_bazaarbench_5b8e7fa1"},"modality":null,"name":"BazaarBench","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.06748","source_url":"https://arxiv.org/abs/2610.06748"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-06748","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06748","topics":["agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":578,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_e66f6bfdea070623"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Big-Bench Hard (BBH) is a suite of 23 challenging tasks selected from BIG-Bench for which prior language model evaluations did not outperform the average human-rater. These tasks require multi-step reasoning across diverse domains including arithmetic, logical reasoning, reading comprehension, and commonsense reasoning. The benchmark was designed to test capabilities believed to be beyond current language models and focuses on evaluating complex reasoning skills including temporal understanding, spatial reasoning, causal understanding, and deductive logical reasoning.","descriptions":{"en":"Big-Bench Hard (BBH) is a suite of 23 challenging tasks selected from BIG-Bench for which prior language model evaluations did not outperform the average human-rater. These tasks require multi-step reasoning across diverse domains including arithmetic, logical reasoning, reading comprehension, and commonsense reasoning. The benchmark was designed to test capabilities believed to be beyond current language models and focuses on evaluating complex reasoning skills including temporal understanding, spatial reasoning, causal understanding, and deductive logical reasoning."},"evidence_summary":{"document_count":1,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bbh:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bbh?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bbh","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_e66f6bfdea070623"},"modality":"text","name":"BBH","openness":"unknown","publication":{"evidence":[{"note":"The authors' repository links the introducing paper and defines BBH as 23 challenging tasks selected from BIG-Bench. It supplies task data and chain-of-thought prompts, matching this record's hard-task subset without treating all BIG-Bench results as BBH.","url":"https://github.com/suzgunmirac/BIG-Bench-Hard/blob/main/README.md"}],"id":"big-bench-hard-original","links":[{"kind":"paper","title":"Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them","url":"https://arxiv.org/abs/2210.09261"},{"kind":"repo","title":"BIG-Bench Hard authors' repository","url":"https://github.com/suzgunmirac/BIG-Bench-Hard"}],"record_keys":["llm-stats:bbh"],"reviewed_at":"2026-10-26","scope":{"en":"BIG-Bench Hard's original selection of 23 challenging BIG-Bench tasks. Direct-answer and chain-of-thought prompting settings remain distinct; the broader BIG-Bench collection is not this subset.","zh":"BIG-Bench Hard 最初选出的 23 项高难度 BIG-Bench 任务。直接作答与思维链提示属于不同设置，本子集也区别于完整 BIG-Bench 题库。"}},"publisher":null,"released":"2022-10-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:bbh","source_url":"https://arxiv.org/abs/2210.09261"},"repo_kind":null,"repo_resolution_status":null,"score_count":12,"score_direction":"higher_is_better","score_summary":{"display_max":88.87,"display_multiplier":100,"model_count":12,"model_count_basis":"source_model_id","numeric_count":12,"raw_max":0.8887,"raw_min":0.304,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bbh:qwen3-235b-a22b","reported_date":"2025-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bbh?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"}],"slug":"llm-stats-bbh","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bbh?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":578,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:2210.09261","repository":null,"source_record":"catalog_ad505a3aa6e37229"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BIG-Bench Hard (BBH) is a subset of the BIG-Bench, a diverse evaluation suite for language models. BBH focuses on a suite of 23 challenging tasks from BIG-Bench that were found to be beyond the capabilities of current language models. BIG Bench-Hard（BBH）是BIG Bench的一个子集，它是一个用于语言模型的多样化评估套件。BBH专注于BIG Bench的23项具有挑战性的任务，这些任务被发现超出了当前语言模型的能力。","descriptions":{"en":"BIG-Bench Hard (BBH) is a subset of the BIG-Bench, a diverse evaluation suite for language models. BBH focuses on a suite of 23 challenging tasks from BIG-Bench that were found to be beyond the capabilities of current language models.","zh":"BIG Bench-Hard（BBH）是BIG Bench的一个子集，它是一个用于语言模型的多样化评估套件。BBH专注于BIG Bench的23项具有挑战性的任务，这些任务被发现超出了当前语言模型的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:539","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"artifact:arxiv:2210.09261","primary":"abstract-reasoning","source_record":"catalog_ad505a3aa6e37229"},"modality":null,"name":"BBH","openness":"unknown","publisher":null,"released":"2022-10-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-539-bbh","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BBH","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":20046,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":144,"join":"llm_stats_id","repository":null,"source_record":"catalog_9019868a487d165b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","knowledge","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BC-VL is a vision-language benchmark for knowledge-grounded multimodal question answering.","descriptions":{"en":"BC-VL is a vision-language benchmark for knowledge-grounded multimodal question answering."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bc-vl:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bc-vl?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bc-vl","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_9019868a487d165b"},"modality":"multimodal","name":"BC-VL","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":51.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.511,"raw_min":0.511,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bc-vl:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bc-vl?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-bc-vl","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bc-vl?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":158,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:mohammadtavakoli78/beam","source_record":"catalog_7bcc639830899dd7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Beam 128K evaluates reasoning over long inputs at a 128K-token context length.","descriptions":{"en":"Beam 128K evaluates reasoning over long inputs at a 128K-token context length."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:beam-128k:muse-glimmer-30b","reported_at":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/beam-128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:beam-128k","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","long-context"],"join":"llm_stats_id","primary":"agent-memory","source_record":"catalog_7bcc639830899dd7"},"modality":"text","name":"Beam 128K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":65.10000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.651,"raw_min":0.651,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:beam-128k:muse-glimmer-30b","reported_date":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/beam-128k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"}],"slug":"llm-stats-beam-128k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/beam-128k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Bilingual reasoning","Finance","Multimodal","Multimodal reasoning","OCR-grounded reasoning","Professional document reasoning","Reasoning","Scientific Research & AI for Science"],"collected_at":"2026-10-14T09:44:51Z","description":"BEAR-Bench evaluates multimodal models on bilingual English-Russian text-dense business and scientific documents with 1,000 human-annotated questions.","descriptions":{"en":"BEAR-Bench evaluates multimodal models on bilingual English-Russian text-dense business and scientific documents with 1,000 human-annotated questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17895","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["document-ocr","science"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_bear-bench_6aff8566"},"modality":null,"name":"BEAR-Bench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17895","source_url":"https://arxiv.org/abs/2608.17895"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17895","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17895","topics":["multimodal","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":12,"github_stars_30d":5,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:nubot-nudt/behavior-skill","source_record":"bm_behavior-skill_187dc006"},"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language-action policies on 50 long-horizon mobile manipulation tasks via 235,492 skill instances, with trajectory-level and skill-level metrics.","descriptions":{"en":"Evaluates vision-language-action policies on 50 long-horizon mobile manipulation tasks via 235,492 skill instances, with trajectory-level and skill-level metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.30536","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_behavior-skill_187dc006"},"modality":null,"name":"Behavior-Skill","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30536","source_url":"https://arxiv.org/abs/2608.30536"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30536","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30536","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:buaa-irip-llm/behavior2trip","source_record":"bm_behavior2trip_b70cb42e"},"categories":["General AI","Language & Knowledge","Planning","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Travel planning agents assist users in generating personalized travel plans by modeling their individual preferences.","descriptions":{"en":"Travel planning agents assist users in generating personalized travel plans by modeling their individual preferences."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.26807","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","agent-memory"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_behavior2trip_b70cb42e"},"modality":null,"name":"Behavior2Trip","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26807","source_url":"https://arxiv.org/abs/2608.26807"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26807","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26807","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_behaviorbench_fe357f72"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"BehaviorBench evaluates personalized decision modeling from real-world behavioral traces, with belief and trade prediction tasks from prediction-market and on-chain records across 2,000 wallets.","descriptions":{"en":"BehaviorBench evaluates personalized decision modeling from real-world behavioral traces, with belief and trade prediction tasks from prediction-market and on-chain records across 2,000 wallets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02798","languages":[],"library_direction":{"basis":"model-review-2","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_behaviorbench_fe357f72"},"modality":null,"name":"BehaviorBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02798","source_url":"https://arxiv.org/abs/2606.02798"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02798","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02798","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:umich-foreseer/behaviorbench_eval","source_record":"bm_behaviorbench_01c04f85"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates foundation models on four behavioral science capabilities: behavior prediction and simulation, strategic decision-making, subject-trait inference, and behavioral knowledge application, assessing both individual-level and distributional alignment.","descriptions":{"en":"Evaluates foundation models on four behavioral science capabilities: behavior prediction and simulation, strategic decision-making, subject-trait inference, and behavioral knowledge application, assessing both individual-level and distributional alignment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.24162","languages":[],"library_direction":null,"modality":null,"name":"BehaviorBench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24162","source_url":"https://arxiv.org/abs/2606.24162"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24162","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24162","topics":[],"unit":null},{"aliases":["BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"BekchiAI-Benchmark evaluates LLM agent skills using 2,057 deterministic tool-using ReAct tasks across 7 categories, scoring accuracy plus tool-call adherence, URL hallucination, source-match, and token cost.","descriptions":{"en":"BekchiAI-Benchmark evaluates LLM agent skills using 2,057 deterministic tool-using ReAct tasks across 7 categories, scoring accuracy plus tool-call adherence, URL hallucination, source-match, and token cost."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.26867","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_bekchiai-measuring-observing-and-controlli_e45a38cc"},"modality":null,"name":"BekchiAI-Benchmark","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26867","source_url":"https://arxiv.org/abs/2608.26867"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26867","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26867","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BELEBELE, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset enables the evaluation of text models in\nhigh-, medium-, and low-resource languages. BELEBELE 是一个多项选择机器阅读理解（MRC）数据集，涵盖 122 种语言变体。该数据集显著扩展了自然语言理解（NLU）基准的语言覆盖范围，使得可以在高、中、低资源语言中评估文本模型。","descriptions":{"en":"BELEBELE, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset enables the evaluation of text models in\nhigh-, medium-, and low-resource languages.","zh":"BELEBELE 是一个多项选择机器阅读理解（MRC）数据集，涵盖 122 种语言变体。该数据集显著扩展了自然语言理解（NLU）基准的语言覆盖范围，使得可以在高、中、低资源语言中评估文本模型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1086","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"artifact:gh:facebookresearch/belebele","primary":"multilingual","source_record":"catalog_001a3cc8c6d7519a"},"modality":null,"name":"Belebele","openness":"unknown","publisher":"FaceBook","released":"2024-07-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1086-belebele","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Belebele","topics":["language"],"unit":null},{"aliases":["BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-09-25T04:40:22Z","description":"An operational benchmark comparing moderation filters, jailbreak detectors, and general-purpose LLM supervisors across effectiveness, false positives, latency, and cost.","descriptions":{"en":"An operational benchmark comparing moderation filters, jailbreak detectors, and general-purpose LLM supervisors across effectiveness, false positives, latency, and cost."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.20668","languages":[],"library_direction":null,"modality":null,"name":"BELLS-O","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20668","source_url":"https://arxiv.org/abs/2606.20668"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20668","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20668","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"This entry describes a simulation benchmark for visuo-tactile bimanual manipulation, but no task dataset, scorer, or public reuse path is evidenced.","descriptions":{"en":"This entry describes a simulation benchmark for visuo-tactile bimanual manipulation, but no task dataset, scorer, or public reuse path is evidenced."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15726","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_bench2dex_de55dd66"},"modality":null,"name":"Bench2Dex","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15726","source_url":"https://arxiv.org/abs/2609.15726"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15726","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15726","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on 149 protocol-modification tasks reconstructed from real changes scientists made to published wet-lab protocols, using weighted rubric scoring.","descriptions":{"en":"Evaluates LLMs on 149 protocol-modification tasks reconstructed from real changes scientists made to published wet-lab protocols, using weighted rubric scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23898","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_benchbench-protocol_1d14c082"},"modality":null,"name":"BenchBench-Protocol","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23898","source_url":"https://arxiv.org/abs/2608.23898"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23898","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23898","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_fdc3093e1cbc6340"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","code","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BenchCAD is a benchmark for programmatic CAD reasoning built from 17,900 execution-verified CadQuery programs spanning 106 industrial part families, roughly half anchored to real ISO, DIN, EN, ASME, and IEC specification tables. It decomposes CAD capability into matched tasks; the Vision2Code task requires models to generate CadQuery code from multi-view renders, scored by voxel IoU against the reference geometry.","descriptions":{"en":"BenchCAD is a benchmark for programmatic CAD reasoning built from 17,900 execution-verified CadQuery programs spanning 106 industrial part families, roughly half anchored to real ISO, DIN, EN, ASME, and IEC specification tables. It decomposes CAD capability into matched tasks; the Vision2Code task requires models to generate CadQuery code from multi-view renders, scored by voxel IoU against the reference geometry."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:benchcad:claude-sonnet-5","reported_at":"2026-06-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:benchcad","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","spatial-3d"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_fdc3093e1cbc6340"},"modality":"multimodal","name":"BenchCAD","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":70.6,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.706,"raw_min":0.373,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:benchcad:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-benchcad","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad?top_n=500","topics":["coding","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":80,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:benchcad/benchcad-main","source_record":"catalog_a94b219f9fd71820"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","code","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BenchCAD variant evaluated with access to a Python tool for programmatic CAD reasoning.","descriptions":{"en":"BenchCAD variant evaluated with access to a Python tool for programmatic CAD reasoning."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:benchcad-with-python-tool:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad-with-python-tool?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:benchcad-with-python-tool","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","spatial-3d"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_a94b219f9fd71820"},"modality":"multimodal","name":"BenchCAD (with Python tool)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":83.39999999999999,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.834,"raw_min":0.739,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:benchcad-with-python-tool:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad-with-python-tool?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-benchcad-with-python-tool","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad-with-python-tool?top_n=500","topics":["coding","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ant-intl/benchcompass","source_record":"bm_benchcompass_fe663cc7"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A payment-domain benchmark with 306 items (MCQ, open QA, and attack variants) that tests models' ability to use domain context correctly, distinguish parametric from context-grounded knowledge, and remain robust under controlled context/question attacks.","descriptions":{"en":"A payment-domain benchmark with 306 items (MCQ, open QA, and attack variants) that tests models' ability to use domain context correctly, distinguish parametric from context-grounded knowledge, and remain robust under controlled context/question attacks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.18270","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","factuality"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_benchcompass_fe663cc7"},"modality":null,"name":"BENCHCOMPASS","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18270","source_url":"https://arxiv.org/abs/2609.18270"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18270","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18270","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BenchMAX is a comprehensive, high-quality, and multiway parallel multilingual benchmark comprising 10 tasks designed to assess crucial capabilities across 17 diverse language. BenchMAX 是一个全面、高质量的多向并行多语言基准，包含 10 个任务，旨在评估 17 种不同语言的关键能力。","descriptions":{"en":"BenchMAX is a comprehensive, high-quality, and multiway parallel multilingual benchmark comprising 10 tasks designed to assess crucial capabilities across 17 diverse language.","zh":"BenchMAX 是一个全面、高质量的多向并行多语言基准，包含 10 个任务，旨在评估 17 种不同语言的关键能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1562","languages":["English","Chinese","Japanese","Korean","French","German","Spanish","Arabic","Russian","Vietnamese","Thai","Multilingual"],"library_direction":null,"modality":null,"name":"BenchMAX","openness":"unknown","publisher":"National Key Laboratory for Novel Software Technology, Nanjing University, etc.","released":"2025-02-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1562-benchmax","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BenchMAX","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:senssei/benchrig","source_record":"bm_benchrig_361559d9"},"categories":["Language & Knowledge","Manufacturing & Process Control","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A cross-platform evaluation and hardware profiling rig for local LLMs, focusing on speed, coding, reasoning, and context scaling.","descriptions":{"en":"A cross-platform evaluation and hardware profiling rig for local LLMs, focusing on speed, coding, reasoning, and context scaling."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:senssei/benchrig","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_benchrig_361559d9"},"modality":null,"name":"BenchRig","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:senssei/benchrig","source_url":"https://github.com/senssei/benchrig"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-senssei-benchrig","source":"claire_radar","source_url":"https://github.com/senssei/benchrig","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_bensyc_50720d15"},"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"BenSyc is a benchmark for studying conversational sycophancy in Bengali social contexts, with binary labels and a five-level taxonomy from invalidation to escalation.","descriptions":{"en":"BenSyc is a benchmark for studying conversational sycophancy in Bengali social contexts, with binary labels and a five-level taxonomy from invalidation to escalation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.10061","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_bensyc_50720d15"},"modality":null,"name":"BenSyc","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10061","source_url":"https://arxiv.org/abs/2606.10061"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10061","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10061","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":3516,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_ccaa76994404b451"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Beyond AIME is a difficult mathematical reasoning benchmark designed to test deeper reasoning chains and harder decomposition than standard AIME-style problem sets.","descriptions":{"en":"Beyond AIME is a difficult mathematical reasoning benchmark designed to test deeper reasoning chains and harder decomposition than standard AIME-style problem sets."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:beyond-aime:sarvam-105b","reported_at":"2026-03-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/beyond-aime?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:beyond-aime","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_ccaa76994404b451"},"modality":"text","name":"Beyond AIME","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":88.0,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.88,"raw_min":0.583,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:beyond-aime:seed-2.1-turbo","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/beyond-aime?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-beyond-aime","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/beyond-aime?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates large language models on 31 expert-authored prompts covering Saudi dialect phenomena, with expert-established ground truths and rubric-based scoring.","descriptions":{"en":"Evaluates large language models on 31 expert-authored prompts covering Saudi dialect phenomena, with expert-established ground truths and rubric-based scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29990","languages":[],"library_direction":null,"modality":null,"name":"Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29990","source_url":"https://arxiv.org/abs/2608.29990"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29990","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29990","topics":[],"unit":null},{"aliases":["BeyondAIME"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":3516,"hf_paper_upvotes":null,"join":"artifact:hf:bytedance-seed/beyondaime","repository":null,"source_record":"catalog_ccaa76994404b451"},"categories":["math"],"collected_at":null,"description":"Beyond AIME difficulty competition math; most models score low and gaps are dominated by run-to-run variance.","descriptions":{"en":"Beyond AIME difficulty competition math; most models score low and gaps are dominated by run-to-run variance."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000beyond_aime\u0000bytedance_seed2_0_model_card\u0000beyond_aime\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000beyond_aime\u0000bytedance_seed2_0_model_card\u0000beyond_aime\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:beyond_aime","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"artifact:hf:bytedance-seed/beyondaime","primary":"math","source_record":"catalog_ccaa76994404b451"},"modality":null,"name":"BeyondAIME","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":86.5,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":86.5,"raw_min":69.0,"source_reference":{"obs_id":"curated\u0000beyond_aime\u0000bytedance_seed2_0_model_card\u0000beyond_aime\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000beyond_aime\u0000bytedance_seed2_0_model_card\u0000beyond_aime\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"beyond_aime","source":"model_reports","source_url":"https://huggingface.co/datasets/ByteDance-Seed/BeyondAIME","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":9,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yigitekin/beyondmasks","source_record":"bm_beyondmasks_787c429f"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates causally consistent video object removal using paired synthetic and real videos with aligned references, and includes the CORE metric for after-effect consistency.","descriptions":{"en":"Evaluates causally consistent video object removal using paired synthetic and real videos with aligned references, and includes the CORE metric for after-effect consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.20107","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_beyondmasks_787c429f"},"modality":null,"name":"BeyondMasks","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.20107","source_url":"https://arxiv.org/abs/2608.20107"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-20107","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20107","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":13051,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:shishirpatil/gorilla","source_record":"lib_bfcl"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The Berkeley Function Calling Leaderboard (BFCL) is the first comprehensive and executable function call evaluation dedicated to assessing Large Language Models' ability to invoke functions. It evaluates serial and parallel function calls across multiple programming languages (Python, Java, JavaScript, REST API) using a novel Abstract Syntax Tree (AST) evaluation method. The benchmark consists of over 2,000 question-function-answer pairs covering diverse application domains and complex use cases including multiple function calls, parallel function calls, and multi-turn interactions.","descriptions":{"en":"The Berkeley Function Calling Leaderboard (BFCL) is the first comprehensive and executable function call evaluation dedicated to assessing Large Language Models' ability to invoke functions. It evaluates serial and parallel function calls across multiple programming languages (Python, Java, JavaScript, REST API) using a novel Abstract Syntax Tree (AST) evaluation method. The benchmark consists of over 2,000 question-function-answer pairs covering diverse application domains and complex use cases including multiple function calls, parallel function calls, and multi-turn interactions."},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bfcl:llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bfcl","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"lib_bfcl"},"modality":"text","name":"BFCL","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":88.5,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.885,"raw_min":0.562,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bfcl:llama-3.1-405b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Sonic"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 30B A3B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 32B"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ-32B"}],"slug":"llm-stats-bfcl","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["BFCL","Berkeley Function Calling Leaderboard"],"attention_signals":null,"categories":["tool_use"],"collected_at":null,"description":"Schema complexity and execution checking vary by version.","descriptions":{"en":"Schema complexity and execution checking vary by version."},"evidence_summary":{"document_count":4,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000bfcl\u0000qwen3_technical_report\u0000bfcl_v3\u0000v3\u0000Qwen3-235B-A22B (Thinking)","reported_at":"2025-05-14","source_url":"https://arxiv.org/abs/2505.09388"},"first_score_reported_at":"2025-05-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000bfcl\u0000qwen3_technical_report\u0000bfcl_v3\u0000v3\u0000Qwen3-235B-A22B (Thinking)","reported_at":"2025-05-14","source_url":"https://arxiv.org/abs/2505.09388"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:bfcl","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["tool-use"],"join":"review:14H034160212:2026-10-15","primary":"tool-use","source_record":null},"modality":null,"name":"BFCL","openness":"unknown","publisher":null,"released":"2024-02-26","released_reference":{"basis":"benchmark_release","source_key":"model-reports:bfcl","source_url":"https://gorilla.cs.berkeley.edu/leaderboard.html"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":72.9,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":72.9,"raw_min":70.8,"source_reference":{"obs_id":"curated\u0000bfcl\u0000qwen3_5_model_card\u0000bfcl_v4\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000bfcl\u0000qwen3_5_model_card\u0000bfcl_v4\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2025-05-14","date_precision":"document_publication","name":"Qwen3-235B-A22B (Thinking)"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"bfcl","source":"model_reports","source_url":"https://gorilla.cs.berkeley.edu/leaderboard.html","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":13051,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:shishirpatil/gorilla","source_record":"catalog_5c197b338806921e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Berkeley Function Calling Leaderboard (BFCL) v2 is a comprehensive benchmark for evaluating large language models' function calling capabilities. It features 2,251 question-function-answer pairs with enterprise and OSS-contributed functions, addressing data contamination and bias through live, user-contributed scenarios. The benchmark evaluates AST accuracy, executable accuracy, irrelevance detection, and relevance detection across multiple programming languages (Python, Java, JavaScript) and includes complex real-world function calling scenarios with multi-lingual prompts.","descriptions":{"en":"Berkeley Function Calling Leaderboard (BFCL) v2 is a comprehensive benchmark for evaluating large language models' function calling capabilities. It features 2,251 question-function-answer pairs with enterprise and OSS-contributed functions, addressing data contamination and bias through live, user-contributed scenarios. The benchmark evaluates AST accuracy, executable accuracy, irrelevance detection, and relevance detection across multiple programming languages (Python, Java, JavaScript) and includes complex real-world function calling scenarios with multi-lingual prompts."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bfcl-v2:llama-3.2-3b-instruct","reported_at":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bfcl-v2","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_5c197b338806921e"},"modality":"text","name":"BFCL v2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":77.3,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.773,"raw_min":0.636,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bfcl-v2:llama-3.3-70b-instruct","reported_date":"2024-12-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 8B V1"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 70B Instruct"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama-3.3 Nemotron Super 49B v1"}],"slug":"llm-stats-bfcl-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v2?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":13051,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_d0ca5bf6b14c4f58"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","structured_output","finance","general","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Berkeley Function Calling Leaderboard v3 (BFCL-v3) is an advanced benchmark that evaluates large language models' function calling capabilities through multi-turn and multi-step interactions. It introduces extended conversational exchanges where models must retain contextual information across turns and execute multiple internal function calls for complex user requests. The benchmark includes 1000 test cases across domains like vehicle control, trading bots, travel booking, and file system management, using state-based evaluation to verify both system state changes and execution path correctness.","descriptions":{"en":"Berkeley Function Calling Leaderboard v3 (BFCL-v3) is an advanced benchmark that evaluates large language models' function calling capabilities through multi-turn and multi-step interactions. It introduces extended conversational exchanges where models must retain contextual information across turns and execute multiple internal function calls for complex user requests. The benchmark includes 1000 test cases across domains like vehicle control, trading bots, travel booking, and file system management, using state-based evaluation to verify both system state changes and execution path correctness."},"evidence_summary":{"document_count":1,"model_count":19,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bfcl-v3:qwen3-coder-480b-a35b-instruct","reported_at":"2025-01-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bfcl-v3","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_d0ca5bf6b14c4f58"},"modality":"text","name":"BFCL-v3","openness":"unknown","publication":{"evidence":[{"note":"The Berkeley author release explicitly names BFCL V3, dates its release 2024-09-19, and explains the multi-turn/multi-step tasks, state verification, vehicle, trading, travel and file-system domains named in the source record. Its notice distinguishes the later V4 leaderboard composition.","url":"https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html"}],"id":"bfcl-v3-publication","links":[{"kind":"website","title":"BFCL V3 official release and methodology","url":"https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html"}],"record_keys":["llm-stats:bfcl-v3"],"reviewed_at":"2026-10-26","scope":{"en":"BFCL V3's original release and multi-turn methodology. Later BFCL V4 results and weighting are not applied to this record.","zh":"BFCL V3 的原始发布及多轮工具调用方法；不将后续 V4 的成绩或权重应用到这条记录。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":19,"score_direction":"higher_is_better","score_summary":{"display_max":77.8,"display_multiplier":100,"model_count":19,"model_count_basis":"source_model_id","numeric_count":19,"raw_max":0.778,"raw_min":0.63,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bfcl-v3:glm-4.5","reported_date":"2025-07-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-bfcl-v3","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3?top_n=500","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":13051,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:shishirpatil/gorilla","source_record":"catalog_072cecd64f67d822"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Berkeley Function Calling Leaderboard V4 (BFCL-V4) evaluates LLMs on their ability to accurately call functions and APIs, including simple, multiple, parallel, and nested function calls across diverse programming scenarios.","descriptions":{"en":"Berkeley Function Calling Leaderboard V4 (BFCL-V4) evaluates LLMs on their ability to accurately call functions and APIs, including simple, multiple, parallel, and nested function calls across diverse programming scenarios."},"evidence_summary":{"document_count":1,"model_count":15,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bfcl-v4:nova-2-lite","reported_at":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v4?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bfcl-v4","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_072cecd64f67d822"},"modality":"text","name":"BFCL-V4","openness":"unknown","publication":{"evidence":[{"note":"The Berkeley page expands BFCL to Berkeley Function Calling Leaderboard and explicitly identifies V4, matching the exact source record's expanded name. It links separate release blogs for V1, V2, V3 and V4.","url":"https://gorilla.cs.berkeley.edu/leaderboard.html"},{"note":"The authors' V4 launch page records release on 2025-07-17 and describes the agentic extension with web search, memory and format sensitivity. It distinguishes earlier versions and links the other V4 methodology parts.","url":"https://gorilla.cs.berkeley.edu/blogs/15_bfcl_v4_web_search.html"}],"id":"bfcl-v4-publication","links":[{"kind":"website","title":"BFCL V4 official release and methodology","url":"https://gorilla.cs.berkeley.edu/blogs/15_bfcl_v4_web_search.html"},{"kind":"website","title":"BFCL official leaderboard and version references","url":"https://gorilla.cs.berkeley.edu/leaderboard.html"}],"record_keys":["llm-stats:bfcl-v4"],"reviewed_at":"2026-10-26","scope":{"en":"Berkeley Function Calling Leaderboard V4's author release. Its agentic extension and version-specific methodology remain distinct from V3.","zh":"Berkeley Function Calling Leaderboard V4 的作者发布页；该版智能体任务扩展与评测方法单独保留，不与 V3 混用。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":15,"score_direction":"higher_is_better","score_summary":{"display_max":75.0,"display_multiplier":100,"model_count":15,"model_count_basis":"source_model_id","numeric_count":15,"raw_max":0.75,"raw_min":0.253,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bfcl-v4:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v4?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-bfcl-v4","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v4?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":13051,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_30e8ebdf60c3b1f9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Berkeley Function Calling Leaderboard (BFCL) V3 MultiTurn benchmark that evaluates large language models' ability to handle multi-turn and multi-step function calling scenarios. The benchmark introduces complex interactions requiring models to manage sequential function calls, handle conversational context across multiple turns, and make dynamic decisions about when and how to use available functions. BFCL V3 uses state-based evaluation by verifying the actual state of API systems after function execution, providing more realistic assessment of function calling capabilities in agentic applications.","descriptions":{"en":"Berkeley Function Calling Leaderboard (BFCL) V3 MultiTurn benchmark that evaluates large language models' ability to handle multi-turn and multi-step function calling scenarios. The benchmark introduces complex interactions requiring models to manage sequential function calls, handle conversational context across multiple turns, and make dynamic decisions about when and how to use available functions. BFCL V3 uses state-based evaluation by verifying the actual state of API systems after function execution, providing more realistic assessment of function calling capabilities in agentic applications."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bfcl-v3-multiturn:nvidia-nemotron-nano-9b-v2","reported_at":"2025-08-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3-multiturn?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bfcl-v3-multiturn","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_30e8ebdf60c3b1f9"},"modality":"text","name":"BFCL_v3_MultiTurn","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":76.8,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.768,"raw_min":0.669,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bfcl-v3-multiturn:minimax-m2.5","reported_date":"2026-02-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3-multiturn?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax M2.5"},{"date":"2025-08-18","date_precision":"model_announcement","name":"Nemotron Nano 9B v2"}],"slug":"llm-stats-bfcl-v3-multiturn","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3-multiturn?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:sci-m-wang/bfi-adapt","source_record":"bm_do-ai-personas-grow-analyzing-and-benchmar_ac2a3c14"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates directional fidelity of event-induced personality change in LLM agents across 11 major life events, scored against Big Five trait directions from human longitudinal evidence.","descriptions":{"en":"Evaluates directional fidelity of event-induced personality change in LLM agents across 11 major life events, scored against Big Five trait directions from human longitudinal evidence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06485","languages":[],"library_direction":null,"modality":null,"name":"BFI-Adapt","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06485","source_url":"https://arxiv.org/abs/2608.06485"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06485","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06485","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_bg-real_ba34eeb5"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"BG-REAL is a benchmark for background manipulation detection and localization in images. It contains 7,000 processed samples (6,000 public-data anchored, 1,000 synthetic) over six edit families with matched authentic controls, source-group splits, and quality control.","descriptions":{"en":"BG-REAL is a benchmark for background manipulation detection and localization in images. It contains 7,000 processed samples (6,000 public-data anchored, 1,000 synthetic) over six edit families with matched authentic controls, source-group splits, and quality control."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.26232","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_bg-real_ba34eeb5"},"modality":null,"name":"BG-REAL","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26232","source_url":"https://arxiv.org/abs/2607.26232"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26232","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26232","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":7,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:biabench/biabench","source_record":"bm_biabench_69f19558"},"categories":["Language & Knowledge","Materials & Chemistry","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Artificial-intelligence (AI) agents hold promise for automating bioimage analysis, yet no benchmark evaluates whether they can carry out real-world analyses end to end.","descriptions":{"en":"Artificial-intelligence (AI) agents hold promise for automating bioimage analysis, yet no benchmark evaluates whether they can carry out real-world analyses end to end."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.34274","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_biabench_69f19558"},"modality":null,"name":"BIABench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34274","source_url":"https://arxiv.org/abs/2609.34274"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34274","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34274","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3543,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_efc9be1bb3b5bc96"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Big Bench Audio is an audio reasoning benchmark adapted from a subset of Big Bench Hard, with text questions converted to spoken audio. It evaluates the reasoning ability of speech-to-speech and audio language models on tasks delivered as audio input, with accuracy scored by an independent evaluation (Artificial Analysis).","descriptions":{"en":"Big Bench Audio is an audio reasoning benchmark adapted from a subset of Big Bench Hard, with text questions converted to spoken audio. It evaluates the reasoning ability of speech-to-speech and audio language models on tasks delivered as audio input, with accuracy scored by an independent evaluation (Artificial Analysis)."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:big-bench-audio:nova-2-sonic","reported_at":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-audio?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:big-bench-audio","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","abstract-reasoning"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_efc9be1bb3b5bc96"},"modality":"audio","name":"Big Bench Audio","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":87.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.87,"raw_min":0.87,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:big-bench-audio:nova-2-sonic","reported_date":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-audio?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Sonic"}],"slug":"llm-stats-big-bench-audio","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-audio?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"bm_bigfinancebench_c8640763"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","finance","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Big Finance Bench evaluates models on complex financial-analysis tasks that require retrieving and reasoning over financial documents and performing multi-step quantitative work.","descriptions":{"en":"Big Finance Bench evaluates models on complex financial-analysis tasks that require retrieving and reasoning over financial documents and performing multi-step quantitative work."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:big-finance-bench:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-finance-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:big-finance-bench","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","deep-research"],"join":"llm_stats_id","primary":"finance-legal","source_record":"bm_bigfinancebench_c8640763"},"modality":"text","name":"Big Finance Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":53.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.53,"raw_min":0.36,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:big-finance-bench:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-finance-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-big-finance-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-finance-bench?top_n=500","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3244,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":"gh:google/big-bench","source_record":"catalog_c240e59be7f89744"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark consisting of 204+ tasks designed to probe large language models and extrapolate their future capabilities. It covers diverse domains including linguistics, mathematics, common-sense reasoning, biology, physics, social bias, software development, and more. The benchmark focuses on tasks believed to be beyond current language model capabilities and includes both English and non-English tasks across multiple languages.","descriptions":{"en":"Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark consisting of 204+ tasks designed to probe large language models and extrapolate their future capabilities. It covers diverse domains including linguistics, mathematics, common-sense reasoning, biology, physics, social bias, software development, and more. The benchmark focuses on tasks believed to be beyond current language model capabilities and includes both English and non-English tasks across multiple languages."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:big-bench:gemini-1.0-pro","reported_at":"2024-02-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:big-bench","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_c240e59be7f89744"},"modality":"text","name":"BIG-Bench","openness":"unknown","publisher":null,"released":"2022-06-09","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:big-bench","source_url":"https://arxiv.org/abs/2206.04615"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":75.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.75,"raw_min":0.682,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:big-bench:gemini-1.0-pro","reported_date":"2024-02-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-02-15","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"}],"slug":"llm-stats-big-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":127,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"llm_stats_id","repository":null,"source_record":"catalog_c4b9e72d07024860"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BIG-Bench Extra Hard (BBEH) is a challenging benchmark that replaces each task in BIG-Bench Hard with a novel task that probes similar reasoning capabilities but exhibits significantly increased difficulty. The benchmark contains 23 tasks testing diverse reasoning skills including many-hop reasoning, causal understanding, spatial reasoning, temporal arithmetic, geometric reasoning, linguistic reasoning, logic puzzles, and humor understanding. Designed to address saturation on existing benchmarks where state-of-the-art models achieve near-perfect scores, BBEH shows substantial room for improvement with best models achieving only 9.8-44.8% average accuracy.","descriptions":{"en":"BIG-Bench Extra Hard (BBEH) is a challenging benchmark that replaces each task in BIG-Bench Hard with a novel task that probes similar reasoning capabilities but exhibits significantly increased difficulty. The benchmark contains 23 tasks testing diverse reasoning skills including many-hop reasoning, causal understanding, spatial reasoning, temporal arithmetic, geometric reasoning, linguistic reasoning, logic puzzles, and humor understanding. Designed to address saturation on existing benchmarks where state-of-the-art models achieve near-perfect scores, BBEH shows substantial room for improvement with best models achieving only 9.8-44.8% average accuracy."},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:big-bench-extra-hard:gemma-3-12b-it","reported_at":"2025-03-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-extra-hard?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:big-bench-extra-hard","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_c4b9e72d07024860"},"modality":"text","name":"BIG-Bench Extra Hard","openness":"unknown","publisher":null,"released":"2025-02-26","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: BIG-Bench Extra Hard. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:big-bench-extra-hard","source_url":"https://arxiv.org/abs/2502.19187"},"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":74.4,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.744,"raw_min":0.072,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:big-bench-extra-hard:gemma-4-31b-it","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-extra-hard?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"}],"slug":"llm-stats-big-bench-extra-hard","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-extra-hard?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":["BigBench Extra Hard","BBEH","BIG-Bench Extra Hard"],"attention_signals":{"as_of":"2026-10-11","github_stars":127,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"artifact:arxiv:2502.19187","repository":null,"source_record":"catalog_c4b9e72d07024860"},"categories":["reasoning"],"collected_at":null,"description":"Successor to BBH after saturation; per-task variance is high.","descriptions":{"en":"Successor to BBH after saturation; per-task variance is high."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000bigbench_extra_hard\u0000google_gemma_4_model_card\u0000bigbench_extra_hard\u0000as reported\u0000Gemma 4 (31B)","reported_at":"2026-03-11","source_url":"https://huggingface.co/google/gemma-4-31B-it"},"first_score_reported_at":"2026-03-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000bigbench_extra_hard\u0000google_gemma_4_model_card\u0000bigbench_extra_hard\u0000as reported\u0000Gemma 4 (31B)","reported_at":"2026-03-11","source_url":"https://huggingface.co/google/gemma-4-31B-it"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:bigbench_extra_hard","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"artifact:arxiv:2502.19187","primary":"abstract-reasoning","source_record":"catalog_c4b9e72d07024860"},"modality":null,"name":"BIG-Bench Extra Hard","openness":"unknown","publisher":null,"released":"2025-02-26","released_reference":{"basis":"benchmark_release","source_key":"model-reports:bigbench_extra_hard","source_url":"https://arxiv.org/abs/2502.19187"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":74.4,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":74.4,"raw_min":74.4,"source_reference":{"obs_id":"curated\u0000bigbench_extra_hard\u0000google_gemma_4_model_card\u0000bigbench_extra_hard\u0000as reported\u0000Gemma 4 (31B)","observation_id":"curated\u0000bigbench_extra_hard\u0000google_gemma_4_model_card\u0000bigbench_extra_hard\u0000as reported\u0000Gemma 4 (31B)","reported_at":"2026-03-11","reported_date":"2026-03-11","source_id":"google_gemma_4_model_card","source_url":"https://huggingface.co/google/gemma-4-31B-it"},"unit":"percent"},"scored_models":[{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"}],"slug":"bigbench_extra_hard","source":"model_reports","source_url":"https://arxiv.org/abs/2502.19187","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":578,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_ad505a3aa6e37229"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BIG-Bench Hard (BBH) is a subset of 23 challenging BIG-Bench tasks selected because prior language model evaluations did not outperform average human-rater performance. The benchmark contains 6,511 evaluation examples testing various forms of multi-step reasoning including arithmetic, logical reasoning (Boolean expressions, logical deduction), geometric reasoning, temporal reasoning, and language understanding. Tasks require capabilities such as causal judgment, object counting, navigation, pattern recognition, and complex problem solving.","descriptions":{"en":"BIG-Bench Hard (BBH) is a subset of 23 challenging BIG-Bench tasks selected because prior language model evaluations did not outperform average human-rater performance. The benchmark contains 6,511 evaluation examples testing various forms of multi-step reasoning including arithmetic, logical reasoning (Boolean expressions, logical deduction), geometric reasoning, temporal reasoning, and language understanding. Tasks require capabilities such as causal judgment, object counting, navigation, pattern recognition, and complex problem solving."},"evidence_summary":{"document_count":1,"model_count":21,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:big-bench-hard:claude-3-opus-20240229","reported_at":"2024-02-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-hard?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:big-bench-hard","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_ad505a3aa6e37229"},"modality":"text","name":"BIG-Bench Hard","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":21,"score_direction":"higher_is_better","score_summary":{"display_max":93.10000000000001,"display_multiplier":100,"model_count":21,"model_count_basis":"source_model_id","numeric_count":21,"raw_max":0.931,"raw_min":0.391,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:big-bench-hard:claude-3-5-sonnet-20240620","reported_date":"2024-06-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-hard?top_n=500"},"unit":null},"scored_models":[{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-big-bench-hard","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-hard?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":516,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":50,"join":"llm_stats_id","repository":null,"source_record":"catalog_de0a9cde8ad3201c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained programming tasks. Evaluates code generation with diverse function calls and complex instructions, featuring two variants: Complete (code completion based on comprehensive docstrings) and Instruct (generating code from natural language instructions).","descriptions":{"en":"A benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained programming tasks. Evaluates code generation with diverse function calls and complex instructions, featuring two variants: Complete (code completion based on comprehensive docstrings) and Instruct (generating code from natural language instructions)."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bigcodebench:qwen-2.5-coder-7b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bigcodebench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_de0a9cde8ad3201c"},"modality":"text","name":"BigCodeBench","openness":"unknown","publisher":null,"released":"2024-06-22","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:bigcodebench","source_url":"https://arxiv.org/abs/2406.15877"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":45.4,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.454,"raw_min":0.41,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bigcodebench:gemini-diffusion","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-bigcodebench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","代码","Code","大语言模型","LLM","逻辑推理","Reasoning","代码工程","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BigCodeBench is a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. BigCodeBench用于评估LLM的代码生成能力，包含1140个可以调用139个库和7个域的多个函数来完成的细粒度任务。","descriptions":{"en":"BigCodeBench is a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks.","zh":"BigCodeBench用于评估LLM的代码生成能力，包含1140个可以调用139个库和7个域的多个函数来完成的细粒度任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1253","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"artifact:arxiv:2406.15877","primary":"code-generation","source_record":"catalog_3d7c9b2f8573753d"},"modality":null,"name":"BigCodeBench","openness":"open","publisher":"Monash University","released":"2024-06-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1253-bigcodebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BigCodeBench","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":516,"github_stars_30d":null,"hf_dataset_downloads":165589,"hf_paper_upvotes":50,"join":"llm_stats_id","repository":"gh:bigcode-project/bigcodebench","source_record":"catalog_3d7c9b2f8573753d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive benchmark that evaluates large language models' ability to solve complex, practical programming tasks via code generation. Contains 1,140 fine-grained tasks across 7 domains using function calls from 139 libraries. Challenges LLMs to invoke multiple function calls as tools and handle complex instructions for realistic software engineering and general-purpose reasoning tasks.","descriptions":{"en":"A comprehensive benchmark that evaluates large language models' ability to solve complex, practical programming tasks via code generation. Contains 1,140 fine-grained tasks across 7 domains using function calls from 139 libraries. Challenges LLMs to invoke multiple function calls as tools and handle complex instructions for realistic software engineering and general-purpose reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bigcodebench-full:qwen-2.5-coder-32b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-full?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bigcodebench-full","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_3d7c9b2f8573753d"},"modality":"text","name":"BigCodeBench-Full","openness":"unknown","publisher":null,"released":"2024-06-22","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:bigcodebench-full","source_url":"https://arxiv.org/abs/2406.15877"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":49.6,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.496,"raw_min":0.496,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bigcodebench-full:qwen-2.5-coder-32b-instruct","reported_date":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-full?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"}],"slug":"llm-stats-bigcodebench-full","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-full?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":516,"github_stars_30d":null,"hf_dataset_downloads":158882,"hf_paper_upvotes":50,"join":"llm_stats_id","repository":"gh:bigcode-project/bigcodebench","source_record":"catalog_d95852dd185a1bb1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BigCodeBench-Hard is a subset of 148 challenging programming tasks from BigCodeBench, designed to evaluate large language models' ability to solve complex, real-world programming problems. These tasks require diverse function calls from multiple libraries across 7 domains including computation, networking, data analysis, and visualization. The benchmark tests compositional reasoning and the ability to implement complex instructions that span 139 libraries with an average of 2.8 libraries per task.","descriptions":{"en":"BigCodeBench-Hard is a subset of 148 challenging programming tasks from BigCodeBench, designed to evaluate large language models' ability to solve complex, real-world programming problems. These tasks require diverse function calls from multiple libraries across 7 domains including computation, networking, data analysis, and visualization. The benchmark tests compositional reasoning and the ability to implement complex instructions that span 139 libraries with an average of 2.8 libraries per task."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bigcodebench-hard:qwen-2.5-coder-32b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-hard?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bigcodebench-hard","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_d95852dd185a1bb1"},"modality":"text","name":"BigCodeBench-Hard","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":27.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.27,"raw_min":0.27,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bigcodebench-hard:qwen-2.5-coder-32b-instruct","reported_date":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-hard?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"}],"slug":"llm-stats-bigcodebench-hard","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-hard?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_bigfinancebench_c8640763"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"BigFinanceBench evaluates financial-research agents on open-ended tasks with 928 items, each paired with a ground-truth answer and a point-weighted rubric decomposing the derivation into steps. Supports partial-credit scoring across 36,241 rubric points.","descriptions":{"en":"BigFinanceBench evaluates financial-research agents on open-ended tasks with 928 items, each paired with a ground-truth answer and a point-weighted rubric decomposing the derivation into steps. Supports partial-credit scoring across 36,241 rubric points."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03829","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","deep-research"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_bigfinancebench_c8640763"},"modality":null,"name":"BigFinanceBench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03829","source_url":"https://arxiv.org/abs/2606.03829"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03829","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03829","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BigO(Bench)是一个包含约 300 个需要用 Python 解决的代码问题的基准测试，以及 3,105 个编码问题和 1,190,250 个解决方案用于训练，以评估LLMs能否找到代码解决方案的时间-空间复杂度，或者生成符合时间-空间复杂度要求的代码解决方案。 BigO(Bench)是一个包含约 300 个需要用 Python 解决的代码问题的基准测试，以及 3,105 个编码问题和 1,190,250 个解决方案用于训练，以评估LLMs能否找到代码解决方案的时间-空间复杂度，或者生成符合时间-空间复杂度要求的代码解决方案。","descriptions":{"en":"BigO(Bench)是一个包含约 300 个需要用 Python 解决的代码问题的基准测试，以及 3,105 个编码问题和 1,190,250 个解决方案用于训练，以评估LLMs能否找到代码解决方案的时间-空间复杂度，或者生成符合时间-空间复杂度要求的代码解决方案。","zh":"BigO(Bench)是一个包含约 300 个需要用 Python 解决的代码问题的基准测试，以及 3,105 个编码问题和 1,190,250 个解决方案用于训练，以评估LLMs能否找到代码解决方案的时间-空间复杂度，或者生成符合时间-空间复杂度要求的代码解决方案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1680","languages":[],"library_direction":null,"modality":null,"name":"BigOBench","openness":"restricted","publisher":"facebook","released":"2025-03-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1680-bigobench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BigOBench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates physical reasoning in synthetic billiards environments, testing collision prediction, wall bounce reasoning, and final position estimation for multimodal LLMs.","descriptions":{"en":"Evaluates physical reasoning in synthetic billiards environments, testing collision prediction, wall bounce reasoning, and final position estimation for multimodal LLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30900","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","visual-reasoning"],"join":"claire_record_id","primary":"world-models","source_record":"bm_billiardphys-bench_142a32a2"},"modality":null,"name":"BilliardPhys-Bench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30900","source_url":"https://arxiv.org/abs/2605.30900"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30900","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30900","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"BIM-Edit evaluates large language models on natural-language editing of Industry Foundation Classes (IFC) building models. The benchmark includes 324 editing tasks across 11 realistic building models and 36 synthetic scenes. Tasks are categorized as direct, spatial, or topological instructions, and outputs are scored on geometric accuracy, semantic validity, and topological consistency.","descriptions":{"en":"BIM-Edit evaluates large language models on natural-language editing of Industry Foundation Classes (IFC) building models. The benchmark includes 324 editing tasks across 11 realistic building models and 36 synthetic scenes. Tasks are categorized as direct, spatial, or topological instructions, and outputs are scored on geometric accuracy, semantic validity, and topological consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.20146","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","code-generation"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_bim-edit_44064775"},"modality":null,"name":"BIM-Edit","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20146","source_url":"https://arxiv.org/abs/2606.20146"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20146","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20146","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"BinJudgeBench evaluates LLM-as-a-Judge for human-oriented binary reverse engineering, covering function name recovery, code summarization, and decompilation optimization, with correlation to human judgment as the metric.","descriptions":{"en":"BinJudgeBench evaluates LLM-as-a-Judge for human-oriented binary reverse engineering, covering function name recovery, code summarization, and decompilation optimization, with correlation to human judgment as the metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07038","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","code-generation"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_binjudgebench_a8dd4ae5"},"modality":null,"name":"BinJudgeBench","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07038","source_url":"https://arxiv.org/abs/2608.07038"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07038","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07038","topics":["coding","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":7,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:baceolus/biolp-bench","source_record":"catalog_7a6be30323cef9fb"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","healthcare","biology"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BioLP-Bench is a model-graded evaluation measuring ability to find and correct mistakes in common biological laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development.","descriptions":{"en":"BioLP-Bench is a model-graded evaluation measuring ability to find and correct mistakes in common biological laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:biolp-bench:grok-4.1-thinking-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/biolp-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:biolp-bench","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"llm_stats_id","primary":"safety-alignment","source_record":"catalog_7a6be30323cef9fb"},"modality":"text","name":"BioLP-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":37.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.37,"raw_min":0.37,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:biolp-bench:grok-4.1-thinking-2025-11-17","reported_date":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/biolp-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-biolp-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/biolp-bench?top_n=500","topics":["safety","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","biology"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BioMysteryBench evaluates a model's ability to reason through challenging molecular biology problems, reporting performance on a hard subset and on the subset of problems solved by human experts.","descriptions":{"en":"BioMysteryBench evaluates a model's ability to reason through challenging molecular biology problems, reporting performance on a hard subset and on the subset of problems solved by human experts."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:biomysterybench:claude-fable-5","reported_at":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/biomysterybench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:biomysterybench","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_9c973121d43bf296"},"modality":"text","name":"BioMysteryBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":90.10000000000001,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.901,"raw_min":0.435,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:biomysterybench:claude-opus-5","reported_date":"2026-07-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/biomysterybench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"}],"slug":"llm-stats-biomysterybench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/biomysterybench?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":["BioMysteryBench"],"attention_signals":null,"categories":["biology"],"collected_at":null,"description":"Reported in two splits (\"hard\" and \"human solved\") that differ by more than 35 points, so a bare score is unreadable without its split. Anthropic notes its own safety refusals depress this number, which means the score mixes capability with policy. First-party to Anthropic, which built it and publishes the dataset. The task set was revised after an answer-key audit, so the item count moved and scores are not comparable across dataset versions.","descriptions":{"en":"Reported in two splits (\"hard\" and \"human solved\") that differ by more than 35 points, so a bare score is unreadable without its split. Anthropic notes its own safety refusals depress this number, which means the score mixes capability with policy. First-party to Anthropic, which built it and publishes the dataset. The task set was revised after an answer-key audit, so the item count moved and scores are not comparable across dataset versions."},"evidence_summary":{"document_count":2,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000biomysterybench\u0000tencent_hy4_preview\u0000biomysterybench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000biomysterybench\u0000tencent_hy4_preview\u0000biomysterybench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:biomysterybench","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"artifact:hf:anthropic/biomysterybench-full","primary":"science","source_record":"catalog_109e010c1ce17e77"},"modality":null,"name":"BioMysteryBench","openness":"unknown","publisher":null,"released":"2026-04-29","released_reference":{"basis":"benchmark_release","source_key":"model-reports:biomysterybench","source_url":"https://huggingface.co/datasets/Anthropic/BioMysteryBench-full"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":71.3,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":71.3,"raw_min":71.3,"source_reference":{"obs_id":"curated\u0000biomysterybench\u0000tencent_hy4_preview\u0000biomysterybench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000biomysterybench\u0000tencent_hy4_preview\u0000biomysterybench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"biomysterybench","source":"model_reports","source_url":"https://huggingface.co/datasets/Anthropic/BioMysteryBench-full","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Dataset for evidence-grounded scientific reasoning over biophysical literature.","descriptions":{"en":"Dataset for evidence-grounded scientific reasoning over biophysical literature."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.19180","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_biophys-bridge_4adbdd47"},"modality":null,"name":"BioPhys-Bridge","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19180","source_url":"https://arxiv.org/abs/2609.19180"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19180","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19180","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"BioSecBench-Refusal evaluates AI agents on 61 Routine and 46 Red-Team biosecurity-related tasks, measuring refusal rates and risk identification across multiple model configurations.","descriptions":{"en":"BioSecBench-Refusal evaluates AI agents on 61 Routine and 46 Red-Team biosecurity-related tasks, measuring refusal rates and risk identification across multiple model configurations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.05462","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_biosecbench-refusal_d798db06"},"modality":null,"name":"BioSecBench-Refusal","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.05462","source_url":"https://arxiv.org/abs/2607.05462"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05462","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05462","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"BioSecBench-Surveillance evaluates AI agents on pathogen genomic surveillance across 100 tasks spanning seven categories, including taxonomic classification and genetic-engineering detection. Agents receive raw sequencing data and surveillance context, and their structured answers are graded deterministically against ground truth.","descriptions":{"en":"BioSecBench-Surveillance evaluates AI agents on pathogen genomic surveillance across 100 tasks spanning seven categories, including taxonomic classification and genetic-engineering detection. Agents receive raw sequencing data and surveillance context, and their structured answers are graded deterministically against ground truth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19262","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_biosecbench-surveillance_edf01c2c"},"modality":null,"name":"BioSecBench-Surveillance","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19262","source_url":"https://arxiv.org/abs/2607.19262"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19262","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19262","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Factuality","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"We evaluate whether AI agents can match the reported findings of published biomedical studies using public data.","descriptions":{"en":"We evaluate whether AI agents can match the reported findings of published biomedical studies using public data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.07614","languages":[],"library_direction":{"basis":"model-review-daily","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_biostudybench_387cfcfa"},"modality":null,"name":"BioStudyBench","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.07614","source_url":"https://arxiv.org/abs/2610.07614"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-07614","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07614","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zjuidg/bird-history","source_record":"bm_bird-history_67c9a405"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates text-to-SQL systems on 1,393 tasks that require leveraging historical SQL logs, with fine-grained annotations for retrieval and knowledge utilization.","descriptions":{"en":"Evaluates text-to-SQL systems on 1,393 tasks that require leveraging historical SQL logs, with fine-grained annotations for retrieval and knowledge utilization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29345","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_bird-history_67c9a405"},"modality":null,"name":"BIRD-History","openness":"unknown","publisher":null,"released":"2026-08-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29345","source_url":"https://arxiv.org/abs/2608.29345"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29345","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29345","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":189,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"llm_stats_id","repository":null,"source_record":"catalog_e8085090a8ed817a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQLs) is a comprehensive text-to-SQL benchmark containing 12,751 question-SQL pairs across 95 databases (33.4 GB total) spanning 37+ professional domains. It evaluates large language models' ability to convert natural language to executable SQL queries in real-world scenarios with complex database schemas and dirty data.","descriptions":{"en":"BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQLs) is a comprehensive text-to-SQL benchmark containing 12,751 question-SQL pairs across 95 databases (33.4 GB total) spanning 37+ professional domains. It evaluates large language models' ability to convert natural language to executable SQL queries in real-world scenarios with complex database schemas and dirty data."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bird-sql-(dev):gemini-2.0-flash","reported_at":"2024-12-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bird-sql-%28dev%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bird-sql-(dev)","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_e8085090a8ed817a"},"modality":"text","name":"Bird-SQL (dev)","openness":"unknown","publisher":null,"released":"2023-05-04","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:bird-sql-(dev)","source_url":"https://arxiv.org/abs/2305.03111"},"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":57.4,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.574,"raw_min":0.064,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bird-sql-(dev):gemini-2.0-flash-lite","reported_date":"2025-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bird-sql-%28dev%29?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"}],"slug":"llm-stats-bird-sql-dev","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bird-sql-%28dev%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_7175aad2c4803478"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BixBench is a benchmark for real-world bioinformatics and computational biology data analysis. It evaluates AI models on multi-step scientific workflows that require code execution, statistical reasoning, and biological domain knowledge to interpret experimental data.","descriptions":{"en":"BixBench is a benchmark for real-world bioinformatics and computational biology data analysis. It evaluates AI models on multi-step scientific workflows that require code execution, statistical reasoning, and biological domain knowledge to interpret experimental data."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:bixbench:gpt-5.5","reported_at":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bixbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:bixbench","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_7175aad2c4803478"},"modality":"text","name":"BixBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":80.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.805,"raw_min":0.805,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:bixbench:gpt-5.5","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bixbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"}],"slug":"llm-stats-bixbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bixbench?top_n=500","topics":["agents","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents on 20 research-study-scale computational biology tasks that require processing raw data into scientific results, scored by matching generated artifacts against study-derived targets.","descriptions":{"en":"Evaluates AI agents on 20 research-study-scale computational biology tasks that require processing raw data into scientific results, scored by matching generated artifacts against study-derived targets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25286","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_bixbench3_ede996e4"},"modality":null,"name":"BixBench3","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25286","source_url":"https://arxiv.org/abs/2608.25286"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25286","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25286","topics":["science"],"unit":null},{"aliases":["Blender World"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Agents create editable Blender scenes from text, image or video briefs across 223 tasks. Overall is a 0–100 score combining requirement satisfaction and visual quality equally, not the full-rubric pass rate. The publisher runs each task once with an automatic judge and different generation harnesses across models; compare the complete setup, not an isolated model.","descriptions":{"en":"Agents create editable Blender scenes from text, image or video briefs across 223 tasks. Overall is a 0–100 score combining requirement satisfaction and visual quality equally, not the full-rubric pass rate. The publisher runs each task once with an automatic judge and different generation harnesses across models; compare the complete setup, not an isolated model."},"editorial_guide":{"id":"blenderworld","record_keys":["model-reports:blenderworld"],"reviewed_at":"2026-10-25","scope":{"en":"The 223-task benchmark and publisher runs in the October 14, 2026 paper v1. Main-table values were checked against the official leaderboard on October 25, 2026.","zh":"对应 2026 年 10 月 14 日论文 v1 的 223 道任务及官方实验。主表数值已于 2026 年 10 月 25 日与官网榜单核对。"},"sections":[{"id":"tasks","source_ids":["paper","leaderboard"],"text":{"en":"Agents deliver editable Blender projects and requested media across nine creative domains. The 223 tasks comprise 78 text, 130 image and 15 video tasks; 59 also include specification attachments. The task suite has 2,651 binary rubric criteria.","zh":"Agent 需要在九个创作领域交付可编辑的 Blender 工程及指定媒体产物。223 道任务包括 78 道文本、130 道图片和 15 道视频任务；其中 59 道另附规格文件。全套任务共有 2,651 条二元评分条件。"},"title":{"en":"What the agent creates","zh":"Agent 要交付什么"}},{"id":"metrics","source_ids":["paper","runtime"],"text":{"en":"Overall equally combines weighted requirement satisfaction and visual quality, then averages all 223 tasks on a 0–100 scale. Evaluation uses scene inspection, eight adaptive rendered views and eight quality dimensions. Failed evaluations score zero. Full pass counts tasks satisfying every rubric criterion; it is a separate percentage, not the Overall score.","zh":"Overall 将加权需求满足程度与视觉质量各按一半权重合成，再对全部 223 道任务取平均，范围为 0–100 分。评估结合场景检查、八个自适应渲染视角和八项质量维度。评估失败记零分。Full pass 是满足全部评分条件的任务比例，与 Overall 不是同一指标。"},"title":{"en":"How the scores work","zh":"分数怎样计算"}},{"id":"completion","source_ids":["paper","leaderboard"],"table":{"caption":{"en":"Paper v1, Table 2 · Completion on all 223 tasks","zh":"论文 v1 表 2 · 全部 223 道任务的完成质量"},"columns":[{"en":"Model","zh":"模型"},{"en":"Overall","zh":"总分"},{"en":"Text","zh":"文本"},{"en":"MM","zh":"多模态"},{"en":"Rubric","zh":"需求评分"},{"en":"Dimension","zh":"质量评分"},{"en":"Full pass %","zh":"全部条件达标率 %"}],"note":{"en":"Overall, Text, MM, Rubric and Dimension are scores on a 0–100 scale. Full pass is the percentage of tasks passing all rubric criteria. Values and displayed precision follow the paper; failed evaluations remain in the denominator.","zh":"Overall、Text、MM、Rubric 和 Dimension 均为 0–100 分。Full pass 表示全部评分条件达标的任务比例。数值与精度按论文原表保留；失败评估仍计入分母。"},"rows":[["GPT-6-Astra","79.9","78.6","80.7","87.3","72.6","42.2"],["Opus-5","73.0","74.8","72.1","80.9","65.1","25.6"],["GPT-5.6-Sol","67.7","71.7","65.5","74.7","60.7","15.7"],["DeepSeek-V4.1-Flash","65.7","66.0","65.5","72.0","59.4","11.7"],["Qwen3.8-Max","60.6","59.6","61.1","65.9","55.3","10.3"],["GLM-5.3-Flash","59.7","63.1","57.8","64.8","54.4","6.28"],["Grok-4.6","57.0","60.0","55.4","61.4","52.7","6.73"],["Opus-4.8","57.0","58.6","56.1","60.2","53.7","5.38"],["Seed2.1pro-0915","51.3","54.7","49.5","54.9","47.7","3.59"],["Qwen3.8-27B","47.7","46.1","48.6","50.0","45.5","1.35"],["Kimi-K3","46.5","41.3","49.4","47.9","45.2","1.79"],["Gemini-3.8-Flash","34.2","35.9","33.2","29.8","38.5","0.45"],["MiniMax-M3","31.8","34.1","30.6","28.9","34.7","0.45"]]},"text":{"en":"The publisher reports these 13 agents on the same task suite. Read average quality alongside Full pass: a strong average does not establish that every requested property was delivered.","zh":"官方在同一任务集上报告了以下 13 个 Agent 的结果。阅读平均分时也应查看 Full pass：平均分较高，并不代表每项交付要求都已满足。"},"title":{"en":"Published completion results","zh":"官方完成质量结果"}},{"id":"resources","source_ids":["paper","leaderboard"],"table":{"caption":{"en":"Paper v1, Table 2 · Mean rollout resources per task","zh":"论文 v1 表 2 · 每任务平均运行资源"},"columns":[{"en":"Model","zh":"模型"},{"en":"Turns","zh":"交互轮数"},{"en":"Output tokens K","zh":"输出 token 数（千）"},{"en":"Tool calls","zh":"工具调用数"}],"note":{"en":"K means thousands of output tokens. A turn is one model-response round. Different harnesses and model accounting limit direct efficiency comparisons.","zh":"K 表示千个输出 token；一轮指一次模型响应。不同运行框架与模型统计方式会限制直接的效率比较。"},"rows":[["GPT-6-Astra","45.6","84.5","79.1"],["Opus-5","158","241","155"],["GPT-5.6-Sol","44.9","39.9","44.0"],["DeepSeek-V4.1-Flash","171","209","168"],["Qwen3.8-Max","75.1","90.6","72.8"],["GLM-5.3-Flash","102","145","101"],["Grok-4.6","93.4","120","89.8"],["Opus-4.8","97.7","164","95.6"],["Seed2.1pro-0915","196","122","194"],["Qwen3.8-27B","90.0","112","87.4"],["Kimi-K3","60.5","54.3","58.3"],["Gemini-3.8-Flash","56.6","35.9","54.5"],["MiniMax-M3","132","72.2","127"]]},"text":{"en":"These are per-task means across the full suite, including unsuccessful calls. They describe generated output and requested operations; they are not matched compute budgets or monetary costs.","zh":"以下是按全部任务计算的每任务均值，包含未成功的调用。它们描述生成输出和请求的操作次数，并不代表相同计算预算或实际费用。"},"title":{"en":"Rollout resources","zh":"运行资源统计"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Each model runs once per task at its highest available reasoning setting. GPT-6-Astra and GPT-5.6-Sol use Codex; the other agents use Claude Code. GPT-5.6-terra performs automatic evaluation. Rankings therefore compare complete agent setups, with no repeated-run variance estimate; they do not isolate base-model ability or certify production readiness.","zh":"每个模型按其最高可用推理设置运行，每道任务只尝试一次。GPT-6-Astra 与 GPT-5.6-Sol 使用 Codex，其余 Agent 使用 Claude Code；自动评估由 GPT-5.6-terra 完成。因此榜单比较的是完整 Agent 配置，未给出重复运行的方差估计；不能据此单独判定基础模型能力或认证生产可用性。"},"title":{"en":"What comparisons can establish","zh":"比较结果的边界"}},{"id":"reproduce","source_ids":["paper","runtime"],"text":{"en":"Pin the task revision, Blender 4.2.10, harness, reasoning configuration and judge. The reference runtime uses Codex CLI 0.147.0, Claude Code 2.1.220 and Node.js 22.17.0; the paper notes that a 48-task MiniMax-M3 batch uses Claude Code 2.1.241. The public evaluator accepts submission ZIPs from other harnesses, so changed setups need separate protocol labels.","zh":"应固定任务版本、Blender 4.2.10、运行框架、推理配置和裁判。参考环境采用 Codex CLI 0.147.0、Claude Code 2.1.220 与 Node.js 22.17.0；论文注明 MiniMax-M3 的一个 48 道任务批次使用 Claude Code 2.1.241。公开评估器也接受其他框架生成的 ZIP 提交，因此配置改变时应单独标记评测协议。"},"title":{"en":"Before reproducing a result","zh":"复现前应记录什么"}}],"sources":[{"id":"paper","title":"BlenderWorld paper v1 · Methods, Table 2 and Appendix D","url":"https://arxiv.org/html/2610.23986v1"},{"id":"leaderboard","title":"BlenderWorld · Official benchmark and leaderboard","url":"https://blenderworld.github.io/"},{"id":"runtime","title":"BlenderWorld · Pinned evaluation protocol and runtime","url":"https://github.com/QwenLM/BlenderWorld/blob/dd4f3571faed4ed45cc720ac94835534b9f67836/docs/evaluation.md"}],"summary":{"en":"Can an agent turn a brief and visual references into a reusable Blender project? BlenderWorld checks the delivered scene as well as its rendered appearance.","zh":"Agent 能否把需求与视觉参考变成可继续编辑的 Blender 工程？BlenderWorld 同时检查交付场景和渲染效果。"},"title":"BlenderWorld"},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000blenderworld\u0000blenderworld_v1_results_table\u0000BlenderWorld paper v1 (223 tasks)\u0000Claude Code 2.1.220 (48-task batch: 2.1.241); Blender 4.2.10; highest available reasoning; one run per task; GPT-5.6-terra judge; failed evaluations score zero and remain in the denominator\u0000MiniMax-M3","reported_at":"2026-10-14","source_url":"https://arxiv.org/html/2610.23986v1#S5.T2"},"first_score_reported_at":"2026-10-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000blenderworld\u0000blenderworld_v1_results_table\u0000BlenderWorld paper v1 (223 tasks)\u0000Claude Code 2.1.220 (48-task batch: 2.1.241); Blender 4.2.10; highest available reasoning; one run per task; GPT-5.6-terra judge; failed evaluations score zero and remain in the denominator\u0000MiniMax-M3","reported_at":"2026-10-14","source_url":"https://arxiv.org/html/2610.23986v1#S5.T2"},"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"model-reports:blenderworld","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["spatial-3d"],"join":"review:Equality Machine:2026-10-25","primary":"spatial-3d","source_record":null},"modality":null,"name":"BlenderWorld","openness":"unknown","publisher":null,"released":"2026-10-14","released_reference":{"basis":"benchmark_release","source_key":"model-reports:blenderworld","source_url":"https://blenderworld.github.io/"},"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":79.9,"display_multiplier":1,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":79.9,"raw_min":31.8,"source_reference":{"obs_id":"curated\u0000blenderworld\u0000blenderworld_v1_results_table\u0000BlenderWorld paper v1 (223 tasks)\u0000Codex CLI 0.147.0; Blender 4.2.10; highest available reasoning; one run per task; GPT-5.6-terra judge; failed evaluations score zero and remain in the denominator\u0000GPT-6-Astra","observation_id":"curated\u0000blenderworld\u0000blenderworld_v1_results_table\u0000BlenderWorld paper v1 (223 tasks)\u0000Codex CLI 0.147.0; Blender 4.2.10; highest available reasoning; one run per task; GPT-5.6-terra judge; failed evaluations score zero and remain in the denominator\u0000GPT-6-Astra","reported_at":"2026-10-14","reported_date":"2026-10-14","source_id":"blenderworld_v1_results_table","source_url":"https://arxiv.org/html/2610.23986v1#S5.T2"},"unit":"points"},"scored_models":[{"date":"2026-10-14","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-10-14","date_precision":"document_publication","name":"Gemini-3.8-Flash"},{"date":"2026-10-14","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-10-14","date_precision":"document_publication","name":"GPT-5.6-Sol"},{"date":"2026-10-14","date_precision":"document_publication","name":"GPT-6-Astra"},{"date":"2026-10-14","date_precision":"document_publication","name":"Grok-4.6"},{"date":"2026-10-14","date_precision":"document_publication","name":"Kimi-K3"},{"date":"2026-10-14","date_precision":"document_publication","name":"MiniMax-M3"},{"date":"2026-10-14","date_precision":"document_publication","name":"Opus-4.8"},{"date":"2026-10-14","date_precision":"document_publication","name":"Opus-5"},{"date":"2026-10-14","date_precision":"document_publication","name":"Qwen3.8-27B"},{"date":"2026-10-14","date_precision":"document_publication","name":"Qwen3.8-Max"},{"date":"2026-10-14","date_precision":"document_publication","name":"Seed2.1pro-0915"}],"slug":"blenderworld","source":"model_reports","source_url":"https://blenderworld.github.io/","topics":["agents","multimodal"],"unit":"points"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"claire_record_id","repository":"gh:matteosantelmo/reasoning-blind-spots","source_record":"bm_blind-spots-bench_1b74311c"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates reasoning blind spots in language, vision-language, and image-generation models across 235 samples with structured reference solutions and taxonomy.","descriptions":{"en":"Evaluates reasoning blind spots in language, vision-language, and image-generation models across 235 samples with structured reference solutions and taxonomy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.08317","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","visual-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_blind-spots-bench_1b74311c"},"modality":null,"name":"Blind-Spots-Bench","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08317","source_url":"https://arxiv.org/abs/2607.08317"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08317","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08317","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"This entry describes a benchmark for trajectory-level safety calibration of tool-using agents, but the provided abstract and links do not include released task data or scoring code.","descriptions":{"en":"This entry describes a benchmark for trajectory-level safety calibration of tool-using agents, but the provided abstract and links do not include released task data or scoring code."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16305","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["factuality","safety-alignment"],"join":"claire_record_id","primary":"factuality","source_record":"bm_blindspot_233efd04"},"modality":null,"name":"BLINDSPOT","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16305","source_url":"https://arxiv.org/abs/2609.16305"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16305","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16305","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":27,"join":"llm_stats_id","repository":null,"source_record":"catalog_e14a51fded1fb9b4"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","spatial_reasoning","3d","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BLINK: Multimodal Large Language Models Can See but Not Perceive. A benchmark for multimodal language models focusing on core visual perception abilities. Reformats 14 classic computer vision tasks into 3,807 multiple-choice questions paired with single or multiple images and visual prompting. Tasks include relative depth estimation, visual correspondence, forensics detection, multi-view reasoning, counting, object localization, and spatial reasoning that humans can solve 'within a blink'.","descriptions":{"en":"BLINK: Multimodal Large Language Models Can See but Not Perceive. A benchmark for multimodal language models focusing on core visual perception abilities. Reformats 14 classic computer vision tasks into 3,807 multiple-choice questions paired with single or multiple images and visual prompting. Tasks include relative depth estimation, visual correspondence, forensics detection, multi-view reasoning, counting, object localization, and spatial reasoning that humans can solve 'within a blink'."},"evidence_summary":{"document_count":1,"model_count":15,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:blink:phi-4-multimodal-instruct","reported_at":"2025-02-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/blink?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:blink","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_e14a51fded1fb9b4"},"modality":"multimodal","name":"BLINK","openness":"unknown","publication":{"evidence":[{"note":"The author repository identifies itself with the exact BLINK paper title and describes the same 14 visual-perception tasks and 3,807 questions as the source record. It links the introducing paper and distinguishes validation from test data.","url":"https://github.com/zeyofu/BLINK_Benchmark/blob/main/README.md"}],"id":"blink-visual-perception-original","links":[{"kind":"paper","title":"BLINK: Multimodal Large Language Models Can See but Not Perceive","url":"https://arxiv.org/abs/2404.12390"},{"kind":"repo","title":"BLINK official repository","url":"https://github.com/zeyofu/BLINK_Benchmark"}],"record_keys":["llm-stats:blink"],"reviewed_at":"2026-10-26","scope":{"en":"BLINK's original multimodal visual-perception benchmark across 14 task types. The project provides validation and test data; the score source determines the evaluated split.","zh":"BLINK 原始多模态视觉感知评测，涵盖 14 类任务。项目分别提供验证与测试数据，成绩对应的数据划分以来源为准。"}},"publisher":null,"released":"2024-04-18","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: BLINK: Multimodal Large Language Models Can See but Not Perceive. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:blink","source_url":"https://arxiv.org/abs/2404.12390"},"repo_kind":null,"repo_resolution_status":null,"score_count":15,"score_direction":"higher_is_better","score_summary":{"display_max":81.39999999999999,"display_multiplier":100,"model_count":15,"model_count_basis":"source_model_id","numeric_count":15,"raw_max":0.814,"raw_min":0.527,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:blink:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/blink?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-blink","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/blink?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":27,"join":"artifact:arxiv:2404.12390","repository":null,"source_record":"catalog_e14a51fded1fb9b4"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BLINK focuses on MLLMs' core visual perception abilities. It contains 3,807 multiple-choice questions spanning 14 classic computer vision tasks. BLINK用于评估多模态大模型的视觉感知能力，包含来自14个经典计算机视觉任务的3807道多项选择题。","descriptions":{"en":"BLINK focuses on MLLMs' core visual perception abilities. It contains 3,807 multiple-choice questions spanning 14 classic computer vision tasks.","zh":"BLINK用于评估多模态大模型的视觉感知能力，包含来自14个经典计算机视觉任务的3807道多项选择题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1365","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"artifact:arxiv:2404.12390","primary":"visual-reasoning","source_record":"catalog_e14a51fded1fb9b4"},"modality":"multimodal","name":"BLINK","openness":"open","publisher":"University of Pensylvania","released":"2024-04-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1365-blink","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BLINK","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_bluefin_2fd6c32a"},"categories":["Finance","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agents on synthesis, manipulation, and comprehension tasks over financial spreadsheet workbooks, with 131 tasks and granular rubric criteria validated by expert annotators.","descriptions":{"en":"Evaluates LLM agents on synthesis, manipulation, and comprehension tasks over financial spreadsheet workbooks, with 131 tasks and granular rubric criteria validated by expert annotators."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30907","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","finance-legal"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_bluefin_2fd6c32a"},"modality":null,"name":"BlueFin","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30907","source_url":"https://arxiv.org/abs/2605.30907"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30907","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30907","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Blueprint-Bench 2 is an agentic spatial reasoning benchmark that evaluates a model's ability to understand, plan, and reason over architectural blueprints and other structured spatial documents. Scores are reported as a normalized score.","descriptions":{"en":"Blueprint-Bench 2 is an agentic spatial reasoning benchmark that evaluates a model's ability to understand, plan, and reason over architectural blueprints and other structured spatial documents. Scores are reported as a normalized score."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:blueprint-bench-2:gemini-3.5-flash","reported_at":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/blueprint-bench-2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:blueprint-bench-2","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_c828085ad9338ced"},"modality":"multimodal","name":"Blueprint-Bench 2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":38.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.386,"raw_min":0.336,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:blueprint-bench-2:claude-fable-5","reported_date":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/blueprint-bench-2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"}],"slug":"llm-stats-blueprint-bench-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/blueprint-bench-2?top_n=500","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":["Blueprint-Bench 2","Blueprint-Bench","BlueprintBench 2"],"attention_signals":null,"categories":["spatial_reasoning"],"collected_at":null,"description":"Spatial-reasoning set: 50 apartments, ~20 photos each, scored by a connectivity-graph grader, with a public leaderboard on the Andon Labs eval page (run in-house; dataset not openly downloadable). Builds on the original Blueprint-Bench paper (arXiv:2509.25229), a distinct instrument released 2025-09-24 whose scores must not be compared across versions.","descriptions":{"en":"Spatial-reasoning set: 50 apartments, ~20 photos each, scored by a connectivity-graph grader, with a public leaderboard on the Andon Labs eval page (run in-house; dataset not openly downloadable). Builds on the original Blueprint-Bench paper (arXiv:2509.25229), a distinct instrument released 2025-09-24 whose scores must not be compared across versions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:blueprint_bench_2","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["spatial-3d"],"join":"review:14H034160212:2026-10-15","primary":"spatial-3d","source_record":null},"modality":null,"name":"Blueprint-Bench 2","openness":"unknown","publisher":null,"released":"2026-05-04","released_reference":{"basis":"benchmark_release","source_key":"model-reports:blueprint_bench_2","source_url":"https://andonlabs.com/evals/blueprint-bench-2"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"blueprint_bench_2","source":"model_reports","source_url":"https://andonlabs.com/evals/blueprint-bench-2","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":73,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:tropicai-research/bluexv2","source_record":"bm_bluex-v2_e4bf2712"},"categories":["Language & Knowledge","Mathematics & Formal Science","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"BLUEX v2 evaluates large language models on open-ended, discursive questions from the second-phase entrance exams of UNICAMP and USP (2022–2025). The dataset includes 395 questions with 919 subquestions, covering nine subjects, with 55.7% of questions containing images represented as context-aware captions. Scoring uses an LLM-as-a-judge protocol with binary rubric criteria based on official reference answers, yielding a 0–10 score.","descriptions":{"en":"BLUEX v2 evaluates large language models on open-ended, discursive questions from the second-phase entrance exams of UNICAMP and USP (2022–2025). The dataset includes 395 questions with 919 subquestions, covering nine subjects, with 55.7% of questions containing images represented as context-aware captions. Scoring uses an LLM-as-a-judge protocol with binary rubric criteria based on official reference answers, yielding a 0–10 score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22723","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_bluex-v2_e4bf2712"},"modality":null,"name":"BLUEX v2","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22723","source_url":"https://arxiv.org/abs/2606.22723"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22723","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22723","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Boiling the Frog evaluates whether tool-using AI models in corporate settings are susceptible to incremental attacks through multi-turn scenarios with persistent workspaces. It includes a three-level operational risk taxonomy and scores attack success rate (ASR) on resulting artifact state.","descriptions":{"en":"Boiling the Frog evaluates whether tool-using AI models in corporate settings are susceptible to incremental attacks through multi-turn scenarios with persistent workspaces. It includes a three-level operational risk taxonomy and scores attack success rate (ASR) on resulting artifact state."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22643","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","tool-use"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_boiling-the-frog_8414ba51"},"modality":null,"name":"Boiling the Frog","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22643","source_url":"https://arxiv.org/abs/2605.22643"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22643","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22643","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":177,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_df39c63037634966"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BoolQ is a reading comprehension dataset for yes/no questions containing 15,942 naturally occurring examples. Each example consists of a question, passage, and boolean answer, where questions are generated in unprompted and unconstrained settings. The dataset challenges models with complex, non-factoid information requiring entailment-like inference to solve.","descriptions":{"en":"BoolQ is a reading comprehension dataset for yes/no questions containing 15,942 naturally occurring examples. Each example consists of a question, passage, and boolean answer, where questions are generated in unprompted and unconstrained settings. The dataset challenges models with complex, non-factoid information requiring entailment-like inference to solve."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:boolq:gemma-2-27b-it","reported_at":"2024-06-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/boolq?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:boolq","languages":[],"library_direction":{"basis":"model-review-2","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_df39c63037634966"},"modality":"text","name":"BoolQ","openness":"unknown","publication":{"evidence":[{"note":"Google Research's original release links the BoolQ paper and defines 15,942 naturally occurring question/passage/answer examples, matching the exact record. The repository is archived but its primary release documentation remains accessible.","url":"https://github.com/google-research-datasets/boolean-questions/blob/master/README.md"},{"note":"The accessible NAACL 2019 paper introduces BoolQ as natural yes/no reading comprehension and lists the original authors; no leaderboard scores are imported.","url":"https://arxiv.org/abs/1905.10044"}],"id":"boolq-original","links":[{"kind":"paper","title":"BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions","url":"https://arxiv.org/abs/1905.10044"},{"kind":"repo","title":"BoolQ original dataset repository","url":"https://github.com/google-research-datasets/boolean-questions"}],"record_keys":["llm-stats:boolq"],"reviewed_at":"2026-10-26","scope":{"en":"Original BoolQ natural yes/no reading-comprehension dataset and authors' release. The source record does not identify a specific evaluation split or prompting setup.","zh":"原始 BoolQ 自然是非问答阅读理解数据集及作者发布仓库。当前来源记录未明确具体评测划分或提示设置。"}},"publisher":null,"released":"2019-05-24","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:boolq","source_url":"https://arxiv.org/abs/1905.10044"},"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":88.03999999999999,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.8804,"raw_min":0.764,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:boolq:hermes-3-70b","reported_date":"2024-08-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/boolq?top_n=500"},"unit":null},"scored_models":[{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-boolq","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/boolq?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":177,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:1905.10044","repository":null,"source_record":"catalog_df39c63037634966"},"categories":["知识","Knowledge","大语言模型","LLM","知识储备","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BoolQ is a question answering dataset for yes/no questions containing 15942 examples. These questions are naturally occurring ---they are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context. BoolQ是一个包含15942个示例的是/否问题的问答数据集。这些问题是自然生成的——即在无prompt和无约束的环境中产生的。每个例子都是一个三元组(问题、段落、答案)，页面标题是可选的附加上下文。","descriptions":{"en":"BoolQ is a question answering dataset for yes/no questions containing 15942 examples. These questions are naturally occurring ---they are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context.","zh":"BoolQ是一个包含15942个示例的是/否问题的问答数据集。这些问题是自然生成的——即在无prompt和无约束的环境中产生的。每个例子都是一个三元组(问题、段落、答案)，页面标题是可选的附加上下文。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:510","languages":[],"library_direction":{"basis":"model-review-2","categories":["knowledge-expert-qa"],"join":"artifact:arxiv:1905.10044","primary":"knowledge-expert-qa","source_record":"catalog_df39c63037634966"},"modality":null,"name":"BoolQ","openness":"unknown","publisher":null,"released":"2019-05-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-510-boolq","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BoolQ","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_boundary-bench_5415f6d9"},"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Boundary-Bench is an open-source plugin that adds configurable security policy levels to Terminal-Bench, enabling evaluation of coding agents under constraints like scoped credentials, restricted egress, read-only filesystems, and non-root execution.","descriptions":{"en":"Boundary-Bench is an open-source plugin that adds configurable security policy levels to Terminal-Bench, enabling evaluation of coding agents under constraints like scoped credentials, restricted egress, read-only filesystems, and non-root execution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02670","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agentic-coding"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_boundary-bench_5415f6d9"},"modality":null,"name":"Boundary-Bench","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02670","source_url":"https://arxiv.org/abs/2608.02670"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02670","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02670","topics":["coding","agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way.","descriptions":{"en":"Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27268","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_braillebench_3cc849b8"},"modality":null,"name":"BrailleBench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27268","source_url":"https://arxiv.org/abs/2608.27268"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27268","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27268","topics":["language"],"unit":null},{"aliases":["Brain-to-text '25","BtoT25"],"attention_signals":null,"categories":["science"],"collected_at":null,"description":"Evaluates one participant (T15), with about two thirds of 1,450 held-out sentences used for the private leaderboard. Speaking strategy, language model and corpus mix affect word error rate, so these scores are not directly comparable to Brain-to-Text '24.","descriptions":{"en":"Evaluates one participant (T15), with about two thirds of 1,450 held-out sentences used for the private leaderboard. Speaking strategy, language model and corpus mix affect word error rate, so these scores are not directly comparable to Brain-to-Text '24."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000brain_to_text_25\u0000brain_to_text_25_kaggle_leaderboard\u0000brain_to_text_25_t15_private_leaderboard\u0000approximately 67% of 1,450 held-out test sentences; aggregate word error rate\u0000BIT submission","reported_at":"2026-09-19","source_url":"https://www.kaggle.com/competitions/brain-to-text-25/leaderboard"},"first_score_reported_at":"2026-09-19","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000brain_to_text_25\u0000brain_to_text_25_kaggle_leaderboard\u0000brain_to_text_25_t15_private_leaderboard\u0000approximately 67% of 1,450 held-out test sentences; aggregate word error rate\u0000BIT submission","reported_at":"2026-09-19","source_url":"https://www.kaggle.com/competitions/brain-to-text-25/leaderboard"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:brain_to_text_25","languages":[],"library_direction":null,"modality":null,"name":"Brain-to-Text '25","openness":"unknown","publisher":null,"released":"2025-07-07","released_reference":{"basis":"benchmark_release","source_key":"model-reports:brain_to_text_25","source_url":"https://www.kaggle.com/competitions/brain-to-text-25"},"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"lower_is_better","score_summary":{"display_max":0.06999,"display_multiplier":1,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.06999,"raw_min":0.01537,"source_reference":{"obs_id":"curated\u0000brain_to_text_25\u0000brain_to_text_25_kaggle_leaderboard\u0000brain_to_text_25_t15_private_leaderboard\u0000approximately 67% of 1,450 held-out test sentences; aggregate word error rate\u0000UCD-NPL causal RNN + 5gram","observation_id":"curated\u0000brain_to_text_25\u0000brain_to_text_25_kaggle_leaderboard\u0000brain_to_text_25_t15_private_leaderboard\u0000approximately 67% of 1,450 held-out test sentences; aggregate word error rate\u0000UCD-NPL causal RNN + 5gram","reported_at":"2026-09-19","reported_date":"2026-09-19","source_id":"brain_to_text_25_kaggle_leaderboard","source_url":"https://www.kaggle.com/competitions/brain-to-text-25/leaderboard"},"unit":"ratio"},"scored_models":[{"date":"2026-09-19","date_precision":"document_publication","name":"BIT submission"},{"date":"2026-09-19","date_precision":"document_publication","name":"EPFL-INL submission"},{"date":"2026-09-19","date_precision":"document_publication","name":"Infera-Neuro submission"},{"date":"2026-09-19","date_precision":"document_publication","name":"Stanford-NPTL causal RNN Ensemble + 5gram"},{"date":"2026-09-19","date_precision":"document_publication","name":"Stanford-NPTL causal RNN TTA-Ensemble + 5gram"},{"date":"2026-09-19","date_precision":"document_publication","name":"UCD-NPL causal RNN + 5gram"}],"slug":"brain_to_text_25","source":"model_reports","source_url":"https://www.kaggle.com/competitions/brain-to-text-25","topics":["science"],"unit":"ratio"},{"aliases":["Brain-to-Text '24","BtoT24"],"attention_signals":null,"categories":["science"],"collected_at":null,"description":"Evaluates 1,200 held-out sentences from one participant (T12). Language model post-processing is part of each system, and results do not establish transfer to another participant or to the different Brain-to-Text '25 dataset.","descriptions":{"en":"Evaluates 1,200 held-out sentences from one participant (T12). Language model post-processing is part of each system, and results do not establish transfer to another participant or to the different Brain-to-Text '25 dataset."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000brain_to_text_24\u0000brain_to_text_24_evalai_leaderboard\u0000brain_to_text_24_t12_1200_holdout_sentences\u00001,200 competitionHoldOut sentences; aggregate word error rate\u0000DCoND-LIFT","reported_at":"2024-06-01","source_url":"https://eval.ai/web/challenges/challenge-page/2099/overview"},"first_score_reported_at":"2024-06-01","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000brain_to_text_24\u0000brain_to_text_24_evalai_leaderboard\u0000brain_to_text_24_t12_1200_holdout_sentences\u00001,200 competitionHoldOut sentences; aggregate word error rate\u0000DCoND-LIFT","reported_at":"2024-06-01","source_url":"https://eval.ai/web/challenges/challenge-page/2099/overview"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:brain_to_text_24","languages":[],"library_direction":null,"modality":null,"name":"Brain-to-Text Benchmark '24","openness":"unknown","publisher":null,"released":"2023-06-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:brain_to_text_24","source_url":"https://eval.ai/web/challenges/challenge-page/2099/overview"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"lower_is_better","score_summary":{"display_max":9.76,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":9.76,"raw_min":5.81,"source_reference":{"obs_id":"curated\u0000brain_to_text_24\u0000brain_to_text_24_evalai_leaderboard\u0000brain_to_text_24_t12_1200_holdout_sentences\u00001,200 competitionHoldOut sentences; aggregate word error rate\u0000PyTorch baseline","observation_id":"curated\u0000brain_to_text_24\u0000brain_to_text_24_evalai_leaderboard\u0000brain_to_text_24_t12_1200_holdout_sentences\u00001,200 competitionHoldOut sentences; aggregate word error rate\u0000PyTorch baseline","reported_at":"2024-06-01","reported_date":"2024-06-01","source_id":"brain_to_text_24_evalai_leaderboard","source_url":"https://eval.ai/web/challenges/challenge-page/2099/overview"},"unit":"percent"},"scored_models":[{"date":"2024-06-01","date_precision":"document_publication","name":"DCoND-LIFT"},{"date":"2024-06-01","date_precision":"document_publication","name":"PyTorch baseline"},{"date":"2024-06-01","date_precision":"document_publication","name":"TeamCyber submission"},{"date":"2024-06-01","date_precision":"document_publication","name":"Tyler Benster et al. submission"}],"slug":"brain_to_text_24","source":"model_reports","source_url":"https://eval.ai/web/challenges/challenge-page/2099/overview","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_brainbench_1fd81276"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"BrainBench is a benchmark for instruction-conditioned EEG understanding covering four subsets across 17 datasets. It evaluates LLMs on producing scientific reports and artifacts from EEG recordings.","descriptions":{"en":"BrainBench is a benchmark for instruction-conditioned EEG understanding covering four subsets across 17 datasets. It evaluates LLMs on producing scientific reports and artifacts from EEG recordings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04156","languages":[],"library_direction":{"basis":"model-review","categories":["science","health-medicine"],"join":"claire_record_id","primary":"science","source_record":"bm_brainbench_1fd81276"},"modality":null,"name":"BrainBench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04156","source_url":"https://arxiv.org/abs/2608.04156"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04156","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04156","topics":["medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences, localize findings within volumetric anatomy, and translate this evidence into report-gr…","descriptions":{"en":"Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences, localize findings within volumetric anatomy, and translate this evidence into report-gr…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.06571","languages":[],"library_direction":{"basis":"model-review-daily","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_braintrace_03ddb3d7"},"modality":null,"name":"BrainTRACE","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.06571","source_url":"https://arxiv.org/abs/2610.06571"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-06571","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06571","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Proposed benchmark for across-animal transfer on multi-region neural recordings from the International Brain Laboratory Brainwide Map dataset.","descriptions":{"en":"Proposed benchmark for across-animal transfer on multi-region neural recordings from the International Brain Laboratory Brainwide Map dataset."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.22064","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_brainwidebench_067d814e"},"modality":null,"name":"BrainWideBench","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.22064","source_url":"https://arxiv.org/abs/2609.22064"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-22064","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22064","topics":[],"unit":null},{"aliases":["BranchDrive: A Branch-Structured Dataset for Action-Conditioned Driving Prediction"],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"BranchDrive is a branch-structured CARLA dataset that pairs one canonical pre-decision history with one nominal expert future and twelve physically executed intervention futures spanning acceleration, braking, and left- and right-steering policies at three magnitudes.","descriptions":{"en":"BranchDrive is a branch-structured CARLA dataset that pairs one canonical pre-decision history with one nominal expert future and twelve physically executed intervention futures spanning acceleration, braking, and left- and right-steering policies at three magnitudes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27275","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_branchdrive-a-branch-structured-dataset-fo_6dd293d1"},"modality":null,"name":"BranchDrive","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27275","source_url":"https://arxiv.org/abs/2609.27275"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27275","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27275","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","Vision & 3D","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark offering 3DGS scenes and tools for evaluating robotic active vision systems in 6DOF object pose estimation tasks.","descriptions":{"en":"A benchmark offering 3DGS scenes and tools for evaluating robotic active vision systems in 6DOF object pose estimation tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17106","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","spatial-3d"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_brave-6d_025d25b5"},"modality":null,"name":"BRAVE-6D","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17106","source_url":"https://arxiv.org/abs/2609.17106"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17106","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17106","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","大语言模型","LLM","检索能力","Retrieval","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BRIGHT is the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. BRIGHT 是第一个需要大量推理来检索相关文档的文本检索基准。","descriptions":{"en":"BRIGHT is the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.","zh":"BRIGHT 是第一个需要大量推理来检索相关文档的文本检索基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1571","languages":[],"library_direction":null,"modality":null,"name":"BRIGHT","openness":"unknown","publisher":"The University of Hong Kong, etc.","released":"2024-10-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1571-bright","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BRIGHT","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["math"],"collected_at":null,"description":"arXiv proofs with planted errors; tests whether a model catches a broken argument instead of reproducing it.","descriptions":{"en":"arXiv proofs with planted errors; tests whether a model catches a broken argument instead of reproducing it."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000brokenarxiv\u0000tencent_hy4_preview\u0000brokenarxiv\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000brokenarxiv\u0000tencent_hy4_preview\u0000brokenarxiv\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:brokenarxiv","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["math"],"join":"review:14H034160212:2026-10-15","primary":"math","source_record":null},"modality":null,"name":"BrokenArXiv","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":54.6,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":54.6,"raw_min":54.6,"source_reference":{"obs_id":"curated\u0000brokenarxiv\u0000tencent_hy4_preview\u0000brokenarxiv\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000brokenarxiv\u0000tencent_hy4_preview\u0000brokenarxiv\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"brokenarxiv","source":"model_reports","source_url":"https://matharena.ai/brokenarxiv","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4665,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:openai/simple-evals","source_record":"lib_browsecomp"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BrowseComp is a benchmark comprising 1,266 questions that challenge AI agents to persistently navigate the internet in search of hard-to-find, entangled information. The benchmark measures agents' ability to exercise persistence in information gathering, demonstrate creativity in web navigation, and find concise, verifiable answers. Despite the difficulty of the questions, BrowseComp is simple and easy-to-use, as predicted answers are short and easily verifiable against reference answers.","descriptions":{"en":"BrowseComp is a benchmark comprising 1,266 questions that challenge AI agents to persistently navigate the internet in search of hard-to-find, entangled information. The benchmark measures agents' ability to exercise persistence in information gathering, demonstrate creativity in web navigation, and find concise, verifiable answers. Despite the difficulty of the questions, BrowseComp is simple and easy-to-use, as predicted answers are short and easily verifiable against reference answers."},"evidence_summary":{"document_count":1,"model_count":62,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:browsecomp:deepseek-v3.1","reported_at":"2025-01-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:browsecomp","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"llm_stats_id","primary":"deep-research","source_record":"lib_browsecomp"},"modality":"text","name":"BrowseComp","openness":"restricted","publisher":"OpenAI","released":"2025-04-10","released_reference":null,"repo_kind":"harness_only","repo_resolution_status":"resolved","score_count":62,"score_direction":"higher_is_better","score_summary":{"display_max":91.2,"display_multiplier":100,"model_count":62,"model_count_basis":"source_model_id","numeric_count":62,"raw_max":0.912,"raw_min":0.089,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:browsecomp:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Pro"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 Pro"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok 4 Fast"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax M2.5"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step-3.5-Flash"}],"slug":"llm-stats-browsecomp","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["BrowseComp"],"attention_signals":null,"categories":["agent"],"collected_at":null,"description":"Live web. The result depends on what the internet contained on the day of the run.","descriptions":{"en":"Live web. The result depends on what the internet contained on the day of the run."},"evidence_summary":{"document_count":12,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000browsecomp\u0000zai_glm_5_model_card\u0000browsecomp\u0000no context management, retain most recent 5 turns\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"first_score_reported_at":"2026-02-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000browsecomp\u0000zai_glm_5_model_card\u0000browsecomp\u0000no context management, retain most recent 5 turns\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:browsecomp","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["deep-research"],"join":"review:14H034160212:2026-10-15","primary":"deep-research","source_record":null},"modality":null,"name":"BrowseComp","openness":"unknown","publisher":null,"released":"2025-04-10","released_reference":{"basis":"benchmark_release","source_key":"model-reports:browsecomp","source_url":"https://openai.com/index/browsecomp/"},"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":91.2,"display_multiplier":1,"model_count":7,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":91.2,"raw_min":62.0,"source_reference":{"obs_id":"curated\u0000browsecomp\u0000moonshot_kimi_k3_model_card\u0000browsecomp\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000browsecomp\u0000moonshot_kimi_k3_model_card\u0000browsecomp\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"browsecomp","source":"model_reports","source_url":"https://openai.com/index/browsecomp/","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4665,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:openai/simple-evals","source_record":"catalog_01284e3412ceb8cb"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A challenging benchmark for evaluating web browsing agents' ability to persistently navigate the internet and find hard-to-locate, entangled information. Comprises 1,266 questions requiring strategic reasoning, creative search, and interpretation of retrieved content, with short and easily verifiable answers.","descriptions":{"en":"A challenging benchmark for evaluating web browsing agents' ability to persistently navigate the internet and find hard-to-locate, entangled information. Comprises 1,266 questions requiring strategic reasoning, creative search, and interpretation of retrieved content, with short and easily verifiable answers."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:browsecomp-long-128k:gpt-5-2025-08-07","reported_at":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:browsecomp-long-128k","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","long-context"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_01284e3412ceb8cb"},"modality":"text","name":"BrowseComp Long Context 128k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":92.0,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.92,"raw_min":0.9,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:browsecomp-long-128k:gpt-5.2-2025-12-11","reported_date":"2025-12-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-128k?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Instant"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Thinking"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"}],"slug":"llm-stats-browsecomp-long-128k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-128k?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4665,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:openai/simple-evals","source_record":"catalog_624989cc95dc50bc"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BrowseComp is a benchmark for measuring the ability of agents to browse the web, comprising 1,266 questions that require persistently navigating the internet in search of hard-to-find, entangled information. Despite the difficulty of the questions, BrowseComp is simple and easy-to-use, as predicted answers are short and easily verifiable against reference answers. The benchmark focuses on questions where answers are obscure, time-invariant, and well-supported by evidence scattered across the open web.","descriptions":{"en":"BrowseComp is a benchmark for measuring the ability of agents to browse the web, comprising 1,266 questions that require persistently navigating the internet in search of hard-to-find, entangled information. Despite the difficulty of the questions, BrowseComp is simple and easy-to-use, as predicted answers are short and easily verifiable against reference answers. The benchmark focuses on questions where answers are obscure, time-invariant, and well-supported by evidence scattered across the open web."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:browsecomp-long-256k:gpt-5-2025-08-07","reported_at":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-256k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:browsecomp-long-256k","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","long-context"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_624989cc95dc50bc"},"modality":"text","name":"BrowseComp Long Context 256k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":89.8,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.898,"raw_min":0.888,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:browsecomp-long-256k:gpt-5.2-2025-12-11","reported_date":"2025-12-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-256k?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"}],"slug":"llm-stats-browsecomp-long-256k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-256k?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":20046,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":144,"join":"llm_stats_id","repository":"gh:alibaba-nlp/deepresearch","source_record":"catalog_c6c0e2526edde416"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","search","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"BrowseComp-VL is the vision-language variant of BrowseComp, evaluating multimodal models on web browsing comprehension tasks that require processing visual web page content alongside text.","descriptions":{"en":"BrowseComp-VL is the vision-language variant of BrowseComp, evaluating multimodal models on web browsing comprehension tasks that require processing visual web page content alongside text."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:browsecomp-vl:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-vl?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:browsecomp-vl","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_c6c0e2526edde416"},"modality":"multimodal","name":"BrowseComp-VL","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":51.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.519,"raw_min":0.519,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:browsecomp-vl:glm-5v-turbo","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-vl?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"}],"slug":"llm-stats-browsecomp-vl","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-vl?top_n=500","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":165,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":null,"source_record":"catalog_4cbf3f4140d31be6"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents on the Chinese web, consisting of 289 multi-hop questions spanning 11 diverse domains including Film & TV, Technology, Medicine, and History. Questions are reverse-engineered from short, objective, and easily verifiable answers, requiring sophisticated reasoning and information reconciliation beyond basic retrieval. The benchmark addresses linguistic, infrastructural, and censorship-related complexities in Chinese web environments.","descriptions":{"en":"A high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents on the Chinese web, consisting of 289 multi-hop questions spanning 11 diverse domains including Film & TV, Technology, Medicine, and History. Questions are reverse-engineered from short, objective, and easily verifiable answers, requiring sophisticated reasoning and information reconciliation beyond basic retrieval. The benchmark addresses linguistic, infrastructural, and censorship-related complexities in Chinese web environments."},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:browsecomp-zh:deepseek-v3.1","reported_at":"2025-01-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-zh?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:browsecomp-zh","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","multilingual"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_4cbf3f4140d31be6"},"modality":"text","name":"BrowseComp-zh","openness":"unknown","publication":{"evidence":[{"note":"The authors' repository links the BrowseComp-ZH paper, describes 289 questions across 11 domains and the reverse-engineered answer design, and explicitly states that the Chinese questions are independently constructed rather than translated. These details match the exact source record.","url":"https://github.com/PALIN2018/BrowseComp-ZH/blob/main/README.md"}],"id":"browsecomp-zh-original","links":[{"kind":"paper","title":"BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese","url":"https://arxiv.org/abs/2504.19314"},{"kind":"repo","title":"BrowseComp-ZH official repository","url":"https://github.com/PALIN2018/BrowseComp-ZH"}],"record_keys":["llm-stats:browsecomp-zh"],"reviewed_at":"2026-10-26","scope":{"en":"BrowseComp-ZH's original Chinese web-browsing benchmark. It contains questions authored for Chinese browsing evaluation, rather than a translation or result subset of the English BrowseComp benchmark.","zh":"BrowseComp-ZH 原始中文网页检索评测，问题为中文检索场景专门编写，并非英文 BrowseComp 的翻译版或成绩子集。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":70.3,"display_multiplier":100,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":0.703,"raw_min":0.357,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:browsecomp-zh:qwen3.5-397b-a17b","reported_date":"2026-02-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-zh?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"}],"slug":"llm-stats-browsecomp-zh","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-zh?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["BrowseComp-ZH","BrowseComp-zh","BrowseComp-Zh"],"attention_signals":{"as_of":"2026-10-13","github_stars":165,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"artifact:arxiv:2504.19314","repository":null,"source_record":"catalog_4cbf3f4140d31be6"},"categories":["agent"],"collected_at":null,"description":"Chinese-language live web. Same day-to-day web drift as BrowseComp, over a different index.","descriptions":{"en":"Chinese-language live web. Same day-to-day web drift as BrowseComp, over a different index."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000browsecomp_zh\u0000zai_glm_5_model_card\u0000browsecomp_zh\u0000no context management, retain most recent 5 turns\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"first_score_reported_at":"2026-02-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000browsecomp_zh\u0000zai_glm_5_model_card\u0000browsecomp_zh\u0000no context management, retain most recent 5 turns\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:browsecomp_zh","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","multilingual"],"join":"artifact:arxiv:2504.19314","primary":"deep-research","source_record":"catalog_4cbf3f4140d31be6"},"modality":null,"name":"BrowseComp-ZH","openness":"unknown","publisher":null,"released":"2025-04-27","released_reference":{"basis":"benchmark_release","source_key":"model-reports:browsecomp_zh","source_url":"https://arxiv.org/abs/2504.19314"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":72.7,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":72.7,"raw_min":70.3,"source_reference":{"obs_id":"curated\u0000browsecomp_zh\u0000zai_glm_5_model_card\u0000browsecomp_zh\u0000no context management, retain most recent 5 turns\u0000GLM-5","observation_id":"curated\u0000browsecomp_zh\u0000zai_glm_5_model_card\u0000browsecomp_zh\u0000no context management, retain most recent 5 turns\u0000GLM-5","reported_at":"2026-02-11","reported_date":"2026-02-11","source_id":"zai_glm_5_model_card","source_url":"https://huggingface.co/zai-org/GLM-5"},"unit":"percent"},"scored_models":[{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"browsecomp_zh","source":"model_reports","source_url":"https://arxiv.org/abs/2504.19314","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kjy7567/bts-agentbench","source_record":"bm_bts-agentbench_70852571"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-turn agent performance on 532 telemetry-derived tasks across train/dev/test splits, with additional XAI4HEAT episodes, using deterministic replayable construction and verifiable gold answers.","descriptions":{"en":"Evaluates multi-turn agent performance on 532 telemetry-derived tasks across train/dev/test splits, with additional XAI4HEAT episodes, using deterministic replayable construction and verifiable gold answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.27334","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_bts-agentbench_70852571"},"modality":null,"name":"BTS-AgentBench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27334","source_url":"https://arxiv.org/abs/2608.27334"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27334","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27334","topics":["agents"],"unit":null},{"aliases":["Companion-style QA Assistance in Ego-Vision"],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:qhuni/buddyvqa","source_record":"bm_companion-style-qa-assistance-in-ego-visio_aecb3225"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"BuddyVQA evaluates model performance on companion-style question answering in streaming egocentric video, including ego-deictic and chained questions.","descriptions":{"en":"BuddyVQA evaluates model performance on companion-style question answering in streaming egocentric video, including ego-deictic and chained questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.06721","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_companion-style-qa-assistance-in-ego-visio_aecb3225"},"modality":null,"name":"BuddyVQA","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06721","source_url":"https://arxiv.org/abs/2609.06721"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06721","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06721","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"The work introduces BugSourceBench, a code repair benchmark with bugs from human-written, LM-generated, and human-edited LM-generated code. The benchmark evaluates the fix rate of language models on these bug sources. No scoring contract, dataset, or public access path is provided.","descriptions":{"en":"The work introduces BugSourceBench, a code repair benchmark with bugs from human-written, LM-generated, and human-edited LM-generated code. The benchmark evaluates the fix rate of language models on these bug sources. No scoring contract, dataset, or public access path is provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.03523","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_bugsourcebench_7e234238"},"modality":null,"name":"BugSourceBench","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03523","source_url":"https://arxiv.org/abs/2607.03523"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03523","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03523","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:eureka246/bulkpr-bench-release","source_record":"bm_bulkpr-bench_1253579f"},"categories":["Agents","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"BulkPR-Bench evaluates governance of interacting pull requests. It includes 581 candidate PRs on 18 repositories, with metrics RDS and Global-SGY measuring safe delivery. The benchmark uses executable repository execution with hidden safety checks.","descriptions":{"en":"BulkPR-Bench evaluates governance of interacting pull requests. It includes 581 candidate PRs on 18 repositories, with metrics RDS and Global-SGY measuring safe delivery. The benchmark uses executable repository execution with hidden safety checks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.02685","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_bulkpr-bench_1253579f"},"modality":null,"name":"BulkPR-Bench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02685","source_url":"https://arxiv.org/abs/2608.02685"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02685","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02685","topics":["coding","agents"],"unit":null},{"aliases":["BullingerDB: A Dataset for Handwritten Text Recognition and Writer Retrieval"],"attention_signals":null,"categories":["General AI","Information retrieval","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A large-scale historical-document benchmark for multilingual handwritten text recognition and time-aware writer retrieval.","descriptions":{"en":"A large-scale historical-document benchmark for multilingual handwritten text recognition and time-aware writer retrieval."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30235","languages":[],"library_direction":null,"modality":null,"name":"BullingerDB","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30235","source_url":"https://arxiv.org/abs/2605.30235"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30235","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30235","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","NAACL 2024","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"BUST is a comprehensive benchmark for evaluating synthetic text detectors, focusing on their effectiveness against outputs from various Large Language Models (LLMs). BUST 是一个综合基准，旨在评估合成文本检测器，BUST 使用多种指标来评估检测器，包括语言特征、可读性和作者态度。","descriptions":{"en":"BUST is a comprehensive benchmark for evaluating synthetic text detectors, focusing on their effectiveness against outputs from various Large Language Models (LLMs).","zh":"BUST 是一个综合基准，旨在评估合成文本检测器，BUST 使用多种指标来评估检测器，包括语言特征、可读性和作者态度。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1146","languages":[],"library_direction":null,"modality":null,"name":"BUST","openness":"unknown","publisher":"Dalle Molle Institute for Artificial Intelligence Research (IDSIA)","released":"2024-06-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1146-bust","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BUST","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["创作","Creation","指令跟随","Instruct","多模态模型","VLM","视觉生成","Visual Generation","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ByteMorph is a benchmark for instruction-guided image editing, focusing on evaluating models’ capabilities in handling non-rigid motions such as camera viewpoint changes, object deformations, human articulations, and complex interactions. ByteMorph 是一个面向指令驱动图像编辑的基准，专注于评估模型在处理非刚性运动（如相机视角变化、物体变形、人类动作和复杂交互）方面的能力。 该基准包括超过 600 万对高分辨率图像编辑样本，涵盖多种动态编辑场景，支持对模型在多种非刚性运动类型下的表现进行细粒度评估。","descriptions":{"en":"ByteMorph is a benchmark for instruction-guided image editing, focusing on evaluating models’ capabilities in handling non-rigid motions such as camera viewpoint changes, object deformations, human articulations, and complex interactions.","zh":"ByteMorph 是一个面向指令驱动图像编辑的基准，专注于评估模型在处理非刚性运动（如相机视角变化、物体变形、人类动作和复杂交互）方面的能力。 该基准包括超过 600 万对高分辨率图像编辑样本，涵盖多种动态编辑场景，支持对模型在多种非刚性运动类型下的表现进行细粒度评估。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1983","languages":[],"library_direction":null,"modality":null,"name":"ByteMorph","openness":"open","publisher":"ByteDance Seed , University of Southern California , University of Tokyo , etc.","released":"2025-06-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1983-bytemorph","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ByteMorph","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1873,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:hkust-nlp/ceval","source_record":"catalog_e32e7c8c89936ac7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"C-Eval is a comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. It comprises 13,948 multiple-choice questions across 52 diverse disciplines spanning humanities, science, and engineering, with four difficulty levels: middle school, high school, college, and professional. The benchmark includes C-Eval Hard, a subset of very challenging subjects requiring advanced reasoning abilities.","descriptions":{"en":"C-Eval is a comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. It comprises 13,948 multiple-choice questions across 52 diverse disciplines spanning humanities, science, and engineering, with four difficulty levels: middle school, high school, college, and professional. The benchmark includes C-Eval Hard, a subset of very challenging subjects requiring advanced reasoning abilities."},"evidence_summary":{"document_count":1,"model_count":18,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:c-eval:qwen2-72b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/c-eval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:c-eval","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_e32e7c8c89936ac7"},"modality":"text","name":"C-Eval","openness":"unknown","publication":{"evidence":[{"note":"The official repository links the paper and project site, describes the Chinese examination suite, and distinguishes C-Eval Hard. The task and subject coverage match this exact record. The linked project homepage was excluded from reader links on 2026-10-26 because a normal browser reported ERR_CERT_DATE_INVALID; the paper and author repository remain available.","url":"https://github.com/hkust-nlp/ceval/blob/main/README.md"}],"id":"c-eval-original","links":[{"kind":"paper","title":"C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models","url":"https://arxiv.org/abs/2305.08322"},{"kind":"repo","title":"C-Eval official repository","url":"https://github.com/hkust-nlp/ceval"}],"record_keys":["llm-stats:c-eval"],"reviewed_at":"2026-10-26","scope":{"en":"Original C-Eval suite. The official project separately documents C-Eval Hard and zero-shot, few-shot and validation results; the publication links do not change the recorded evaluation setting.","zh":"原始 C-Eval 评测套件；官方项目另列 C-Eval Hard、零样本、少样本和验证集结果，出处链接不会改变已收录成绩的评测设置。"}},"publisher":null,"released":"2023-05-15","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:c-eval","source_url":"https://arxiv.org/abs/2305.08322"},"repo_kind":null,"repo_resolution_status":null,"score_count":18,"score_direction":"higher_is_better","score_summary":{"display_max":93.30000000000001,"display_multiplier":100,"model_count":18,"model_count_basis":"source_model_id","numeric_count":18,"raw_max":0.933,"raw_min":0.407,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:c-eval:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/c-eval?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2025-01-20","date_precision":"model_announcement","name":"Kimi-k1.5"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-c-eval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/c-eval?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1873,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2305.08322","repository":"gh:hkust-nlp/ceval","source_record":"catalog_e32e7c8c89936ac7"},"categories":["学科","Examination","大语言模型","LLM","知识储备","Knowledge","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"C-Eval is a comprehensive Chinese evaluation suite for foundation models. It consists of 13948 multi-choice questions spanning 52 diverse disciplines and four difficulty levels. C-Eval 是一个全面的中文基础模型评估套件。它包含了13948个多项选择题，涵盖了52个不同的学科和四个难度级别。","descriptions":{"en":"C-Eval is a comprehensive Chinese evaluation suite for foundation models. It consists of 13948 multi-choice questions spanning 52 diverse disciplines and four difficulty levels.","zh":"C-Eval 是一个全面的中文基础模型评估套件。它包含了13948个多项选择题，涵盖了52个不同的学科和四个难度级别。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:496","languages":["English","Chinese"],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"artifact:arxiv:2305.08322","primary":"knowledge-expert-qa","source_record":"catalog_e32e7c8c89936ac7"},"modality":null,"name":"C-Eval","openness":"unknown","publisher":null,"released":"2023-05-15","released_reference":{"basis":"paper_first_version","note":"First version of the paper introducing C-Eval.","source_key":"opencompass:496","source_url":"https://arxiv.org/abs/2305.08322"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-496-c-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/C-Eval","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","事实可靠性","Factual Reliability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"C-FAITH, a Chinese QA hallucination benchmark created from 1,399 knowledge documents obtained from web scraping, totaling 60,702 entries. C-FAITH，这是一个中国 QA 幻觉基准，由从网络抓取中获得的 1,399 份知识文档创建，总共 60,702 个条目。","descriptions":{"en":"C-FAITH, a Chinese QA hallucination benchmark created from 1,399 knowledge documents obtained from web scraping, totaling 60,702 entries.","zh":"C-FAITH，这是一个中国 QA 幻觉基准，由从网络抓取中获得的 1,399 份知识文档创建，总共 60,702 个条目。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1780","languages":["Chinese"],"library_direction":null,"modality":null,"name":"C-FAITH","openness":"unknown","publisher":"PKU","released":"2025-04-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1780-c-faith","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/C-FAITH","topics":["language"],"unit":null},{"aliases":["Who Judges the Judges? A Chinese Safety QA Benchmark for Evaluating LLM Responses and Safety Judges"],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"A policy-grounded benchmark for response-level Chinese safety evaluation, assessing LLM responses and safety judges.","descriptions":{"en":"A policy-grounded benchmark for response-level Chinese safety evaluation, assessing LLM responses and safety judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01210","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_who-judges-the-judges-a-chinese-safety-qa-_0295f156"},"modality":null,"name":"C-SafeQA","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01210","source_url":"https://arxiv.org/abs/2609.01210"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01210","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01210","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"C-SUITEBENCH evaluates multimodal LLMs as CEOs on five decision tasks under paired text-only and multimodal conditions across 50 scenarios, focusing on evidence-centric reasoning and constraint satisfaction.","descriptions":{"en":"C-SUITEBENCH evaluates multimodal LLMs as CEOs on five decision tasks under paired text-only and multimodal conditions across 50 scenarios, focusing on evidence-centric reasoning and constraint satisfaction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05864","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","visual-reasoning"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_c-suitebench_9e6fa218"},"modality":null,"name":"C-SUITEBENCH","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05864","source_url":"https://arxiv.org/abs/2608.05864"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05864","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05864","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A free-form multiple-Choice Chinese machine reading Comprehension dataset (C3), containing 13,369 documents (dialogues or more formally written mixed-genre texts) and their associated 19,577 multiple-choice free-form questions collected from Chinese-as-a-second-language examinations 一个自由形式的多项选择中文机器阅读理解数据集（C3），包含13369篇文献（对话或更正式的混合体裁文本）及其相关的19577道自由选择题，这些问题都是从汉语作为第二语言的考试中收集到的","descriptions":{"en":"A free-form multiple-Choice Chinese machine reading Comprehension dataset (C3), containing 13,369 documents (dialogues or more formally written mixed-genre texts) and their associated 19,577 multiple-choice free-form questions collected from Chinese-as-a-second-language examinations","zh":"一个自由形式的多项选择中文机器阅读理解数据集（C3），包含13369篇文献（对话或更正式的混合体裁文本）及其相关的19577道自由选择题，这些问题都是从汉语作为第二语言的考试中收集到的"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:514","languages":["Chinese"],"library_direction":null,"modality":null,"name":"C3","openness":"unknown","publisher":null,"released":"2019-04-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-514-c3","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/C3","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"C3-Bench evaluates context-aware change captioning with 4,996 human-labeled image pairs across 51 real-world contexts, using an LLM-as-Judge framework scoring correctness, specificity, fluency, relevance, and a reversibility metric.","descriptions":{"en":"C3-Bench evaluates context-aware change captioning with 4,996 human-labeled image pairs across 51 real-world contexts, using an LLM-as-Judge framework scoring correctness, specificity, fluency, relevance, and a reversibility metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.25445","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_c3-bench_c82d4fcd"},"modality":null,"name":"C3-Bench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25445","source_url":"https://arxiv.org/abs/2606.25445"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25445","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25445","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_cadengbench_ba1f84a8"},"categories":["CAD","General AI","Geometric reasoning","Reasoning","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"CADEngBench evaluates parametric CAD part generation and editing through B-Rep validity, engineering checks, parameter perturbation, and linear-static FEA, plus assembly reasoning via joint retrieval, grounding, and kinematic verification.","descriptions":{"en":"CADEngBench evaluates parametric CAD part generation and editing through B-Rep validity, engineering checks, parameter perturbation, and linear-static FEA, plus assembly reasoning via joint retrieval, grounding, and kinematic verification."},"editorial_guide":{"id":"cadengbench","record_keys":["claire-radar:2608.09296"],"reviewed_at":"2026-10-25","scope":{"en":"August 2026 paper v1; parametric-part and assembly tracks retain separate metrics and denominators.","zh":"依据 2026 年 8 月论文 v1；参数化零件与装配赛道分别保留指标和分母。"},"sections":[{"id":"tasks","source_ids":["paper"],"text":{"en":"P uses 300 parts for 600 generation/editing tasks. A contains 150 body pairs; its reported comparison excludes 30 development pairs and evaluates 120.","zh":"P 赛道用 300 个零件构成 600 道生成／编辑任务。A 赛道含 150 对实体，论文比较排除 30 对开发样本，评测其余 120 对。"},"title":{"en":"Two tracks, different deliverables","zh":"两条赛道，两类交付物"}},{"id":"part-checks","source_ids":["paper"],"text":{"en":"L0 checks execution and STEP round-tripping; L1 checks engineering requirements; L2-Z tests parameter behaviour; L2-E tests the requested edit while preserving protected geometry.","zh":"L0 检查执行及 STEP 导出后重载；L1 检查工程要求；L2-Z 检查参数行为；L2-E 检查目标编辑及受保护几何是否保持。"},"title":{"en":"From valid solids to controlled edits","zh":"从有效实体到受控编辑"}},{"id":"physics","source_ids":["paper"],"text":{"en":"L3 covers 164 eligible parts. CalculiX compares generated and reference responses under matched conditions; simulation reach and successful matched-state pairs use different denominators.","zh":"L3 覆盖 164 个适用零件。CalculiX 在一致条件下比较生成与参考响应；可进入仿真的比例和匹配状态对的通过率使用不同分母。"},"title":{"en":"A solved simulation is only the start","zh":"仿真算完并不等于通过"}},{"id":"assembly","source_ids":["paper"],"text":{"en":"Assembly scoring distinguishes entity retrieval from joint-type correctness. End-to-end success also requires a correct frame and verified permitted and blocked motions.","zh":"装配指标区分实体检索与关节类型正确性。端到端通过还要求坐标框架正确，并验证允许和禁止的运动。"},"title":{"en":"Identify the joint, entities and motion","zh":"找对连接、实体与运动"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Tests cover bounded edits, body pairs and linear-static analysis. They do not certify manufacturing readiness or full-assembly recovery.","zh":"测试限于受控编辑、实体对和线性静力分析，不构成制造就绪认证或完整装配重建证明。"},"title":{"en":"Read the scope before comparing scores","zh":"比分前先看范围"}},{"id":"public-package","source_ids":["code"],"text":{"en":"The authors’ repository provides ten P development items and seven A development pairs, not the sealed full corpora. It includes evaluator code, sample controls and saved reference summaries. Full-paper aggregates require the frozen full datasets and original model responses.","zh":"作者仓库提供 10 个 P 开发样本和 7 对 A 开发样本，并非封存的完整题集；同时提供评估代码、对照检查和参考摘要。复现论文总体结果仍需固定的完整数据与原始模型响应。"},"title":{"en":"What can be reproduced from the public package?","zh":"公开资料能复现到哪一步？"}},{"id":"runtime","source_ids":["code"],"text":{"en":"The package specifies Python 3.11, CalculiX 2.21 and an Ubuntu/Linux reference environment. Its verification and control scripts check the installation before model calls; Apple Silicon needs an additional conda-compatible environment for PyBullet.","zh":"项目指定 Python 3.11、CalculiX 2.21，并以 Ubuntu/Linux 为参考环境。先执行安装验证与对照检查，再调用模型；Apple Silicon 还需兼容 conda 的环境安装 PyBullet。"},"title":{"en":"Reproduction prerequisites","zh":"复现环境要求"}},{"id":"published-p","source_ids":["paper"],"table":{"caption":{"en":"Paper Table 2(a) · CADEngBench-P","zh":"论文表 2(a) · CADEngBench-P"},"columns":[{"en":"System","zh":"系统"},{"en":"L0 %","zh":"L0 %"},{"en":"L1 %","zh":"L1 %"},{"en":"L2-Z % (evaluable)","zh":"L2-Z %（可评测数）"},{"en":"L2-E %","zh":"L2-E %"},{"en":"L3 reach % (parts/164)","zh":"L3 进入率 %（零件数/164）"},{"en":"L3 pair pass % (passed/pairs)","zh":"L3 状态对通过率 %（通过数/对数）"}],"note":{"en":"L0, L1 and L2-E each use 300 items; L2-Z uses the shown per-model denominator. L3 reach counts parts; L3 pair pass counts comparable generated/reference parameter states. Do not average these columns into a single score.","zh":"L0、L1、L2-E 各使用 300 个样本；L2-Z 使用所列模型分母。L3 进入率按零件计数，L3 状态对通过率按可比较的生成／参考参数状态计数，不将各列平均为总分。"},"rows":[["GPT-5.2","41.3","22.0","31.6 (291)","70.3","37.8 (62/164)","32.8 (65/198)"],["Claude 4.5","58.0","28.3","41.4 (290)","66.7","54.3 (89/164)","35.8 (101/282)"],["Gemini 3 Flash","52.7","30.0","39.8 (289)","72.3","46.3 (76/164)","46.3 (119/257)"],["GLM-4.6V","47.3","13.3","30.7 (287)","59.3","44.5 (73/164)","27.0 (64/237)"],["Kimi K2.5","53.3","24.3","41.0 (288)","67.0","51.8 (85/164)","34.8 (94/270)"],["Mistral 3.5","29.3","8.3","16.5 (297)","68.3","26.8 (44/164)","34.5 (49/142)"],["Llama 4 Maverick","48.0","12.7","34.0 (288)","60.7","47.6 (78/164)","27.5 (71/258)"],["Qwen3.5-35B","13.3","5.0","6.3 (300)","61.7","15.9 (26/164)","36.6 (30/82)"]]},"text":{"en":"Part-generation, editing and simulation columns describe different checks. Parentheses retain the paper’s evaluable counts.","zh":"零件生成、编辑和仿真列对应不同验收项目；括号内保留论文给出的可评测数量。"},"title":{"en":"Published results: parametric parts","zh":"论文成绩：参数化零件"}},{"id":"published-a","source_ids":["paper"],"table":{"caption":{"en":"Paper Table 2(b) · CADEngBench-A, 120 comparison pairs","zh":"论文表 2(b) · CADEngBench-A，120 对比较样本"},"columns":[{"en":"System","zh":"系统"},{"en":"Valid %","zh":"格式有效 %"},{"en":"Entity@1 %","zh":"Entity@1 %"},{"en":"Typed@1 %","zh":"Typed@1 %"},{"en":"Typed@3 %","zh":"Typed@3 %"},{"en":"MRR","zh":"MRR"},{"en":"A2 E2E % (passed/120)","zh":"A2 端到端 %（通过数/120）"}],"note":{"en":"The remaining 30 of the 150 source pairs were reserved for evaluator development. Entity ignores joint type; Typed also requires joint family and body roles. The final column retains the paper’s percentage and parenthesized pass count.","zh":"全部 150 对实体中，另 30 对用于评估器开发。Entity 不要求关节类型正确，Typed 还要求关节类别与实体角色正确。末列保留论文的百分比和括号内通过数量。"},"rows":[["GPT-5.2","100.0","61.7","12.5","30.0",".197","11.7 (14)"],["Claude 4.5","100.0","57.5","14.2","32.5",".221","12.5 (15)"],["Gemini 3 Flash","100.0","62.5","23.3","41.7",".310","15.8 (19)"],["GLM-4.6V","99.2","50.0","10.0","27.5",".174","8.3 (10)"],["Kimi K2.5","78.3","56.7","15.0","32.5",".228","7.5 (9)"],["Mistral 3.5","100.0","52.5","15.8","25.8",".200","14.2 (17)"],["Llama 4 Maverick","100.0","55.8","14.2","26.7",".192","10.8 (13)"],["Qwen3.5-35B","100.0","55.0","14.2","24.2",".190","10.8 (13)"]]},"text":{"en":"Entity retrieval, typed retrieval and end-to-end motion checks remain separate. MRR uses a different scale from percentages.","zh":"实体检索、带类型检索及端到端运动检查分别列出；MRR 与百分比的量纲不同。"},"title":{"en":"Published results: assembly relations","zh":"论文成绩：装配关系"}}],"sources":[{"id":"paper","title":"CADEngBench introducing paper · tracks, metrics and limitations (arXiv v1)","url":"https://arxiv.org/html/2608.09296v1"},{"id":"code","title":"Authors’ code and development-data supplement","url":"https://github.com/coderjs-lab/cadengbench"}],"summary":{"en":"Test whether generated CAD remains usable under parameter changes, edits, physical analysis and assembly constraints.","zh":"检查生成的 CAD 在参数修改、功能编辑、物理分析与装配约束下是否仍然可用。"},"title":"CADEngBench"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09296","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_cadengbench_ba1f84a8"},"modality":null,"name":"CADEngBench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09296","source_url":"https://arxiv.org/abs/2608.09296"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09296","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09296","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ariking11/cadp-bench","source_record":"bm_cadp-bench_c4c34058"},"categories":["Academic document parsing","Chart-to-code generation","Multimodal","Multimodal code generation","Scientific Research & AI for Science","Structured document reconstruction","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CADP-Bench evaluates multimodal LLMs on compilable academic document parsing. It includes expert-verified full academic pages with tightly coupled text and structured elements, assessed through a re-injection compilation protocol.","descriptions":{"en":"CADP-Bench evaluates multimodal LLMs on compilable academic document parsing. It includes expert-verified full academic pages with tightly coupled text and structured elements, assessed through a re-injection compilation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.17550","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_cadp-bench_c4c34058"},"modality":null,"name":"CADP-Bench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17550","source_url":"https://arxiv.org/abs/2608.17550"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17550","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17550","topics":["multimodal"],"unit":null},{"aliases":["CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":14,"join":"claire_record_id","repository":null,"source_record":"bm_cadworld-computer-use-benchmark-for-long-h_103de6e3"},"categories":["CAD","Geometric reasoning","Manufacturing & Process Control","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-horizon computer use in FreeCAD through 200 tasks spanning 11 mechanical-CAD workflow categories, with success determined by executable checks over saved artifacts.","descriptions":{"en":"Evaluates long-horizon computer use in FreeCAD through 200 tasks spanning 11 mechanical-CAD workflow categories, with success determined by executable checks over saved artifacts."},"editorial_guide":{"id":"cadworld","record_keys":["claire-radar:2609.16251"],"reviewed_at":"2026-10-25","scope":{"en":"September 2026 paper v1 and the authors’ public task site; the main experiment uses GUI interaction.","zh":"依据 2026 年 9 月论文 v1 与作者任务网站；主实验采用图形界面操作。"},"sections":[{"id":"coverage","source_ids":["project"],"text":{"en":"There are 200 tasks across 11 workflows, including sketches, parts, assemblies, CAM, FEM and technical drawings. The suite is largest in part modelling and sketching; small categories should be read with their task counts.","zh":"共 200 道任务、11 类工作流，覆盖草图、零件、装配、CAM、FEM 和工程图。零件建模与草图占比最高；小类别的成绩需结合题量理解。"},"title":{"en":"What work does it cover?","zh":"覆盖哪些工作？"}},{"id":"interaction","source_ids":["project","paper"],"text":{"en":"The main setup permits screenshots and GUI actions, with 100 GUI steps per task. A separate terminal experiment uses 50 tasks and should not be pooled with these results.","zh":"主实验允许截图观察与 GUI 操作，每题最多 100 个 GUI 步骤。另有 50 题终端实验，不应混入主实验成绩。"},"title":{"en":"How the agent works","zh":"Agent 怎样操作？"}},{"id":"grading","source_ids":["paper"],"text":{"en":"Executable checks inspect native .FCStd projects and required auxiliary files. Geometry, constraints, feature structure or downstream outputs must satisfy every task requirement; diagnostics do not award partial credit.","zh":"程序检查原生 .FCStd 工程与指定辅助文件。几何、约束、特征结构或后续输出必须满足全部题目要求；诊断指标不折算为部分分数。"},"title":{"en":"The saved artifact decides success","zh":"保存的成果决定是否通过"}},{"id":"findings","source_ids":["project"],"text":{"en":"The best reported agent succeeds on 17.5% of tasks, versus an 87.0% expert reference. The expert row represents one reference pass, not a study of average engineer performance.","zh":"论文最优 Agent 的任务成功率为 17.5%，专家参考为 87.0%。专家行来自一次参考执行，并非对工程师平均能力的统计研究。"},"title":{"en":"What the published comparison shows","zh":"论文比较说明什么？"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Tasks provide explicit goals and assets. FreeCAD-only results do not establish performance in other CAD products, open-ended client projects, or requirements discovery.","zh":"题目提供明确目标与素材。FreeCAD 成绩不能直接代表其他 CAD 软件、开放式客户项目或需求发现能力。"},"title":{"en":"What remains outside the test","zh":"哪些能力仍在测试之外？"}},{"id":"reproduction","source_ids":["project"],"text":{"en":"The official site links the tasks, VM and installation guide. Reproduction needs Linux, KVM, Docker and the FreeCAD VM; preserve the harness, budget and trace alongside each reported result.","zh":"官方网站提供题集、虚拟机和安装指南。复现需要 Linux、KVM、Docker 与 FreeCAD 虚拟机；报告成绩时应保留运行框架、预算与轨迹。"},"title":{"en":"Start from a task and its checks","zh":"从任务与验收条件开始复现"}},{"id":"published-success","source_ids":["paper"],"table":{"caption":{"en":"Paper Table 3 · success column, full 200-task GUI experiment","zh":"论文表 3 · 成功率列，完整 200 题 GUI 实验"},"columns":[{"en":"Agent / reference","zh":"Agent／参考"},{"en":"Task success","zh":"任务成功率"}],"note":{"en":"Names and percentages follow the paper. The human row is a single expert reference pass; it is not an average across engineers. The terminal-only 50-task ablation is excluded.","zh":"名称与百分比保留论文写法。人类行是单次专家参考执行，并非工程师群体平均值；不含 50 题终端实验。"},"rows":[["GPT5.4 (CUA)","17.5%"],["Opus4.8 (CUA)","16.0%"],["Kimi K2.6","7.5%"],["OpenCUA","1.5%"],["Holo 3.1","0.5%"],["Qwen3.6","0.0%"],["MiniMax M3","0.0%"],["Human Expert reference","87.0%"]]},"text":{"en":"Success requires all task checks to pass. These are the success values from Table 3, distinct from its finish, cost and action-count columns.","zh":"成功要求通过任务全部检查。以下摘录表 3 的成功率，与表中的结束率、费用和动作数分别理解。"},"title":{"en":"Published task-success results","zh":"论文中的任务成功率"}}],"sources":[{"id":"paper","title":"CADWorld introducing paper · protocol, evaluation and limitations (arXiv v1)","url":"https://arxiv.org/html/2609.16251v1"},{"id":"project","title":"Authors’ project · tasks, evaluation and reproduction links","url":"https://cad-world.github.io/"},{"id":"code","title":"Authors’ task definitions and evaluation implementation","url":"https://github.com/Zdong104/CADWORLD"}],"summary":{"en":"A computer-use agent must leave a valid, editable FreeCAD project—not merely a plausible screenshot.","zh":"电脑操作 Agent 需要交付有效、可编辑的 FreeCAD 工程文件，让最终成果经得起程序检查。"},"title":"CADWorld"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16251","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_cadworld-computer-use-benchmark-for-long-h_103de6e3"},"modality":null,"name":"CADWorld","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16251","source_url":"https://arxiv.org/abs/2609.16251"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16251","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16251","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"CAFOSat is a dataset of over 45,000 image patches with infrastructure-level annotations for CAFO mapping across 20 states, benchmarking models for infrastructure-aware agricultural monitoring.","descriptions":{"en":"CAFOSat is a dataset of over 45,000 image patches with infrastructure-level annotations for CAFO mapping across 20 states, benchmarking models for infrastructure-aware agricultural monitoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00548","languages":[],"library_direction":null,"modality":null,"name":"CAFOSat","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00548","source_url":"https://arxiv.org/abs/2606.00548"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00548","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00548","topics":["multimodal"],"unit":null},{"aliases":["CalArena: A Large-Scale Post-Hoc Calibration Benchmark"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"CalArena is a standardized benchmark for comparing post-hoc calibration methods across tabular and computer-vision classification settings.","descriptions":{"en":"CalArena is a standardized benchmark for comparing post-hoc calibration methods across tabular and computer-vision classification settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30188","languages":[],"library_direction":null,"modality":null,"name":"CalArena","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30188","source_url":"https://arxiv.org/abs/2605.30188"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30188","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30188","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CallBench is a Chinese benchmark for evaluating dual-goal coordination in phone call assistants, with 50,000 multi-turn dialogues across six scenarios and a preset-aware turn-level scoring protocol.","descriptions":{"en":"CallBench is a Chinese benchmark for evaluating dual-goal coordination in phone call assistants, with 50,000 multi-turn dialogues across six scenarios and a preset-aware turn-level scoring protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.22635","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_callbench_09a4fc23"},"modality":null,"name":"CallBench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22635","source_url":"https://arxiv.org/abs/2607.22635"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22635","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22635","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CaLM is the first comprehensive benchmark for evaluating the causal reasoning capabilities of language models. The CaLM framework establishes a foundational taxonomy consisting of four modules: causal target, adaptation, metric, and error. CaLM是上海人工智能实验室联合同济大学、上海交通大学、北京大学及商汤科技发布首个大模型因果推理开放评测体系。首次从因果推理角度提出评估框架，为AI研究者打造可靠评测工具，从而为推进大模型认知能力向人类水平看齐提供指标参考。","descriptions":{"en":"CaLM is the first comprehensive benchmark for evaluating the causal reasoning capabilities of language models. The CaLM framework establishes a foundational taxonomy consisting of four modules: causal target, adaptation, metric, and error.","zh":"CaLM是上海人工智能实验室联合同济大学、上海交通大学、北京大学及商汤科技发布首个大模型因果推理开放评测体系。首次从因果推理角度提出评估框架，为AI研究者打造可靠评测工具，从而为推进大模型认知能力向人类水平看齐提供指标参考。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1052","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"CaLM","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-05-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1052-calm","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CaLM","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"CalorieBench-80K is a food image benchmark with calorie labels and dietary advice annotations, used to evaluate Food-R1 and other models for food analysis tasks.","descriptions":{"en":"CalorieBench-80K is a food image benchmark with calorie labels and dietary advice annotations, used to evaluate Food-R1 and other models for food analysis tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04986","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_caloriebench-80k_97ce8906"},"modality":null,"name":"CalorieBench-80K","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04986","source_url":"https://arxiv.org/abs/2606.04986"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04986","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04986","topics":["multimodal","medical"],"unit":null},{"aliases":["Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation"],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_temporally-grounded-compositional-camera-m_d9aa8053"},"categories":["Factuality","General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates temporally grounded compositional camera motion understanding using 4,229 single-shot clips with expert-annotated segments and 20 direction-aware labels.","descriptions":{"en":"Evaluates temporally grounded compositional camera motion understanding using 4,229 single-shot clips with expert-annotated segments and 20 direction-aware labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10932","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_temporally-grounded-compositional-camera-m_d9aa8053"},"modality":null,"name":"CamChoreo","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10932","source_url":"https://arxiv.org/abs/2608.10932"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10932","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10932","topics":[],"unit":null},{"aliases":["Can AI Make Money in Crypto? Measuring the Gap from Backtests to Real Markets"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:starlien95/awesome-tradingai","source_record":"bm_can-ai-make-money-in-crypto-measuring-the-_f6e196af"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates machine learning, reinforcement learning, LLM-based, and agent-based trading methods in cryptocurrency markets across three stages: historical backtesting, prospective exchange-based paper trading, and real-money live trading.","descriptions":{"en":"Evaluates machine learning, reinforcement learning, LLM-based, and agent-based trading methods in cryptocurrency markets across three stages: historical backtesting, prospective exchange-based paper trading, and real-money live trading."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.34510","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_can-ai-make-money-in-crypto-measuring-the-_f6e196af"},"modality":null,"name":"Can AI Make Money in Crypto?","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34510","source_url":"https://arxiv.org/abs/2609.34510"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34510","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34510","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","eess.AS"],"collected_at":"2026-10-14T09:44:51Z","description":"Audio-visual speech recognition on natural dyadic videoconference conversations.","descriptions":{"en":"Audio-visual speech recognition on natural dyadic videoconference conversations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10394","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_candor-lr_c4d723cf"},"modality":null,"name":"Candor-LR","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10394","source_url":"https://arxiv.org/abs/2609.10394"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10394","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10394","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"CanLegalRAGBench evaluates retrieval-augmented generation for Canadian case law with realistic queries and expert-annotated answers. It tests retrieval and answer generation grounded in legal documents.","descriptions":{"en":"CanLegalRAGBench evaluates retrieval-augmented generation for Canadian case law with realistic queries and expert-annotated answers. It tests retrieval and answer generation grounded in legal documents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30497","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","deep-research"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_canlegalragbench_9244f079"},"modality":null,"name":"CanLegalRAGBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30497","source_url":"https://arxiv.org/abs/2605.30497"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30497","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30497","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Proposes a benchmark for AI-generated operator code on Huawei's Ascend NPU, covering 53 operators and 1060 test cases. Evaluation uses a three-dimensional weighted composite score for compilation, functional correctness, and performance.","descriptions":{"en":"Proposes a benchmark for AI-generated operator code on Huawei's Ascend NPU, covering 53 operators and 1060 test cases. Evaluation uses a three-dimensional weighted composite score for compilation, functional correctness, and performance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20518","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","code-generation"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_cann-bench_f89ecd9f"},"modality":null,"name":"CANN Bench","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20518","source_url":"https://arxiv.org/abs/2607.20518"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20518","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20518","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CAP evaluates cross-site browser agents on 420 tasks across 108 real websites and 24 domains. Tasks require complex UI interactions and visual perception, with scoring via an agent-as-a-judge framework.","descriptions":{"en":"CAP evaluates cross-site browser agents on 420 tasks across 108 real websites and 24 domains. Tasks require complex UI interactions and visual perception, with scoring via an agent-as-a-judge framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08392","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_cap_c96b1f01"},"modality":null,"name":"CAP","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08392","source_url":"https://arxiv.org/abs/2608.08392"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08392","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08392","topics":["computer-use","agents"],"unit":null},{"aliases":["How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates semantic correctness of open-domain QA answers using 8.8k examples derived from widely used QA datasets, with a framework for answer verification.","descriptions":{"en":"Evaluates semantic correctness of open-domain QA answers using 8.8k examples derived from widely used QA datasets, with a framework for answer verification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01369","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_how-correct-is-your-answer-a-semantic-corr_98ea61f9"},"modality":null,"name":"CAP-Correctness","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01369","source_url":"https://arxiv.org/abs/2609.01369"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01369","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01369","topics":[],"unit":null},{"aliases":["capability-overload"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bizhanchik/capability-overload","source_record":"bm_capability-overload_41269a03"},"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates tool-selection reliability, task success, unnecessary tool calls, and token cost in LLM agents under varying catalog sizes (6-48) and semantic overlap levels (low/medium/high) across 40 tasks, with deterministic grading and multiple provider adapters.","descriptions":{"en":"Evaluates tool-selection reliability, task success, unnecessary tool calls, and token cost in LLM agents under varying catalog sizes (6-48) and semantic overlap levels (low/medium/high) across 40 tasks, with deterministic grading and multiple provider adapters."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:bizhanchik/capability-overload","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_capability-overload_41269a03"},"modality":null,"name":"Capability Overload Benchmark","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:bizhanchik/capability-overload","source_url":"https://github.com/bizhanchik/capability-overload"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-bizhanchik-capability-overload","source":"claire_radar","source_url":"https://github.com/bizhanchik/capability-overload","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CapMem is a benchmark for caption-based episodic memory in egocentric video with 1,000 multiple-choice questions.","descriptions":{"en":"CapMem is a benchmark for caption-based episodic memory in egocentric video with 1,000 multiple-choice questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17688","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","agent-memory","video-understanding"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_capmem_f25df26a"},"modality":null,"name":"CapMem","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17688","source_url":"https://arxiv.org/abs/2609.17688"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17688","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17688","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CapProbe is a full-scene dense QA benchmark for evaluating detailed image captions from Vision-Language Models. It decomposes images into semantic regions and generates multiple-choice questions across 10 semantic categories, with a language judge answering from captions. The benchmark comprises 346 images, 1,868 regions, and 25,650 questions.","descriptions":{"en":"CapProbe is a full-scene dense QA benchmark for evaluating detailed image captions from Vision-Language Models. It decomposes images into semantic regions and generates multiple-choice questions across 10 semantic categories, with a language judge answering from captions. The benchmark comprises 346 images, 1,868 regions, and 25,650 questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.11074","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_capprobe_99438ec2"},"modality":null,"name":"CapProbe","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11074","source_url":"https://arxiv.org/abs/2608.11074"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11074","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11074","topics":[],"unit":null},{"aliases":["Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video caption quality through human-verified multiple-choice questions across 10 types and 24 domains, measuring information fidelity as coverage and factuality.","descriptions":{"en":"Evaluates video caption quality through human-verified multiple-choice questions across 10 types and 24 domains, measuring information fidelity as coverage and factuality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09973","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_putting-captions-to-the-test-evaluating-vi_efc92f10"},"modality":null,"name":"CapQuiz","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09973","source_url":"https://arxiv.org/abs/2609.09973"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09973","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09973","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:xlchen0205/capricorn-1k","source_record":"bm_capricorn-1k_86185f2f"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CapRiCorn-1K evaluates video captioning quality and subject referential consistency across long videos (15s-10min) with audiovisual and visual-only settings. It uses LLM judge to compute accuracy, coverage, and referential consistency metrics based on manual annotations.","descriptions":{"en":"CapRiCorn-1K evaluates video captioning quality and subject referential consistency across long videos (15s-10min) with audiovisual and visual-only settings. It uses LLM judge to compute accuracy, coverage, and referential consistency metrics based on manual annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.21949","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_capricorn-1k_86185f2f"},"modality":null,"name":"CapRiCorn-1K","openness":"unknown","publisher":null,"released":"2026-06-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21949","source_url":"https://arxiv.org/abs/2606.21949"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21949","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21949","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Capture-the-Flag Challenges is OpenAI's internal expansion of competitive, professional-level cybersecurity capture-the-flag tasks used to evaluate vulnerability identification and exploitation capability under the Preparedness Framework.","descriptions":{"en":"Capture-the-Flag Challenges is OpenAI's internal expansion of competitive, professional-level cybersecurity capture-the-flag tasks used to evaluate vulnerability identification and exploitation capability under the Preparedness Framework."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:capture-the-flag-challenges:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/capture-the-flag-challenges?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:capture-the-flag-challenges","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"llm_stats_id","primary":"cybersecurity","source_record":"catalog_9d542e1869409972"},"modality":"text","name":"Capture-the-Flag Challenges (Internal)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":96.7,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.967,"raw_min":0.852,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:capture-the-flag-challenges:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/capture-the-flag-challenges?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-capture-the-flag-challenges","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/capture-the-flag-challenges?top_n=500","topics":["coding","agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":113,"github_stars_30d":9,"hf_dataset_downloads":null,"hf_paper_upvotes":46,"join":"claire_record_id","repository":"gh:lijayutnt/shuttermuse","source_record":"bm_captureguide-bench_fb3878c6"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CaptureGuide-Bench evaluates capture-time photography guidance with two tasks: photographer-side composition decision/refinement and subject-side scene-conditioned pose recommendation, using metrics like IoU and plausibility.","descriptions":{"en":"CaptureGuide-Bench evaluates capture-time photography guidance with two tasks: photographer-side composition decision/refinement and subject-side scene-conditioned pose recommendation, using metrics like IoU and plausibility."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.25763","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_captureguide-bench_fb3878c6"},"modality":null,"name":"CaptureGuide-Bench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25763","source_url":"https://arxiv.org/abs/2606.25763"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25763","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25763","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CardioLens evaluates MLLMs on multi-sequence cardiac MRI interpretation, covering image understanding, report generation, and disease diagnosis using QA pairs from private hospital archives.","descriptions":{"en":"CardioLens evaluates MLLMs on multi-sequence cardiac MRI interpretation, covering image understanding, report generation, and disease diagnosis using QA pairs from private hospital archives."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00123","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_cardiolens_7981754b"},"modality":null,"name":"CardioLens","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00123","source_url":"https://arxiv.org/abs/2606.00123"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00123","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00123","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CARE-Bench is a source-grounded benchmark for patient-facing triage evaluation with 500 cases and 1,059 patient-disclosure prefixes. It evaluates LLMs on a four-label current-action task across 269 held-out rounds.","descriptions":{"en":"CARE-Bench is a source-grounded benchmark for patient-facing triage evaluation with 500 cases and 1,059 patient-disclosure prefixes. It evaluates LLMs on a four-label current-action task across 269 held-out rounds."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03731","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_care-bench_02aa6f70"},"modality":null,"name":"CARE-Bench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03731","source_url":"https://arxiv.org/abs/2608.03731"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03731","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03731","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_carebench_48d01c35"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"CAREBench evaluates language models on upstream child-safety risks with 500 prompts across twelve categories, assessing recognition, refusal, de-escalation, and redirection.","descriptions":{"en":"CAREBench evaluates language models on upstream child-safety risks with 500 prompts across twelve categories, assessing recognition, refusal, de-escalation, and redirection."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.29685","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_carebench_48d01c35"},"modality":null,"name":"CAREBench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29685","source_url":"https://arxiv.org/abs/2606.29685"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29685","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29685","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_cast-bench_2c1a3e97"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"CaST-Bench evaluates causal chain-grounded spatio-temporal reasoning in video question answering. It contains 2,066 questions over 1,015 videos with causal chains annotated as temporal segments and bounding-box tracks. The evaluation suite includes metrics for answer correctness and visual evidence grounding.","descriptions":{"en":"CaST-Bench evaluates causal chain-grounded spatio-temporal reasoning in video question answering. It contains 2,066 questions over 1,015 videos with causal chains annotated as temporal segments and bounding-box tracks. The evaluation suite includes metrics for answer correctness and visual evidence grounding."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.23216","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","abstract-reasoning"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_cast-bench_2c1a3e97"},"modality":null,"name":"CaST-Bench","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23216","source_url":"https://arxiv.org/abs/2605.23216"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23216","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23216","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_caster-bench_6f29d748"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CASTER-Bench evaluates whether user-generated content achieves positive community resonance based on multimodal attributes, using human annotations across diverse categories.","descriptions":{"en":"CASTER-Bench evaluates whether user-generated content achieves positive community resonance based on multimodal attributes, using human annotations across diverse categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01897","languages":[],"library_direction":null,"modality":null,"name":"CASTER-Bench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01897","source_url":"https://arxiv.org/abs/2606.01897"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01897","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01897","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:thuais-lab/catch","source_record":"bm_catch_1310e8c8"},"categories":["Cybersecurity","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"During reinforcement learning with verifiable rewards (RLVR), large language models (LLMs) can exploit loopholes in their environments to obtain high rewards without improving the intended capabilities, i.e., reward hacking.","descriptions":{"en":"During reinforcement learning with verifiable rewards (RLVR), large language models (LLMs) can exploit loopholes in their environments to obtain high rewards without improving the intended capabilities, i.e., reward hacking."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.39533","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_catch_1310e8c8"},"modality":null,"name":"CATCH","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39533","source_url":"https://arxiv.org/abs/2609.39533"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39533","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39533","topics":["safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":11,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:yzhao062/catchbench","source_record":"bm_catchbench_f09872b4"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Assesses agent failure auditing across PRE, LIVE, and POST information states with seven task contracts covering evidential and Gold-derived diagnostics.","descriptions":{"en":"Assesses agent failure auditing across PRE, LIVE, and POST information states with seven task contracts covering evidential and Gold-derived diagnostics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.22808","languages":[],"library_direction":null,"modality":null,"name":"CatchBench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22808","source_url":"http://arxiv.org/abs/2608.22808v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22808","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22808v1","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":6,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:thusi-lab/causal-reasoner","source_record":"bm_causal-plan-bench_5fd0e720"},"categories":["Planning","Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Causal-Plan-Bench evaluates embodied planning across four causal dimensions (executability, effects, composition, robustness) using 1,200 instances from 12 tasks, with MCQ and rubric-based scoring.","descriptions":{"en":"Causal-Plan-Bench evaluates embodied planning across four causal dimensions (executability, effects, composition, robustness) using 1,200 instances from 12 tasks, with MCQ and rubric-based scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01810","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","abstract-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_causal-plan-bench_5fd0e720"},"modality":null,"name":"Causal-Plan-Bench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01810","source_url":"https://arxiv.org/abs/2606.01810"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01810","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01810","topics":["reasoning-math"],"unit":null},{"aliases":["A Plausible Future Is Not a Causal Future: Can Video Models Generate How the World Unfolds?"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sjtu-deepdeltalab/causalcascade","source_record":"bm_a-plausible-future-is-not-a-causal-future-_b86aa025"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A simulator-grounded benchmark for assessing causal cascade generation in image-to-video models through event-rich scenes, motion metrics, and diagnostic question answering.","descriptions":{"en":"A simulator-grounded benchmark for assessing causal cascade generation in image-to-video models through event-rich scenes, motion metrics, and diagnostic question answering."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.15562","languages":[],"library_direction":{"basis":"model-review-daily","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_a-plausible-future-is-not-a-causal-future-_b86aa025"},"modality":null,"name":"CausalCascade","openness":"unknown","publisher":null,"released":"2026-10-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.15562","source_url":"https://arxiv.org/abs/2610.15562"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-15562","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15562","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:andleb/causalds","source_record":"bm_causalds_e9c95fdf"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates causal reasoning in data-science workflows using synthetic scenes with hidden structural causal models, covering Pearl's three rungs and abstention scoring.","descriptions":{"en":"Evaluates causal reasoning in data-science workflows using synthetic scenes with hidden structural causal models, covering Pearl's three rungs and abstention scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.08093","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","professional-work"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_causalds_e9c95fdf"},"modality":null,"name":"CausalDS","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08093","source_url":"https://arxiv.org/abs/2607.08093"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08093","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08093","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"CausalGame is a benchmark for evaluating causal thinking in LLM agents through interactive games. It includes 14 scenarios with selection bias, measurement error, and hidden confounders. Agents design experimental protocols, collect data, and produce solutions with explanations, scored against analytical optima and causal-reasoning rubrics.","descriptions":{"en":"CausalGame is a benchmark for evaluating causal thinking in LLM agents through interactive games. It includes 14 scenarios with selection bias, measurement error, and hidden confounders. Agents design experimental protocols, collect data, and produce solutions with explanations, scored against analytical optima and causal-reasoning rubrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.04293","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","multi-agent-games"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_causalgame_db1d5cd5"},"modality":null,"name":"CausalGame","openness":"unknown","publisher":null,"released":"2026-07-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04293","source_url":"https://arxiv.org/abs/2607.04293"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04293","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04293","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","VQA","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CausalVQA tests causal reasoning in videos across five question types, and state-of-the-art multimodal models still trail human performance. CausalVQA 是面向视频问答的因果推理基准，涵盖反事实、假设、预判、规划、描述五类问题，强调真实物理场景。","descriptions":{"en":"CausalVQA tests causal reasoning in videos across five question types, and state-of-the-art multimodal models still trail human performance.","zh":"CausalVQA 是面向视频问答的因果推理基准，涵盖反事实、假设、预判、规划、描述五类问题，强调真实物理场景。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1975","languages":[],"library_direction":null,"modality":"multimodal","name":"CausalVQA","openness":"unknown","publisher":"FAIR at Meta","released":"2025-06-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1975-causalvqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CausalVQA","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Robustness","Safety & Trustworthiness","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"AI voice assistants often use Automatic Speech Recognition (ASR) with LLM-based reasoning, yet existing systems struggle with regional British accents, including Scottish, Irish, and Welsh accents, since most ASR models are trained predomi…","descriptions":{"en":"AI voice assistants often use Automatic Speech Recognition (ASR) with LLM-based reasoning, yet existing systems struggle with regional British accents, including Scottish, Irish, and Welsh accents, since most ASR models are trained predomi…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.06587","languages":[],"library_direction":{"basis":"model-review-daily","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_cavabench_d5de77f0"},"modality":null,"name":"CavaBench","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.06587","source_url":"https://arxiv.org/abs/2610.06587"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-06587","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06587","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LLM agents increasingly keep working after a task succeeds as they resume after compaction or take over handoffs.","descriptions":{"en":"LLM agents increasingly keep working after a task succeeds as they resume after compaction or take over handoffs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32616","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_cave-bench_0b7f70f1"},"modality":null,"name":"CAVE-Bench","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32616","source_url":"https://arxiv.org/abs/2609.32616"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32616","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32616","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for computer-use agents in incentive-misaligned environments with nine marketplace environments and eight steering mechanisms.","descriptions":{"en":"Benchmark for computer-use agents in incentive-misaligned environments with nine marketplace environments and eight steering mechanisms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27273","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_caveat_461c1b44"},"modality":null,"name":"CAVEAT","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27273","source_url":"https://arxiv.org/abs/2609.27273"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27273","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27273","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Curriculum Learning of Bayesian Network Structures (CBNSL) benchmark for evaluating algorithms that learn Bayesian network structures from data using curriculum learning techniques. The benchmark uses networks from the bnlearn repository and evaluates structure learning performance using BDeu scoring metrics.","descriptions":{"en":"Curriculum Learning of Bayesian Network Structures (CBNSL) benchmark for evaluating algorithms that learn Bayesian network structures from data using curriculum learning techniques. The benchmark uses networks from the bnlearn repository and evaluates structure learning performance using BDeu scoring metrics."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cbnsl:kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cbnsl?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cbnsl","languages":[],"library_direction":null,"modality":"text","name":"CBNSL","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":95.6,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.956,"raw_min":0.956,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cbnsl:kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cbnsl?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"}],"slug":"llm-stats-cbnsl","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cbnsl?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:meric-karadag/cbx-bench","source_record":"bm_cbx-bench_5fdbe064"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CBX-Bench is a benchmark for quantitatively measuring the quality of Concept Bottleneck Model (CBM) explanations. It uses a council of five open-weight multimodal LLMs to score explanations given an image and class, validated against human preferences. The benchmark maintains a leaderboard for CBM explanation quality.","descriptions":{"en":"CBX-Bench is a benchmark for quantitatively measuring the quality of Concept Bottleneck Model (CBM) explanations. It uses a council of five open-weight multimodal LLMs to score explanations given an image and class, validated against human preferences. The benchmark maintains a leaderboard for CBM explanation quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.15404","languages":[],"library_direction":null,"modality":null,"name":"CBX-Bench","openness":"unknown","publisher":null,"released":"2026-08-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15404","source_url":"https://arxiv.org/abs/2608.15404"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15404","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15404","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CC-Bench-V2 Backend evaluates coding agents on backend development tasks, measuring practical engineering ability to implement server-side logic, APIs, and system components.","descriptions":{"en":"CC-Bench-V2 Backend evaluates coding agents on backend development tasks, measuring practical engineering ability to implement server-side logic, APIs, and system components."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cc-bench-v2-backend:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-backend?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cc-bench-v2-backend","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_4ad13fc4bbc17737"},"modality":"text","name":"CC-Bench-V2 Backend","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":22.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.228,"raw_min":0.228,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cc-bench-v2-backend:glm-5v-turbo","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-backend?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"}],"slug":"llm-stats-cc-bench-v2-backend","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-backend?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CC-Bench-V2 Frontend evaluates coding agents on frontend development tasks, measuring ability to build UI components, handle styling, and implement client-side logic.","descriptions":{"en":"CC-Bench-V2 Frontend evaluates coding agents on frontend development tasks, measuring ability to build UI components, handle styling, and implement client-side logic."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cc-bench-v2-frontend:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-frontend?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cc-bench-v2-frontend","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_f4ff6092bc298091"},"modality":"text","name":"CC-Bench-V2 Frontend","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":68.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.684,"raw_min":0.684,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cc-bench-v2-frontend:glm-5v-turbo","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-frontend?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"}],"slug":"llm-stats-cc-bench-v2-frontend","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-frontend?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CC-Bench-V2 Repo Exploration evaluates coding agents on repository-level understanding and navigation, measuring ability to explore, comprehend, and work across entire codebases.","descriptions":{"en":"CC-Bench-V2 Repo Exploration evaluates coding agents on repository-level understanding and navigation, measuring ability to explore, comprehend, and work across entire codebases."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cc-bench-v2-repo:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-repo?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cc-bench-v2-repo","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_30468f05c76e5998"},"modality":"text","name":"CC-Bench-V2 Repo Exploration","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":72.2,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.722,"raw_min":0.722,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cc-bench-v2-repo:glm-5v-turbo","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-repo?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"}],"slug":"llm-stats-cc-bench-v2-repo","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-repo?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_47c588cfffae91bf"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","structured_output","text-to-image","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive OCR benchmark for evaluating Large Multimodal Models (LMMs) in literacy. Comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. Contains 39 subsets with 7,058 fully annotated images, 41% sourced from real applications. Tests capabilities including text grounding, multi-orientation text recognition, and detecting hallucination/repetition across diverse visual challenges.","descriptions":{"en":"A comprehensive OCR benchmark for evaluating Large Multimodal Models (LMMs) in literacy. Comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. Contains 39 subsets with 7,058 fully annotated images, 41% sourced from real applications. Tests capabilities including text grounding, multi-orientation text recognition, and detecting hallucination/repetition across diverse visual challenges."},"evidence_summary":{"document_count":1,"model_count":18,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cc-ocr:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-ocr?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cc-ocr","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_47c588cfffae91bf"},"modality":"multimodal","name":"CC-OCR","openness":"unknown","publication":{"evidence":[{"note":"The introducing paper specifies the four OCR tracks, 39 subsets and 7,058 annotated images, matching the exact source record's description.","url":"https://arxiv.org/abs/2412.02210"},{"note":"The benchmark directory identifies itself as the dataset and evaluation code for this paper, links arXiv:2412.02210 and the wulipc/CC-OCR dataset, and records its 2024-12-26 release.","url":"https://github.com/AlibabaResearch/AdvancedLiterateMachinery/blob/main/Benchmarks/CC-OCR/README.md"}],"id":"cc-ocr-publication","links":[{"kind":"paper","title":"CC-OCR introducing paper","url":"https://arxiv.org/abs/2412.02210"},{"kind":"repo","title":"CC-OCR official code and data instructions","url":"https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/Benchmarks/CC-OCR"},{"kind":"dataset","title":"CC-OCR dataset linked by the authors","url":"https://huggingface.co/datasets/wulipc/CC-OCR"}],"record_keys":["llm-stats:cc-ocr"],"reviewed_at":"2026-10-26","scope":{"en":"The original CC-OCR benchmark with four OCR tracks, not CC-OCR V2. The code link opens the benchmark's own directory within the authors' shared repository.","zh":"原始 CC-OCR 四类 OCR 任务评测，不是 CC-OCR V2。代码链接直接打开作者共享仓库中的该评测专用目录。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":18,"score_direction":"higher_is_better","score_summary":{"display_max":83.39999999999999,"display_multiplier":100,"model_count":18,"model_count_basis":"source_model_id","numeric_count":18,"raw_max":0.834,"raw_min":0.738,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cc-ocr:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-ocr?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-cc-ocr","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-ocr?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:0815linyuan/ccrv-bench-constraint-based-evaluation-of-causal-reasoning-in-vision-language-models","source_record":"bm_ccrv-bench_1387b4d1"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates visual causal reasoning under constraints in single-image physical scenarios.","descriptions":{"en":"Evaluates visual causal reasoning under constraints in single-image physical scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.30979","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning","instruction-following","visual-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_ccrv-bench_1387b4d1"},"modality":null,"name":"CCRV-Bench","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30979","source_url":"https://arxiv.org/abs/2609.30979"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30979","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30979","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["cdaf"],"attention_signals":{"as_of":"2026-10-11","github_stars":219,"github_stars_30d":92,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:uditakhourii/cdaf","source_record":"bm_cdaf_17e7d375"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark compares sidecar-based video understanding against direct analysis on accuracy, prompt token usage, and latency using a reproducible setup with Gemini.","descriptions":{"en":"Benchmark compares sidecar-based video understanding against direct analysis on accuracy, prompt token usage, and latency using a reproducible setup with Gemini."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:uditakhourii/cdaf","languages":[],"library_direction":{"basis":"model-review-2","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_cdaf_17e7d375"},"modality":null,"name":"CDAF — Cached Descriptive Asset Files","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:uditakhourii/cdaf","source_url":"https://github.com/UditAkhourii/cdaf"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-uditakhourii-cdaf","source":"claire_radar","source_url":"https://github.com/UditAkhourii/cdaf","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_cdr-bench_40384718"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CDR-Bench is a benchmark of 3,462 tasks for evaluating large language models on faithful execution of compositional, order-sensitive data refinement recipes across four domains and 29 operators, with deterministic reference outputs for exact evaluation.","descriptions":{"en":"CDR-Bench is a benchmark of 3,462 tasks for evaluating large language models on faithful execution of compositional, order-sensitive data refinement recipes across four domains and 29 operators, with deterministic reference outputs for exact evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.31435","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","professional-work"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_cdr-bench_40384718"},"modality":null,"name":"CDR-Bench","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31435","source_url":"https://arxiv.org/abs/2606.31435"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31435","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31435","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CEB evaluates LLM bias compositionally, featuring 11k samples characterized across bias types, social groups, and tasks. CEB是一个用于大型语言模型偏差的组成评估基准，引入了包含 11,004 个样本的组成评估基准，从偏差类型、社会群体和任务三个维度描述每个数据集。","descriptions":{"en":"CEB evaluates LLM bias compositionally, featuring 11k samples characterized across bias types, social groups, and tasks.","zh":"CEB是一个用于大型语言模型偏差的组成评估基准，引入了包含 11,004 个样本的组成评估基准，从偏差类型、社会群体和任务三个维度描述每个数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1552","languages":[],"library_direction":null,"modality":null,"name":"CEB","openness":"restricted","publisher":"University of Virginia, Arizona State University, etc.","released":"2024-07-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1552-ceb","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CEB","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CEO-Bench evaluates LLM agents on strategic resource allocation in multi-round organizational simulations with conflicting advisor inputs.","descriptions":{"en":"CEO-Bench evaluates LLM agents on strategic resource allocation in multi-round organizational simulations with conflicting advisor inputs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17459","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_ceo-bench_8522cfe6"},"modality":null,"name":"CEO-Bench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17459","source_url":"https://arxiv.org/abs/2606.17459"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17459","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17459","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":82,"github_stars_30d":10,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":"gh:zlab-princeton/ceobench-src","source_record":"bm_ceo-bench_5cae7f96"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CEO-Bench evaluates long-horizon agent capabilities by simulating a startup over 500 days. Agents manage pricing, marketing, budgeting, and other business aspects through a programmable Python interface, facing noisy data and changing market conditions. Performance is measured by final company balance against a rule-based baseline.","descriptions":{"en":"CEO-Bench evaluates long-horizon agent capabilities by simulating a startup over 500 days. Agents manage pricing, marketing, budgeting, and other business aspects through a programmable Python interface, facing noisy data and changing market conditions. Performance is measured by final company balance against a rule-based baseline."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.18543","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","professional-work"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_ceo-bench_5cae7f96"},"modality":null,"name":"CEO-Bench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18543","source_url":"https://arxiv.org/abs/2606.18543"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18543","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18543","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Lists 380 expert-written items across six sub-domains of cryptographic engineering security for IoT devices.","descriptions":{"en":"Lists 380 expert-written items across six sub-domains of cryptographic engineering security for IoT devices."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.21344","languages":[],"library_direction":null,"modality":null,"name":"CESBench","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21344","source_url":"https://arxiv.org/abs/2609.21344"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21344","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21344","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","Finance"],"collected_at":"2026-10-14T09:44:51Z","description":"CFAgentBench is an executable environment for autonomous construction-finance agents, with 1,014 task specifications across 8 domains and 77 families. A subset of 40 tasks (54 with PM extension) has oracle-validated evaluators. Grading uses state diffs, forbidden-side-effect checks, and required-output regexes, with an LLM judge only for reply quality. A public split of 711 tasks is available, and a private split of 303 is reserved for remote scoring.","descriptions":{"en":"CFAgentBench is an executable environment for autonomous construction-finance agents, with 1,014 task specifications across 8 domains and 77 families. A subset of 40 tasks (54 with PM extension) has oracle-validated evaluators. Grading uses state diffs, forbidden-side-effect checks, and required-output regexes, with an LLM judge only for reply quality. A public split of 711 tasks is available, and a private split of 303 is reserved for remote scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.22000","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","finance-legal"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_cfagentbench_9ccf4efc"},"modality":null,"name":"CFAgentBench","openness":"unknown","publisher":null,"released":"2026-06-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22000","source_url":"https://arxiv.org/abs/2606.22000"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22000","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22000","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CFEval benchmark for evaluating code generation and problem-solving capabilities","descriptions":{"en":"CFEval benchmark for evaluating code generation and problem-solving capabilities"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cfeval:qwen3-235b-a22b-thinking-2507","reported_at":"2025-07-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cfeval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cfeval","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_27a86d8712b30998"},"modality":"text","name":"CFEval","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":2134.0,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":2134.0,"raw_min":2071.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cfeval:qwen3-235b-a22b-thinking-2507","reported_date":"2025-07-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cfeval?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-cfeval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cfeval?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","NAACL 2025","金融","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"We present CFinBench: a meticulously crafted, the most comprehensive evaluation benchmark to date, for assessing the financial knowledge of LLMs under Chinese context. 为了更加全面地探究大语言模型在中文财经领域的能力，本工作提出了目前为止量级最大的中文财经评测基准（CFinBench）。该数据集共包含99,100个评测样本，并包含单选题、多选题和判断题在内的三种题型。该工作对当前主流的大模型从四个维度进行了详细评测：财经学科基础、财经资格认证、财经从业实践、财经法律法规。数据集和测评代码均已开源。","descriptions":{"en":"We present CFinBench: a meticulously crafted, the most comprehensive evaluation benchmark to date, for assessing the financial knowledge of LLMs under Chinese context.","zh":"为了更加全面地探究大语言模型在中文财经领域的能力，本工作提出了目前为止量级最大的中文财经评测基准（CFinBench）。该数据集共包含99,100个评测样本，并包含单选题、多选题和判断题在内的三种题型。该工作对当前主流的大模型从四个维度进行了详细评测：财经学科基础、财经资格认证、财经从业实践、财经法律法规。数据集和测评代码均已开源。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":true,"key":"opencompass:1907","languages":["Chinese"],"library_direction":null,"modality":null,"name":"CFinBench","openness":"unknown","publisher":"华为, 新加坡南洋理工","released":"2024-10-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1907-cfinbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CFinBench","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","ACL 2024","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CFLUE is the Chinese Financial Language Understanding Evaluation benchmark, designed to assess the capability of LLMs across various dimensions. CFLUE 是中国金融语言理解评估基准，旨在评估大型语言模型（LLMs）在各个维度上的能力。具体而言，CFLUE 提供了针对知识评估和应用评估量身定制的数据集。在知识评估方面，它包含超过 38,000 道选择题及相关的解决方案解释。","descriptions":{"en":"CFLUE is the Chinese Financial Language Understanding Evaluation benchmark, designed to assess the capability of LLMs across various dimensions.","zh":"CFLUE 是中国金融语言理解评估基准，旨在评估大型语言模型（LLMs）在各个维度上的能力。具体而言，CFLUE 提供了针对知识评估和应用评估量身定制的数据集。在知识评估方面，它包含超过 38,000 道选择题及相关的解决方案解释。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1079","languages":[],"library_direction":null,"modality":null,"name":"CFLUE","openness":"unknown","publisher":"Alibaba","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1079-cflue","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CFLUE","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object is unclear from the provided information.","descriptions":{"en":"The evaluation object is unclear from the provided information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14975","languages":[],"library_direction":null,"modality":null,"name":"CFM-Bench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14975","source_url":"https://arxiv.org/abs/2607.14975"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14975","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14975","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CG-Bench is meant for evaluating MLLMs' long video understanding, including 12,129 QA pairs from 1219 videos in 3 major question types: perception, reasoning, and hallucination. CG-Bench用于评估多模态大模型的长视频理解能力，基于1219个视频设计了12129个涵盖感知、推理和幻觉三种问题类型的QA对。","descriptions":{"en":"CG-Bench is meant for evaluating MLLMs' long video understanding, including 12,129 QA pairs from 1219 videos in 3 major question types: perception, reasoning, and hallucination.","zh":"CG-Bench用于评估多模态大模型的长视频理解能力，基于1219个视频设计了12129个涵盖感知、推理和幻觉三种问题类型的QA对。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1513","languages":[],"library_direction":null,"modality":"multimodal","name":"CG-Bench","openness":"restricted","publisher":"Nanjing University","released":"2024-12-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1513-cg-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CG-Bench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"ChainSWE evaluates coding agents on sequential, dependent bug fixes within a shared codebase. It includes 304 issues across 54 Python projects, forming chronological chains, and measures performance drop as chain length increases.","descriptions":{"en":"ChainSWE evaluates coding agents on sequential, dependent bug fixes within a shared codebase. It includes 304 issues across 54 Python projects, forming chronological chains, and measures performance drop as chain length increases."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02606","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_chainswe_b0efa40a"},"modality":null,"name":"ChainSWE","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02606","source_url":"https://arxiv.org/abs/2607.02606"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02606","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02606","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"ChaosBench-Logic v2 evaluates LLM logical reasoning over dynamical systems with 40,886 questions across 165 systems, 27 first-order logic predicates, and 78 axiom edges. The CARE protocol measures calibration and adversarial robustness, reporting metrics such as MCC.","descriptions":{"en":"ChaosBench-Logic v2 evaluates LLM logical reasoning over dynamical systems with 40,886 questions across 165 systems, 27 first-order logic predicates, and 78 axiom edges. The CARE protocol measures calibration and adversarial robustness, reporting metrics such as MCC."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24305","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","science"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_chaosbench-logic_c5cd44be"},"modality":null,"name":"ChaosBench-Logic","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24305","source_url":"https://arxiv.org/abs/2605.24305"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24305","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24305","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_262332403b3cb325"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","language","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Charades-STA is a benchmark dataset for temporal activity localization via language queries, extending the Charades dataset with sentence temporal annotations. It contains 12,408 training and 3,720 testing segment-sentence pairs from videos with natural language descriptions and precise temporal boundaries for localizing activities based on language queries.","descriptions":{"en":"Charades-STA is a benchmark dataset for temporal activity localization via language queries, extending the Charades dataset with sentence temporal annotations. It contains 12,408 training and 3,720 testing segment-sentence pairs from videos with natural language descriptions and precise temporal boundaries for localizing activities based on language queries."},"evidence_summary":{"document_count":1,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:charadessta:qwen2.5-vl-7b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charadessta?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:charadessta","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_262332403b3cb325"},"modality":"multimodal","name":"CharadesSTA","openness":"unknown","publisher":null,"released":"2017-05-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: TALL: Temporal Activity Localization via Language Query. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:charadessta","source_url":"https://arxiv.org/abs/1705.02101"},"repo_kind":null,"repo_resolution_status":null,"score_count":12,"score_direction":"higher_is_better","score_summary":{"display_max":64.8,"display_multiplier":100,"model_count":12,"model_count_basis":"source_model_id","numeric_count":12,"raw_max":0.648,"raw_min":0.436,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:charadessta:qwen3-vl-235b-a22b-instruct","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charadessta?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-charadessta","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charadessta?top_n=500","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","ACL 2024","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CHARM is the first benchmark for comprehensively and in-depth evaluating the commonsense reasoning ability of large language models (LLMs) in Chinese, which covers both globally known and Chinese-specific commonsense. CHARM 是首个全面深入评估大语言模型（LLMs）在中文中的常识推理能力的基准，涵盖了全球通用的常识和特定于中国的常识。","descriptions":{"en":"CHARM is the first benchmark for comprehensively and in-depth evaluating the commonsense reasoning ability of large language models (LLMs) in Chinese, which covers both globally known and Chinese-specific commonsense.","zh":"CHARM 是首个全面深入评估大语言模型（LLMs）在中文中的常识推理能力的基准，涵盖了全球通用的常识和特定于中国的常识。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1141","languages":["Chinese"],"library_direction":null,"modality":null,"name":"CHARM","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1141-charm","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CHARM","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":14,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"claire_record_id","repository":"gh:wulin-wulin/osworld_chart","source_record":"bm_chartact_f82f233a"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Interactive benchmark for dynamic chart understanding requiring GUI actions to obtain evidence, with 673 charts and 1,440 QA pairs.","descriptions":{"en":"Interactive benchmark for dynamic chart understanding requiring GUI actions to obtain evidence, with 673 charts and 1,440 QA pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26994","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","computer-use"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_chartact_f82f233a"},"modality":null,"name":"ChartAct","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26994","source_url":"https://arxiv.org/abs/2605.26994"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26994","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26994","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_chartanno_e949f852"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ChartAnno is a benchmark for evaluating MLLMs on chart annotation generation with 1,200 real-world charts and paired code/instructions. It evaluates annotation quality under different instruction specificity and input settings.","descriptions":{"en":"ChartAnno is a benchmark for evaluating MLLMs on chart annotation generation with 1,200 real-world charts and paired code/instructions. It evaluates annotation quality under different instruction specificity and input settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03464","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","code-generation"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_chartanno_e949f852"},"modality":null,"name":"ChartAnno","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03464","source_url":"https://arxiv.org/abs/2608.03464"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03464","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03464","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":8,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":"gh:pspdada/chartarena","source_record":"bm_chartarena_4eca7a75"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ChartArena is a bilingual benchmark for chart parsing covering eight chart families (bar, line, pie, radar, box plot, combination, flowchart, mind map) across three visual scenarios (digital, printed, hand-drawn). It uses a format-agnostic evaluation protocol with structure-aware metrics.","descriptions":{"en":"ChartArena is a bilingual benchmark for chart parsing covering eight chart families (bar, line, pie, radar, box plot, combination, flowchart, mind map) across three visual scenarios (digital, printed, hand-drawn). It uses a format-agnostic evaluation protocol with structure-aware metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01348","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","visual-reasoning"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_chartarena_4eca7a75"},"modality":null,"name":"ChartArena","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01348","source_url":"https://arxiv.org/abs/2606.01348"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01348","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01348","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code generation","Multimodal","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LMM judges in chart-to-code workflows.","descriptions":{"en":"Evaluates LMM judges in chart-to-code workflows."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24210","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_chartjudgebench_09ef41ac"},"modality":null,"name":"ChartJudgeBench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24210","source_url":"https://arxiv.org/abs/2609.24210"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24210","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24210","topics":["coding","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":25,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"llm_stats_id","repository":null,"source_record":"catalog_85df86ec25b07e1b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ChartMuseum is a chart question-answering benchmark of 1,162 expert-annotated questions over real-world chart images drawn from 184 sources, including academic figures, infographics, and unconventional chart designs. It specifically targets questions that require visual reasoning, such as comparing unlabeled visual elements, tracking trajectories, and judging spatial relationships.","descriptions":{"en":"ChartMuseum is a chart question-answering benchmark of 1,162 expert-annotated questions over real-world chart images drawn from 184 sources, including academic figures, infographics, and unconventional chart designs. It specifically targets questions that require visual reasoning, such as comparing unlabeled visual elements, tracking trajectories, and judging spatial relationships."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:chartmuseum:claude-sonnet-5","reported_at":"2026-06-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartmuseum?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:chartmuseum","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_85df86ec25b07e1b"},"modality":"multimodal","name":"ChartMuseum","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":86.7,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.867,"raw_min":0.867,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:chartmuseum:claude-sonnet-5","reported_date":"2026-06-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartmuseum?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"}],"slug":"llm-stats-chartmuseum","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartmuseum?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Manufacturing & Process Control","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Chartography evaluates vision-language models on 100 chart-understanding tasks sourced from professional practice, with questions authored by domain experts and triple-verified. The benchmark uses pass@1 accuracy as the scoring metric across 30 frontier model configurations.","descriptions":{"en":"Chartography evaluates vision-language models on 100 chart-understanding tasks sourced from professional practice, with questions authored by domain experts and triple-verified. The benchmark uses pass@1 accuracy as the scoring metric across 30 frontier model configurations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10677","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_chartography_26c993c6"},"modality":null,"name":"Chartography","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10677","source_url":"https://arxiv.org/abs/2608.10677"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10677","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10677","topics":["finance"],"unit":null},{"aliases":["Chartography"],"attention_signals":null,"categories":["vision"],"collected_at":null,"description":"Chart comprehension benchmark with tool access. Scores depend on context length and tool configuration.","descriptions":{"en":"Chart comprehension benchmark with tool access. Scores depend on context length and tool configuration."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000chartography\u0000zai_glm_5_3_flash_model_card\u0000chartography\u0000Chartography, w/ tools, temp 1.0, top_p 0.95, max context 256K\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000chartography\u0000zai_glm_5_3_flash_model_card\u0000chartography\u0000Chartography, w/ tools, temp 1.0, top_p 0.95, max context 256K\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:chartography","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["visual-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"visual-reasoning","source_record":null},"modality":null,"name":"Chartography","openness":"unknown","publisher":null,"released":"2025-06-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:chartography","source_url":"https://github.com/Chartography/Chartography"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":78.9,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":78.9,"raw_min":78.0,"source_reference":{"obs_id":"curated\u0000chartography\u0000deepseek_v4_1_flash_technical_report\u0000chartography\u0000think max, Pass@1, with tools\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000chartography\u0000deepseek_v4_1_flash_technical_report\u0000chartography\u0000think max, Pass@1, with tools\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"}],"slug":"chartography","source":"model_reports","source_url":"https://github.com/Chartography/Chartography","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":266,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_0a5d08ac3f6ebcb5"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ChartQA is a large-scale benchmark comprising 9.6K human-written questions and 23.1K questions generated from human-written chart summaries, designed to evaluate models' abilities in visual and logical reasoning over charts.","descriptions":{"en":"ChartQA is a large-scale benchmark comprising 9.6K human-written questions and 23.1K questions generated from human-written chart summaries, designed to evaluate models' abilities in visual and logical reasoning over charts."},"evidence_summary":{"document_count":1,"model_count":26,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:chartqa:grok-1.5v","reported_at":"2024-04-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:chartqa","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_0a5d08ac3f6ebcb5"},"modality":"multimodal","name":"ChartQA","openness":"unknown","publisher":null,"released":"2022-03-19","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:chartqa","source_url":"https://arxiv.org/abs/2203.10244"},"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":26,"score_direction":"higher_is_better","score_summary":{"display_max":90.8,"display_multiplier":100,"model_count":26,"model_count_basis":"source_model_id","numeric_count":26,"raw_max":0.908,"raw_min":0.688,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:chartqa:claude-3-5-sonnet-20241022","reported_date":"2024-10-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-04-12","date_precision":"model_announcement","name":"Grok-1.5V"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-vision-instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-chartqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqa?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["ChartQA"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Largely saturated; relaxed-accuracy tolerance affects the reported figure.","descriptions":{"en":"Largely saturated; relaxed-accuracy tolerance affects the reported figure."},"evidence_summary":{"document_count":2,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:chartqa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["visual-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"visual-reasoning","source_record":null},"modality":null,"name":"ChartQA","openness":"unknown","publisher":null,"released":"2022-03-19","released_reference":{"basis":"benchmark_release","source_key":"model-reports:chartqa","source_url":"https://github.com/vis-nlp/ChartQA"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"chartqa","source":"model_reports","source_url":"https://github.com/vis-nlp/ChartQA","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":47,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":25,"join":"llm_stats_id","repository":null,"source_record":"catalog_76c1795b33807a83"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ChartQAPro is a challenging benchmark for question answering over diverse, real-world charts and infographics.","descriptions":{"en":"ChartQAPro is a challenging benchmark for question answering over diverse, real-world charts and infographics."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:chartqapro:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqapro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:chartqapro","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_76c1795b33807a83"},"modality":"multimodal","name":"ChartQAPro","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":70.89999999999999,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.709,"raw_min":0.709,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:chartqapro:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqapro?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-chartqapro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqapro?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":6,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:kaka-yjk/chartsynccodebase","source_record":"bm_chartsync_6261046a"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ChartSync evaluates image editing models on chart editing tasks, including text-only edits and visuo-logical cascading edits (VLCE) requiring synchronized text and geometry changes. The benchmark contains 870 expert-validated triplets across nine chart families, with objective visual metrics and a vision-language model judge for evaluation.","descriptions":{"en":"ChartSync evaluates image editing models on chart editing tasks, including text-only edits and visuo-logical cascading edits (VLCE) requiring synchronized text and geometry changes. The benchmark contains 870 expert-validated triplets across nine chart families, with objective visual metrics and a vision-language model judge for evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.10301","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","visual-reasoning"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_chartsync_6261046a"},"modality":null,"name":"ChartSync","openness":"unknown","publisher":null,"released":"2026-07-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10301","source_url":"https://arxiv.org/abs/2607.10301"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10301","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10301","topics":[],"unit":null},{"aliases":["CharXiv Reasoning","CharXiv"],"attention_signals":null,"categories":["vision"],"collected_at":null,"description":"Chart reasoning benchmark with tool access. Scores depend on context length and tool configuration.","descriptions":{"en":"Chart reasoning benchmark with tool access. Scores depend on context length and tool configuration."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000charxiv_reasoning\u0000zai_glm_5_3_flash_model_card\u0000charxiv_reasoning\u0000CharXiv Reasoning, w/ tools, temp 1.0, top_p 0.95, max context 256K\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000charxiv_reasoning\u0000zai_glm_5_3_flash_model_card\u0000charxiv_reasoning\u0000CharXiv Reasoning, w/ tools, temp 1.0, top_p 0.95, max context 256K\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:charxiv_reasoning","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["visual-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"visual-reasoning","source_record":null},"modality":null,"name":"CharXiv Reasoning","openness":"unknown","publisher":null,"released":"2025-06-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:charxiv_reasoning","source_url":"https://github.com/CharXiv/CharXiv"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":89.4,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":89.4,"raw_min":89.4,"source_reference":{"obs_id":"curated\u0000charxiv_reasoning\u0000zai_glm_5_3_flash_model_card\u0000charxiv_reasoning\u0000CharXiv Reasoning, w/ tools, temp 1.0, top_p 0.95, max context 256K\u0000GLM-5.3-Flash","observation_id":"curated\u0000charxiv_reasoning\u0000zai_glm_5_3_flash_model_card\u0000charxiv_reasoning\u0000CharXiv Reasoning, w/ tools, temp 1.0, top_p 0.95, max context 256K\u0000GLM-5.3-Flash","reported_at":"2026-08-27","reported_date":"2026-08-27","source_id":"zai_glm_5_3_flash_model_card","source_url":"https://z.ai/blog/glm-5.3-flash"},"unit":"percent"},"scored_models":[{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"}],"slug":"charxiv_reasoning","source":"model_reports","source_url":"https://github.com/CharXiv/CharXiv","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":160,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"llm_stats_id","repository":null,"source_record":"catalog_f33650566edd9798"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","structured_output","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CharXiv-D is the descriptive questions subset of the CharXiv benchmark, designed to assess multimodal large language models' ability to extract basic information from scientific charts. It contains descriptive questions covering information extraction, enumeration, pattern recognition, and counting across 2,323 diverse charts from arXiv papers, all curated and verified by human experts.","descriptions":{"en":"CharXiv-D is the descriptive questions subset of the CharXiv benchmark, designed to assess multimodal large language models' ability to extract basic information from scientific charts. It contains descriptive questions covering information extraction, enumeration, pattern recognition, and counting across 2,323 diverse charts from arXiv papers, all curated and verified by human experts."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:charxiv-d:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-d?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:charxiv-d","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_f33650566edd9798"},"modality":"multimodal","name":"CharXiv-D","openness":"unknown","publication":{"evidence":[{"note":"The authors' repository links the introducing paper and documents separate descriptive and reasoning evaluation modes plus validation/test splits. Its expert-curated chart description matches the record, whose task explicitly selects descriptive questions.","url":"https://github.com/princeton-nlp/CharXiv/blob/main/README.md"}],"id":"charxiv-descriptive-original","links":[{"kind":"paper","title":"CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs","url":"https://arxiv.org/abs/2406.18521"},{"kind":"repo","title":"CharXiv official repository","url":"https://github.com/princeton-nlp/CharXiv"}],"record_keys":["llm-stats:charxiv-d"],"reviewed_at":"2026-10-26","scope":{"en":"CharXiv's descriptive-question subset. The paper and repository cover both descriptive and reasoning questions; this record refers to descriptive evaluation, with the split left as reported by the score source.","zh":"CharXiv 的描述性问题子集。论文和仓库同时涵盖描述与推理问题；本条对应描述性评测，具体数据划分仍以成绩来源为准。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":95.5,"display_multiplier":100,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":0.955,"raw_min":0.6,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:charxiv-d:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-d?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-charxiv-d","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-d?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":160,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"llm_stats_id","repository":null,"source_record":"catalog_65ac5d8988d5755d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CharXiv-R is the reasoning component of the CharXiv benchmark, focusing on complex reasoning questions that require synthesizing information across visual chart elements. It evaluates multimodal large language models on their ability to understand and reason about scientific charts from arXiv papers through various reasoning tasks.","descriptions":{"en":"CharXiv-R is the reasoning component of the CharXiv benchmark, focusing on complex reasoning questions that require synthesizing information across visual chart elements. It evaluates multimodal large language models on their ability to understand and reason about scientific charts from arXiv papers through various reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":51,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:charxiv-r:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-r?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:charxiv-r","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_65ac5d8988d5755d"},"modality":"multimodal","name":"CharXiv-R","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":51,"score_direction":"higher_is_better","score_summary":{"display_max":93.2,"display_multiplier":100,"model_count":51,"model_count_basis":"source_model_id","numeric_count":51,"raw_max":0.932,"raw_min":0.397,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:charxiv-r:claude-mythos-preview","reported_date":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-r?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-charxiv-r","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-r?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","数学","Math","大语言模型","LLM","代码工程","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CHASE is a unified framework to synthetically generate challenging problems using LLMs without human involvement CHASE是一个无需人工参与的统一框架，用于合成生成具有挑战性的问题","descriptions":{"en":"CHASE is a unified framework to synthetically generate challenging problems using LLMs without human involvement","zh":"CHASE是一个无需人工参与的统一框架，用于合成生成具有挑战性的问题"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1534","languages":[],"library_direction":null,"modality":null,"name":"CHASE-Code","openness":"open","publisher":"Mila and McGill University","released":"2025-02-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1534-chase-code","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CHASE-Code","topics":["coding","reasoning-math"],"unit":null},{"aliases":["Chatbot Arena","LMArena","LMSYS Arena"],"attention_signals":null,"categories":["human_preference"],"collected_at":null,"description":"Elo from real user traffic; sampling and prompt distribution are outside any vendor's control.","descriptions":{"en":"Elo from real user traffic; sampling and prompt distribution are outside any vendor's control."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:chatbot_arena","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["chat-writing"],"join":"review:14H034160212:2026-10-15","primary":"chat-writing","source_record":null},"modality":null,"name":"Chatbot Arena","openness":"unknown","publisher":null,"released":"2023-05-03","released_reference":{"basis":"benchmark_release","source_key":"model-reports:chatbot_arena","source_url":"https://lmarena.ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"chatbot_arena","source":"model_reports","source_url":"https://lmarena.ai/","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":16,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:centerforaisafety/cheatbench","source_record":"bm_cheatbench_3973fc67"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Reinforcement learning has helped AI agents solve increasingly difficult tasks, but high rewards do not always reflect the work users intended.","descriptions":{"en":"Reinforcement learning has helped AI agents solve increasingly difficult tasks, but high rewards do not always reflect the work users intended."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.36308","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_cheatbench_3973fc67"},"modality":null,"name":"CheatBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36308","source_url":"https://arxiv.org/abs/2609.36308"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36308","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36308","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":71,"join":"claire_record_id","repository":null,"source_record":"bm_checkerbench_6ee992f4"},"categories":["Code & Software","Cybersecurity","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback.","descriptions":{"en":"Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.07557","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agentic-coding","cybersecurity"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_checkerbench_6ee992f4"},"modality":null,"name":"CheckerBench","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.07557","source_url":"https://arxiv.org/abs/2610.07557"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-07557","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07557","topics":["coding","agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Chehre is a video dataset of 2,111 facial expressions prompted by 40 emojis, with annotations transferred to synthetic faces. It defines two tasks: dominant expression recognition and distributional expression recognition, evaluating models' ability to predict human-rated labels and capture response diversity.","descriptions":{"en":"Chehre is a video dataset of 2,111 facial expressions prompted by 40 emojis, with annotations transferred to synthetic faces. It defines two tasks: dominant expression recognition and distributional expression recognition, evaluating models' ability to predict human-rated labels and capture response diversity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21657","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_chehre_a4cc7955"},"modality":null,"name":"Chehre","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21657","source_url":"https://arxiv.org/abs/2606.21657"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21657","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21657","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Materials & Chemistry","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Chem World integrates 17 chemical datasets with 800,000+ molecules for property prediction, offering a unified evaluation platform.","descriptions":{"en":"Chem World integrates 17 chemical datasets with 800,000+ molecules for property prediction, offering a unified evaluation platform."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.28079","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_chem-world_1eaa9499"},"modality":null,"name":"Chem World","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28079","source_url":"https://arxiv.org/abs/2607.28079"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28079","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28079","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Information retrieval","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"Chem2Gen-Bench evaluates chemical-to-genetic translation using 260,084 chemical and 1,099,045 genetic perturbation profiles in cell-target contexts. It measures pairwise alignment, retrieval success, and representation quality across matched perturbation settings.","descriptions":{"en":"Chem2Gen-Bench evaluates chemical-to-genetic translation using 260,084 chemical and 1,099,045 genetic perturbation profiles in cell-target contexts. It measures pairwise alignment, retrieval success, and representation quality across matched perturbation settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21109","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_chem2gen-bench_dc691880"},"modality":null,"name":"Chem2Gen-Bench","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21109","source_url":"https://arxiv.org/abs/2606.21109"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21109","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21109","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","科学智能","AI for Science","知识储备","科学推理","Scientific Reasoning","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ChemBench is a large-scale chemistry competency evaluation benchmark for language models, which includes nine chemistry core tasks and 4100 high-quality single-choice questions and answers. ChemBench是一个包含了九项化学核心任务，4100个高质量单选问答的大语言模型化学能力评测基准.","descriptions":{"en":"ChemBench is a large-scale chemistry competency evaluation benchmark for language models, which includes nine chemistry core tasks and 4100 high-quality single-choice questions and answers.","zh":"ChemBench是一个包含了九项化学核心任务，4100个高质量单选问答的大语言模型化学能力评测基准."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:692","languages":[],"library_direction":null,"modality":null,"name":"ChemBench","openness":"unknown","publisher":null,"released":"2024-02-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-692-chembench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ChemBench","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mohammadkhodadad/multi-lingual-qac","source_record":"bm_chemclir-bench_76439ed7"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates cross-lingual retrieval models on a multilingual chemical patent corpus with generated question-answer pairs; performance is measured by Recall@10 and MRR@10.","descriptions":{"en":"Evaluates cross-lingual retrieval models on a multilingual chemical patent corpus with generated question-answer pairs; performance is measured by Recall@10 and MRR@10."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.23231","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","science","multilingual"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_chemclir-bench_76439ed7"},"modality":null,"name":"ChemCLIR-Bench","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23231","source_url":"https://arxiv.org/abs/2609.23231"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23231","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23231","topics":["language","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_chemcotbench-v2_5360db80"},"categories":["Language & Knowledge","Materials & Chemistry","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"ChemCoTBench-V2 evaluates chemical reasoning at the process level, with 5,620 samples across molecular understanding, editing, optimization, and reaction prediction. It checks intermediate steps using deterministic chemistry rules and reference traces.","descriptions":{"en":"ChemCoTBench-V2 evaluates chemical reasoning at the process level, with 5,620 samples across molecular understanding, editing, optimization, and reaction prediction. It checks intermediate steps using deterministic chemistry rules and reference traces."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03660","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_chemcotbench-v2_5360db80"},"modality":null,"name":"ChemCoTBench-V2","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03660","source_url":"https://arxiv.org/abs/2606.03660"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03660","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03660","topics":["reasoning-math","science"],"unit":null},{"aliases":["cheme-ml-benchmarks"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yukevindai/cheme-ml-benchmarks","source_record":"bm_cheme-ml-benchmarks_af62de72"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A locally runnable evaluation suite for machine learning methods on chemical engineering tasks, including concrete strength and airfoil self-noise datasets plus synthetic fixtures, with fixed splits, preprocessing, scoring metrics, and leaderboard generation.","descriptions":{"en":"A locally runnable evaluation suite for machine learning methods on chemical engineering tasks, including concrete strength and airfoil self-noise datasets plus synthetic fixtures, with fixed splits, preprocessing, scoring metrics, and leaderboard generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:yukevindai/cheme-ml-benchmarks","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_cheme-ml-benchmarks_af62de72"},"modality":null,"name":"ChemE ML Benchmarks","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:yukevindai/cheme-ml-benchmarks","source_url":"https://github.com/yukevindai/cheme-ml-benchmarks"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-yukevindai-cheme-ml-benchmarks","source":"claire_radar","source_url":"https://github.com/yukevindai/cheme-ml-benchmarks","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Chess-World-Model evaluates state tracking by predicting the exact board state after sequences of legal moves from 10 million chess games, with accuracy as the metric.","descriptions":{"en":"Chess-World-Model evaluates state tracking by predicting the exact board state after sequences of legal moves from 10 million chess games, with accuracy as the metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30100","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","abstract-reasoning"],"join":"claire_record_id","primary":"world-models","source_record":"bm_chess-world-model_85ede047"},"modality":null,"name":"Chess-World-Model","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30100","source_url":"https://arxiv.org/abs/2605.30100"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30100","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30100","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_db08c7c91811297e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CheXpert is a large dataset of 224,316 chest radiographs from 65,240 patients for automated chest X-ray interpretation. The dataset includes uncertainty labels for 14 medical observations extracted from radiology reports. It serves as a benchmark for developing and evaluating automated chest radiograph interpretation models.","descriptions":{"en":"CheXpert is a large dataset of 224,316 chest radiographs from 65,240 patients for automated chest X-ray interpretation. The dataset includes uncertainty labels for 14 medical observations extracted from radiology reports. It serves as a benchmark for developing and evaluating automated chest radiograph interpretation models."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:chexpert-cxr:medgemma-4b-it","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chexpert-cxr?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:chexpert-cxr","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_db08c7c91811297e"},"modality":"image","name":"CheXpert CXR","openness":"unknown","publisher":null,"released":"2019-01-21","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:chexpert-cxr","source_url":"https://arxiv.org/abs/1901.07031"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":48.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.481,"raw_min":0.481,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:chexpert-cxr:medgemma-4b-it","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chexpert-cxr?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-20","date_precision":"model_announcement","name":"MedGemma 4B IT"}],"slug":"llm-stats-chexpert-cxr","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chexpert-cxr?top_n=500","topics":["medical"],"unit":null},{"aliases":["Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:cincynlp/chiaro","source_record":"bm_chiaroscuro-for-emotions-a-contrastive-emo_5279a229"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Human-annotated benchmark of 1,000 sentences where each scene elicits opposing emotions in two agents, evaluated via macro-F1 for both LLMs and classifiers.","descriptions":{"en":"Human-annotated benchmark of 1,000 sentences where each scene elicits opposing emotions in two agents, evaluated via macro-F1 for both LLMs and classifiers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.03394","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_chiaroscuro-for-emotions-a-contrastive-emo_5279a229"},"modality":null,"name":"CHIARO","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03394","source_url":"https://arxiv.org/abs/2609.03394"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03394","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03394","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CHID is a chinese idiom reading comprehension task, which requires to select the correct idiom to fill in the blank according to the context, with 10 candidate idioms. CHID是一个中文成语阅读理解任务，要求根据上下文选择正确的成语填空，共有10个候选成语。","descriptions":{"en":"CHID is a chinese idiom reading comprehension task, which requires to select the correct idiom to fill in the blank according to the context, with 10 candidate idioms.","zh":"CHID是一个中文成语阅读理解任务，要求根据上下文选择正确的成语填空，共有10个候选成语。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:505","languages":["Chinese"],"library_direction":null,"modality":null,"name":"CHID","openness":"unknown","publisher":null,"released":"2019-06-04","released_reference":{"basis":"paper_first_version","note":"First version of the paper introducing the Chinese idiom cloze dataset.","source_key":"opencompass:505","source_url":"https://arxiv.org/abs/1906.01265"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-505-chid","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CHID","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ziyanluo/childeval","source_record":"bm_childeval_9775399e"},"categories":["General AI","Language & Knowledge","Long Context"],"collected_at":"2026-10-14T09:44:51Z","description":"ChildEval is a benchmark for evaluating LLMs' ability to infer and follow child-centered preferences in long-context conversations. It contains 29K synthesized persona profiles of children aged 3-6, with explicit and implicit preference expressions across five top-level and fourteen sub-level categories.","descriptions":{"en":"ChildEval is a benchmark for evaluating LLMs' ability to infer and follow child-centered preferences in long-context conversations. It contains 29K synthesized persona profiles of children aged 3-6, with explicit and implicit preference expressions across five top-level and fourteen sub-level categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.27805","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","long-context"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_childeval_9775399e"},"modality":null,"name":"ChildEval","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27805","source_url":"https://arxiv.org/abs/2605.27805"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27805","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27805","topics":["agents","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark dataset proposed for collaborative behavior understanding in children during naturalistic interactions.","descriptions":{"en":"Benchmark dataset proposed for collaborative behavior understanding in children during naturalistic interactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.06353","languages":[],"library_direction":null,"modality":null,"name":"ChildGaze","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06353","source_url":"https://arxiv.org/abs/2609.06353"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06353","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06353","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Intended to evaluate LLM understanding of competitive Chinese-language debate at match, stage, and speaker levels.","descriptions":{"en":"Intended to evaluate LLM understanding of competitive Chinese-language debate at match, stage, and speaker levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.21637","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_chinese-competitive-debating-dataset-and-b_d8ea1aa0"},"modality":null,"name":"Chinese Competitive Debating Dataset and Benchmark","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21637","source_url":"https://arxiv.org/abs/2609.21637"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21637","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21637","topics":[],"unit":null},{"aliases":["ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning","ChitraMiti-12.8k"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":35,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_chitramiti-benchmarking-visual-grounding-a_190d3422"},"categories":["Geometric reasoning","Mathematics & Formal Science","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"ChitraMiti-12.8k is a synthetic Bengali planar geometry benchmark of 12,874 problems with structured 15-attribute diagram descriptions, complemented by NCTB-500, 500 textbook-extracted diagrams, targeting visual grounding and modality reliance.","descriptions":{"en":"ChitraMiti-12.8k is a synthetic Bengali planar geometry benchmark of 12,874 problems with structured 15-attribute diagram descriptions, complemented by NCTB-500, 500 textbook-extracted diagrams, targeting visual grounding and modality reliance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.12509","languages":[],"library_direction":{"basis":"model-review","categories":["math","visual-reasoning","multilingual"],"join":"claire_record_id","primary":"math","source_record":"bm_chitramiti-benchmarking-visual-grounding-a_190d3422"},"modality":null,"name":"ChitraMiti-12.8k","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12509","source_url":"https://arxiv.org/abs/2609.12509"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12509","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12509","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":"gh:0328zpx/chlogic","source_record":"bm_chlogic_3b9dd8db"},"categories":["General AI","Reasoning","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"ChLogic is an English–Chinese aligned benchmark for evaluating logical reasoning robustness across surface realizations. It includes 3,000 general, 2,000 difficult, and 1,500 Chinese-only items derived from formal logical templates, with each aligned item pairing one English reference with five Chinese variants.","descriptions":{"en":"ChLogic is an English–Chinese aligned benchmark for evaluating logical reasoning robustness across surface realizations. It includes 3,000 general, 2,000 difficult, and 1,500 Chinese-only items derived from formal logical templates, with each aligned item pairing one English reference with five Chinese variants."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.17905","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","multilingual"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_chlogic_3b9dd8db"},"modality":null,"name":"ChLogic","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17905","source_url":"https://arxiv.org/abs/2606.17905"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17905","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17905","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ChoroplethMap-Bench evaluates spatial understanding of foundation models with 2,400 synthetic choropleth maps, corresponding GeoJSON data, and 12,000 questions across five cognitive dimensions (Identify, Spatial Recognition, Compare, Rank, Delineate). Models are assessed under Data Only, Map Only, and Data + Map conditions.","descriptions":{"en":"ChoroplethMap-Bench evaluates spatial understanding of foundation models with 2,400 synthetic choropleth maps, corresponding GeoJSON data, and 12,000 questions across five cognitive dimensions (Identify, Spatial Recognition, Compare, Rank, Delineate). Models are assessed under Data Only, Map Only, and Data + Map conditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.17999","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_choroplethmap-bench_3460d576"},"modality":null,"name":"ChoroplethMap-Bench","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.17999","source_url":"https://arxiv.org/abs/2607.17999"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-17999","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17999","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":11,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:intellisensing/geochrono","source_record":"bm_chronobench_44b7b76d"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"ChronoBench evaluates long-term temporal understanding in remote sensing across four progressive cognitive levels: land cover perception, temporal recognition, long-term memory, and spatio-temporal reasoning. It comprises 12 sub-tasks and 17,689 QA pairs over 3,469 images spanning 500 regions across 39 U.S. cities.","descriptions":{"en":"ChronoBench evaluates long-term temporal understanding in remote sensing across four progressive cognitive levels: land cover perception, temporal recognition, long-term memory, and spatio-temporal reasoning. It comprises 12 sub-tasks and 17,689 QA pairs over 3,469 images spanning 500 regions across 39 U.S. cities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.15768","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_chronobench_44b7b76d"},"modality":null,"name":"ChronoBench","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15768","source_url":"https://arxiv.org/abs/2607.15768"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15768","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15768","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["创作","Creation","NeurIPS 2024","多模态模型","VLM","视觉生成","Visual Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ChronoMagic-Bench can evaluate the temporal and metamorphic capabilities of the T2V (text-to-video) models in time-lapse video generation, introducing 1,649 prompts and real-world videos as references. ChronoMagic-Bench用来评估 T2V （文本到视频 ）模型在延时视频生成中的时间和变形能力，引入了1649个提示和真实世界的视频作为参考。","descriptions":{"en":"ChronoMagic-Bench can evaluate the temporal and metamorphic capabilities of the T2V (text-to-video) models in time-lapse video generation, introducing 1,649 prompts and real-world videos as references.","zh":"ChronoMagic-Bench用来评估 T2V （文本到视频 ）模型在延时视频生成中的时间和变形能力，引入了1649个提示和真实世界的视频作为参考。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1278","languages":[],"library_direction":null,"modality":null,"name":"ChronoMagic-Bench","openness":"open","publisher":"Peking University","released":"2024-06-26","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1278-chronomagic-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ChronoMagic-Bench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Multimodal","Reasoning","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"ChronoPhyBench evaluates multimodal LLMs on chronological physical dynamics reasoning via next-state prediction and VQA, using video frames and captions for single-image selection and multi-frame sorting.","descriptions":{"en":"ChronoPhyBench evaluates multimodal LLMs on chronological physical dynamics reasoning via next-state prediction and VQA, using video frames and captions for single-image selection and multi-frame sorting."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07962","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","video-understanding"],"join":"claire_record_id","primary":"world-models","source_record":"bm_chronophybench_c3f107b4"},"modality":null,"name":"ChronoPhyBench","openness":"unknown","publisher":null,"released":"2026-06-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07962","source_url":"https://arxiv.org/abs/2606.07962"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07962","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07962","topics":["multimodal","reasoning-math","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object is unclear from the provided information.","descriptions":{"en":"The evaluation object is unclear from the provided information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14770","languages":[],"library_direction":null,"modality":null,"name":"ChronoQG","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14770","source_url":"https://arxiv.org/abs/2607.14770"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14770","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14770","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ChronoState evaluates whether a frozen language model can compose hidden elapsed-time scalars with symbolic task state to select temporal actions, using forced-choice accuracy under direct supervision.","descriptions":{"en":"ChronoState evaluates whether a frozen language model can compose hidden elapsed-time scalars with symbolic task state to select temporal actions, using forced-choice accuracy under direct supervision."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09124","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_chronostate_2f5e6d2f"},"modality":null,"name":"ChronoState","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09124","source_url":"https://arxiv.org/abs/2608.09124"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09124","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09124","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:vsingh45/churnbench","source_record":"bm_churnbench_8f71ce1d"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Measures staleness errors in agentic AI answers under temporal drift across heterogeneous enterprise data sources.","descriptions":{"en":"Measures staleness errors in agentic AI answers under temporal drift across heterogeneous enterprise data sources."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.11515","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","professional-work"],"join":"claire_record_id","primary":"factuality","source_record":"bm_churnbench_8f71ce1d"},"modality":null,"name":"ChurnBench","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11515","source_url":"https://arxiv.org/abs/2609.11515"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11515","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11515","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":11,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:facebookresearch/cimemories","source_record":"catalog_c80d5333e2d792e7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["memory","privacy","safety","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CI Memories measures privacy behavior in memory-enabled agents using contextual-integrity scenarios. This metric reports evaluation coverage.","descriptions":{"en":"CI Memories measures privacy behavior in memory-enabled agents using contextual-integrity scenarios. This metric reports evaluation coverage."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ci-memories-coverage:muse-glimmer-30b","reported_at":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-coverage?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ci-memories-coverage","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agent-memory"],"join":"llm_stats_id","primary":"agent-security","source_record":"catalog_c80d5333e2d792e7"},"modality":"text","name":"CI Memories Coverage","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":64.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.648,"raw_min":0.648,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ci-memories-coverage:muse-glimmer-30b","reported_date":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-coverage?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"}],"slug":"llm-stats-ci-memories-coverage","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-coverage?top_n=500","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":11,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:facebookresearch/cimemories","source_record":"catalog_851343005b4a282e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["memory","privacy","safety","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CI Memories measures privacy failures in memory-enabled agents using contextual-integrity scenarios. This metric is the violation rate; lower is better.","descriptions":{"en":"CI Memories measures privacy failures in memory-enabled agents using contextual-integrity scenarios. This metric is the violation rate; lower is better."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ci-memories-violation:muse-glimmer-30b","reported_at":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-violation?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ci-memories-violation","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agent-memory"],"join":"llm_stats_id","primary":"agent-security","source_record":"catalog_851343005b4a282e"},"modality":"text","name":"CI Memories Violation Rate","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":26.400000000000002,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.264,"raw_min":0.264,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ci-memories-violation:muse-glimmer-30b","reported_date":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-violation?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"}],"slug":"llm-stats-ci-memories-violation","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-violation?top_n=500","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CIAware-Bench measures control intervention awareness in language models through four task domains: essay writing, BigCodeBench, Bash Arena, and SHADE-Arena. Models are tested on distinguishing their own trajectories from those modified by a control protocol, with variations in watermarking, side-task presence, and protocol.","descriptions":{"en":"CIAware-Bench measures control intervention awareness in language models through four task domains: essay writing, BigCodeBench, Bash Arena, and SHADE-Arena. Models are tested on distinguishing their own trajectories from those modified by a control protocol, with variations in watermarking, side-task presence, and protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.11063","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_ciaware-bench_e29a610f"},"modality":null,"name":"CIAware-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11063","source_url":"https://arxiv.org/abs/2606.11063"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11063","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11063","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:superyfan/cibuzzbench","source_record":"bm_cibuzzbench_d5bcdecb"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates Chinese-to-English cross-lingual understanding of internet buzzwords through explanation, equivalence selection, and harmfulness classification.","descriptions":{"en":"Evaluates Chinese-to-English cross-lingual understanding of internet buzzwords through explanation, equivalence selection, and harmfulness classification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.21722","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_cibuzzbench_d5bcdecb"},"modality":null,"name":"CIBuzzBench","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21722","source_url":"https://arxiv.org/abs/2609.21722"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21722","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21722","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:tafseer-nayeem/cinesubbench","source_record":"bm_cinesubbench_eb31f9c6"},"categories":["General AI","Language & Knowledge","Long Context"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering, and cybersecurity, yet film remains comparatively underexplored despite requiring long-form narrative integration, multili…","descriptions":{"en":"Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering, and cybersecurity, yet film remains comparatively underexplored despite requiring long-form narrative integration, multili…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.36218","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_cinesubbench_eb31f9c6"},"modality":null,"name":"CineSubBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36218","source_url":"https://arxiv.org/abs/2609.36218"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36218","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36218","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["CAD","Geometric reasoning","Manufacturing & Process Control","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"The two-dimensional irregular knapsack problem in a fixed circular container is an important combinatorial optimization problem for maximizing material utilization in manufacturing.","descriptions":{"en":"The two-dimensional irregular knapsack problem in a fixed circular container is an important combinatorial optimization problem for maximizing material utilization in manufacturing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38863","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_circlenest-bench_c285eb41"},"modality":null,"name":"CircleNest-Bench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38863","source_url":"https://arxiv.org/abs/2609.38863"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38863","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38863","topics":["reasoning-math"],"unit":null},{"aliases":["CircuitReason-1k: Benchmarking Long-Horizon Visual-to-Symbolic Reasoning inElectrical Circuits"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-horizon visual-to-symbolic reasoning on 1,000 authentic circuit problems, combining typed scoring with multi-model semantic consensus, covering topology grounding, physical modeling, and symbolic computation.","descriptions":{"en":"Evaluates long-horizon visual-to-symbolic reasoning on 1,000 authentic circuit problems, combining typed scoring with multi-model semantic consensus, covering topology grounding, physical modeling, and symbolic computation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09374","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_circuitreason-1k-benchmarking-long-horizon_0cd98566"},"modality":null,"name":"CircuitReason-1k","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09374","source_url":"https://arxiv.org/abs/2608.09374"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09374","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09374","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DB"],"collected_at":"2026-10-14T09:44:51Z","description":"CITBench evaluates LLMs on interactive tabular data processing, covering table matching, cleaning, augmentation, and transformation across 18 task types and 1,296 instances.","descriptions":{"en":"CITBench evaluates LLMs on interactive tabular data processing, covering table matching, cleaning, augmentation, and transformation across 18 task types and 1,296 instances."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00018","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_citbench_cd2dd37c"},"modality":null,"name":"CITBench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00018","source_url":"https://arxiv.org/abs/2608.00018"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00018","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00018","topics":[],"unit":null},{"aliases":["GridFlow: Structured Latent Flow for Seamless City-Scale 3D Point Cloud Generation"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":50,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_gridflow-structured-latent-flow-for-seamle_f65faabd"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"City3D-MultiGen provides 163K densely annotated tiles with aligned point clouds, satellite images, semantic maps, and elevation data for city-scale 3D point cloud generation evaluation.","descriptions":{"en":"City3D-MultiGen provides 163K densely annotated tiles with aligned point clouds, satellite images, semantic maps, and elevation data for city-scale 3D point cloud generation evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29793","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_gridflow-structured-latent-flow-for-seamle_f65faabd"},"modality":null,"name":"City3D-MultiGen","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29793","source_url":"https://arxiv.org/abs/2608.29793"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29793","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29793","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CityRep is a benchmark for urban representation learning across 8 cities and 8 tasks with spatially structured splits to mitigate leakage.","descriptions":{"en":"CityRep is a benchmark for urban representation learning across 8 cities and 8 tasks with spatially structured splits to mitigate leakage."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26036","languages":[],"library_direction":null,"modality":null,"name":"CITYREP","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26036","source_url":"https://arxiv.org/abs/2605.26036"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26036","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26036","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"CityTrajBench evaluates city-scale vehicle trajectory generation methods with standardized preprocessing, model adaptation, and multi-level evaluation across statistical, VAE, GAN, diffusion, and flow-matching models.","descriptions":{"en":"CityTrajBench evaluates city-scale vehicle trajectory generation methods with standardized preprocessing, model adaptation, and multi-level evaluation across statistical, VAE, GAN, diffusion, and flow-matching models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02287","languages":[],"library_direction":null,"modality":null,"name":"CityTrajBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02287","source_url":"https://arxiv.org/abs/2606.02287"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02287","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02287","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":214,"github_stars_30d":26,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:lmwilki/civ6-mcp","source_record":"bm_civbench_b296ca93"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates language model agents in long-horizon, tool-mediated Civilization VI gameplay via 76 MCP tools.","descriptions":{"en":"Evaluates language model agents in long-horizon, tool-mediated Civilization VI gameplay via 76 MCP tools."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.02459","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["tool-use","multi-agent-games"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_civbench_b296ca93"},"modality":null,"name":"CivBench","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02459","source_url":"https://arxiv.org/abs/2609.02459"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02459","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02459","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":584,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":"gh:tencent-hunyuan/cl-bench","source_record":"catalog_62c91317432638c2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CL-bench is an open-source benchmark with its own data and rubrics for evaluating models on coding and agentic tasks, scored using a setup fully aligned with the official procedure.","descriptions":{"en":"CL-bench is an open-source benchmark with its own data and rubrics for evaluating models on coding and agentic tasks, scored using a setup fully aligned with the official procedure."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cl-bench:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cl-bench","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","long-context"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_62c91317432638c2"},"modality":"text","name":"CL-bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":23.799999999999997,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.238,"raw_min":0.2048,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cl-bench:hy3","reported_date":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-cl-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CL-bench Life variant.","descriptions":{"en":"CL-bench Life variant."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cl-bench-(life):hy3","reported_at":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench-%28life%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cl-bench-(life)","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","long-context"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_bcdfd03bd948d941"},"modality":"text","name":"CL-bench (Life)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":17.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.17,"raw_min":0.17,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cl-bench-(life):hy3","reported_date":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench-%28life%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"}],"slug":"llm-stats-cl-bench-life","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench-%28life%29?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fangz-cs/clarifycodebench","source_record":"bm_clarifycodebench_aa8d2946"},"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"ClarifyCodeBench evaluates LLMs on clarifying ambiguous requirements for code generation through interactive dialogues. It includes manual annotations of ambiguity types, clarification questions, and ground-truth answers, with metrics for interaction quality.","descriptions":{"en":"ClarifyCodeBench evaluates LLMs on clarifying ambiguous requirements for code generation through interactive dialogues. It includes manual annotations of ambiguity types, clarification questions, and ground-truth answers, with metrics for interaction quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.00711","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","instruction-following"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_clarifycodebench_aa8d2946"},"modality":null,"name":"ClarifyCodeBench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00711","source_url":"https://arxiv.org/abs/2607.00711"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00711","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00711","topics":["coding","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"ClashBench comprises 268 validated conflict cases across 55 resource types for evaluating agent behavior.","descriptions":{"en":"ClashBench comprises 268 validated conflict cases across 55 resource types for evaluating agent behavior."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.19892","languages":[],"library_direction":null,"modality":null,"name":"ClashBench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19892","source_url":"https://arxiv.org/abs/2609.19892"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19892","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19892","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Factuality","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"ClassicLogic is a benchmark suite of four classic logic puzzles (Sudoku, KenKen, Kakuro, Futoshiki) with a hierarchical knowledge base that defines complex strategies as compositions of simpler ones. It evaluates an agent's compositional generalization in problem-solving from basic rules to multi-step strategies across increasing puzzle difficulty.","descriptions":{"en":"ClassicLogic is a benchmark suite of four classic logic puzzles (Sudoku, KenKen, Kakuro, Futoshiki) with a hierarchical knowledge base that defines complex strategies as compositions of simpler ones. It evaluates an agent's compositional generalization in problem-solving from basic rules to multi-step strategies across increasing puzzle difficulty."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.05185","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_classiclogic_43434cd2"},"modality":null,"name":"ClassicLogic","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.05185","source_url":"https://arxiv.org/abs/2607.05185"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05185","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05185","topics":["agents","reasoning-math"],"unit":null},{"aliases":["Cross-lingual Biography Enrichment via Claim Extraction and Alignment"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates cross-lingual biography enrichment on 300 Wikipedia biography pairs with claim annotations and a fine-grained claim-pair relation corpus.","descriptions":{"en":"Evaluates cross-lingual biography enrichment on 300 Wikipedia biography pairs with claim annotations and a fine-grained claim-pair relation corpus."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23390","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","factuality"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_cross-lingual-biography-enrichment-via-cla_8d142b84"},"modality":null,"name":"CLAW-4L","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23390","source_url":"http://arxiv.org/abs/2608.23390v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23390","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23390v1","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":48,"github_stars_30d":-1,"hf_dataset_downloads":97,"hf_paper_upvotes":26,"join":"claire_record_id","repository":"gh:libercoders/claw-anything","source_record":"bm_claw-anything_3c56b6b4"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Claw-Anything evaluates always-on LLM personal assistants in simulated environments with long-horizon activity histories, interdependent backend services, and GUI/CLI across devices. It includes 200 human-verified tasks scored on completion, robustness, communication, and safety, with a live leaderboard.","descriptions":{"en":"Claw-Anything evaluates always-on LLM personal assistants in simulated environments with long-horizon activity histories, interdependent backend services, and GUI/CLI across devices. It includes 200 human-verified tasks scored on completion, robustness, communication, and safety, with a live leaderboard."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26086","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","agent-memory"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_claw-anything_3c56b6b4"},"modality":null,"name":"Claw-Anything","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"first_public","source_key":"claire-radar:2605.26086","source_url":"https://arxiv.org/abs/2605.26086"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26086","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26086","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":778,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":117,"join":"llm_stats_id","repository":"gh:claw-eval/claw-eval","source_record":"catalog_837a019e46144f8d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Claw-Eval tests real-world agentic task completion across complex multi-step scenarios, evaluating a model's ability to use tools, navigate environments, and complete end-to-end tasks autonomously.","descriptions":{"en":"Claw-Eval tests real-world agentic task completion across complex multi-step scenarios, evaluating a model's ability to use tools, navigate environments, and complete end-to-end tasks autonomously."},"evidence_summary":{"document_count":1,"model_count":14,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:claw-eval:mimo-v2-omni","reported_at":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:claw-eval","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_837a019e46144f8d"},"modality":"text","name":"Claw-Eval","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":14,"score_direction":"higher_is_better","score_summary":{"display_max":80.9,"display_multiplier":100,"model_count":14,"model_count_basis":"source_model_id","numeric_count":14,"raw_max":0.809,"raw_min":0.5,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:claw-eval:kimi-k2.6","reported_date":"2026-04-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-claw-eval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":111,"github_stars_30d":3,"hf_dataset_downloads":813,"hf_paper_upvotes":40,"join":"claire_record_id","repository":"gh:opensquilla/claw-swe-bench","source_record":"bm_claw-swe-bench_ea69cc9d"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Claw-SWE-Bench is a multilingual SWE-bench-style benchmark and adapter protocol for comparing agent harnesses on coding tasks, with 350 GitHub issue-resolution instances across 8 languages and 43 repositories. Score is Pass@1 on patch correctness.","descriptions":{"en":"Claw-SWE-Bench is a multilingual SWE-bench-style benchmark and adapter protocol for comparing agent harnesses on coding tasks, with 350 GitHub issue-resolution instances across 8 languages and 43 repositories. Score is Pass@1 on patch correctness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.12344","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_claw-swe-bench_ea69cc9d"},"modality":null,"name":"Claw-SWE-Bench","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12344","source_url":"https://arxiv.org/abs/2606.12344"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12344","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12344","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":69,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:aiming-lab/clawarena","source_record":"bm_clawarena-team_8c93773c"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"ClawArena-Team evaluates a single text-only LLM's ability to manage a fixed, locally served pool of subagents (LLM, VLM, omni) across 41 multi-turn, multimodal, multi-directory scenarios with 258 evaluation rounds and 72 staged updates. Scoring is execution-based via shell commands, producing a Subagent-Management Score (SMS) that multiplies task correctness by a least-privilege and modality-routing factor, without LLM judges.","descriptions":{"en":"ClawArena-Team evaluates a single text-only LLM's ability to manage a fixed, locally served pool of subagents (LLM, VLM, omni) across 41 multi-turn, multimodal, multi-directory scenarios with 258 evaluation rounds and 72 staged updates. Scoring is execution-based via shell commands, producing a Subagent-Management Score (SMS) that multiplies task correctness by a least-privilege and modality-routing factor, without LLM judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.31174","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","tool-use"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_clawarena-team_8c93773c"},"modality":null,"name":"ClawArena-Team","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31174","source_url":"https://arxiv.org/abs/2606.31174"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31174","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31174","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ClawEval-MM is the multimodal variant of ClawEval, evaluating agentic problem solving with visual inputs.","descriptions":{"en":"ClawEval-MM is the multimodal variant of ClawEval, evaluating agentic problem solving with visual inputs."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:claw-eval-mm:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval-mm?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:claw-eval-mm","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","visual-reasoning"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_2c9490b45364dd97"},"modality":"multimodal","name":"ClawEval-MM","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":56.89999999999999,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.569,"raw_min":0.46,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:claw-eval-mm:qwen3.8-27b","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval-mm?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-claw-eval-mm","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval-mm?top_n=500","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":825,"github_stars_30d":1,"hf_dataset_downloads":88,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:suyoumo/clawprobench","source_record":"bm_clawprobench_7f331d78"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates agent configurations on 102 live-runtime and 68 frozen holdout scenarios, scoring execution traces with a safety-gated formula covering correctness, process quality, and efficiency.","descriptions":{"en":"Evaluates agent configurations on 102 live-runtime and 68 frozen holdout scenarios, scoring execution traces with a safety-gated formula covering correctness, process quality, and efficiency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22510","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","safety-alignment"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_clawprobench_7f331d78"},"modality":null,"name":"ClawProBench","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22510","source_url":"http://arxiv.org/abs/2608.22510v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22510","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22510v1","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_clawtrack_7bde407b"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ClawTrack is a dual-assessment benchmark for agents, measuring task outcomes and process quality across 320 tasks in 8 domains with 25+ mock services.","descriptions":{"en":"ClawTrack is a dual-assessment benchmark for agents, measuring task outcomes and process quality across 320 tasks in 8 domains with 25+ mock services."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.28037","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_clawtrack_7bde407b"},"modality":null,"name":"ClawTrack","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28037","source_url":"https://arxiv.org/abs/2607.28037"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28037","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28037","topics":["agents"],"unit":null},{"aliases":["From Prompt Injection to Persistent Control: Defending Agentic Harness Against Trojan Backdoors"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-09-25T04:40:22Z","description":"A reusable benchmark for measuring multi-step trojan attacks that establish persistent control through local agent workspace state.","descriptions":{"en":"A reusable benchmark for measuring multi-step trojan attacks that establish persistent control through local agent workspace state."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.31042","languages":[],"library_direction":null,"modality":null,"name":"ClawTrojan","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31042","source_url":"https://arxiv.org/abs/2605.31042"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31042","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31042","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":46,"join":"claire_record_id","repository":"gh:iamlihua/clbench-v","source_record":"bm_clbench-v_d435f082"},"categories":["Factuality","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"CLBench-V evaluates multimodal context learning across three dimensions: context grounding, new information application, and new knowledge learning. It includes 3,443 instances across 14 subdatasets spanning science, finance, long-document understanding, spatial reasoning, and web-based VQA.","descriptions":{"en":"CLBench-V evaluates multimodal context learning across three dimensions: context grounding, new information application, and new knowledge learning. It includes 3,443 instances across 14 subdatasets spanning science, finance, long-document understanding, spatial reasoning, and web-based VQA."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.25294","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","abstract-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_clbench-v_d435f082"},"modality":null,"name":"CLBench-V","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25294","source_url":"https://arxiv.org/abs/2607.25294"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25294","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25294","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","安全","Safety","大语言模型","LLM","语言理解","Comprehension","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CleanPatrick is a large-scale, real-world image data-cleaning benchmark with 496,377 binary annotations from 933 medical crowd workers for ranking off-topic, near-duplicate, and label-error issues. CleanPatrick是首个大规模图像数据清洗基准，基于公开的Fitzpatrick17k皮肤科数据集构建。该基准包含超过50万条来自933名医学众包工人的二元注释，涵盖三种数据质量问题：离题样本、近似重复样本和标签错误。通过医学专家验证，CleanPatrick提供了高质量的基准数据，用于评估图像数据清洗策略。基准测试结果表明，现有的数据清洗方法在近似重复检测中表现出色，但在标签错误检测方面仍面临挑战。CleanPatrick为数据清洗方法提供了标准化的评估框架，推动了更可靠的数据中心人工智能的发展。","descriptions":{"en":"CleanPatrick is a large-scale, real-world image data-cleaning benchmark with 496,377 binary annotations from 933 medical crowd workers for ranking off-topic, near-duplicate, and label-error issues.","zh":"CleanPatrick是首个大规模图像数据清洗基准，基于公开的Fitzpatrick17k皮肤科数据集构建。该基准包含超过50万条来自933名医学众包工人的二元注释，涵盖三种数据质量问题：离题样本、近似重复样本和标签错误。通过医学专家验证，CleanPatrick提供了高质量的基准数据，用于评估图像数据清洗策略。基准测试结果表明，现有的数据清洗方法在近似重复检测中表现出色，但在标签错误检测方面仍面临挑战。CleanPatrick为数据清洗方法提供了标准化的评估框架，推动了更可靠的数据中心人工智能的发展。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1849","languages":[],"library_direction":null,"modality":null,"name":"CleanPatrick","openness":"restricted","publisher":"University of Basel,etc","released":"2025-05-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1849-cleanpatrick","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CleanPatrick","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["创作","Creation","大语言模型","LLM","语言生成","Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"clembench is a benchmark framework for evaluating LLMs through dialogue game-based interactions, assessing chat-optimized models as conversational agents via multi-turn interactive game scenario. clembench是一个基于对话游戏的大语言模型评测框架，通过多回合交互式游戏场景评测聊天优化模型的会话智能体能力，包含Wordle、Taboo等多种游戏任务，评测指令遵循、目标导向行为和精细化交互理解等核心维度，提供可重复、可控制的自玩评测环境和综合得分机制，支持英语及多语言扩展的开源评测平台。","descriptions":{"en":"clembench is a benchmark framework for evaluating LLMs through dialogue game-based interactions, assessing chat-optimized models as conversational agents via multi-turn interactive game scenario.","zh":"clembench是一个基于对话游戏的大语言模型评测框架，通过多回合交互式游戏场景评测聊天优化模型的会话智能体能力，包含Wordle、Taboo等多种游戏任务，评测指令遵循、目标导向行为和精细化交互理解等核心维度，提供可重复、可控制的自玩评测环境和综合得分机制，支持英语及多语言扩展的开源评测平台。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2127","languages":[],"library_direction":null,"modality":null,"name":"clembench","openness":"unknown","publisher":"Computational Linguistics ,etc","released":"2025-07-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2127-clembench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/clembench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","科学智能","AI for Science","代码工程","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CLEVER is a benchmark suite for end-to-end code generation and formal verification in Lean 4, adapted from the HumanEval dataset. It requires models to generate implementations, formal specifications, and proofs—all verifiable by Lean's type checker, moving beyond test-case-driven evaluation. CLEVER is a benchmark suite for end-to-end code generation and formal verification in Lean 4, adapted from the HumanEval dataset. It requires models to generate implementations, formal specifications, and proofs—all verifiable by Lean's type checker, moving beyond test-case-driven evaluation.","descriptions":{"en":"CLEVER is a benchmark suite for end-to-end code generation and formal verification in Lean 4, adapted from the HumanEval dataset. It requires models to generate implementations, formal specifications, and proofs—all verifiable by Lean's type checker, moving beyond test-case-driven evaluation.","zh":"CLEVER is a benchmark suite for end-to-end code generation and formal verification in Lean 4, adapted from the HumanEval dataset. It requires models to generate implementations, formal specifications, and proofs—all verifiable by Lean's type checker, moving beyond test-case-driven evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1834","languages":[],"library_direction":null,"modality":null,"name":"CLEVER","openness":"open","publisher":"University of Texas at Austin ,etc","released":"2025-05-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1834-clever","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CLEVER","topics":["coding","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:arjunkshah12345-hash/cli-bench","source_record":"bm_cli-bench_bf617c5b"},"categories":["Agents","Cybersecurity","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"CLI-Bench is a benchmark framework for coding-agent CLI harnesses, featuring deterministic verifiers, seeded tasks, and resource accounting metrics.","descriptions":{"en":"CLI-Bench is a benchmark framework for coding-agent CLI harnesses, featuring deterministic verifiers, seeded tasks, and resource accounting metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:arjunkshah12345-hash/cli-bench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-skills-evolution","spatial-3d","finance-legal"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_cli-bench_bf617c5b"},"modality":null,"name":"CLI-Bench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:arjunkshah12345-hash/cli-bench","source_url":"https://github.com/arjunkshah12345-hash/cli-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-arjunkshah12345-hash-cli-bench","source":"claire_radar","source_url":"https://github.com/arjunkshah12345-hash/cli-bench","topics":["agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","知识","Knowledge","Multimodal Reasoning","Fact-Checking","Charts","科学智能","AI for Science","逻辑推理","知识储备","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning","OCR与文档理解","OCR and Document Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ClimateViz is a large-scale multimodal benchmark designed to evaluate the scientific fact-checking and statistical reasoning capabilities of large language and vision-language models. It focuses on real-world climate science data, with over 49,000 high quality natural language claims. ClimateViz 是一个多模态基准数据集，用于评估大模型在气候科学图表上的事实核查与统计推理能力。数据来源于 NOAA、英国气象局等权威机构，共包含约 2,800 张科学图表与近 5 万条主张，标注为支持（support）、反驳（refute）或信息不足（NEI）。\n\n该数据集支持三种输入格式：图表+主张、表格+主张、图表标题+表格+主张，涵盖趋势识别、时空推理与科学对比等核心任务。ClimateViz 适用于多模态大模型和语言模型的系统性评估。\n图表转表格 + 图表标题 + 主张（Caption + Table","descriptions":{"en":"ClimateViz is a large-scale multimodal benchmark designed to evaluate the scientific fact-checking and statistical reasoning capabilities of large language and vision-language models. It focuses on real-world climate science data, with over 49,000 high quality natural language claims.","zh":"ClimateViz 是一个多模态基准数据集，用于评估大模型在气候科学图表上的事实核查与统计推理能力。数据来源于 NOAA、英国气象局等权威机构，共包含约 2,800 张科学图表与近 5 万条主张，标注为支持（support）、反驳（refute）或信息不足（NEI）。\n\n该数据集支持三种输入格式：图表+主张、表格+主张、图表标题+表格+主张，涵盖趋势识别、时空推理与科学对比等核心任务。ClimateViz 适用于多模态大模型和语言模型的系统性评估。\n图表转表格 + 图表标题 + 主张（Caption + Table"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1954","languages":[],"library_direction":null,"modality":"multimodal","name":"ClimateViz","openness":"unknown","publisher":"University of Oxford","released":"2025-06-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1954-climateviz","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ClimateViz","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Biology & Drug Discovery","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ClinEnv evaluates LLMs as physicians in an interactive multi-stage EHR simulation, requiring queries to four specialized agents before committing to medical decisions, scored via ontology-grounded matching.","descriptions":{"en":"ClinEnv evaluates LLMs as physicians in an interactive multi-stage EHR simulation, requiring queries to four specialized agents before committing to medical decisions, scored via ontology-grounded matching."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02568","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_clinenv_3008fd36"},"modality":null,"name":"ClinEnv","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02568","source_url":"https://arxiv.org/abs/2606.02568"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02568","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02568","topics":["agents","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":10,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"claire_record_id","repository":"gh:alibaba-damo-academy/clinhallu","source_record":"bm_clinhallu_85f948a0"},"categories":["Biology & Drug Discovery","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"ClinHallu is a benchmark for diagnosing stage-wise hallucinations in medical multimodal large language models. It contains 7,031 instances with structured reasoning traces decomposed into visual recognition, knowledge recall, and reasoning integration, along with stage-replacement interventions for measuring final answer changes.","descriptions":{"en":"ClinHallu is a benchmark for diagnosing stage-wise hallucinations in medical multimodal large language models. It contains 7,031 instances with structured reasoning traces decomposed into visual recognition, knowledge recall, and reasoning integration, along with stage-replacement interventions for measuring final answer changes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.14697","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","factuality","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_clinhallu_85f948a0"},"modality":null,"name":"ClinHallu","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14697","source_url":"https://arxiv.org/abs/2606.14697"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14697","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14697","topics":["reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ClinicalMC evaluates LLM clinical decision-making across multi-course patient trajectories, with 1,275 Chinese and 5,804 English samples spanning four stages from admission to discharge. Includes triage, examination, diagnosis, treatment, and final diagnosis.","descriptions":{"en":"ClinicalMC evaluates LLM clinical decision-making across multi-course patient trajectories, with 1,275 Chinese and 5,804 English samples spanning four stages from admission to discharge. Includes triage, examination, diagnosis, treatment, and final diagnosis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03157","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_clinicalmc_22427e9b"},"modality":null,"name":"ClinicalMC","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03157","source_url":"https://arxiv.org/abs/2606.03157"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03157","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03157","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"CliniCARE-Bench evaluates clinical agents on retrospective audit tasks over longitudinal EHR data, measuring verdict accuracy, evidence grounding, process adherence, calibrated abstention, reliability, and efficiency.","descriptions":{"en":"CliniCARE-Bench evaluates clinical agents on retrospective audit tasks over longitudinal EHR data, measuring verdict accuracy, evidence grounding, process adherence, calibrated abstention, reliability, and efficiency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07796","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_clinicare-bench_6faa931e"},"modality":null,"name":"CliniCARE-Bench","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07796","source_url":"https://arxiv.org/abs/2608.07796"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07796","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07796","topics":["reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"ClinLens evaluates clinical data-science agents on 200 executable tasks over five linked MIMIC resources (EHR, notes, ECG, chest X-rays, echocardiograms), organized by a 4x5 taxonomy of patient-time scopes and analysis capabilities. Scoring uses a STRICTPASS metric requiring correct artifacts, semantics, and final answers.","descriptions":{"en":"ClinLens evaluates clinical data-science agents on 200 executable tasks over five linked MIMIC resources (EHR, notes, ECG, chest X-rays, echocardiograms), organized by a 4x5 taxonomy of patient-time scopes and analysis capabilities. Scoring uses a STRICTPASS metric requiring correct artifacts, semantics, and final answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.26155","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","professional-work"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_clinlens_3ca0c7ce"},"modality":null,"name":"ClinLens","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26155","source_url":"https://arxiv.org/abs/2607.26155"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26155","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:clinocr-bench/clinocr-bench","source_record":"bm_clinocr-bench_8be5c7a7"},"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ClinOCR-Bench is a public dataset of 384 scanned clinical documents across six artifact subsets (normal, handwriting, poor quality, rotation, tables, mixed). It evaluates OCR systems on clinical text extraction, with ground truth transcriptions and template-aware train/test splits supporting 0-shot and 1-shot evaluation.","descriptions":{"en":"ClinOCR-Bench is a public dataset of 384 scanned clinical documents across six artifact subsets (normal, handwriting, poor quality, rotation, tables, mixed). It evaluates OCR systems on clinical text extraction, with ground truth transcriptions and template-aware train/test splits supporting 0-shot and 1-shot evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.03650","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","health-medicine"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_clinocr-bench_8be5c7a7"},"modality":null,"name":"ClinOCR-Bench","openness":"unknown","publisher":null,"released":"2026-07-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03650","source_url":"https://arxiv.org/abs/2607.03650"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03650","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03650","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":"gh:multimodal-intelligence-lab/clip-cc-bench","source_record":"bm_clip-cc-bench_6e2be6c1"},"categories":["General AI","Multimodal","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CLIP-CC-Bench evaluates paragraph-level video description quality using 5-hour movie content and expert-written references. It scores 17 VLMs via coarse- and fine-grained semantic matching with five LLM-based embedding judges.","descriptions":{"en":"CLIP-CC-Bench evaluates paragraph-level video description quality using 5-hour movie content and expert-written references. It scores 17 VLMs via coarse- and fine-grained semantic matching with five LLM-based embedding judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04302","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_clip-cc-bench_6e2be6c1"},"modality":null,"name":"CLIP-CC-Bench","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04302","source_url":"https://arxiv.org/abs/2608.04302"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04302","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04302","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":151,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_clir-bench_e1c31f75"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for question answering over irregular clinical time series from ICU records, containing 6,600 QA instances across 11 clinical variables and 11 tasks. It evaluates answer accuracy and evidence use through explicit temporal evidence and answer derivation rules.","descriptions":{"en":"A benchmark for question answering over irregular clinical time series from ICU records, containing 6,600 QA instances across 11 clinical variables and 11 tasks. It evaluates answer accuracy and evidence use through explicit temporal evidence and answer derivation rules."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09880","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_clir-bench_e1c31f75"},"modality":null,"name":"CLIR-Bench","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09880","source_url":"https://arxiv.org/abs/2607.09880"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09880","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09880","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":133,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":"gh:future-house/lab-bench","source_record":"catalog_3102d317f87189a6"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","safety","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CloningScenarios is an expert-level multi-step reasoning benchmark about difficult genetic cloning scenarios in multiple-choice format. It evaluates dual-use biological knowledge relevant to bioweapons development.","descriptions":{"en":"CloningScenarios is an expert-level multi-step reasoning benchmark about difficult genetic cloning scenarios in multiple-choice format. It evaluates dual-use biological knowledge relevant to bioweapons development."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cloningscenarios:grok-4.1-thinking-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cloningscenarios?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cloningscenarios","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"llm_stats_id","primary":"safety-alignment","source_record":"catalog_3102d317f87189a6"},"modality":"text","name":"CloningScenarios","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":46.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.46,"raw_min":0.46,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cloningscenarios:grok-4.1-thinking-2025-11-17","reported_date":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cloningscenarios?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-cloningscenarios","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cloningscenarios?top_n=500","topics":["reasoning-math","safety","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"CLOSER-Bench evaluates budgeted cross-stage design closure for hardware agents with spec-to-RTL, RTL-to-GDS, and spec-to-GDS tasks, using open-source tools and recording quality, progress, tool cost, and recovery.","descriptions":{"en":"CLOSER-Bench evaluates budgeted cross-stage design closure for hardware agents with spec-to-RTL, RTL-to-GDS, and spec-to-GDS tasks, using open-source tools and recording quality, progress, tool cost, and recovery."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.16632","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_closer-bench_7847c9e5"},"modality":null,"name":"CLOSER-Bench","openness":"unknown","publisher":null,"released":"2026-07-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.16632","source_url":"https://arxiv.org/abs/2607.16632"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-16632","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16632","topics":[],"unit":null},{"aliases":["clouda-ocr-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sahrasayed3-crypto/clouda-ocr-benchmark","source_record":"bm_clouda-ocr-benchmark_4bff2fb7"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates distorted Arabic document OCR on 177 page images with reference transcriptions using normalized Arabic character error rate and a primary leaderboard for models with full coverage.","descriptions":{"en":"Evaluates distorted Arabic document OCR on 177 page images with reference transcriptions using normalized Arabic character error rate and a primary leaderboard for models with full coverage."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:sahrasayed3-crypto/clouda-ocr-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","multilingual"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_clouda-ocr-benchmark_4bff2fb7"},"modality":null,"name":"Clouda OCR Benchmark","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:sahrasayed3-crypto/clouda-ocr-benchmark","source_url":"https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-sahrasayed3-crypto-clouda-ocr-benchmark","source":"claire_radar","source_url":"https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CloudCons is an end-to-end benchmark for evaluating forecasting models in cloud resource consolidation. It encompasses datasets from Huawei Cloud, Microsoft Azure, and Google Borg with diverse workload patterns, and evaluates statistical, deep learning, and time series foundation models. The evaluation includes resource efficiency and service reliability metrics under a forecast-then-optimize paradigm.","descriptions":{"en":"CloudCons is an end-to-end benchmark for evaluating forecasting models in cloud resource consolidation. It encompasses datasets from Huawei Cloud, Microsoft Azure, and Google Borg with diverse workload patterns, and evaluates statistical, deep learning, and time series foundation models. The evaluation includes resource efficiency and service reliability metrics under a forecast-then-optimize paradigm."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13513","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_cloudcons_9f1d2495"},"modality":null,"name":"CloudCons","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13513","source_url":"https://arxiv.org/abs/2606.13513"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13513","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13513","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"CLUBench evaluates clustering algorithms across 131 datasets (tabular, text, image) using 24 algorithms, measuring clustering performance via standard metrics like NMI, ARI, etc.","descriptions":{"en":"CLUBench evaluates clustering algorithms across 131 datasets (tabular, text, image) using 24 algorithms, measuring clustering performance via standard metrics like NMI, ARI, etc."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29933","languages":[],"library_direction":null,"modality":null,"name":"CLUBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29933","source_url":"https://arxiv.org/abs/2605.29933"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29933","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29933","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark decomposing synthetic image detection into annotated forensic cognitive skills, used to evaluate a proposed detection framework.","descriptions":{"en":"A benchmark decomposing synthetic image detection into annotated forensic cognitive skills, used to evaluate a proposed detection framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.25009","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_clueaegis-bench_30a26f44"},"modality":null,"name":"ClueAegis-Bench","openness":"unknown","publisher":null,"released":"2026-05-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25009","source_url":"https://arxiv.org/abs/2605.25009"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25009","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25009","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4286,"github_stars_30d":null,"hf_dataset_downloads":7904,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:cluebenchmark/clue","source_record":"catalog_f44ad3b35d2776f3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CLUEWSC2020 is the Chinese version of the Winograd Schema Challenge, part of the CLUE benchmark. It focuses on pronoun disambiguation and coreference resolution, requiring models to determine which noun a pronoun refers to in a sentence. The dataset contains 1,244 training samples and 304 development samples extracted from contemporary Chinese literature.","descriptions":{"en":"CLUEWSC2020 is the Chinese version of the Winograd Schema Challenge, part of the CLUE benchmark. It focuses on pronoun disambiguation and coreference resolution, requiring models to determine which noun a pronoun refers to in a sentence. The dataset contains 1,244 training samples and 304 development samples extracted from contemporary Chinese literature."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cluewsc:deepseek-v3","reported_at":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cluewsc?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cluewsc","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","abstract-reasoning"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_f44ad3b35d2776f3"},"modality":"text","name":"CLUEWSC","openness":"unknown","publisher":null,"released":"2020-04-13","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CLUE: A Chinese Language Understanding Evaluation Benchmark. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:cluewsc","source_url":"https://arxiv.org/abs/2004.05986"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":91.4,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.914,"raw_min":0.486,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cluewsc:kimi-k1.5","reported_date":"2025-01-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cluewsc?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2025-01-20","date_precision":"model_announcement","name":"Kimi-k1.5"}],"slug":"llm-stats-cluewsc","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cluewsc?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","NAACL 2024","大语言模型","LLM","知识储备","Knowledge"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CMB is Comprehensive Medical Benchmark in Chinese, designed and rooted entirely within the native Chinese linguistic and cultural framework. CMB 是一个综合医学基准，专为中文而设计，并完全依赖于本土的中文语言和文化框架中。","descriptions":{"en":"CMB is Comprehensive Medical Benchmark in Chinese, designed and rooted entirely within the native Chinese linguistic and cultural framework.","zh":"CMB 是一个综合医学基准，专为中文而设计，并完全依赖于本土的中文语言和文化框架中。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1137","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"CMB","openness":"open","publisher":"The Chinese University of Hong Kong & Shenzhen Research Institute of Big Data","released":"2024-04-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1137-cmb","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CMB","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_5694f82dafbace0e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CMMLU (Chinese Massive Multitask Language Understanding) is a comprehensive Chinese benchmark that evaluates the knowledge and reasoning capabilities of large language models across 67 different subject topics. The benchmark covers natural sciences, social sciences, engineering, and humanities with multiple-choice questions ranging from basic to advanced professional levels.","descriptions":{"en":"CMMLU (Chinese Massive Multitask Language Understanding) is a comprehensive Chinese benchmark that evaluates the knowledge and reasoning capabilities of large language models across 67 different subject topics. The benchmark covers natural sciences, social sciences, engineering, and humanities with multiple-choice questions ranging from basic to advanced professional levels."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cmmlu:qwen2-72b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cmmlu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cmmlu","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_5694f82dafbace0e"},"modality":"text","name":"CMMLU","openness":"unknown","publisher":null,"released":"2023-06-15","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CMMLU: Measuring massive multitask language understanding in Chinese. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:cmmlu","source_url":"https://arxiv.org/abs/2306.09212"},"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":90.2,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.902,"raw_min":0.398,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cmmlu:mimo-v2.5-pro","reported_date":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cmmlu?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"}],"slug":"llm-stats-cmmlu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cmmlu?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2306.09212","repository":null,"source_record":"catalog_5694f82dafbace0e"},"categories":["学科","Examination","大语言模型","LLM","知识储备","Knowledge","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CMMLU is a comprehensive Chinese evaluation benchmark specifically designed to assess the knowledge and reasoning abilities of language models in the context of the Chinese language. CMMLU covers 67 topics ranging from basic subjects to advanced professional levels. It includes tasks that require calculations and reasoning in natural sciences, as well as tasks involving knowledge from humanities, social sciences, and practical aspects like Chinese driving rules. Moreover, many tasks within CMMLU have answers specific to China, which might not be universally applicable in other regions or languages. As a result, CMMLU serves as a fully localized Chinese evaluation benchmark. CMMLU是一个综合性的中文评估基准，专门用于评估语言模型在中文语境下的知识和推理能力。CMMLU涵盖了从基础学科到高级专业水平的67个主题。它包括：需要计算和推理的自然科学，需要知识的人文科学和社会科学,以及需要生活常识的中国驾驶规则等。此外，CMMLU中的许多任务具有中国特定的答案，可能在其他地区或语言中并不普遍适用。因此是一个完全中国化的中文测试基准。","descriptions":{"en":"CMMLU is a comprehensive Chinese evaluation benchmark specifically designed to assess the knowledge and reasoning abilities of language models in the context of the Chinese language. CMMLU covers 67 topics ranging from basic subjects to advanced professional levels. It includes tasks that require calculations and reasoning in natural sciences, as well as tasks involving knowledge from humanities, social sciences, and practical aspects like Chinese driving rules. Moreover, many tasks within CMMLU have answers specific to China, which might not be universally applicable in other regions or languages. As a result, CMMLU serves as a fully localized Chinese evaluation benchmark.","zh":"CMMLU是一个综合性的中文评估基准，专门用于评估语言模型在中文语境下的知识和推理能力。CMMLU涵盖了从基础学科到高级专业水平的67个主题。它包括：需要计算和推理的自然科学，需要知识的人文科学和社会科学,以及需要生活常识的中国驾驶规则等。此外，CMMLU中的许多任务具有中国特定的答案，可能在其他地区或语言中并不普遍适用。因此是一个完全中国化的中文测试基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:499","languages":["Chinese"],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"artifact:arxiv:2306.09212","primary":"knowledge-expert-qa","source_record":"catalog_5694f82dafbace0e"},"modality":null,"name":"CMMLU","openness":"unknown","publisher":null,"released":"2023-06-15","released_reference":{"basis":"paper_first_version","note":"First version introducing CMMLU; the 2024 revision is not its release.","source_key":"opencompass:499","source_url":"https://arxiv.org/abs/2306.09212"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-499-cmmlu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CMMLU","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"CMNIE jointly annotates event triggers, event arguments, named entities, and entity relations in 13,000 Chinese military news instances for structured extraction evaluation.","descriptions":{"en":"CMNIE jointly annotates event triggers, event arguments, named entities, and entity relations in 13,000 Chinese military news instances for structured extraction evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10722","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_cmnie_4eeb6461"},"modality":null,"name":"CMNIE","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10722","source_url":"https://arxiv.org/abs/2609.10722"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10722","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10722","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4286,"github_stars_30d":null,"hf_dataset_downloads":7904,"hf_paper_upvotes":0,"join":"artifact:arxiv:2004.05986","repository":"gh:cluebenchmark/clue","source_record":"catalog_f44ad3b35d2776f3"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CMNLI  is a Chinese natural language inference task, which requires to determine the logical relation between two sentences, with three relations: entailment, contradiction and neutral. CMNLI是一个中文自然语言推理任务，要求根据两个句子判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。","descriptions":{"en":"CMNLI  is a Chinese natural language inference task, which requires to determine the logical relation between two sentences, with three relations: entailment, contradiction and neutral.","zh":"CMNLI是一个中文自然语言推理任务，要求根据两个句子判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:524","languages":["Chinese"],"library_direction":{"basis":"model-review","categories":["multilingual","abstract-reasoning"],"join":"artifact:arxiv:2004.05986","primary":"multilingual","source_record":"catalog_f44ad3b35d2776f3"},"modality":null,"name":"CMNLI","openness":"unknown","publisher":null,"released":"2020-04-13","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CLUE: A Chinese Language Understanding Evaluation Benchmark. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"opencompass:524","source_url":"https://arxiv.org/abs/2004.05986"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-524-cmnli","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CMNLI","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_ba5232cc57812540"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["physics","reasoning","science"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CMT-Benchmark evaluates models on condensed matter theory problems, testing advanced physics reasoning across areas such as many-body systems, quantum field theory, and statistical mechanics.","descriptions":{"en":"CMT-Benchmark evaluates models on condensed matter theory problems, testing advanced physics reasoning across areas such as many-body systems, quantum field theory, and statistical mechanics."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cmt-benchmark:hy3","reported_at":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cmt-benchmark?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cmt-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_ba5232cc57812540"},"modality":"text","name":"CMT-Benchmark","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":37.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.379,"raw_min":0.379,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cmt-benchmark:hy3","reported_date":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cmt-benchmark?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"}],"slug":"llm-stats-cmt-benchmark","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cmt-benchmark?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"CN-NewsTTS Bench v0.1 evaluates Chinese news TTS pronunciation of dense written forms (scores, model names, ranges, units, percentages, abbreviations) from raw text, using 800 public test records and a 992-target auto-evaluable subset with ASR-ensemble transcripts and an automatic target scorer.","descriptions":{"en":"CN-NewsTTS Bench v0.1 evaluates Chinese news TTS pronunciation of dense written forms (scores, model names, ranges, units, percentages, abbreviations) from raw text, using 800 public test records and a 992-target auto-evaluable subset with ASR-ensemble transcripts and an automatic target scorer."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.24714","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_cn-newstts-bench_e74cef36"},"modality":null,"name":"CN-NewsTTS Bench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24714","source_url":"https://arxiv.org/abs/2606.24714"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24714","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24714","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","Robotics & Embodied AI","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark of 4,759 Chinese neologisms with reference definitions and a two-tier evaluation separating definition generation from mechanism-based manipulation tasks.","descriptions":{"en":"Benchmark of 4,759 Chinese neologisms with reference definitions and a two-tier evaluation separating definition generation from mechanism-based manipulation tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.28053","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","embodied-robotics"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_cneo-bench_bbdf3de1"},"modality":null,"name":"CNeo-Bench","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28053","source_url":"http://arxiv.org/abs/2608.28053v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28053","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28053v1","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"China Mathematical Olympiad 2024 - A challenging mathematics competition.","descriptions":{"en":"China Mathematical Olympiad 2024 - A challenging mathematics competition."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cnmo-2024:deepseek-v3","reported_at":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cnmo-2024?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cnmo-2024","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_ab6fd1363b38386d"},"modality":"text","name":"CNMO 2024","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":74.3,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.743,"raw_min":0.432,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cnmo-2024:kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cnmo-2024?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"}],"slug":"llm-stats-cnmo-2024","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cnmo-2024?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Planning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Construct-level benchmark with 897 oracle-validated household task instances evaluating four coordination constructs (allocation, ordering, mutual exclusion, handoff) via task success and construct-level scores.","descriptions":{"en":"Construct-level benchmark with 897 oracle-validated household task instances evaluating four coordination constructs (allocation, ordering, mutual exclusion, handoff) via task success and construct-level scores."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.28266","languages":[],"library_direction":null,"modality":null,"name":"CoCoBench","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28266","source_url":"http://arxiv.org/abs/2608.28266v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28266","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28266v1","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:melonkick3090/cocotree","source_record":"bm_cocotree_73046ddd"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"COCOTree evaluates open tree-structured visual decomposition, segmenting images into hierarchical trees of visual components with unconstrained granularity. It includes over 21K images and 1.8M structural nodes, with 3.5K unique labels, and uses the Open Tree Quality (OTQ) metric for scoring.","descriptions":{"en":"COCOTree evaluates open tree-structured visual decomposition, segmenting images into hierarchical trees of visual components with unconstrained granularity. It includes over 21K images and 1.8M structural nodes, with 3.5K unique labels, and uses the Open Tree Quality (OTQ) metric for scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.22068","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_cocotree_73046ddd"},"modality":null,"name":"COCOTree","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22068","source_url":"https://arxiv.org/abs/2605.22068"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22068","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22068","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robustness","Safety & Trustworthiness","Software & AI Compute","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"COD10K-C evaluates camouflaged object detection models under 8 corruption types at 5 severity levels, yielding 40 conditions and 81,040 image pairs based on COD10K. Models are scored on Dice and other standard metrics for segmentation robustness.","descriptions":{"en":"COD10K-C evaluates camouflaged object detection models under 8 corruption types at 5 severity levels, yielding 40 conditions and 81,040 image pairs based on COD10K. Models are scored on Dice and other standard metrics for segmentation robustness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02603","languages":[],"library_direction":null,"modality":null,"name":"COD10K-C","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02603","source_url":"https://arxiv.org/abs/2606.02603"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02603","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02603","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":48,"github_stars_30d":1,"hf_dataset_downloads":557,"hf_paper_upvotes":13,"join":"claire_record_id","repository":"gh:ruc-datalab/coda-bench","source_record":"bm_coda-bench_a3ad2891"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CODA-Bench is a benchmark for evaluating AI agents on data-intensive analytical tasks in a Linux sandbox with 1,009 tasks across 31 communities, requiring data discovery, code generation, and correct answers.","descriptions":{"en":"CODA-Bench is a benchmark for evaluating AI agents on data-intensive analytical tasks in a Linux sandbox with 1,009 tasks across 31 communities, requiring data discovery, code generation, and correct answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.15300","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","science"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_coda-bench_a3ad2891"},"modality":null,"name":"CODA-BENCH","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15300","source_url":"https://arxiv.org/abs/2606.15300"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15300","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15300","topics":["agents","science"],"unit":null},{"aliases":["code-taste-bench"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:dymoo/code-taste-bench","source_record":"bm_code-taste-bench_29d4bf7c"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates code-generating models on code taste across five rubric dimensions and pairwise duels, producing Taste Elo and per-dimension diagnostics.","descriptions":{"en":"Evaluates code-generating models on code taste across five rubric dimensions and pairwise duels, producing Taste Elo and per-dimension diagnostics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:dymoo/code-taste-bench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_code-taste-bench_29d4bf7c"},"modality":null,"name":"Code Taste Bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:dymoo/code-taste-bench","source_url":"https://github.com/dymoo/code-taste-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-dymoo-code-taste-bench","source":"claire_radar","source_url":"https://github.com/dymoo/code-taste-bench","topics":["coding"],"unit":null},{"aliases":["Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA"],"attention_signals":null,"categories":["Code & Software","Reasoning","Software & AI Compute"],"collected_at":"2026-09-25T04:40:22Z","description":"An open-source framework for generating repository-level question-answering benchmarks under closed-book, code-only, and documented conditions.","descriptions":{"en":"An open-source framework for generating repository-level question-answering benchmarks under closed-book, code-only, and documented conditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29277","languages":[],"library_direction":null,"modality":null,"name":"Code-QA-Bench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29277","source_url":"https://arxiv.org/abs/2605.29277"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29277","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29277","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_code4scene_e5c49d30"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely control scene state remains unclear.","descriptions":{"en":"Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely control scene state remains unclear."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.36777","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","spatial-3d"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_code4scene_e5c49d30"},"modality":null,"name":"Code4Scene","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36777","source_url":"https://arxiv.org/abs/2609.36777"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36777","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36777","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_codeactionbench_e22f3b62"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code?","descriptions":{"en":"How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code?"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33807","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","code-generation"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_codeactionbench_e22f3b62"},"modality":null,"name":"CodeActionBench","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33807","source_url":"https://arxiv.org/abs/2609.33807"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33807","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33807","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"CodeAssay is a benchmark of 185 Python tasks across ten software-engineering categories with audited ground truth, public and hidden tests, and code-property measures. It evaluates LLM code generation correctness and other code properties.","descriptions":{"en":"CodeAssay is a benchmark of 185 Python tasks across ten software-engineering categories with audited ground truth, public and hidden tests, and code-property measures. It evaluates LLM code generation correctness and other code properties."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03535","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_codeassay_bb1ca7aa"},"modality":null,"name":"CodeAssay","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03535","source_url":"https://arxiv.org/abs/2608.03535"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03535","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03535","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CodeCriticBench assesses LLMs' critiquing ability in code generation and QA tasks. Covering 10 criteria, it features a 4.3k-samples dataset with three difficulty levels and balanced distribution. CodeCriticBench ，旨在系统地评估LLMs在代码生成和代码问答任务中的批评能力。其涵盖 10 个不同的标准，数据集根据难度分为三个等级，共包含4.3k个样本，确保了难度级别的平衡分布。","descriptions":{"en":"CodeCriticBench assesses LLMs' critiquing ability in code generation and QA tasks. Covering 10 criteria, it features a 4.3k-samples dataset with three difficulty levels and balanced distribution.","zh":"CodeCriticBench ，旨在系统地评估LLMs在代码生成和代码问答任务中的批评能力。其涵盖 10 个不同的标准，数据集根据难度分为三个等级，共包含4.3k个样本，确保了难度级别的平衡分布。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1546","languages":[],"library_direction":null,"modality":null,"name":"CodeCriticBench","openness":"restricted","publisher":"NJU, M-A-P, Alibaba, etc.","released":"2025-02-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1546-codecriticbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CodeCriticBench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Researchers introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible. CodeElo，这是一个标准化的竞赛级代码生成基准测试，有效解决了所有这些挑战。CodeElo 基准测试主要基于官方 CodeForces 平台，并尽可能与该平台保持一致。","descriptions":{"en":"Researchers introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible.","zh":"CodeElo，这是一个标准化的竞赛级代码生成基准测试，有效解决了所有这些挑战。CodeElo 基准测试主要基于官方 CodeForces 平台，并尽可能与该平台保持一致。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1624","languages":[],"library_direction":null,"modality":null,"name":"CodeElo","openness":"open","publisher":"QwenTeam, Alibaba Group","released":"2025-01-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1624-codeelo","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CodeElo","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A competitive programming benchmark using problems from the CodeForces platform. The benchmark evaluates code generation capabilities of LLMs on algorithmic problems with difficulty ratings ranging from 800 to 2400. Problems cover diverse algorithmic categories including dynamic programming, graph algorithms, data structures, and mathematical problems with standardized evaluation through direct platform submission.","descriptions":{"en":"A competitive programming benchmark using problems from the CodeForces platform. The benchmark evaluates code generation capabilities of LLMs on algorithmic problems with difficulty ratings ranging from 800 to 2400. Problems cover diverse algorithmic categories including dynamic programming, graph algorithms, data structures, and mathematical problems with standardized evaluation through direct platform submission."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:codeforces:deepseek-v3.1","reported_at":"2025-01-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/codeforces?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:codeforces","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_e13a9b6e8ac54db6"},"modality":"text","name":"CodeForces","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":1.0,"display_multiplier":1,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":1.0,"raw_min":0.4763,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:codeforces:deepseek-v4-flash-max","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/codeforces?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2-Speciale"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 20B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 32B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-codeforces","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/codeforces?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["Codeforces","Codeforces ELO","Codeforces Rating"],"attention_signals":null,"categories":["coding"],"collected_at":null,"description":"An Elo estimate against human contestants, not a fixed dataset. The problem set moves continuously and rating conversion differs by vendor.","descriptions":{"en":"An Elo estimate against human contestants, not a fixed dataset. The problem set moves continuously and rating conversion differs by vendor."},"evidence_summary":{"document_count":5,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000codeforces\u0000bytedance_seed2_0_model_card\u0000codeforces\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000codeforces\u0000bytedance_seed2_0_model_card\u0000codeforces\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:codeforces","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["code-generation"],"join":"review:14H034160212:2026-10-15","primary":"code-generation","source_record":null},"modality":null,"name":"Codeforces","openness":"unknown","publisher":null,"released":"2010-02-19","released_reference":{"basis":"benchmark_release","source_key":"model-reports:codeforces","source_url":"https://codeforces.com/"},"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":3471.0,"display_multiplier":1,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":3471.0,"raw_min":1943.0,"source_reference":{"obs_id":"curated\u0000codeforces\u0000deepseek_v4_1_flash_technical_report\u0000codeforces\u0000think max, Rating\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000codeforces\u0000deepseek_v4_1_flash_technical_report\u0000codeforces\u0000think max, Rating\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"elo"},"scored_models":[{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"codeforces","source":"model_reports","source_url":"https://codeforces.com/","topics":["coding"],"unit":"elo"},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"CodegenBench evaluates LLM-generated parallel code across x86_64, Sunway, and Kunpeng architectures using BLAS routines and specialized kernels. Scoring measures efficiency on each platform.","descriptions":{"en":"CodegenBench evaluates LLM-generated parallel code across x86_64, Sunway, and Kunpeng architectures using BLAS routines and specialized kernels. Scoring measures efficiency on each platform."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04023","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","code-generation"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_codegenbench_b6866f0c"},"modality":null,"name":"CodegenBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04023","source_url":"https://arxiv.org/abs/2606.04023"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04023","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04023","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"CodeGolf Bench evaluates concise code generation across 60 programming languages, using code golf platform problems and human performance baselines for scoring.","descriptions":{"en":"CodeGolf Bench evaluates concise code generation across 60 programming languages, using code golf platform problems and human performance baselines for scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30394","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_codegolf-bench_db502fb1"},"modality":null,"name":"CodeGolf Bench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30394","source_url":"https://arxiv.org/abs/2605.30394"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30394","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30394","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Codegolf v2.2 benchmark","descriptions":{"en":"Codegolf v2.2 benchmark"},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:codegolf-v2.2:gemma-3n-e2b-it-litert-preview","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/codegolf-v2.2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:codegolf-v2.2","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_e261ed54dd2f1c07"},"modality":"text","name":"Codegolf v2.2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":16.8,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.168,"raw_min":0.11,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:codegolf-v2.2:gemma-3n-e4b-it","reported_date":"2025-06-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/codegolf-v2.2?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"}],"slug":"llm-stats-codegolf-v2-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/codegolf-v2.2?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CodeMMLU is a comprehensive benchmark designed to evaluate the capabilities of large language models (LLMs) in coding and software knowledge. It builds upon the structure of multiple-choice question answering (MCQA) to cover a wide range of programming tasks and domains. CodeMMLU 是一个旨在评估大型语言模型（LLMs）在编码和软件知识方面能力的全面基准。它基于多项选择题回答（MCQA）的结构，涵盖了广泛的编程任务和领域，包括代码生成、缺陷检测、软件工程原则等。","descriptions":{"en":"CodeMMLU is a comprehensive benchmark designed to evaluate the capabilities of large language models (LLMs) in coding and software knowledge. It builds upon the structure of multiple-choice question answering (MCQA) to cover a wide range of programming tasks and domains.","zh":"CodeMMLU 是一个旨在评估大型语言模型（LLMs）在编码和软件知识方面能力的全面基准。它基于多项选择题回答（MCQA）的结构，涵盖了广泛的编程任务和领域，包括代码生成、缺陷检测、软件工程原则等。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1582","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"artifact:arxiv:2406.15877","primary":"code-generation","source_record":"catalog_3d7c9b2f8573753d"},"modality":null,"name":"CodeMMLU","openness":"open","publisher":"Monash University, CSIRO’s Data61, etc.","released":"2024-06-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1582-codemmlu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CodeMMLU","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-09-25T04:40:22Z","description":"CodeRouterBench is an evaluation environment for comparing model routers on streaming coding tasks.","descriptions":{"en":"CodeRouterBench is an evaluation environment for comparing model routers on streaming coding tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22902","languages":[],"library_direction":null,"modality":null,"name":"CodeRouterBench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22902","source_url":"https://arxiv.org/abs/2606.22902"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22902","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22902","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sipratt-p/coding-agent-bench","source_record":"bm_coding-agent-bench_ed751df8"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Nine Python implementation tasks covering data structures, concurrency, parsing and regex work, scored by executing hidden test programs against extracted model answers, with a separate text-only code-quality heuristic and an agent-harness arena.","descriptions":{"en":"Nine Python implementation tasks covering data structures, concurrency, parsing and regex work, scored by executing hidden test programs against extracted model answers, with a separate text-only code-quality heuristic and an agent-harness arena."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:sipratt-p/coding-agent-bench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_coding-agent-bench_ed751df8"},"modality":null,"name":"coding-agent-bench","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:sipratt-p/coding-agent-bench","source_url":"https://github.com/sipratt-p/coding-agent-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-sipratt-p-coding-agent-bench","source":"claire_radar","source_url":"https://github.com/sipratt-p/coding-agent-bench","topics":["coding","agents"],"unit":null},{"aliases":["Codoku: Renewable Program-Reasoning Challenges for Frontier Coding Agents"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:connglli/codoku","source_record":"bm_codoku-renewable-program-reasoning-challen_64dfb75f"},"categories":["Code & Software","General AI","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on solving Codoku puzzles, where typed cells in partial programs must be filled to satisfy global static and dynamic constraints.","descriptions":{"en":"Evaluates coding agents on solving Codoku puzzles, where typed cells in partial programs must be filled to satisfy global static and dynamic constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.34661","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","abstract-reasoning"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_codoku-renewable-program-reasoning-challen_64dfb75f"},"modality":null,"name":"Codoku","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34661","source_url":"https://arxiv.org/abs/2609.34661"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34661","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34661","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Real-world driving is inherently multi-agent, yet most existing driving world models generate observations from a single ego vehicle.","descriptions":{"en":"Real-world driving is inherently multi-agent, yet most existing driving world models generate observations from a single ego vehicle."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34749","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","world-models"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_codrive-bench_99632fb2"},"modality":null,"name":"CoDrive-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34749","source_url":"https://arxiv.org/abs/2609.34749"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34749","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34749","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":41,"github_stars_30d":8,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:sakanaai/coffeebench","source_record":"bm_coffeebench_22526372"},"categories":["Agents","Language & Knowledge","Logistics & Operations"],"collected_at":"2026-10-14T09:44:51Z","description":"CoffeeBench evaluates LLM agents as a coffee roaster in a 90-day multi-agent economy with fixed reference agents, measuring cumulative net income through autonomous communication, negotiation, and transactions.","descriptions":{"en":"CoffeeBench evaluates LLM agents as a coffee roaster in a 90-day multi-agent economy with fixed reference agents, measuring cumulative net income through autonomous communication, negotiation, and transactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.16613","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","professional-work"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_coffeebench_22526372"},"modality":null,"name":"CoffeeBench","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16613","source_url":"https://arxiv.org/abs/2606.16613"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16613","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16613","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CogCanvas is a benchmark for multi-subject reference-based image generation, with 1,952 curated reference images, 1,361 compositional prompts, and a unified six-axis evaluation protocol. It includes tasks for reference-based generation, text-to-image composition, and reference retrieval, with metrics BG-Sim and Attr-VQA.","descriptions":{"en":"CogCanvas is a benchmark for multi-subject reference-based image generation, with 1,952 curated reference images, 1,361 compositional prompts, and a unified six-axis evaluation protocol. It includes tasks for reference-based generation, text-to-image composition, and reference retrieval, with metrics BG-Sim and Attr-VQA."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15867","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_cogcanvas_d3de924a"},"modality":null,"name":"CogCanvas","openness":"unknown","publisher":null,"released":"2026-06-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15867","source_url":"https://arxiv.org/abs/2606.15867"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15867","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15867","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates 15 manipulative behavior strategies in 1,000 multi-turn LLM interaction scenarios.","descriptions":{"en":"Evaluates 15 manipulative behavior strategies in 1,000 multi-turn LLM interaction scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06099","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_cogmanip_d6109b16"},"modality":null,"name":"CogManip","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06099","source_url":"https://arxiv.org/abs/2606.06099"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06099","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06099","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["question_answering","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Cohere's internal North evaluation for measuring how well a model answers enterprise questions using MCP-connected cloud file systems. Scores are reported with LLM-as-a-judge techniques.","descriptions":{"en":"Cohere's internal North evaluation for measuring how well a model answers enterprise questions using MCP-connected cloud file systems. Scores are reported with LLM-as-a-judge techniques."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cohere-agentic-question-answering:command-a-plus-05-2026","reported_at":"2026-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-agentic-question-answering?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cohere-agentic-question-answering","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","professional-work"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_a0b527c3bcba968d"},"modality":"text","name":"Cohere Agentic Question Answering","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":65.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.65,"raw_min":0.65,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cohere-agentic-question-answering:command-a-plus-05-2026","reported_date":"2026-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-agentic-question-answering?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"}],"slug":"llm-stats-cohere-agentic-question-answering","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-agentic-question-answering?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","data_analysis","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Cohere's internal North evaluation for measuring a model's ability to perform data science tasks over uploaded spreadsheets. Scores are reported with LLM-as-a-judge techniques.","descriptions":{"en":"Cohere's internal North evaluation for measuring a model's ability to perform data science tasks over uploaded spreadsheets. Scores are reported with LLM-as-a-judge techniques."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cohere-data-analysis:command-a-plus-05-2026","reported_at":"2026-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-data-analysis?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cohere-data-analysis","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_e9ba1beedb22161e"},"modality":"text","name":"Cohere Data Analysis","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":45.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.45,"raw_min":0.45,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cohere-data-analysis:command-a-plus-05-2026","reported_date":"2026-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-data-analysis?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"}],"slug":"llm-stats-cohere-data-analysis","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-data-analysis?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["memory","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Cohere's internal North evaluation for measuring how well an agent uses information from North's memory system across sessions. Scores are reported with LLM-as-a-judge techniques.","descriptions":{"en":"Cohere's internal North evaluation for measuring how well an agent uses information from North's memory system across sessions. Scores are reported with LLM-as-a-judge techniques."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cohere-memory-usage-quality:command-a-plus-05-2026","reported_at":"2026-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-memory-usage-quality?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cohere-memory-usage-quality","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"llm_stats_id","primary":"agent-memory","source_record":"catalog_b0e2f30b12a4f61b"},"modality":"text","name":"Cohere Memory Usage Quality","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":54.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.54,"raw_min":0.54,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cohere-memory-usage-quality:command-a-plus-05-2026","reported_date":"2026-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-memory-usage-quality?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"}],"slug":"llm-stats-cohere-memory-usage-quality","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-memory-usage-quality?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-humanoid collaboration policies in simulation under egocentric visual observations across 10 tasks with two or three humanoids.","descriptions":{"en":"Evaluates multi-humanoid collaboration policies in simulation under egocentric visual observations across 10 tasks with two or three humanoids."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34782","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","multi-agent-games"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_cohub-a-simulation-benchmark-for-multi-hum_fa7f9f59"},"modality":null,"name":"CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34782","source_url":"https://arxiv.org/abs/2609.34782"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34782","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34782","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":35,"github_stars_30d":3,"hf_dataset_downloads":2195,"hf_paper_upvotes":16,"join":"claire_record_id","repository":"gh:coinve200k/coinve-200k","source_record":"bm_coinve-bench_5d91b406"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CoinVE-Bench provides 361 multi-instruction test cases for compositional video editing, with four-dimensional evaluation metrics covering instruction following, editing accuracy, visual quality, and temporal consistency.","descriptions":{"en":"CoinVE-Bench provides 361 multi-instruction test cases for compositional video editing, with four-dimensional evaluation metrics covering instruction following, editing accuracy, visual quality, and temporal consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.17566","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_coinve-bench_5d91b406"},"modality":null,"name":"CoinVE-Bench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17566","source_url":"https://arxiv.org/abs/2608.17566"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17566","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17566","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":20,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:0217ljh/coldddi-neurips2026","source_record":"bm_coldddi_2ae3cf9a"},"categories":["Factuality","Language & Knowledge","Materials & Chemistry","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Cold-start drug-drug interaction (DDI) prediction tests whether models can identify clinically significant interactions for drugs without training-time interaction history.","descriptions":{"en":"Cold-start drug-drug interaction (DDI) prediction tests whether models can identify clinically significant interactions for drugs without training-time interaction history."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.05590","languages":[],"library_direction":{"basis":"model-review-daily","categories":["health-medicine","science"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_coldddi_2ae3cf9a"},"modality":null,"name":"ColdDDI","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05590","source_url":"https://arxiv.org/abs/2610.05590"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05590","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05590","topics":["medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_collabbench_1ee76547"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates collaborative ability of LLM agents in cooperative game environments with diverse player profiles and proactive engagement.","descriptions":{"en":"Evaluates collaborative ability of LLM agents in cooperative game environments with diverse player profiles and proactive engagement."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05793","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_collabbench_1ee76547"},"modality":null,"name":"CollabBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05793","source_url":"https://arxiv.org/abs/2606.05793"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05793","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05793","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":64,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_fcb758b78074e827"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language","writing"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"COLLIE is a grammar-based framework for systematic construction of constrained text generation tasks. It allows specification of rich, compositional constraints across diverse generation levels and modeling challenges including language understanding, logical reasoning, and semantic planning. The COLLIE-v1 dataset contains 2,080 instances across 13 constraint structures.","descriptions":{"en":"COLLIE is a grammar-based framework for systematic construction of constrained text generation tasks. It allows specification of rich, compositional constraints across diverse generation levels and modeling challenges including language understanding, logical reasoning, and semantic planning. The COLLIE-v1 dataset contains 2,080 instances across 13 constraint structures."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:collie:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/collie?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:collie","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_fcb758b78074e827"},"modality":"text","name":"COLLIE","openness":"unknown","publisher":null,"released":"2023-07-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: COLLIE: Systematic Construction of Constrained Text Generation Tasks. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:collie","source_url":"https://arxiv.org/abs/2307.08689"},"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":99.0,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.99,"raw_min":0.425,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:collie:gpt-5-2025-08-07","reported_date":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/collie?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"}],"slug":"llm-stats-collie","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/collie?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":["COLLIE"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Systematic construction of constrained text generation tasks; evaluates rich, compositional constraints across diverse generation levels.","descriptions":{"en":"Systematic construction of constrained text generation tasks; evaluates rich, compositional constraints across diverse generation levels."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000collie\u0000bytedance_seed2_0_model_card\u0000collie\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000collie\u0000bytedance_seed2_0_model_card\u0000collie\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:collie","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["instruction-following"],"join":"review:14H034160212:2026-10-15","primary":"instruction-following","source_record":null},"modality":null,"name":"COLLIE","openness":"unknown","publisher":null,"released":"2023-07-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: COLLIE: Systematic Construction of Constrained Text Generation Tasks. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"model-reports:collie","source_url":"https://arxiv.org/abs/2307.08689"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":94.0,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":94.0,"raw_min":91.2,"source_reference":{"obs_id":"curated\u0000collie\u0000bytedance_seed2_0_model_card\u0000collie\u0000as printed in Table 4 (Efficient Models)\u0000Seed2.0 Lite","observation_id":"curated\u0000collie\u0000bytedance_seed2_0_model_card\u0000collie\u0000as printed in Table 4 (Efficient Models)\u0000Seed2.0 Lite","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"collie","source":"model_reports","source_url":"https://collie-benchmark.github.io/","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":64,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2307.08689","repository":null,"source_record":"catalog_fcb758b78074e827"},"categories":["强推理","Strong Reasoning","大语言模型","LLM","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"COLLIE is a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage) and modeling challenges (e.g.,language understanding, logical reasoning, counting, semantic planning). COLLIE用于评估大模型在约束性文本生成任务中的表现，可指定具有不同生成级别（单词、句子、段落、段落）和建模挑战（例如，语言理解、逻辑推理、计数、语义规划）的丰富组合。","descriptions":{"en":"COLLIE is a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage) and modeling challenges (e.g.,language understanding, logical reasoning, counting, semantic planning).","zh":"COLLIE用于评估大模型在约束性文本生成任务中的表现，可指定具有不同生成级别（单词、句子、段落、段落）和建模挑战（例如，语言理解、逻辑推理、计数、语义规划）的丰富组合。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1250","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"artifact:arxiv:2307.08689","primary":"instruction-following","source_record":"catalog_fcb758b78074e827"},"modality":null,"name":"Collie","openness":"unknown","publisher":"Princeton University","released":"2023-07-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1250-collie","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Collie","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ColorBench, an innovative benchmark meticulously crafted to assess the capabilities of VLMs in color understanding, including color perception, reasoning, and robustness. ColorBench，这是一个创新且精心设计的基准测试，旨在评估VLMs在颜色理解方面的能力，包括颜色感知、推理和鲁棒性。","descriptions":{"en":"ColorBench, an innovative benchmark meticulously crafted to assess the capabilities of VLMs in color understanding, including color perception, reasoning, and robustness.","zh":"ColorBench，这是一个创新且精心设计的基准测试，旨在评估VLMs在颜色理解方面的能力，包括颜色感知、推理和鲁棒性。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1777","languages":[],"library_direction":null,"modality":"multimodal","name":"ColorBench","openness":"unknown","publisher":"University of Maryland, College Park","released":"2025-04-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1777-colorbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ColorBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Simulation benchmark for VLA generalization with 28 tasks across 13 categories and two robot morphologies, using ManiSkill.","descriptions":{"en":"Simulation benchmark for VLA generalization with 28 tasks across 13 categories and two robot morphologies, using ManiSkill."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27759","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_colosseum-v2_833bffbd"},"modality":null,"name":"Colosseum V2","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27759","source_url":"https://arxiv.org/abs/2605.27759"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27759","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27759","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":13,"github_stars_30d":-1,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"claire_record_id","repository":"gh:simplified-reasoning/combench","source_record":"bm_combench_b27be297"},"categories":["Language & Knowledge","Mathematics & Formal Science","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"ComBench evaluates large language models on 100 human-annotated Olympiad-level combinatorics problems, split into 50 analysis-centric and 50 construction-centric tasks. Scoring combines rubric-guided proof grading with deterministic verification of construction outputs.","descriptions":{"en":"ComBench evaluates large language models on 100 human-annotated Olympiad-level combinatorics problems, split into 50 analysis-centric and 50 construction-centric tasks. Scoring combines rubric-guided proof grading with deterministic verification of construction outputs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.10479","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_combench_b27be297"},"modality":null,"name":"ComBench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10479","source_url":"https://arxiv.org/abs/2606.10479"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10479","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10479","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yuxuzhou-cn/combination-problem-generation","source_record":"bm_combeval_06893a13"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CombEval evaluates combinatorial counting abilities of large language models using problems generated from typed Cofola specifications, with solver-verified answers. It supports systematic variation of object types, entity scales, constraint counts, and reasoning depth.","descriptions":{"en":"CombEval evaluates combinatorial counting abilities of large language models using problems generated from typed Cofola specifications, with solver-verified answers. It supports systematic variation of object types, entity scales, constraint counts, and reasoning depth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.19788","languages":[],"library_direction":{"basis":"model-review","categories":["math","abstract-reasoning"],"join":"claire_record_id","primary":"math","source_record":"bm_combeval_06893a13"},"modality":null,"name":"CombEval","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19788","source_url":"https://arxiv.org/abs/2606.19788"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19788","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19788","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["数学","Math","Combinatorics","Lean4","科学智能","AI for Science","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"We introduce CombiBench, a comprehensive benchmark comprising 100 combinatorial problems, each formalized in Lean4 and paired with its corresponding informal statement. The problem set covers a wide spectrum of difficulty levels, ranging from middle school to IMO and university level. CombiBench是一个包含100个组合问题的综合基准测试，每个问题都用Lean4进行了形式化，并附有其对应的非形式化表述。这些问题涵盖了从中学生到国际数学奥林匹克竞赛（IMO）以及大学水平的广泛难度范围。","descriptions":{"en":"We introduce CombiBench, a comprehensive benchmark comprising 100 combinatorial problems, each formalized in Lean4 and paired with its corresponding informal statement. The problem set covers a wide spectrum of difficulty levels, ranging from middle school to IMO and university level.","zh":"CombiBench是一个包含100个组合问题的综合基准测试，每个问题都用Lean4进行了形式化，并附有其对应的非形式化表述。这些问题涵盖了从中学生到国际数学奥林匹克竞赛（IMO）以及大学水平的广泛难度范围。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1943","languages":[],"library_direction":null,"modality":null,"name":"CombiBench","openness":"unknown","publisher":"AMSS, SYSU, Moonshot AI, Numina","released":"2025-04-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1943-combibench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CombiBench","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ComboShoppingBench is a benchmark for agentic basket shopping with coupons, evaluating LLM agents on constructing feasible baskets with budget and coupon constraints in a simulated environment.","descriptions":{"en":"ComboShoppingBench is a benchmark for agentic basket shopping with coupons, evaluating LLM agents on constructing feasible baskets with budget and coupon constraints in a simulated environment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09282","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_comboshoppingbench_78836e04"},"modality":null,"name":"ComboShoppingBench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09282","source_url":"https://arxiv.org/abs/2608.09282"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09282","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09282","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"CoMedBench evaluates synthetic medical data fidelity and downstream utility across 37 dataset-task pairs from seven public sources, using a common clinical-validity framework and shared training and evaluation engine.","descriptions":{"en":"CoMedBench evaluates synthetic medical data fidelity and downstream utility across 37 dataset-task pairs from seven public sources, using a common clinical-validity framework and shared training and evaluation engine."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12805","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_comedbench_50407db9"},"modality":null,"name":"CoMedBench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12805","source_url":"https://arxiv.org/abs/2608.12805"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12805","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12805","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information must remain isolated.","descriptions":{"en":"Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information must remain isolated."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32192","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","agent-memory"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_comembench_a37c2cfc"},"modality":null,"name":"CoMemBench","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32192","source_url":"https://arxiv.org/abs/2609.32192"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32192","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32192","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_comet-bench_355dabd3"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CoMET-Bench is a benchmark for conditional multi-event temporal grounding in long-form video, with 2,789 queries over 600 videos and a unified evaluation protocol including counting, grounding, and negative-query recognition.","descriptions":{"en":"CoMET-Bench is a benchmark for conditional multi-event temporal grounding in long-form video, with 2,789 queries over 600 videos and a unified evaluation protocol including counting, grounding, and negative-query recognition."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15320","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_comet-bench_355dabd3"},"modality":null,"name":"CoMET-Bench","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15320","source_url":"https://arxiv.org/abs/2606.15320"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15320","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15320","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"COMEX is a benchmark for explainable aesthetic image cropping containing 33,161 quadruples of expanded image, crop box, composition category, and composition-grounded explanation. It supports joint evaluation of crop localization, composition understanding, and explanation generation.","descriptions":{"en":"COMEX is a benchmark for explainable aesthetic image cropping containing 33,161 quadruples of expanded image, crop box, composition category, and composition-grounded explanation. It supports joint evaluation of crop localization, composition understanding, and explanation generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07570","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_comex_9077da26"},"modality":null,"name":"COMEX","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07570","source_url":"https://arxiv.org/abs/2608.07570"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07570","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07570","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":3491,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:common-voice/common-voice","source_record":"catalog_838d14d939d6a079"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["speech_to_text","language","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Common Voice is a massively-multilingual collection of transcribed speech intended for speech technology research and development. Version 15.0 contains 28,750 recorded hours across 114 languages, consisting of crowdsourced voice recordings with corresponding transcriptions.","descriptions":{"en":"Common Voice is a massively-multilingual collection of transcribed speech intended for speech technology research and development. Version 15.0 contains 28,750 recorded hours across 114 languages, consisting of crowdsourced voice recordings with corresponding transcriptions."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:common-voice-15:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/common-voice-15?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:common-voice-15","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_838d14d939d6a079"},"modality":"audio","name":"Common Voice 15","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.076,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.076,"raw_min":0.076,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:common-voice-15:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/common-voice-15?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-common-voice-15","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/common-voice-15?top_n=500","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":170,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_1f306761f559c50a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CommonSenseQA is a multiple-choice question answering dataset that requires different types of commonsense knowledge to predict correct answers. It contains 12,102 questions with one correct answer and four distractors, designed to test semantic reasoning and conceptual relationships. Questions are created based on ConceptNet concepts and require prior world knowledge for accurate reasoning.","descriptions":{"en":"CommonSenseQA is a multiple-choice question answering dataset that requires different types of commonsense knowledge to predict correct answers. It contains 12,102 questions with one correct answer and four distractors, designed to test semantic reasoning and conceptual relationships. Questions are created based on ConceptNet concepts and require prior world knowledge for accurate reasoning."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:commonsenseqa:mistral-nemo-instruct-2407","reported_at":"2024-07-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/commonsenseqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:commonsenseqa","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_1f306761f559c50a"},"modality":"text","name":"CommonSenseQA","openness":"unknown","publisher":null,"released":"2018-11-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:commonsenseqa","source_url":"https://arxiv.org/abs/1811.00937"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":70.39999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.704,"raw_min":0.704,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:commonsenseqa:mistral-nemo-instruct-2407","reported_date":"2024-07-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/commonsenseqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-18","date_precision":"model_announcement","name":"Mistral NeMo Instruct"}],"slug":"llm-stats-commonsenseqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/commonsenseqa?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":170,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:1811.00937","repository":null,"source_record":"catalog_1f306761f559c50a"},"categories":["知识","Knowledge","大语言模型","LLM","知识储备","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CommonsenseQA is a multiple-choice question answering dataset that requires different types of commonsense knowledge to predict the correct answers . It contains 12,102 questions with one correct answer and four distractor answers. CommonsenseQA是一个选择题数据集，它需要不同类型的常识知识来预测正确答案。它包含12,102个问题，有一个正确答案和四个干扰答案。","descriptions":{"en":"CommonsenseQA is a multiple-choice question answering dataset that requires different types of commonsense knowledge to predict the correct answers . It contains 12,102 questions with one correct answer and four distractor answers.","zh":"CommonsenseQA是一个选择题数据集，它需要不同类型的常识知识来预测正确答案。它包含12,102个问题，有一个正确答案和四个干扰答案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:511","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"artifact:arxiv:1811.00937","primary":"knowledge-expert-qa","source_record":"catalog_1f306761f559c50a"},"modality":null,"name":"CommonSenseQA","openness":"unknown","publisher":null,"released":"2018-11-02","released_reference":{"basis":"paper_first_version","note":"First version of the CommonsenseQA dataset paper.","source_key":"opencompass:511","source_url":"https://arxiv.org/abs/1811.00937"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-511-commonsenseqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CommonSenseQA","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_communityfact_765beb88"},"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"CommunityFact evaluates misinformation detection on 15,992 claims across five languages and two domains, using accuracy against human Community Notes ratings as the scoring metric.","descriptions":{"en":"CommunityFact evaluates misinformation detection on 15,992 claims across five languages and two domains, using accuracy against human Community Notes ratings as the scoring metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30241","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","multilingual"],"join":"claire_record_id","primary":"factuality","source_record":"bm_communityfact_765beb88"},"modality":null,"name":"CommunityFact","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30241","source_url":"https://arxiv.org/abs/2605.30241"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30241","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30241","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_companionbench_b876ca73"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"CompanionBench is an interactive bilingual benchmark for AI emotional companionship, grounding scenarios and a user simulator in de-identified real-world data. It evaluates ten capabilities derived from 25 theories, using a rubric and deterministic disclosure measure.","descriptions":{"en":"CompanionBench is an interactive bilingual benchmark for AI emotional companionship, grounding scenarios and a user simulator in de-identified real-world data. It evaluates ten capabilities derived from 25 theories, using a rubric and deterministic disclosure measure."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02046","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","health-medicine"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_companionbench_b876ca73"},"modality":null,"name":"CompanionBench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02046","source_url":"https://arxiv.org/abs/2608.02046"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02046","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02046","topics":["language","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:0xboyu/company-mental-model-benchmark","source_record":"bm_company-mental-model-benchmark_e52dec26"},"categories":["Finance","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Compares three evidence-to-thesis treatments for LLM-based company analysis across eight companies, scored by blind reviewers against reference models on identifying dominant compounding mechanisms.","descriptions":{"en":"Compares three evidence-to-thesis treatments for LLM-based company analysis across eight companies, scored by blind reviewers against reference models on identifying dominant compounding mechanisms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:0xboyu/company-mental-model-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Company Mental Model Benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:0xboyu/company-mental-model-benchmark","source_url":"https://github.com/0xboyu/company-mental-model-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-0xboyu-company-mental-model-benchmark","source":"claire_radar","source_url":"https://github.com/0xboyu/company-mental-model-benchmark","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:1612.00837","repository":null,"source_record":"catalog_afc35271db09b185"},"categories":["多模态","Multimodal","推理","Reasoning","NeurIPS 2024","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CompBench is designed to evaluate the comparative reasoning capability of multimodal large language models, including a collection of around 40K image pairs and visually oriented questions covering 8 dimensions CompBench旨在评估多模态大模型的比较推理能力，包含约4万个图像对及8个维度的视觉配对问题。","descriptions":{"en":"CompBench is designed to evaluate the comparative reasoning capability of multimodal large language models, including a collection of around 40K image pairs and visually oriented questions covering 8 dimensions","zh":"CompBench旨在评估多模态大模型的比较推理能力，包含约4万个图像对及8个维度的视觉配对问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1336","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:1612.00837","primary":"visual-reasoning","source_record":"catalog_afc35271db09b185"},"modality":"multimodal","name":"CompBench","openness":"unknown","publisher":"The Ohio State University","released":"2024-07-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1336-compbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CompBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Long Context"],"collected_at":"2026-09-25T04:40:22Z","description":"COMPINT measures how well context compactors preserve user-issued session constraints across chat, agentic, and research scenarios.","descriptions":{"en":"COMPINT measures how well context compactors preserve user-issued session constraints across chat, agentic, and research scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11242","languages":[],"library_direction":null,"modality":null,"name":"COMPINT","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11242","source_url":"https://arxiv.org/abs/2608.11242"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11242","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11242","topics":["long-context"],"unit":null},{"aliases":["ComplexConstraints and Beyond: Expert Rubrics for RLVR"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"An expert-curated instruction-following benchmark containing 75 public prompts and 1,559 atomic rubric criteria.","descriptions":{"en":"An expert-curated instruction-following benchmark containing 75 public prompts and 1,559 atomic rubric criteria."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09118","languages":[],"library_direction":null,"modality":null,"name":"ComplexConstraints","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09118","source_url":"https://arxiv.org/abs/2606.09118"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09118","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09118","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":181,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":22,"join":"llm_stats_id","repository":null,"source_record":"catalog_3e912c727f61a95c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","structured_output","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ComplexFuncBench is a benchmark designed to evaluate large language models' capabilities in handling complex function calling scenarios. It encompasses multi-step and constrained function calling tasks that require long-parameter filling, parameter value reasoning, and managing contexts up to 128k tokens. The benchmark includes 1,000 samples across five real-world scenarios.","descriptions":{"en":"ComplexFuncBench is a benchmark designed to evaluate large language models' capabilities in handling complex function calling scenarios. It encompasses multi-step and constrained function calling tasks that require long-parameter filling, parameter value reasoning, and managing contexts up to 128k tokens. The benchmark includes 1,000 samples across five real-world scenarios."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:complexfuncbench:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/complexfuncbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:complexfuncbench","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_3e912c727f61a95c"},"modality":"text","name":"ComplexFuncBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":66.5,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.665,"raw_min":0.057,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:complexfuncbench:gpt-4o-2024-08-06","reported_date":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/complexfuncbench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Sonic"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"}],"slug":"llm-stats-complexfuncbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/complexfuncbench?top_n=500","topics":["agents","reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"ComplexityWorld is a benchmark of 390 visual decision-making tasks across 39 worlds, scored by an executable verifier. It evaluates VLMs on tasks requiring global constraint satisfaction.","descriptions":{"en":"ComplexityWorld is a benchmark of 390 visual decision-making tasks across 39 worlds, scored by an executable verifier. It evaluates VLMs on tasks requiring global constraint satisfaction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07584","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","abstract-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_complexityworld_8718400b"},"modality":null,"name":"ComplexityWorld","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07584","source_url":"https://arxiv.org/abs/2608.07584"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07584","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07584","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_compskillbench_264db8aa"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"CompSkillBench evaluates compositional skill routing for LLM agents: given a user query and a library of 2,209 real MCP server skills across 24 categories, systems must decompose the query into sub-tasks, retrieve a skill per sub-task, and produce an executable plan. Scoring uses step-level category recall and decomposition accuracy.","descriptions":{"en":"CompSkillBench evaluates compositional skill routing for LLM agents: given a user query and a library of 2,209 real MCP server skills across 24 categories, systems must decompose the query into sub-tasks, retrieve a skill per sub-task, and produce an executable plan. Scoring uses step-level category recall and decomposition accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18051","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_compskillbench_264db8aa"},"modality":null,"name":"CompSkillBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18051","source_url":"https://arxiv.org/abs/2606.18051"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18051","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18051","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":39,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":50,"join":"claire_record_id","repository":"gh:inclusionai/conceptedit","source_record":"bm_conceptedit-bench_65e0c6a3"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A granular evaluation suite for image editing across over 1,000 fine-grained edit concepts, designed to diagnose model capabilities across real-world scenarios.","descriptions":{"en":"A granular evaluation suite for image editing across over 1,000 fine-grained edit concepts, designed to diagnose model capabilities across real-world scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.16812","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_conceptedit-bench_65e0c6a3"},"modality":null,"name":"ConceptEdit-Bench","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16812","source_url":"https://arxiv.org/abs/2608.16812"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16812","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16812","topics":[],"unit":null},{"aliases":["If, Then, Otherwise: Diagnosing Conditional Branching in Vision-Language Navigation"],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CondVLN evaluates vision-language navigation agents on 11,500 generated conditional instructions across four simulators, using standard VLN metrics plus Branch Selection Accuracy and Conditional Success Rate.","descriptions":{"en":"CondVLN evaluates vision-language navigation agents on 11,500 generated conditional instructions across four simulators, using standard VLN metrics plus Branch Selection Accuracy and Conditional Success Rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17318","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_if-then-otherwise-diagnosing-conditional-b_7e2fe0a2"},"modality":null,"name":"CondVLN","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17318","source_url":"https://arxiv.org/abs/2608.17318"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17318","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17318","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_confbench_a42d7b7f"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"ConfBench is a calibration-specific benchmark for key information extraction from documents. It applies 20 degradation pipelines to create 1,346 variants and over 70K entity-level evaluations, spanning the accuracy spectrum for evaluating confidence estimates of VLMs.","descriptions":{"en":"ConfBench is a calibration-specific benchmark for key information extraction from documents. It applies 20 degradation pipelines to create 1,346 variants and over 70K entity-level evaluations, spanning the accuracy spectrum for evaluating confidence estimates of VLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01792","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_confbench_a42d7b7f"},"modality":null,"name":"ConfBench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01792","source_url":"https://arxiv.org/abs/2608.01792"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01792","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01792","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A calibration benchmark evaluating verbalized confidence estimates in frontier LLMs using Brier scores across 200 multiple-choice questions in four categories. Scores are elicited via prompting without logits, applicable to closed and open models.","descriptions":{"en":"A calibration benchmark evaluating verbalized confidence estimates in frontier LLMs using Brier scores across 200 multiple-choice questions in four categories. Scores are elicited via prompting without logits, applicable to closed and open models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20526","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_confidencebench_21fa8523"},"modality":null,"name":"ConfidenceBench","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20526","source_url":"https://arxiv.org/abs/2607.20526"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20526","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20526","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_conflictbench_d8fc3760"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"ConflictBench is a benchmark with ConflictScore metric to quantify how well models acknowledge conflicting evidence in grounding documents, decomposing responses into claims and labeling them against documents.","descriptions":{"en":"ConflictBench is a benchmark with ConflictScore metric to quantify how well models acknowledge conflicting evidence in grounding documents, decomposing responses into claims and labeling them against documents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26437","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_conflictbench_d8fc3760"},"modality":null,"name":"ConflictBench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26437","source_url":"https://arxiv.org/abs/2606.26437"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26437","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26437","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"ConlangBench evaluates large language models on translation and vocabulary learning across 21 constructed languages, with a corpus of over 21 million conlang-English parallel sentence pairs and 321K vocabulary entries. The benchmark includes bidirectional translation tasks and learning-curve analysis for models trained on eight conlangs with sufficient parallel data.","descriptions":{"en":"ConlangBench evaluates large language models on translation and vocabulary learning across 21 constructed languages, with a corpus of over 21 million conlang-English parallel sentence pairs and 321K vocabulary entries. The benchmark includes bidirectional translation tasks and learning-curve analysis for models trained on eight conlangs with sufficient parallel data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03505","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_conlangbench_b9af66c3"},"modality":null,"name":"ConlangBench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03505","source_url":"https://arxiv.org/abs/2608.03505"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03505","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03505","topics":["language"],"unit":null},{"aliases":["ConnectomeBench"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"ConnectomeBench2 evaluates automated proofreading of connectomic segmentation errors across mouse, human, zebrafish, and fly data.","descriptions":{"en":"ConnectomeBench2 evaluates automated proofreading of connectomic segmentation errors across mouse, human, zebrafish, and fly data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.21116","languages":[],"library_direction":null,"modality":null,"name":"ConnectomeBench2","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21116","source_url":"https://arxiv.org/abs/2606.21116"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21116","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21116","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Connectors is an OpenAI internal production benchmark measuring reliable use of connector-based tools in agentic workflows, reported as a pass rate.","descriptions":{"en":"Connectors is an OpenAI internal production benchmark measuring reliable use of connector-based tools in agentic workflows, reported as a pass rate."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:openai-connectors:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-connectors?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:openai-connectors","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"llm_stats_id","primary":"agent-security","source_record":"catalog_9064fade0bec0331"},"modality":"text","name":"Connectors","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":100.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":1.0,"raw_min":0.999,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:openai-connectors:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-connectors?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-openai-connectors","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-connectors?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_contactworld_92ad6aa1"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ContactWorld is a benchmark and empirical study for vision-tactile world models in contact-rich manipulation, spanning 12 tasks including insertion and screwing, evaluating planning success rates.","descriptions":{"en":"ContactWorld is a benchmark and empirical study for vision-tactile world models in contact-rich manipulation, spanning 12 tasks including insertion and screwing, evaluating planning success rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13877","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","world-models"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_contactworld_92ad6aa1"},"modality":null,"name":"ContactWorld","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13877","source_url":"https://arxiv.org/abs/2606.13877"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13877","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13877","topics":[],"unit":null},{"aliases":["When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models"],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-09-25T04:40:22Z","description":"Context-flip evaluation tests whether aligned language models adjust their actions when a situational change reverses which action is safe.","descriptions":{"en":"Context-flip evaluation tests whether aligned language models adjust their actions when a situational change reverses which action is safe."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27851","languages":[],"library_direction":null,"modality":null,"name":"context-flip evaluation","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27851","source_url":"https://arxiv.org/abs/2605.27851"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27851","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27851","topics":["safety"],"unit":null},{"aliases":["ContextBias: Controlled Evaluation of Bias Persistence Under Context Shift in Text-to-Image Models"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":200,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:sina-emami/contextbias","source_record":"bm_contextbias-controlled-evaluation-of-bias-_978cb4aa"},"categories":["General AI","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates text-to-image models on 92 roles and 1,656 controlled prompts across context-free, related, and unrelated conditions, measuring bias persistence in visual attributes through metrics like Bias Intensity and Context Consistency Score.","descriptions":{"en":"Evaluates text-to-image models on 92 roles and 1,656 controlled prompts across context-free, related, and unrelated conditions, measuring bias persistence in visual attributes through metrics like Bias Intensity and Context Consistency Score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29847","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","safety-alignment"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_contextbias-controlled-evaluation-of-bias-_978cb4aa"},"modality":null,"name":"ContextBench","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29847","source_url":"https://arxiv.org/abs/2608.29847"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29847","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29847","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_contextecho_fd140a2e"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"ContextEcho measures persona drift in long agentic-coding sessions, combining a 25-probe identity suite, snapshot-then-probe protocol, and three anonymized Claude Code sessions.","descriptions":{"en":"ContextEcho measures persona drift in long agentic-coding sessions, combining a 25-probe identity suite, snapshot-then-probe protocol, and three anonymized Claude Code sessions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24279","languages":[],"library_direction":{"basis":"model-review-2","categories":["safety-alignment","agentic-coding"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_contextecho_fd140a2e"},"modality":null,"name":"ContextEcho","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24279","source_url":"https://arxiv.org/abs/2605.24279"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24279","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24279","topics":["coding","agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ContextShift is a controlled benchmark that manipulates object-context relationships in COCO images to isolate context as an independent variable for object detection evaluation.","descriptions":{"en":"ContextShift is a controlled benchmark that manipulates object-context relationships in COCO images to isolate context as an independent variable for object detection evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09495","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_contextshift_bf97ab71"},"modality":null,"name":"ContextShift","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09495","source_url":"https://arxiv.org/abs/2606.09495"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09495","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09495","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ContextualJudgeBench is a pairwise benchmark with 2,000 samples for evaluating LLM-as-judge models in two contextual settings: Contextual QA and summarization. We propose a pairwise evaluation hierarchy and generate splits for our proposed hierarchy. ContextualJudgeBench 是一个包含 2,000 个样本的成对基准，用于评估在两个上下文环境下的LLM-as-judge 模型：上下文问答和摘要。我们提出一个成对评估层次结构，并为我们的层次结构生成分割。","descriptions":{"en":"ContextualJudgeBench is a pairwise benchmark with 2,000 samples for evaluating LLM-as-judge models in two contextual settings: Contextual QA and summarization. We propose a pairwise evaluation hierarchy and generate splits for our proposed hierarchy.","zh":"ContextualJudgeBench 是一个包含 2,000 个样本的成对基准，用于评估在两个上下文环境下的LLM-as-judge 模型：上下文问答和摘要。我们提出一个成对评估层次结构，并为我们的层次结构生成分割。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1679","languages":[],"library_direction":null,"modality":null,"name":"ContextualJudgeBench","openness":"open","publisher":"Salesforce AI Research","released":"2025-03-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1679-contextualjudgebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ContextualJudgeBench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"ContextWeave is a longitudinal benchmark for evaluating memory in office workflows, with 1,005 executable tasks from 14 participants. It measures workspace quality and preference alignment.","descriptions":{"en":"ContextWeave is a longitudinal benchmark for evaluating memory in office workflows, with 1,005 executable tasks from 14 participants. It measures workspace quality and preference alignment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04830","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","professional-work"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_contextweave_c388d0b9"},"modality":null,"name":"ContextWeave","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04830","source_url":"https://arxiv.org/abs/2608.04830"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04830","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04830","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_continual-learning-bench_09825e20"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates continual learning in LLM-based systems across six expert-validated domains with stateful tasks sharing learnable latent structure.","descriptions":{"en":"Evaluates continual learning in LLM-based systems across six expert-validated domains with stateful tasks sharing learnable latent structure."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05661","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","agent-memory"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_continual-learning-bench_09825e20"},"modality":null,"name":"Continual Learning Bench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05661","source_url":"https://arxiv.org/abs/2606.05661"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05661","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05661","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:vishal-sys-code/continuity-bench","source_record":"bm_continuitybench_4edb780a"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ContinuityBench evaluates stateful failover in multi-provider LLM routing. It measures Continuity Preservation Rate (CPR) and Continuity Latency Overhead (CLO) using synthetic conversation graphs under injected provider failures, with an LLM-as-a-judge scoring context preservation.","descriptions":{"en":"ContinuityBench evaluates stateful failover in multi-provider LLM routing. It measures Continuity Preservation Rate (CPR) and Continuity Latency Overhead (CLO) using synthetic conversation graphs under injected provider failures, with an LLM-as-a-judge scoring context preservation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.15899","languages":[],"library_direction":null,"modality":null,"name":"ContinuityBench","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15899","source_url":"https://arxiv.org/abs/2607.15899"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15899","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15899","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:plau666/continuousbencheval","source_record":"bm_continuousbench_3f96ea58"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ContinuousBench evaluates differentially private synthetic text by measuring capability gain on QA sets derived from a new quarterly corpus (Geminon procedural data or News articles), with standardized training and evaluation harness.","descriptions":{"en":"ContinuousBench evaluates differentially private synthetic text by measuring capability gain on QA sets derived from a new quarterly corpus (Geminon procedural data or News articles), with standardized training and evaluation harness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01849","languages":[],"library_direction":null,"modality":null,"name":"ContinuousBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01849","source_url":"https://arxiv.org/abs/2606.01849"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01849","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01849","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_e48bc81d43c698f2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ContPhy is a continuum physical-reasoning benchmark evaluating understanding of physical dynamics in video.","descriptions":{"en":"ContPhy is a continuum physical-reasoning benchmark evaluating understanding of physical dynamics in video."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:contphy:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/contphy?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:contphy","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","video-understanding"],"join":"llm_stats_id","primary":"world-models","source_record":"catalog_e48bc81d43c698f2"},"modality":"multimodal","name":"ContPhy","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":63.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.636,"raw_min":0.611,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:contphy:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/contphy?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-contphy","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/contphy?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_contractscrub_bad94249"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on legal contract scrubbing tasks using hand-crafted contracts covering error categories like defined term misuse and inconsistent language.","descriptions":{"en":"Evaluates LLMs on legal contract scrubbing tasks using hand-crafted contracts covering error categories like defined term misuse and inconsistent language."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.20204","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_contractscrub_bad94249"},"modality":null,"name":"ContractScrub","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.20204","source_url":"https://arxiv.org/abs/2608.20204"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-20204","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20204","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"ConVBench is a vision-centric reasoning benchmark where each image is paired with two logically equivalent questions across six categories (action/state, complex counting, spatial reasoning, causal/intent, commonsense, temporal perception), with metrics for logical consistency and robust accuracy.","descriptions":{"en":"ConVBench is a vision-centric reasoning benchmark where each image is paired with two logically equivalent questions across six categories (action/state, complex counting, spatial reasoning, causal/intent, commonsense, temporal perception), with metrics for logical consistency and robust accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.21722","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_convbench_5f241ad0"},"modality":null,"name":"ConVBench","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21722","source_url":"https://arxiv.org/abs/2607.21722"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21722","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21722","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ConvBench is a novel multi-turn conversation evaluation benchmark tailored for Large Vision-Language Models (LVLMs). It comprises 577 meticulously curated multi-turn conversations encompassing 215 tasks reflective of real-world demands. ConvBench是专用于大型视觉语言模型 （LVLM）的新型多轮对话评估基准，由基于215 个反映实际需求的任务的577个多轮次对话组成。","descriptions":{"en":"ConvBench is a novel multi-turn conversation evaluation benchmark tailored for Large Vision-Language Models (LVLMs). It comprises 577 meticulously curated multi-turn conversations encompassing 215 tasks reflective of real-world demands.","zh":"ConvBench是专用于大型视觉语言模型 （LVLM）的新型多轮对话评估基准，由基于215 个反映实际需求的任务的577个多轮次对话组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1269","languages":[],"library_direction":null,"modality":null,"name":"ConvBench","openness":"open","publisher":"Shanghai AI Laboratory","released":"2024-03-29","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1269-convbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ConvBench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A proposed evaluation suite for geographic implicit neural representations across multiple spatial scales.","descriptions":{"en":"A proposed evaluation suite for geographic implicit neural representations across multiple spatial scales."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.25454","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_coordbench_4444d26d"},"modality":null,"name":"CoordBench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25454","source_url":"https://arxiv.org/abs/2609.25454"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25454","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25454","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"artifact:arxiv:1905.00537","repository":null,"source_record":"catalog_56ba0cb792f6bfce"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"COPA is a causal inference task, which requires to select the correct causal relation based on the given premise. COPA是一个因果推断任务，要求根据给定的前提，选择正确的因果关系。","descriptions":{"en":"COPA is a causal inference task, which requires to select the correct causal relation based on the given premise.","zh":"COPA是一个因果推断任务，要求根据给定的前提，选择正确的因果关系。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":"2019-05-02","first_score_source_reference":{"basis":"score_publication","date_precision":"score_publication","note":"The original benchmark release day is absent from the crawl. Table 2 provides the earliest dated numeric language-model evaluation retained in this date archive for the source-linked COPA task; it is not a benchmark release date.","reported_at":"2019-05-02","score_evidence":{"locator":"Table 2, BERT row, COPA column","metric":"accuracy","model":"BERT-large-cased (fine-tuned)","unit":"percent","value":69},"source_key":"opencompass:529","source_url":"https://arxiv.org/abs/1905.00537v1"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:529","languages":[],"library_direction":null,"modality":null,"name":"COPA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-529-copa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/COPA","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:03grant/sycophancy-and-stubbornness","source_record":"bm_cope-bench_8be06e5e"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Reliable language models should resist unsupported user pressure while effectively using objective contextual information.","descriptions":{"en":"Reliable language models should resist unsupported user pressure while effectively using objective contextual information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.04329","languages":[],"library_direction":{"basis":"model-review-daily","categories":["safety-alignment","instruction-following"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_cope-bench_8be06e5e"},"modality":null,"name":"CoPE-Bench","openness":"unknown","publisher":null,"released":"2026-10-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.04329","source_url":"https://arxiv.org/abs/2610.04329"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-04329","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04329","topics":["language","safety"],"unit":null},{"aliases":["CF-YOLO: Context-Aware Feature Refinement for Camouflaged Industrial Micro-Defect Detection"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates object detection on 1,847 copper tube images with 4,898 bounding-box defect instances for identifying surface micro-defects.","descriptions":{"en":"Evaluates object detection on 1,847 copper tube images with 4,898 bounding-box defect instances for identifying surface micro-defects."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.28070","languages":[],"library_direction":null,"modality":null,"name":"Copper Tube Defect Dataset","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28070","source_url":"https://arxiv.org/abs/2608.28070"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28070","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28070","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Copyright-Bench evaluates LLM agents' compliance with copyright law through realistic commercial tasks—website development, merchandise design, and pitch deck production—where agents choose between public-domain and copyrighted content, with prompt variations and time pressure.","descriptions":{"en":"Copyright-Bench evaluates LLM agents' compliance with copyright law through realistic commercial tasks—website development, merchandise design, and pitch deck production—where agents choose between public-domain and copyrighted content, with prompt variations and time pressure."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.21799","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","finance-legal"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_copyright-bench_ed85deec"},"modality":null,"name":"Copyright-Bench","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21799","source_url":"https://arxiv.org/abs/2607.21799"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21799","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21799","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:spotai-mbzuai/copyshield","source_record":"bm_copyshield_e7fa34f9"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Compares copyright defenses in LLMs across three intervention levels using controlled memorization over five public-domain books and a shared protocol measuring leakage, utility, and degeneracy.","descriptions":{"en":"Compares copyright defenses in LLMs across three intervention levels using controlled memorization over five public-domain books and a shared protocol measuring leakage, utility, and degeneracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.01161","languages":[],"library_direction":null,"modality":null,"name":"CopyShield","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01161","source_url":"https://arxiv.org/abs/2609.01161"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01161","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01161","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates neuron reconstruction from light microscopy at block and whole-brain scales, with structure-aware fiber prediction metrics for topological accuracy.","descriptions":{"en":"Evaluates neuron reconstruction from light microscopy at block and whole-brain scales, with structure-aware fiber prediction metrics for topological accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30768","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_coral_8daa2683"},"modality":null,"name":"CORAL","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30768","source_url":"https://arxiv.org/abs/2608.30768"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30768","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30768","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","RAG","大语言模型","LLM","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Researchers present a large-scale conversational RAG benchmark named CORAL and propose a unified framework for standardizing and evaluating various conversational RAG baselines. CORAL 是一个大规模对话 RAG 基准，包含一个统一框架，用于标准化和评估各种对话 RAG 基线。","descriptions":{"en":"Researchers present a large-scale conversational RAG benchmark named CORAL and propose a unified framework for standardizing and evaluating various conversational RAG baselines.","zh":"CORAL 是一个大规模对话 RAG 基准，包含一个统一框架，用于标准化和评估各种对话 RAG 基线。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1622","languages":[],"library_direction":null,"modality":null,"name":"CORAL","openness":"open","publisher":"RUC, BAAI, etc.","released":"2024-10-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1622-coral","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CORAL","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":89,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sileod/cordis-bench","source_record":"bm_cordisbench_3da54bce"},"categories":["Agents","General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates language models on reasoning about component lifecycle changes in dynamic agent harnesses via 1,200 questions covering localization, prediction, conditions, and reconfiguration.","descriptions":{"en":"Evaluates language models on reasoning about component lifecycle changes in dynamic agent harnesses via 1,200 questions covering localization, prediction, conditions, and reconfiguration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.01600","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_cordisbench_3da54bce"},"modality":null,"name":"CordisBench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01600","source_url":"https://arxiv.org/abs/2609.01600"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01600","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01600","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_corporatebench_6a07b031"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LLMs are increasingly able to answer complex questions about enterprise-scale document collections.","descriptions":{"en":"LLMs are increasingly able to answer complex questions about enterprise-scale document collections."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27391","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","long-context"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_corporatebench_6a07b031"},"modality":null,"name":"CorporateBench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27391","source_url":"https://arxiv.org/abs/2608.27391"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27391","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27391","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CorpusQA is a multi-document, free-form long-context question answering benchmark in which a model must retrieve and reason over information distributed across a large corpus to produce open-ended answers that are scored by an LLM judge.","descriptions":{"en":"CorpusQA is a multi-document, free-form long-context question answering benchmark in which a model must retrieve and reason over information distributed across a large corpus to produce open-ended answers that are scored by an LLM judge."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:corpusqa:mai-thinking-1","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:corpusqa","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_8733ada55656cd09"},"modality":"text","name":"CorpusQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":82.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.82,"raw_min":0.82,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:corpusqa:mai-thinking-1","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"}],"slug":"llm-stats-corpusqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":7,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:tongyi-zhiwen/corpusqa","source_record":"catalog_a03619ad23937b0d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CorpusQA 1M is a long-context question answering benchmark designed to evaluate models at approximately 1 million token contexts. Models are scored on accuracy when retrieving and reasoning over information distributed across an extremely long input corpus.","descriptions":{"en":"CorpusQA 1M is a long-context question answering benchmark designed to evaluate models at approximately 1 million token contexts. Models are scored on accuracy when retrieving and reasoning over information distributed across an extremely long input corpus."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:corpusqa-1m:deepseek-v4-flash-0423","reported_at":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa-1m?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:corpusqa-1m","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_a03619ad23937b0d"},"modality":"text","name":"CorpusQA 1M","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":62.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.62,"raw_min":0.593,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:corpusqa-1m:deepseek-v4-pro-max","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa-1m?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"}],"slug":"llm-stats-corpusqa-1m","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa-1m?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":["CoSec: Benchmarking Agent Security in Communities"],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"CoSec evaluates privacy and authorization enforcement in LLM agent systems across 208 scenarios involving community boundaries, protected information, and attacks via dialogue, environment, memory, and workflows.","descriptions":{"en":"CoSec evaluates privacy and authorization enforcement in LLM agent systems across 208 scenarios involving community boundaries, protected information, and attacks via dialogue, environment, memory, and workflows."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34790","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_cosec-benchmarking-agent-security-in-commu_5f79bf0a"},"modality":null,"name":"CoSec","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34790","source_url":"https://arxiv.org/abs/2609.34790"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34790","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34790","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_de43871974cdc7c1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","spatial_reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CountBench evaluates object counting capabilities in visual understanding.","descriptions":{"en":"CountBench evaluates object counting capabilities in visual understanding."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:countbench:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/countbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:countbench","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_de43871974cdc7c1"},"modality":"image","name":"CountBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":0.978,"display_multiplier":1,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.978,"raw_min":0.725,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:countbench:qwen3.5-27b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/countbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"}],"slug":"llm-stats-countbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/countbench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates defenses against generative engine optimization using 247 queries with rewrites, scoring attack success rate, false positive rate, and answer quality.","descriptions":{"en":"Evaluates defenses against generative engine optimization using 247 queries with rewrites, scoring attack success rate, false positive rate, and answer quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02316","languages":[],"library_direction":null,"modality":null,"name":"Counter-GEO-Bench","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02316","source_url":"https://arxiv.org/abs/2609.02316"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02316","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02316","topics":[],"unit":null},{"aliases":["Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning"],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-09-25T04:40:22Z","description":"A three-level benchmark evaluating whether text-to-image models can follow explicit and implicit counterfactual rules rather than defaulting to real-world visual priors.","descriptions":{"en":"A three-level benchmark evaluating whether text-to-image models can follow explicit and implicit counterfactual rules rather than defaulting to real-world visual priors."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.24548","languages":[],"library_direction":null,"modality":null,"name":"Counterfactual-World (CF-World)","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24548","source_url":"https://arxiv.org/abs/2606.24548"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24548","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24548","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_173d4c77defffb6a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CountQA is a benchmark for visual object counting and quantity reasoning over images.","descriptions":{"en":"CountQA is a benchmark for visual object counting and quantity reasoning over images."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:countqa:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/countqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:countqa","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_173d4c77defffb6a"},"modality":"multimodal","name":"CountQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":77.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.77,"raw_min":0.77,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:countqa:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/countqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-countqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/countqa?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":19,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":53,"join":"claire_record_id","repository":"gh:nju-link/covebench","source_record":"bm_covebench_94152ad0"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CoVEBench evaluates compositional video editing with 416 source videos, 626 multi-point instructions, and 9,990 checklist items, using MLLM judges and objective metrics.","descriptions":{"en":"CoVEBench evaluates compositional video editing with 416 source videos, 626 multi-point instructions, and 9,990 checklist items, using MLLM judges and objective metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.08415","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_covebench_94152ad0"},"modality":null,"name":"CoVEBench","openness":"unknown","publisher":null,"released":"2026-06-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08415","source_url":"https://arxiv.org/abs/2606.08415"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08415","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08415","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_1b29385c15b421f3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["speech_to_text","language","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CoVoST 2 is a large-scale multilingual speech translation corpus derived from Common Voice, covering translations from 21 languages into English and from English into 15 languages. The dataset contains 2,880 hours of speech with 78K speakers for speech translation research.","descriptions":{"en":"CoVoST 2 is a large-scale multilingual speech translation corpus derived from Common Voice, covering translations from 21 languages into English and from English into 15 languages. The dataset contains 2,880 hours of speech with 78K speakers for speech translation research."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:covost2:gemini-2.0-flash","reported_at":"2024-12-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:covost2","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_1b29385c15b421f3"},"modality":"audio","name":"CoVoST2","openness":"unknown","publisher":null,"released":"2020-07-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CoVoST 2 and Massively Multilingual Speech-to-Text Translation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:covost2","source_url":"https://arxiv.org/abs/2007.10310"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":40.699999999999996,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.407,"raw_min":0.384,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:covost2:nova-2-omni","reported_date":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"}],"slug":"llm-stats-covost2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2?top_n=500","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_aec207bcc15d0ca9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["speech_to_text","language","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CoVoST 2 English-to-Chinese subset is part of the large-scale multilingual speech translation corpus derived from Common Voice. This subset focuses specifically on English to Chinese speech translation tasks within the broader CoVoST 2 dataset.","descriptions":{"en":"CoVoST 2 English-to-Chinese subset is part of the large-scale multilingual speech translation corpus derived from Common Voice. This subset focuses specifically on English to Chinese speech translation tasks within the broader CoVoST 2 dataset."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:covost2-en-zh:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2-en-zh?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:covost2-en-zh","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_aec207bcc15d0ca9"},"modality":"audio","name":"CoVoST2 en-zh","openness":"unknown","publisher":null,"released":"2020-07-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CoVoST 2 and Massively Multilingual Speech-to-Text Translation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:covost2-en-zh","source_url":"https://arxiv.org/abs/2007.10310"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.414,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.414,"raw_min":0.414,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:covost2-en-zh:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2-en-zh?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-covost2-en-zh","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2-en-zh?top_n=500","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["productivity","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CoWorkBench is Qwen's internal cowork benchmark for evaluating long-horizon office and productivity agent tasks across domains such as computer science, finance, law, and medicine.","descriptions":{"en":"CoWorkBench is Qwen's internal cowork benchmark for evaluating long-horizon office and productivity agent tasks across domains such as computer science, finance, law, and medicine."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:coworkbench:qwen3.7-max","reported_at":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/coworkbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:coworkbench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_34983b29734ad7b0"},"modality":"text","name":"CoWorkBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":74.8,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.748,"raw_min":0.651,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:coworkbench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/coworkbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-coworkbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/coworkbench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":8,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"claire_record_id","repository":"gh:zqyzzz/cpi-benchmark","source_record":"bm_cpi-bench_2cecc896"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CPI-Bench evaluates image editing models across general, practical, and intelligent tasks with VLM-as-Judge scoring, including multi-image and reasoning-based editing.","descriptions":{"en":"CPI-Bench evaluates image editing models across general, practical, and intelligent tasks with VLM-as-Judge scoring, including multi-image and reasoning-based editing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.14546","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_cpi-bench_2cecc896"},"modality":null,"name":"CPI-Bench","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14546","source_url":"https://arxiv.org/abs/2608.14546"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14546","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14546","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","Problem Similarity","Competitive Programming","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Programming contests are long used to evaluate algorithmic thinking and coding skills, and have recently become benchmarks for assessing large language models (LLMs). However, the rapid expansion of problem sets has led to a surge in duplicate or highly similar problems, compromising fairness ... 编程竞赛长期用于评估算法与编程能力，近年来也被用于大语言模型的评测。但随着题库扩展，重复或相似题激增，影响竞赛公平性与模型评测效果。为此，本文提出“相似题目检索”任务，并构建统一检索基准数据集 CPRet，涵盖题目与代码的四类检索任务，包含自动爬取和人工标注的数据样本。同时，设计并训练了两种检索模型 CPRetriever-Code 与 CPRetriever-Prob，显著提升检索效果。实验还发现相似题会提高模型得分、减小模型差异，强调了评测中引入“相似性感知”的必要性。我们还发布了开源检索平台，支持重复题检测与相似题推荐。项目地址：https://github.com/coldchair/","descriptions":{"en":"Programming contests are long used to evaluate algorithmic thinking and coding skills, and have recently become benchmarks for assessing large language models (LLMs). However, the rapid expansion of problem sets has led to a surge in duplicate or highly similar problems, compromising fairness ...","zh":"编程竞赛长期用于评估算法与编程能力，近年来也被用于大语言模型的评测。但随着题库扩展，重复或相似题激增，影响竞赛公平性与模型评测效果。为此，本文提出“相似题目检索”任务，并构建统一检索基准数据集 CPRet，涵盖题目与代码的四类检索任务，包含自动爬取和人工标注的数据样本。同时，设计并训练了两种检索模型 CPRetriever-Code 与 CPRetriever-Prob，显著提升检索效果。实验还发现相似题会提高模型得分、减小模型差异，强调了评测中引入“相似性感知”的必要性。我们还发布了开源检索平台，支持重复题检测与相似题推荐。项目地址：https://github.com/coldchair/"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1972","languages":[],"library_direction":null,"modality":null,"name":"CPRet","openness":"unknown","publisher":"Tsinghua University, Shanghai AI Laboratory","released":"2025-05-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1972-cpret","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CPRet","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Assesses LLM-assisted ontology generation from competency questions using gold OWL ontologies with provenance.","descriptions":{"en":"Assesses LLM-assisted ontology generation from competency questions using gold OWL ontologies with provenance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.26029","languages":[],"library_direction":null,"modality":null,"name":"CQ4OE","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.26029","source_url":"https://arxiv.org/abs/2609.26029"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-26029","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26029","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A session-layer framework for tracking conversational risk accumulation in multi-turn LLM systems, including CRA-Bench datasets and trajectory-native evaluation protocols.","descriptions":{"en":"A session-layer framework for tracking conversational risk accumulation in multi-turn LLM systems, including CRA-Bench datasets and trajectory-native evaluation protocols."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19361","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_cra-bench_d9f4029f"},"modality":null,"name":"CRA-Bench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19361","source_url":"https://arxiv.org/abs/2607.19361"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19361","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19361","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_crab-bench_a7bbb55e"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"CRAB-Bench evaluates LLM agents on tasks generated via a constraint graph over multiple interdependent entities, using the RUSE user simulator with imperfect behavior. Scoring is pass@1 against valid solutions.","descriptions":{"en":"CRAB-Bench evaluates LLM agents on tasks generated via a constraint graph over multiple interdependent entities, using the RUSE user simulator with imperfect behavior. Scoring is pass@1 against valid solutions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01815","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_crab-bench_a7bbb55e"},"modality":null,"name":"CRAB-Bench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01815","source_url":"https://arxiv.org/abs/2606.01815"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01815","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01815","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":390,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_crackedpdfs_1a52f043"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates hidden prompt injection detection in PDFs through classification and paired ranking tasks, using 29,322 generated PDFs from 4,983 base documents. Includes frozen splits, features, and metrics for reproducibility.","descriptions":{"en":"Evaluates hidden prompt injection detection in PDFs through classification and paired ranking tasks, using 29,322 generated PDFs from 4,983 base documents. Includes frozen splits, features, and metrics for reproducibility."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.19396","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_crackedpdfs_1a52f043"},"modality":null,"name":"CrackedPDFs","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19396","source_url":"https://arxiv.org/abs/2607.19396"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19396","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19396","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":164,"github_stars_30d":7,"hf_dataset_downloads":null,"hf_paper_upvotes":64,"join":"claire_record_id","repository":"gh:haozhezhao/crafter","source_record":"bm_craftbench_a8850dcb"},"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"CraftBench evaluates scientific figure generation across three figure types and four input conditions, with human-drawn targets and a referenced VLM judge for scoring.","descriptions":{"en":"CraftBench evaluates scientific figure generation across three figure types and four input conditions, with human-drawn targets and a referenced VLM judge for scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.30611","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","science"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_craftbench_a8850dcb"},"modality":null,"name":"CraftBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30611","source_url":"https://arxiv.org/abs/2605.30611"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30611","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30611","topics":["agents","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":307,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":46,"join":"llm_stats_id","repository":null,"source_record":"catalog_53852581fcc7cfc2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search","finance","economics"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CRAG (Comprehensive RAG Benchmark) is a factual question answering benchmark consisting of 4,409 question-answer pairs across 5 domains (finance, sports, music, movie, open domain) and 8 question categories. The benchmark includes mock APIs to simulate web and Knowledge Graph search, designed to represent the diverse and dynamic nature of real-world QA tasks with temporal dynamism ranging from years to seconds. It evaluates retrieval-augmented generation systems for trustworthy question answering.","descriptions":{"en":"CRAG (Comprehensive RAG Benchmark) is a factual question answering benchmark consisting of 4,409 question-answer pairs across 5 domains (finance, sports, music, movie, open domain) and 8 question categories. The benchmark includes mock APIs to simulate web and Knowledge Graph search, designed to represent the diverse and dynamic nature of real-world QA tasks with temporal dynamism ranging from years to seconds. It evaluates retrieval-augmented generation systems for trustworthy question answering."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:crag:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crag?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:crag","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","factuality"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_53852581fcc7cfc2"},"modality":"text","name":"CRAG","openness":"unknown","publisher":null,"released":"2024-06-07","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CRAG -- Comprehensive RAG Benchmark. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:crag","source_url":"https://arxiv.org/abs/2406.04744"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":50.3,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.503,"raw_min":0.431,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:crag:nova-pro","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crag?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"}],"slug":"llm-stats-crag","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crag?top_n=500","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":307,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":46,"join":"artifact:arxiv:2406.04744","repository":null,"source_record":"catalog_53852581fcc7cfc2"},"categories":["其他","Other","RAG","大语言模型","LLM","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The Comprehensive RAG Benchmark (CRAG) is a rich and comprehensive factual question answering benchmark designed to advance research in RAG. Besides question-answer pairs, CRAG provides mock APIs to simulate web and knowledge graph search. CRAG是一个丰富且全面的基于事实的问题回答基准，旨在推进 RAG 研究。除了问答对之外，CRAG 还提供了模拟网页和知识图谱搜索的模拟 API。","descriptions":{"en":"The Comprehensive RAG Benchmark (CRAG) is a rich and comprehensive factual question answering benchmark designed to advance research in RAG. Besides question-answer pairs, CRAG provides mock APIs to simulate web and knowledge graph search.","zh":"CRAG是一个丰富且全面的基于事实的问题回答基准，旨在推进 RAG 研究。除了问答对之外，CRAG 还提供了模拟网页和知识图谱搜索的模拟 API。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1626","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","factuality"],"join":"artifact:arxiv:2406.04744","primary":"deep-research","source_record":"catalog_53852581fcc7cfc2"},"modality":null,"name":"CRAG","openness":"restricted","publisher":"Meta Reality Labs,  FAIR, Meta, etc.","released":"2024-06-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1626-crag","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CRAG","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Information retrieval","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"CRAG-MM-Diagnostics is a diagnostic benchmark for knowledge-intensive visual question answering (KI-VQA) with stage-wise annotations to isolate visual grounding, object identification, and knowledge retrieval/reasoning, including metadata like target ROIs and visual complexity scores.","descriptions":{"en":"CRAG-MM-Diagnostics is a diagnostic benchmark for knowledge-intensive visual question answering (KI-VQA) with stage-wise annotations to isolate visual grounding, object identification, and knowledge retrieval/reasoning, including metadata like target ROIs and visual complexity scores."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.21155","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_crag-mm-diagnostics_8195c595"},"modality":null,"name":"CRAG-MM-Diagnostics","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21155","source_url":"https://arxiv.org/abs/2607.21155"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21155","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Manufacturing & Process Control","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"With the advances of coding agents, software developers are spending more time reviewing code than writing code.","descriptions":{"en":"With the advances of coding agents, software developers are spending more time reviewing code than writing code."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13668","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_crbugbench_febe2cb9"},"modality":null,"name":"CRBugBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13668","source_url":"https://arxiv.org/abs/2610.13668"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13668","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13668","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","长文本","Long-Context","创作","Creation","多模态模型","VLM","长上下文","Long Context","语言生成","Generation","官方自建","Official"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A multimodal benchmark specifically designed to evaluate the creative capabilities of MLLMs. It features three main aspects: 1. Comprehensive Creation Benchmark for MLLM and LLM. 2. Robust Evaluation Methodology. 3. Attractive Experiment Insight. 专为评估 多模态大模型 的创作能力而设计的多模态基准。采用两个不同指标对模型的基础感知能力和深层次视觉创作能力进行评估，采用GPT-4o作为评判模型进行评估。","descriptions":{"en":"A multimodal benchmark specifically designed to evaluate the creative capabilities of MLLMs. It features three main aspects: 1. Comprehensive Creation Benchmark for MLLM and LLM. 2. Robust Evaluation Methodology. 3. Attractive Experiment Insight.","zh":"专为评估 多模态大模型 的创作能力而设计的多模态基准。采用两个不同指标对模型的基础感知能力和深层次视觉创作能力进行评估，采用GPT-4o作为评判模型进行评估。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1643","languages":[],"library_direction":null,"modality":"multimodal","name":"Creation-MMBench","openness":"unknown","publisher":"opencompass","released":"2025-03-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1643-creation-mmbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Creation-MMBench","topics":["multimodal","long-context","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":146,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eq-bench/creative-writing-bench","source_record":"catalog_05a6cb704e35a0f8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["creativity","writing"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"EQ-Bench Creative Writing v3 is an LLM-judged creative writing benchmark that evaluates models across 32 writing prompts with 3 iterations per prompt. Uses a hybrid scoring system combining rubric assessment and Elo ratings through pairwise comparisons. Challenges models in areas like humor, romance, spatial awareness, and unique perspectives to assess emotional intelligence and creative writing capabilities.","descriptions":{"en":"EQ-Bench Creative Writing v3 is an LLM-judged creative writing benchmark that evaluates models across 32 writing prompts with 3 iterations per prompt. Uses a hybrid scoring system combining rubric assessment and Elo ratings through pairwise comparisons. Challenges models in areas like humor, romance, spatial awareness, and unique perspectives to assess emotional intelligence and creative writing capabilities."},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:creative-writing-v3:qwen3-235b-a22b-instruct-2507","reported_at":"2025-07-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/creative-writing-v3?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:creative-writing-v3","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_05a6cb704e35a0f8"},"modality":"text","name":"Creative Writing v3","openness":"unknown","publication":{"evidence":[{"note":"The EQ-Bench repository explicitly identifies Creative Writing Benchmark v3 and links the dedicated methodology section. Its 32 prompts, three iterations, humour/romance/spatial-perspective challenges and hybrid rubric/pairwise scoring match this record, not generic EQ-Bench or Creative Writing v2.","url":"https://github.com/EQ-bench/creative-writing-bench/blob/main/README.md"},{"note":"The accessible author-maintained v3 section links the repository and documents the March 2026 Elo judge change. Historical score settings are therefore not inferred from the current judge configuration.","url":"https://eqbench.com/about.html#creative-writing-v3"}],"id":"creative-writing-v3-original","links":[{"kind":"repo","title":"Creative Writing Benchmark v3 official repository","url":"https://github.com/EQ-bench/creative-writing-bench"},{"kind":"website","title":"Creative Writing v3 methodology","url":"https://eqbench.com/about.html#creative-writing-v3"}],"record_keys":["llm-stats:creative-writing-v3"],"reviewed_at":"2026-10-26","scope":{"en":"EQ-Bench's Creative Writing v3 uses 32 prompts over three iterations with rubric and pairwise ranking. Its judges have changed over time; current documentation does not establish the judge used for every historical score.","zh":"EQ-Bench 的 Creative Writing v3 使用 32 道提示、三轮生成及评分量表与成对排名。评审模型曾发生变更，当前文档不能证明每项历史成绩采用的评审模型。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":87.5,"display_multiplier":100,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":0.875,"raw_min":0.761,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:creative-writing-v3:qwen3-235b-a22b-instruct-2507","reported_date":"2025-07-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/creative-writing-v3?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1"},{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"}],"slug":"llm-stats-creative-writing-v3","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/creative-writing-v3?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CreativeWork evaluates agents on open-ended creative production tasks within realistic tool and application environments, measuring the quality and completeness of generated deliverables.","descriptions":{"en":"CreativeWork evaluates agents on open-ended creative production tasks within realistic tool and application environments, measuring the quality and completeness of generated deliverables."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:creativework:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/creativework?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:creativework","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","professional-work","tool-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_d7cac9c36bec7188"},"modality":"multimodal","name":"CreativeWork","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":42.5,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.425,"raw_min":0.345,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:creativework:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/creativework?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-creativework","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/creativework?top_n=500","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":["credit-risk-tabpfn-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mehkh-analysis/credit-risk-tabpfn-benchmark","source_record":"bm_credit-risk-tabpfn-benchmark_744b243f"},"categories":["Chip Design & EDA","Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The repository compares TabPFN with traditional machine learning models on a credit risk dataset, but lacks a formally declared benchmark name or explicit public submission path.","descriptions":{"en":"The repository compares TabPFN with traditional machine learning models on a credit risk dataset, but lacks a formally declared benchmark name or explicit public submission path."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:mehkh-analysis/credit-risk-tabpfn-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_credit-risk-tabpfn-benchmark_744b243f"},"modality":null,"name":"Credit Risk Benchmark: TabPFN vs. Traditional ML","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:mehkh-analysis/credit-risk-tabpfn-benchmark","source_url":"https://github.com/MehKh-Analysis/credit-risk-tabpfn-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-mehkh-analysis-credit-risk-tabpfn-benchmark","source":"claire_radar","source_url":"https://github.com/MehKh-Analysis/credit-risk-tabpfn-benchmark","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CREW-WILDFIRE is a benchmark designed to evaluate large language model-based multi-agent systems’ collaboration capabilities in complex, dynamic tasks, targeting agentic frameworks with perception, planning, and execution abilities. CREW-WILDFIRE 是一个用于评估基于大语言模型的多智能体系统在复杂动态任务中协作能力的基准，面向具备感知、规划与执行能力的智能体框架。","descriptions":{"en":"CREW-WILDFIRE is a benchmark designed to evaluate large language model-based multi-agent systems’ collaboration capabilities in complex, dynamic tasks, targeting agentic frameworks with perception, planning, and execution abilities.","zh":"CREW-WILDFIRE 是一个用于评估基于大语言模型的多智能体系统在复杂动态任务中协作能力的基准，面向具备感知、规划与执行能力的智能体框架。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2046","languages":[],"library_direction":null,"modality":null,"name":"CREW-WILDFIRE","openness":"unknown","publisher":"Duke University , Army Research Laboratory","released":"2025-07-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2046-crew-wildfire","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CREW-WILDFIRE","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:dyshang/dacri-criticalscm-bench","source_record":"bm_criticalscm-bench_de361f05"},"categories":["Factuality","General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"CriticalSCM-Bench v1 is a synthetic supply-chain benchmark with causal ground truth, paired factual/counterfactual rollouts, and an explicit net-value objective. It evaluates intervention ranking policies across semiconductor, critical-material, and digital-infrastructure archetypes under partial and delayed information, with stress tests on fidelity, timing, cost, and held-out disruptions.","descriptions":{"en":"CriticalSCM-Bench v1 is a synthetic supply-chain benchmark with causal ground truth, paired factual/counterfactual rollouts, and an explicit net-value objective. It evaluates intervention ranking policies across semiconductor, critical-material, and digital-infrastructure archetypes under partial and delayed information, with stress tests on fidelity, timing, cost, and held-out disruptions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11154","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_criticalscm-bench_de361f05"},"modality":null,"name":"CriticalSCM-Bench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11154","source_url":"https://arxiv.org/abs/2608.11154"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11154","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11154","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CriticBench, a novel benchmark designed to comprehensively and reliably evaluate four key critique ability dimensions of LLMs: feedback, comparison, refinement and meta-feedback. CriticBench encompasses nine diverse tasks, each assessing the LLMs' ability to critique responses at varying levels of quality granularity. CriticBench是一个新颖的基准，旨在全面可靠地评估LLM的四个关键批判能力维度。CriticBench包括九项不同的任务，每项任务都评估LLM在不同质量粒度水平上对响应进行批评的能力。","descriptions":{"en":"CriticBench, a novel benchmark designed to comprehensively and reliably evaluate four key critique ability dimensions of LLMs: feedback, comparison, refinement and meta-feedback. CriticBench encompasses nine diverse tasks, each assessing the LLMs' ability to critique responses at varying levels of quality granularity.","zh":"CriticBench是一个新颖的基准，旨在全面可靠地评估LLM的四个关键批判能力维度。CriticBench包括九项不同的任务，每项任务都评估LLM在不同质量粒度水平上对响应进行批评的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:568","languages":[],"library_direction":null,"modality":null,"name":"CriticBench","openness":"unknown","publisher":null,"released":"2024-02-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-568-criticbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CriticBench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","reasoning","science"],"collected_at":"2026-08-25T10:41:06Z","description":"Physics reasoning","descriptions":{"en":"Physics reasoning"},"evidence_summary":{"document_count":1,"model_count":492,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:critpt:217b34ec-5920-4fc1-8886-6a70a324837d","reported_at":"2023-09-27","source_url":"https://artificialanalysis.ai/evaluations/critpt"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:critpt","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["science"],"join":"review:14H034160212:2026-10-15","primary":"science","source_record":null},"modality":null,"name":"CritPt","openness":"unknown","publisher":null,"released":"2025-11-21","released_reference":{"basis":"release_announcement","note":"The project announces its public challenge dataset and evaluation pipeline, and its debut on Artificial Analysis, on November 21. Earlier scored model releases do not date this benchmark.","source_key":"artificial-analysis:critpt","source_url":"https://github.com/CritPt-Benchmark/CritPt"},"repo_kind":null,"repo_resolution_status":null,"score_count":492,"score_direction":"higher_is_better","score_summary":{"display_max":32.2857142857143,"display_multiplier":100,"model_count":492,"model_count_basis":"source_model_id","numeric_count":492,"raw_max":0.322857142857143,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:critpt:d93edfe8-bf35-49ad-b56e-b18116142a1c","reported_date":"2026-07-09","source_url":"https://artificialanalysis.ai/evaluations/critpt"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Gemini 3 Deep Think"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 Pro (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 Pro (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-critpt","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/critpt","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":91,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_6093c843839ad76f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CritPT is a challenging reasoning benchmark reported by Qwen for evaluating frontier mathematical and critical problem-solving capability.","descriptions":{"en":"CritPT is a challenging reasoning benchmark reported by Qwen for evaluating frontier mathematical and critical problem-solving capability."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:critpt:qwen3.7-max","reported_at":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/critpt?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:critpt","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_6093c843839ad76f"},"modality":"text","name":"CritPT","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":16.7,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.167,"raw_min":0.031,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:critpt:glm-5.2","reported_date":"2026-06-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/critpt?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-critpt","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/critpt?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["CritPt"],"attention_signals":{"as_of":"2026-10-13","github_stars":91,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:2509.26574","repository":null,"source_record":"catalog_6093c843839ad76f"},"categories":["science"],"collected_at":null,"description":"Research-level physics reasoning; small expert-authored set.","descriptions":{"en":"Research-level physics reasoning; small expert-authored set."},"evidence_summary":{"document_count":3,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000critpt\u0000moonshot_kimi_k3_model_card\u0000critpt\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"first_score_reported_at":"2026-06-13","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000critpt\u0000moonshot_kimi_k3_model_card\u0000critpt\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:critpt","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"artifact:arxiv:2509.26574","primary":"science","source_record":"catalog_6093c843839ad76f"},"modality":null,"name":"CritPt","openness":"unknown","publisher":null,"released":"2025-09-30","released_reference":{"basis":"benchmark_release","source_key":"model-reports:critpt","source_url":"https://arxiv.org/abs/2509.26574"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":23.4,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":23.4,"raw_min":16.9,"source_reference":{"obs_id":"curated\u0000critpt\u0000moonshot_kimi_k3_model_card\u0000critpt\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000critpt\u0000moonshot_kimi_k3_model_card\u0000critpt\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"}],"slug":"critpt","source":"model_reports","source_url":"https://arxiv.org/abs/2509.26574","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":196,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_crjudgebench_6a81b5fc"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers.","descriptions":{"en":"Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37216","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_crjudgebench_6a81b5fc"},"modality":null,"name":"CRJudgeBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37216","source_url":"https://arxiv.org/abs/2609.37216"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37216","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37216","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":9,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":"gh:genintel/cronos-benchmark","source_record":"bm_cronos_36061fdb"},"categories":["Cybersecurity","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CRONOS is an intervention-based benchmark for evaluating counterfactual physical consistency in video prediction models. It provides a photorealistic Unreal Engine environment with controlled videos of physical events (collision, occlusion, fall) while intervening on viewpoint, scene, object category, and object appearance. The evaluation protocol defines six metrics computed via video segmentation, tracking, 3D reconstruction, and VLM-based task performance.","descriptions":{"en":"CRONOS is an intervention-based benchmark for evaluating counterfactual physical consistency in video prediction models. It provides a photorealistic Unreal Engine environment with controlled videos of physical events (collision, occlusion, fall) while intervening on viewpoint, scene, object category, and object appearance. The evaluation protocol defines six metrics computed via video segmentation, tracking, 3D reconstruction, and VLM-based task performance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.23699","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","video-understanding"],"join":"claire_record_id","primary":"world-models","source_record":"bm_cronos_36061fdb"},"modality":null,"name":"CRONOS","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23699","source_url":"https://arxiv.org/abs/2605.23699"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23699","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23699","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Vision-language models increasingly reason through crops, regions, and tool-produced observations.","descriptions":{"en":"Vision-language models increasingly reason through crops, regions, and tool-produced observations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.09550","languages":[],"library_direction":{"basis":"model-review-daily","categories":["visual-reasoning","tool-use"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_cross-bench_f0bc35a3"},"modality":null,"name":"CROSS-Bench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.09550","source_url":"https://arxiv.org/abs/2610.09550"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-09550","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09550","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tiago369/cross-engine-humanoid-benchmark","source_record":"bm_cross-engine-humanoid-benchmark_f797b695"},"categories":["Language & Knowledge","Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Compares MuJoCo and PyBullet on a scripted squat-and-recover task for the Unitree G1 humanoid, measuring joint tracking RMSE, contact forces, CoM deviation, and realtime factor, with PSO-based calibration.","descriptions":{"en":"Compares MuJoCo and PyBullet on a scripted squat-and-recover task for the Unitree G1 humanoid, measuring joint tracking RMSE, contact forces, CoM deviation, and realtime factor, with PSO-based calibration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:tiago369/cross-engine-humanoid-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_cross-engine-humanoid-benchmark_f797b695"},"modality":null,"name":"Cross-Engine Humanoid Benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:tiago369/cross-engine-humanoid-benchmark","source_url":"https://github.com/tiago369/cross-engine-humanoid-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-tiago369-cross-engine-humanoid-benchmark","source":"claire_radar","source_url":"https://github.com/tiago369/cross-engine-humanoid-benchmark","topics":[],"unit":null},{"aliases":["Socialized UAV Cross-Task Learning: Towards Cross-Granularity Collaboration through Hierarchical Interaction"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"UAV benchmark for joint object detection and instance segmentation.","descriptions":{"en":"UAV benchmark for joint object detection and instance segmentation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.19867","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_socialized-uav-cross-task-learning-towards_1cb43b04"},"modality":null,"name":"CrossUAV","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19867","source_url":"https://arxiv.org/abs/2609.19867"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19867","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19867","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":23,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:chuntianli666/crossvid","source_record":"catalog_848579aa1776b705"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CrossVid evaluates cross-video reasoning, requiring models to integrate information across multiple videos.","descriptions":{"en":"CrossVid evaluates cross-video reasoning, requiring models to integrate information across multiple videos."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:crossvid:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crossvid?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:crossvid","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_848579aa1776b705"},"modality":"multimodal","name":"CrossVid","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":65.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.65,"raw_min":0.632,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:crossvid:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crossvid?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-crossvid","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crossvid?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:utaustin-swarmlab/crossview","source_record":"bm_crossview_b52b71d9"},"categories":["Autonomous Driving","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"CrossView is a multi-camera video question-answering benchmark spanning autonomous driving, security surveillance, egocentric/exocentric video, and robotics. It evaluates vision-language models on tasks that require joint reasoning across multiple simultaneous camera views, including resolving occlusions and integrating evidence across perspectives.","descriptions":{"en":"CrossView is a multi-camera video question-answering benchmark spanning autonomous driving, security surveillance, egocentric/exocentric video, and robotics. It evaluates vision-language models on tasks that require joint reasoning across multiple simultaneous camera views, including resolving occlusions and integrating evidence across perspectives."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.15539","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","spatial-3d"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_crossview_b52b71d9"},"modality":null,"name":"CrossView","openness":"unknown","publisher":null,"released":"2026-08-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15539","source_url":"https://arxiv.org/abs/2608.15539"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15539","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15539","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CrossWordBench, a benchmark designed to evaluate the reasoning capabilities of both LLMs and LVLMs through the medium of crossword puzzles. CrossWordBench，这是一个基准测试，旨在通过填字游戏的方式来评估LLMs和LVLMs的推理能力。","descriptions":{"en":"CrossWordBench, a benchmark designed to evaluate the reasoning capabilities of both LLMs and LVLMs through the medium of crossword puzzles.","zh":"CrossWordBench，这是一个基准测试，旨在通过填字游戏的方式来评估LLMs和LVLMs的推理能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1739","languages":[],"library_direction":null,"modality":"multimodal","name":"CrossWordBench","openness":"restricted","publisher":"CMU, WUSTL,UW,UIUC","released":"2025-03-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1739-crosswordbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CrossWordBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping. CrowS-Pairs 包含 1508 个示例，涵盖与九种偏见类型相关的刻板印象，例如种族、宗教和年龄。在 CrowS-Pairs 中，模型会接收到两句话：一句是更具刻板印象的，另一句则是较少刻板印象的。","descriptions":{"en":"CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping.","zh":"CrowS-Pairs 包含 1508 个示例，涵盖与九种偏见类型相关的刻板印象，例如种族、宗教和年龄。在 CrowS-Pairs 中，模型会接收到两句话：一句是更具刻板印象的，另一句则是较少刻板印象的。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1123","languages":[],"library_direction":null,"modality":null,"name":"Crows-Pairs","openness":"unknown","publisher":"New York University","released":"2020-09-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1123-crows-pairs","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Crows-Pairs","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":507,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:2402.19474","repository":"gh:opengvlab/all-seeing","source_record":"catalog_558ad5fcac1a5329"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CRPE is a benchmark designed to quantitatively evaluate the object recognition and relation comprehension ability of models. It consists of four splits, and the evaluation is formulated as single-choice questions. CRPE用于定量评估多模态大模型的对象识别和关系理解能力，分为四部分，以单选题形式呈现。","descriptions":{"en":"CRPE is a benchmark designed to quantitatively evaluate the object recognition and relation comprehension ability of models. It consists of four splits, and the evaluation is formulated as single-choice questions.","zh":"CRPE用于定量评估多模态大模型的对象识别和关系理解能力，分为四部分，以单选题形式呈现。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1500","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2402.19474","primary":"visual-reasoning","source_record":"catalog_558ad5fcac1a5329"},"modality":"multimodal","name":"CRPE","openness":"open","publisher":"Shanghai AI Laboratory","released":"2024-02-29","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1500-crpe","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CRPE","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":507,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:opengvlab/all-seeing","source_record":"catalog_558ad5fcac1a5329"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Clinical reasoning problems evaluation benchmark for assessing diagnostic reasoning and medical knowledge application capabilities.","descriptions":{"en":"Clinical reasoning problems evaluation benchmark for assessing diagnostic reasoning and medical knowledge application capabilities."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:crperelation:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crperelation?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:crperelation","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_558ad5fcac1a5329"},"modality":"text","name":"CRPErelation","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":76.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.765,"raw_min":0.765,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:crperelation:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crperelation?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-crperelation","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crperelation?top_n=500","topics":["reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Vision & 3D","eess.IV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates 15 pretrained image encoder families on ISIC 2019, APTOS 2019, and CheXpert using discrimination, calibration, label efficiency, and robustness, summarized by the Clinical Reliability Score.","descriptions":{"en":"Evaluates 15 pretrained image encoder families on ISIC 2019, APTOS 2019, and CheXpert using discrimination, calibration, label efficiency, and robustness, summarized by the Clinical Reliability Score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22059","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["factuality","visual-reasoning","health-medicine"],"join":"claire_record_id","primary":"factuality","source_record":"bm_crs-bench_17476b11"},"modality":null,"name":"CRS-Bench","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22059","source_url":"http://arxiv.org/abs/2608.22059v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22059","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22059v1","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"CruiseBench evaluates remaining useful life prediction for aircraft engines using a fixed cruise-stage protocol derived from N-CMAPSS. It provides a reproducible sub-benchmark with datasets and baseline results for LSTM, GRU, TCN, and TSMixer models under specific settings.","descriptions":{"en":"CruiseBench evaluates remaining useful life prediction for aircraft engines using a fixed cruise-stage protocol derived from N-CMAPSS. It provides a reproducible sub-benchmark with datasets and baseline results for LSTM, GRU, TCN, and TSMixer models under specific settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19380","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_cruisebench_5f43e5a1"},"modality":null,"name":"CruiseBench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19380","source_url":"https://arxiv.org/abs/2607.19380"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19380","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19380","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"llm_stats_id","repository":null,"source_record":"catalog_08f47d035bbfbc28"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CRUXEval-O (output prediction) is part of the CRUXEval benchmark consisting of 800 Python functions (3-13 lines) designed to evaluate AI models' capabilities in code reasoning, understanding, and execution. The benchmark tests models' ability to predict correct function outputs given function code and inputs, focusing on short problems that a good human programmer should be able to solve in a minute.","descriptions":{"en":"CRUXEval-O (output prediction) is part of the CRUXEval benchmark consisting of 800 Python functions (3-13 lines) designed to evaluate AI models' capabilities in code reasoning, understanding, and execution. The benchmark tests models' ability to predict correct function outputs given function code and inputs, focusing on short problems that a good human programmer should be able to solve in a minute."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:crux-o:qwen3-235b-a22b","reported_at":"2025-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crux-o?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:crux-o","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_08f47d035bbfbc28"},"modality":"text","name":"CRUX-O","openness":"unknown","publisher":null,"released":"2024-01-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:crux-o","source_url":"https://arxiv.org/abs/2401.03065"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.79,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.79,"raw_min":0.79,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:crux-o:qwen3-235b-a22b","reported_date":"2025-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crux-o?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"}],"slug":"llm-stats-crux-o","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crux-o?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"llm_stats_id","repository":null,"source_record":"catalog_181f2f5e13ac9448"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CRUXEval input prediction task with Chain of Thought (CoT) prompting. Part of the CRUXEval benchmark for code reasoning, understanding, and execution evaluation. Given a Python function and its expected output, the task is to predict the appropriate input using chain-of-thought reasoning. Consists of 800 Python functions (3-13 lines) designed to evaluate code comprehension and reasoning capabilities.","descriptions":{"en":"CRUXEval input prediction task with Chain of Thought (CoT) prompting. Part of the CRUXEval benchmark for code reasoning, understanding, and execution evaluation. Given a Python function and its expected output, the task is to predict the appropriate input using chain-of-thought reasoning. Consists of 800 Python functions (3-13 lines) designed to evaluate code comprehension and reasoning capabilities."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cruxeval-input-cot:qwen-2.5-coder-7b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-input-cot?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cruxeval-input-cot","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_181f2f5e13ac9448"},"modality":"text","name":"CRUXEval-Input-CoT","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":56.49999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.565,"raw_min":0.565,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cruxeval-input-cot:qwen-2.5-coder-7b-instruct","reported_date":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-input-cot?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-cruxeval-input-cot","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-input-cot?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"llm_stats_id","repository":null,"source_record":"catalog_981b30ba7456c8df"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CruxEval-O is the output prediction task of the CRUXEval benchmark, designed to evaluate code reasoning, understanding, and execution capabilities. It consists of 800 Python functions (3-13 lines) where models must predict the output given a function and input. The benchmark tests fundamental code execution reasoning abilities and goes beyond simple code generation to assess deeper understanding of program behavior.","descriptions":{"en":"CruxEval-O is the output prediction task of the CRUXEval benchmark, designed to evaluate code reasoning, understanding, and execution capabilities. It consists of 800 Python functions (3-13 lines) where models must predict the output given a function and input. The benchmark tests fundamental code execution reasoning abilities and goes beyond simple code generation to assess deeper understanding of program behavior."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cruxeval-o:codestral-22b","reported_at":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-o?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cruxeval-o","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_981b30ba7456c8df"},"modality":"text","name":"CruxEval-O","openness":"unknown","publisher":null,"released":"2024-01-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:cruxeval-o","source_url":"https://arxiv.org/abs/2401.03065"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":51.300000000000004,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.513,"raw_min":0.513,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cruxeval-o:codestral-22b","reported_date":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-o?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-29","date_precision":"model_announcement","name":"Codestral-22B"}],"slug":"llm-stats-cruxeval-o","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-o?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"llm_stats_id","repository":null,"source_record":"catalog_d7bd94f44513f617"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CRUXEval-O (output prediction) with Chain-of-Thought prompting. Part of the CRUXEval benchmark consisting of 800 Python functions (3-13 lines) designed to evaluate code reasoning, understanding, and execution capabilities. The output prediction task requires models to predict the output of a given Python function with specific inputs, evaluated using chain-of-thought reasoning methodology.","descriptions":{"en":"CRUXEval-O (output prediction) with Chain-of-Thought prompting. Part of the CRUXEval benchmark consisting of 800 Python functions (3-13 lines) designed to evaluate code reasoning, understanding, and execution capabilities. The output prediction task requires models to predict the output of a given Python function with specific inputs, evaluated using chain-of-thought reasoning methodology."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cruxeval-output-cot:qwen-2.5-coder-7b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-output-cot?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cruxeval-output-cot","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_d7bd94f44513f617"},"modality":"text","name":"CRUXEval-Output-CoT","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":56.00000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.56,"raw_min":0.56,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cruxeval-output-cot:qwen-2.5-coder-7b-instruct","reported_date":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-output-cot?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-cruxeval-output-cot","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-output-cot?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":624,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:entendrefinance/crypto-accounting-benchmark","source_record":"bm_crypto-accounting-bench_75761088"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Crypto Accounting Bench tests whether language models can reproduce the exact journal entry recorded for 118 crypto-asset transactions, using transaction evidence and each organization's full chart of accounts.","descriptions":{"en":"Crypto Accounting Bench tests whether language models can reproduce the exact journal entry recorded for 118 crypto-asset transactions, using transaction evidence and each organization's full chart of accounts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.14811","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_crypto-accounting-bench_75761088"},"modality":null,"name":"Crypto Accounting Bench","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.14811","source_url":"https://arxiv.org/abs/2609.14811"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-14811","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14811","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"CrystalXRD-Bench evaluates vision-language models on XRD peak indexing, requiring the model to identify HKL indices from rendered XRD images and CIF text across 250 samples from 10 databases.","descriptions":{"en":"CrystalXRD-Bench evaluates vision-language models on XRD peak indexing, requiring the model to identify HKL indices from rendered XRD images and CIF text across 250 samples from 10 databases."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29446","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_crystalxrd-bench_0f28a22b"},"modality":null,"name":"CrystalXRD-Bench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29446","source_url":"https://arxiv.org/abs/2605.29446"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29446","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29446","topics":["multimodal","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","代码","Code","数学","Math","大语言模型","LLM","逻辑推理","代码工程","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CS-Bench, the first bilingual (Chinese-English) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 5K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. CS-Bench 第一个专门用于评估LLMs在计算机科学中表现的双语（中英文）基准。CS-Bench包括约5,000个精心策划的测试样本，涵盖了计算机科学4个关键领域中的26个子领域，并包括各种任务形式和知识推理的划分。","descriptions":{"en":"CS-Bench, the first bilingual (Chinese-English) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 5K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning.","zh":"CS-Bench 第一个专门用于评估LLMs在计算机科学中表现的双语（中英文）基准。CS-Bench包括约5,000个精心策划的测试样本，涵盖了计算机科学4个关键领域中的26个子领域，并包括各种任务形式和知识推理的划分。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:924","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"CS-Bench","openness":"restricted","publisher":null,"released":"2024-06-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-924-cs-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CS-Bench","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","知识","Knowledge","安全","Safety","大语言模型","LLM","知识储备","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CS-Eval is a large language model cybersecurity capability evaluation suite jointly established by Alibaba Security, Fudan University, and the University of Chinese Academy of Sciences. The dataset encompasses 11 major cybersecurity categories and 42 subcategories, offering comprehensive assessment CS-Eval 是由阿里安全、复旦大学和中国科学院大学联合建立的大模型网络安全能力评测集。数据集覆盖11个网络安全大类领域、42个子类领域，提供知识型和实战型的综合评估任务，支持用户自主评测，同时为大模型落地网络安全提供参考和启发。","descriptions":{"en":"CS-Eval is a large language model cybersecurity capability evaluation suite jointly established by Alibaba Security, Fudan University, and the University of Chinese Academy of Sciences. The dataset encompasses 11 major cybersecurity categories and 42 subcategories, offering comprehensive assessment","zh":"CS-Eval 是由阿里安全、复旦大学和中国科学院大学联合建立的大模型网络安全能力评测集。数据集覆盖11个网络安全大类领域、42个子类领域，提供知识型和实战型的综合评估任务，支持用户自主评测，同时为大模型落地网络安全提供参考和启发。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1219","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"CS-Eval","openness":"restricted","publisher":"阿里巴巴集团安全部，复旦大学，中国科学院大学","released":"2024-05-31","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1219-cs-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CS-Eval","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":79,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":35,"join":"llm_stats_id","repository":"gh:livingfuturelab/chinesesimpleqa","source_record":"catalog_c9d74c7adac2117e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Chinese SimpleQA is the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions. It contains 3,000 high-quality questions spanning 6 major topics with 99 diverse subtopics, designed to assess Chinese factual knowledge across humanities, science, engineering, culture, and society.","descriptions":{"en":"Chinese SimpleQA is the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions. It contains 3,000 high-quality questions spanning 6 major topics with 99 diverse subtopics, designed to assess Chinese factual knowledge across humanities, science, engineering, culture, and society."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:csimpleqa:deepseek-v3","reported_at":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/csimpleqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:csimpleqa","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","multilingual"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_c9d74c7adac2117e"},"modality":"text","name":"CSimpleQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":84.39999999999999,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.844,"raw_min":0.648,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:csimpleqa:deepseek-v4-pro-max","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/csimpleqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"}],"slug":"llm-stats-csimpleqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/csimpleqa?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CSL is a large-scale Chinese Scientific Literature dataset, which contains the titles, abstracts, keywords and academic fields of 396k papers. CSL是一个大规模的中文科技文献数据集，包含 39.6 万篇论文的标题、摘要、关键词和学术领域信息。","descriptions":{"en":"CSL is a large-scale Chinese Scientific Literature dataset, which contains the titles, abstracts, keywords and academic fields of 396k papers.","zh":"CSL是一个大规模的中文科技文献数据集，包含 39.6 万篇论文的标题、摘要、关键词和学术领域信息。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:519","languages":["Chinese"],"library_direction":null,"modality":null,"name":"CSL","openness":"unknown","publisher":null,"released":"2022-09-12","released_reference":{"basis":"paper_first_version","note":"The source describes the 396k-paper scientific literature dataset introduced here.","source_key":"opencompass:519","source_url":"https://arxiv.org/abs/2209.05034"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-519-csl","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CSL","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CSTS is a synthetic benchmark for evaluating correlation structure discovery in multivariate time series. It features 23 distinct correlation structures with systematic data variations (distribution shifts, sparsification, downsampling) and provides ground truth labels for validation. CSTS is a synthetic benchmark for evaluating correlation structure discovery in multivariate time series. It features 23 distinct correlation structures with systematic data variations (distribution shifts, sparsification, downsampling) and provides ground truth labels for validation.","descriptions":{"en":"CSTS is a synthetic benchmark for evaluating correlation structure discovery in multivariate time series. It features 23 distinct correlation structures with systematic data variations (distribution shifts, sparsification, downsampling) and provides ground truth labels for validation.","zh":"CSTS is a synthetic benchmark for evaluating correlation structure discovery in multivariate time series. It features 23 distinct correlation structures with systematic data variations (distribution shifts, sparsification, downsampling) and provides ground truth labels for validation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1833","languages":[],"library_direction":null,"modality":null,"name":"CSTS","openness":"open","publisher":"University of Bristol,University of Nanjing,etc","released":"2025-05-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1833-csts","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CSTS","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Code generation","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CSTutorBench evaluates small language models as tutors in VEX VR block-based programming, with 17 scenario-based questions scored via a rubric and LLM-as-judge.","descriptions":{"en":"CSTutorBench evaluates small language models as tutors in VEX VR block-based programming, with 17 scenario-based questions scored via a rubric and LLM-as-judge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.05571","languages":[],"library_direction":null,"modality":null,"name":"CSTutorBench","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.05571","source_url":"https://arxiv.org/abs/2607.05571"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05571","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05571","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CSVFidelity-Bench evaluates knowledge graph construction from statistical tables, focusing on the effect of extraction schema and format coupling on fidelity. It includes 15 datasets, 11 Type-II and 4 Type-III tables, with 1,892 gold standard facts across 6 domains.","descriptions":{"en":"CSVFidelity-Bench evaluates knowledge graph construction from statistical tables, focusing on the effect of extraction schema and format coupling on fidelity. It includes 15 datasets, 11 Type-II and 4 Type-III tables, with 1,892 gold standard facts across 6 domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.21974","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_csvfidelity-bench_89145848"},"modality":null,"name":"CSVFidelity-Bench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.21974","source_url":"https://arxiv.org/abs/2605.21974"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-21974","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.21974","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents on telecom network troubleshooting tasks, focusing on root cause analysis and path restoration, using expert-grounded metrics.","descriptions":{"en":"Evaluates AI agents on telecom network troubleshooting tasks, focusing on root cause analysis and path restoration, using expert-grounded metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12002","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_ctbench_d210f8ea"},"modality":null,"name":"CTBench","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12002","source_url":"https://arxiv.org/abs/2608.12002"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12002","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12002","topics":["agents"],"unit":null},{"aliases":["No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"CTC-Bench is a 22-task suite released alongside code and data to study corpus task complexity as corpora grow, per the abstract.","descriptions":{"en":"CTC-Bench is a 22-task suite released alongside code and data to study corpus task complexity as corpora grow, per the abstract."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.29245","languages":[],"library_direction":null,"modality":null,"name":"CTC-Bench","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.29245","source_url":"https://arxiv.org/abs/2609.29245"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-29245","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29245","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","NeurIPS 2024","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CTIBench is a benchmark designed to assess LLMs' performance in CTI (Cyber threat intelligence) applications. It includes multiple datasets focused on evaluating knowledge acquired by LLMs in the cyber-threat landscape. CTIBench旨在评估LLM在CTI（网络安全情报）场景下的能力，包含多个数据集，专注于评测LLM在网络威胁环境中获得的知识。","descriptions":{"en":"CTIBench is a benchmark designed to assess LLMs' performance in CTI (Cyber threat intelligence) applications. It includes multiple datasets focused on evaluating knowledge acquired by LLMs in the cyber-threat landscape.","zh":"CTIBench旨在评估LLM在CTI（网络安全情报）场景下的能力，包含多个数据集，专注于评测LLM在网络威胁环境中获得的知识。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1268","languages":[],"library_direction":null,"modality":null,"name":"CTIBench","openness":"restricted","publisher":"Rochester Institute of Technology","released":"2024-06-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1268-ctibench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CTIBench","topics":["safety"],"unit":null},{"aliases":["Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming"],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-09-25T04:40:22Z","description":"A public 793-episode benchmark for evaluating prompt-injection attacks against computer-using agents across web tasks, attack templates, attack families, and system-prompt configurations.","descriptions":{"en":"A public 793-episode benchmark for evaluating prompt-injection attacks against computer-using agents across web tasks, attack templates, attack families, and system-prompt configurations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05233","languages":[],"library_direction":null,"modality":null,"name":"CUA-HandCrafted","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05233","source_url":"https://arxiv.org/abs/2606.05233"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05233","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05233","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":8,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"claire_record_id","repository":"gh:kingofspace0wzz/cua-swe","source_record":"bm_cua-swe_6b25a9b4"},"categories":["Agents & Tool Use","Code","General AI","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Software development requires more than editing code: developers repeatedly run software, interact with its interfaces, visually inspect its behavior, and use these observations to decide what to change next and whether a change works.","descriptions":{"en":"Software development requires more than editing code: developers repeatedly run software, interact with its interfaces, visually inspect its behavior, and use these observations to decide what to change next and whether a change works."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.32600","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","computer-use"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_cua-swe_6b25a9b4"},"modality":null,"name":"CUA-SWE","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32600","source_url":"https://arxiv.org/abs/2609.32600"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32600","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32600","topics":["coding","computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":6,"github_stars_30d":null,"hf_dataset_downloads":2334,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:yulinlp/cue-mem","source_record":"bm_cue-mem_a62047b6"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Long-term memory is essential for multimodal agents that interact with users across sustained conversations.","descriptions":{"en":"Long-term memory is essential for multimodal agents that interact with users across sustained conversations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.32574","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","visual-reasoning"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_cue-mem_a62047b6"},"modality":null,"name":"CUE-Mem","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32574","source_url":"https://arxiv.org/abs/2609.32574"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32574","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32574","topics":["agents","multimodal"],"unit":null},{"aliases":["CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage"],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"CulMind evaluates multimodal understanding across Chinese cultural-heritage tasks, while CulMind-R and ReaScore assess reasoning processes.","descriptions":{"en":"CulMind evaluates multimodal understanding across Chinese cultural-heritage tasks, while CulMind-R and ReaScore assess reasoning processes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.21618","languages":[],"library_direction":null,"modality":null,"name":"CulMind","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21618","source_url":"https://arxiv.org/abs/2606.21618"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21618","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21618","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_cultural-moment-benchmark_aacba1dd"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video cultural reasoning and grounding on 306 expert-curated concepts from Southeast Asia, scoring naming, visual recognition, and temporal localization.","descriptions":{"en":"Evaluates video cultural reasoning and grounding on 306 expert-curated concepts from Southeast Asia, scoring naming, visual recognition, and temporal localization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23065","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_cultural-moment-benchmark_aacba1dd"},"modality":null,"name":"Cultural Moment Benchmark","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23065","source_url":"http://arxiv.org/abs/2608.23065v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23065","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23065v1","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bobtsang-nlp/culturalmenubench","source_record":"bm_culturalmenubench_496caf80"},"categories":["Factuality","General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal models on culinary reasoning across 4,870 items in 10 languages and 18 regions, with tasks from dish recognition to cultural attribution using images and procedural text.","descriptions":{"en":"Evaluates multimodal models on culinary reasoning across 4,870 items in 10 languages and 18 regions, with tasks from dish recognition to cultural attribution using images and procedural text."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.03526","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","multilingual"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_culturalmenubench_496caf80"},"modality":null,"name":"CulturalMenuBench","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03526","source_url":"https://arxiv.org/abs/2609.03526"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03526","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03526","topics":["multimodal","reasoning-math","language"],"unit":null},{"aliases":["Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"CULTURE-MT evaluates social-media translation across cultural transmission, emotion resonance, expression accuracy, and cultural adaptability.","descriptions":{"en":"CULTURE-MT evaluates social-media translation across cultural transmission, emotion resonance, expression accuracy, and cultural adaptability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.25626","languages":[],"library_direction":null,"modality":null,"name":"CULTURE-MT","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25626","source_url":"https://arxiv.org/abs/2605.25626"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25626","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25626","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_cultureforest_c27631e8"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"CultureForest is a benchmark for cultural norm grounded reasoning, featuring 5,378 examples across 8 domains and 53 countries/regions, with progressive evaluation from multiple-choice to open-ended generation.","descriptions":{"en":"CultureForest is a benchmark for cultural norm grounded reasoning, featuring 5,378 examples across 8 domains and 53 countries/regions, with progressive evaluation from multiple-choice to open-ended generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01879","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_cultureforest_c27631e8"},"modality":null,"name":"CultureForest","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01879","source_url":"https://arxiv.org/abs/2606.01879"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01879","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01879","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"CultureTalk-ID is a dialogue-based benchmark for cultural commonsense in Indonesian and local languages, containing 4,496 culturally grounded dialogues across 11 languages and 13 topics, with three tasks: dialogue-based multiple-choice reasoning, culturally faithful translation, and language steering.","descriptions":{"en":"CultureTalk-ID is a dialogue-based benchmark for cultural commonsense in Indonesian and local languages, containing 4,496 culturally grounded dialogues across 11 languages and 13 topics, with three tasks: dialogue-based multiple-choice reasoning, culturally faithful translation, and language steering."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.21016","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_culturetalk-id_d7b2d28e"},"modality":null,"name":"CultureTalk-ID","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21016","source_url":"https://arxiv.org/abs/2607.21016"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21016","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21016","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CultureVidBench evaluates cultural understanding in text-to-video generation with 1,000 curated prompts covering 12 countries, 6 continents, and 14 cultural aspects. It assesses cultural faithfulness, multimodal cultural rendering, semantic adherence, and perceptual quality using human studies and MLLM-based automatic assessment.","descriptions":{"en":"CultureVidBench evaluates cultural understanding in text-to-video generation with 1,000 curated prompts covering 12 countries, 6 continents, and 14 cultural aspects. It assesses cultural faithfulness, multimodal cultural rendering, semantic adherence, and perceptual quality using human studies and MLLM-based automatic assessment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01942","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_culturevidbench_541d6118"},"modality":null,"name":"CultureVidBench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01942","source_url":"https://arxiv.org/abs/2608.01942"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01942","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01942","topics":[],"unit":null},{"aliases":["CursorBench","Cursor Bench","CursorBench 3.2"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Maintained by Cursor and measured inside the Cursor product, so it reflects that scaffold rather than the bare model.","descriptions":{"en":"Maintained by Cursor and measured inside the Cursor product, so it reflects that scaffold rather than the bare model."},"evidence_summary":{"document_count":2,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:cursor_bench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"CursorBench","openness":"unknown","publisher":null,"released":"2025-10-29","released_reference":{"basis":"benchmark_release","source_key":"model-reports:cursor_bench","source_url":"https://cursor.com/blog/cursor-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"cursor_bench","source":"model_reports","source_url":"https://cursor.com/blog/cursor-bench","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CursorBench v3.2 evaluates coding agents on interactive software engineering tasks in the Cursor environment.","descriptions":{"en":"CursorBench v3.2 evaluates coding agents on interactive software engineering tasks in the Cursor environment."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cursorbench-3.2:grok-4.6","reported_at":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cursorbench-3.2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cursorbench-3.2","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_9b3b3ed1cf7144a8"},"modality":"text","name":"CursorBench v3.2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":69.89999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.699,"raw_min":0.699,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cursorbench-3.2:grok-4.6","reported_date":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cursorbench-3.2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"}],"slug":"llm-stats-cursorbench-3-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cursorbench-3.2?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation"],"attention_signals":{"as_of":"2026-10-11","github_stars":42,"github_stars_30d":27,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:alibabaresearch/cut-craft-bench","source_record":"bm_beyond-coherence-benchmarking-professional_2647e597"},"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"CutCraft evaluates editing-technique execution in multi-shot audio-video generation, covering shot structure, transitions, audio-video cuts, and montage.","descriptions":{"en":"CutCraft evaluates editing-technique execution in multi-shot audio-video generation, covering shot structure, transitions, audio-video cuts, and montage."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.08275","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_beyond-coherence-benchmarking-professional_2647e597"},"modality":null,"name":"CutCraft","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08275","source_url":"https://arxiv.org/abs/2609.08275"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08275","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08275","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_cv-arena_3558dfa5"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CV-Arena is a benchmark of 12K high-resolution image instruction pairs across 16 task types, with a human-AI collaborative preference protocol (Active Elo) for evaluating instructional computer vision problem solving.","descriptions":{"en":"CV-Arena is a benchmark of 12K high-resolution image instruction pairs across 16 task types, with a human-AI collaborative preference protocol (Active Elo) for evaluating instructional computer vision problem solving."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00931","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_cv-arena_3558dfa5"},"modality":null,"name":"CV-Arena","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00931","source_url":"https://arxiv.org/abs/2606.00931"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00931","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00931","topics":[],"unit":null},{"aliases":["SeqLoc: Beyond the Single Frame for Cross-View Geo-Localization in Feature-Sparse Scenes"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for cross-view geo-localization in feature-sparse scenes, pairing sequential panoramas from rural regions with aligned OpenStreetMap maps and evaluating position and orientation recall.","descriptions":{"en":"A benchmark for cross-view geo-localization in feature-sparse scenes, pairing sequential panoramas from rural regions with aligned OpenStreetMap maps and evaluating position and orientation recall."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07835","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_seqloc-beyond-the-single-frame-for-cross-v_3d990b1c"},"modality":null,"name":"CV-FSS","openness":"unknown","publisher":null,"released":"2026-08-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07835","source_url":"https://arxiv.org/abs/2608.07835"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07835","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07835","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CVDP is a next-generation benchmark for evaluating large language models (LLMs) and agents in hardware design and verification, comprising 783 problems across 13 task categories, including RTL generation, verification, debugging, specification alignment, and technical Q&A. CVDP 是一个面向大型语言模型（LLM）和智能体的下一代硬件设计与验证评测基准，涵盖 13 类任务共 783 个问题，涉及 RTL 生成、验证、调试、规范对齐和技术问答等。","descriptions":{"en":"CVDP is a next-generation benchmark for evaluating large language models (LLMs) and agents in hardware design and verification, comprising 783 problems across 13 task categories, including RTL generation, verification, debugging, specification alignment, and technical Q&A.","zh":"CVDP 是一个面向大型语言模型（LLM）和智能体的下一代硬件设计与验证评测基准，涵盖 13 类任务共 783 个问题，涉及 RTL 生成、验证、调试、规范对齐和技术问答等。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1985","languages":[],"library_direction":null,"modality":null,"name":"CVDP","openness":"unknown","publisher":"NVIDIA","released":"2025-06-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1985-cvdp","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CVDP","topics":["agents"],"unit":null},{"aliases":["CVE-Bench","CVEBench"],"attention_signals":{"as_of":"2026-10-11","github_stars":298,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:gh:uiuc-kang-lab/cve-bench","repository":"gh:uiuc-kang-lab/cve-bench","source_record":"catalog_c36c554916db16cd"},"categories":["security"],"collected_at":null,"description":"Real CVE exploitation in sandboxes. Vendors report it as a safety ceiling rather than a capability to maximize.","descriptions":{"en":"Real CVE exploitation in sandboxes. Vendors report it as a safety ceiling rather than a capability to maximize."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:cvebench","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"artifact:gh:uiuc-kang-lab/cve-bench","primary":"cybersecurity","source_record":"catalog_c36c554916db16cd"},"modality":null,"name":"CVE-Bench","openness":"unknown","publisher":null,"released":"2025-03-21","released_reference":{"basis":"benchmark_release","source_key":"model-reports:cvebench","source_url":"https://github.com/uiuc-kang-lab/cve-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"cvebench","source":"model_reports","source_url":"https://github.com/uiuc-kang-lab/cve-bench","topics":["agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","知识","Knowledge","NeurIPS 2024","多模态模型","VLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CVQA is a new culturally-diverse multilingual Visual Question Answering benchmark, designed to cover a rich set of languages and cultures. It includes culturally-driven images and 10k questions from across 30 countries on 4 continents. CVQA是一种新的文化多元化多语言视觉问答基准，旨在涵盖丰富的语言和文化，包括来自四大洲30个国家/地区的文化向图像和10000个问题。","descriptions":{"en":"CVQA is a new culturally-diverse multilingual Visual Question Answering benchmark, designed to cover a rich set of languages and cultures. It includes culturally-driven images and 10k questions from across 30 countries on 4 continents.","zh":"CVQA是一种新的文化多元化多语言视觉问答基准，旨在涵盖丰富的语言和文化，包括来自四大洲30个国家/地区的文化向图像和10000个问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1239","languages":["English","Japanese","Multilingual"],"library_direction":null,"modality":"multimodal","name":"CVQA","openness":"restricted","publisher":"MBZUAI","released":"2024-06-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1239-cvqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CVQA","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":12,"github_stars_30d":null,"hf_dataset_downloads":677,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:earth-insights/cvsbench","source_record":"bm_cvsbench_ac22ff1a"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"CVSBench evaluates cross-view spatial reasoning in vision-language models using satellite-street image pairs. It includes tasks for cross-view VQA, grounding, and viewpoint identification, with 3,297 image groups, 9,468 object-level annotations, and 40,679 QA pairs.","descriptions":{"en":"CVSBench evaluates cross-view spatial reasoning in vision-language models using satellite-street image pairs. It includes tasks for cross-view VQA, grounding, and viewpoint identification, with 3,297 image groups, 9,468 object-level annotations, and 40,679 QA pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22476","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_cvsbench_ac22ff1a"},"modality":null,"name":"CVSBench","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22476","source_url":"https://arxiv.org/abs/2606.22476"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22476","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22476","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":97,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":94,"join":"llm_stats_id","repository":"gh:nju-pcalab/textcrafter","source_record":"catalog_0d4bd370b107a31d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["image-generation","language","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CVTG-2K (Chinese Visual Text Generation 2K) is a benchmark for evaluating text-to-image models on their ability to accurately render text within generated images. It measures Word Accuracy, Normalized Edit Distance (NED), and CLIPScore across 2,000 prompts.","descriptions":{"en":"CVTG-2K (Chinese Visual Text Generation 2K) is a benchmark for evaluating text-to-image models on their ability to accurately render text within generated images. It measures Word Accuracy, Normalized Edit Distance (NED), and CLIPScore across 2,000 prompts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cvtg-2k","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"llm_stats_id","primary":"image-video-generation","source_record":"catalog_0d4bd370b107a31d"},"modality":"image","name":"CVTG-2K","openness":"unknown","publisher":null,"released":"2025-03-30","released_reference":{"basis":"paper_first_version","note":"TextCrafter introduces the complex visual text generation benchmark. The source's Chinese-only name expansion is not used to identify a separate instrument.","source_key":"llm-stats:cvtg-2k","source_url":"https://arxiv.org/abs/2503.23461"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":"higher_is_better","score_summary":{"display_max":null,"display_multiplier":1,"model_count":null,"model_count_basis":null,"numeric_count":0,"raw_max":null,"raw_min":null,"source_reference":null,"unit":null},"scored_models":[],"slug":"llm-stats-cvtg-2k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cvtg-2k?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Information retrieval","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"CXR-Retrieve is a benchmark for compositional chest X-ray text-to-image retrieval, with 5,159 MIMIC-CXR-JPG test images and 145 queries over single and conjunction findings with positive and negative assertions.","descriptions":{"en":"CXR-Retrieve is a benchmark for compositional chest X-ray text-to-image retrieval, with 5,159 MIMIC-CXR-JPG test images and 145 queries over single and conjunction findings with positive and negative assertions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27779","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_cxr-retrieve_95ed172c"},"modality":null,"name":"CXR-Retrieve","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27779","source_url":"https://arxiv.org/abs/2607.27779"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27779","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27779","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":341,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":"gh:andyzorigin/cybench","source_record":"catalog_8bcaca6998006997"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CyBench is a suite of Capture-the-Flag (CTF) challenges measuring agentic cyber attack capabilities. It evaluates dual-use cybersecurity knowledge and measures the 'unguided success rate', where agents complete tasks end-to-end without guidance on appropriate subtasks.","descriptions":{"en":"CyBench is a suite of Capture-the-Flag (CTF) challenges measuring agentic cyber attack capabilities. It evaluates dual-use cybersecurity knowledge and measures the 'unguided success rate', where agents complete tasks end-to-end without guidance on appropriate subtasks."},"editorial_guide":{"id":"cybench","record_keys":["llm-stats:cybench"],"reviewed_at":"2026-10-20","scope":{"en":"The original 40-task suite and its unguided, subtask-guided, and subtask metrics.","zh":"原始 40 题任务集，以及无引导、子任务引导和子任务三类指标。"},"sections":[{"id":"overview","source_ids":["paper","blog"],"text":{"en":"Cybench contains 40 challenges drawn from four professional Capture the Flag competitions. The tasks span cryptography, web security, reverse engineering, forensics, binary exploitation, and miscellaneous problems. It evaluates an agent’s ability to investigate a contained challenge and reach a checkable answer through multiple actions.","zh":"Cybench 包含来自四项专业 Capture the Flag 竞赛的 40 道题，覆盖密码学、Web 安全、逆向分析、取证、二进制利用和杂项问题。它考察智能体在受控题目环境中经过多步分析，最终得到可验证答案的能力。"},"title":{"en":"What it measures","zh":"测什么"}},{"id":"tasks","source_ids":["repo","blog"],"text":{"en":"Each task supplies a description, starter files, and an evaluator. The agent works in a Kali Linux container, with local artifacts and task-specific services where needed. In guided runs, intermediate questions are supplied sequentially; unguided runs present the overall objective without that decomposition.","zh":"每道题提供描述、初始文件和评测器。智能体在 Kali Linux 容器中使用本地材料，并按题目需要访问配套服务。引导模式会依次给出中间问题；无引导模式只提供整体目标，不进行这类拆解。"},"title":{"en":"What a task contains","zh":"任务如何设置"}},{"id":"evaluation","source_ids":["home","blog"],"text":{"en":"Unguided success checks the complete task. Subtask-guided success checks the final subtask, while the subtask score measures intermediate completion and is macro-averaged across tasks. First Solve Time describes the fastest human team’s competition solve time for a task; it is a difficulty reference, not the agent’s runtime.","zh":"无引导成功率检查整题目标；子任务引导成功率检查最后一个子任务；子任务分数则衡量中间步骤完成比例，再跨题目做宏平均。First Solve Time 是竞赛中首个人类队伍解出该题所用的时间，用作难度参考，不是智能体运行耗时。"},"title":{"en":"Three different success measures","zh":"如何区分三类分数"}},{"id":"use_cases","source_ids":["blog"],"text":{"en":"Use it to study tool use, sustained problem solving, and the effect of explicit intermediate guidance. The distinction between partial progress and a completed challenge helps identify where an agent stalls, rather than treating every incomplete run as equally informative.","zh":"适合研究工具使用、连续解题能力及明确中间提示的影响。把过程进展与整题完成分开，有助于判断智能体卡在哪个阶段，让尚未完成的运行也能提供有用信息。"},"title":{"en":"When it is useful","zh":"适合用来判断什么"}},{"id":"limitations","source_ids":["home","paper"],"text":{"en":"CTF performance is evidence about these challenge environments, not a direct measurement of real enterprise intrusions. Published results may use subsets of the original suite or different trial aggregation. Compare the task list, guidance mode, agent harness, and budget before interpreting a percentage difference.","zh":"CTF 成绩反映的是这些竞赛环境中的表现，不能直接当作真实企业网络入侵能力。公开结果可能使用原始任务集的不同子集或不同尝试汇总方式；比较百分比前，应核对题目列表、引导模式、智能体框架和预算。"},"title":{"en":"Compare matching protocols","zh":"读分时的限制"}}],"sources":[{"id":"home","title":"Cybench official website and metric definitions","url":"https://cybench.github.io/"},{"id":"paper","title":"Cybench paper","url":"https://arxiv.org/abs/2408.08926"},{"id":"repo","title":"Cybench official repository","url":"https://github.com/andyzorigin/cybench"},{"id":"blog","title":"Stanford CRFM introduction to Cybench","url":"https://crfm.stanford.edu/2024/08/19/cybench.html"}],"summary":{"en":"Cybench evaluates agents on professional CTF challenges, with optional subtasks that reveal progress before the final answer.","zh":"Cybench 使用专业 CTF 竞赛题评估智能体，并通过可选子任务观察最终答案之前的解题进展。"},"title":"Cybench"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cybench:grok-4.1-thinking-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cybench","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"llm_stats_id","primary":"cybersecurity","source_record":"catalog_8bcaca6998006997"},"modality":"text","name":"CyBench","openness":"unknown","publisher":null,"released":"2024-08-15","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:cybench","source_url":"https://arxiv.org/abs/2408.08926"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":100.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":1.0,"raw_min":0.39,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cybench:claude-mythos-preview","reported_date":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-cybench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybench?top_n=500","topics":["coding","agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Vision & 3D","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"CyberChainBench evaluates LLM-based agents on smart contract security across vulnerability detection, exploit generation, and patch synthesis, using 541 real exploit incidents with on-chain evaluation on historical forks.","descriptions":{"en":"CyberChainBench evaluates LLM-based agents on smart contract security across vulnerability detection, exploit generation, and patch synthesis, using 541 real exploit incidents with on-chain evaluation on historical forks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26216","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_cyberchainbench_89e4f090"},"modality":null,"name":"CyberChainBench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26216","source_url":"https://arxiv.org/abs/2606.26216"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26216","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26216","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Cybersecurity","Language & Knowledge","Long Context"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language model agents have demonstrated promising capabilities in cybersecurity tasks, yet their ability to reconstruct complete Advanced Persistent Threat attack campaigns from complex security logs remains largely unexplored.","descriptions":{"en":"Large language model agents have demonstrated promising capabilities in cybersecurity tasks, yet their ability to reconstruct complete Advanced Persistent Threat attack campaigns from complex security logs remains largely unexplored."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32424","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_cyberclear_47a119cd"},"modality":null,"name":"CyberClear","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32424","source_url":"https://arxiv.org/abs/2609.32424"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32424","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32424","topics":["agents","long-context","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_d269420d62ea46a3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CyberGym is a benchmark for evaluating AI agents on cybersecurity tasks, testing their ability to identify vulnerabilities, perform security analysis, and complete security-related challenges in a controlled environment.","descriptions":{"en":"CyberGym is a benchmark for evaluating AI agents on cybersecurity tasks, testing their ability to identify vulnerabilities, perform security analysis, and complete security-related challenges in a controlled environment."},"editorial_guide":{"id":"cybergym","record_keys":["llm-stats:cybergym","model-reports:cybergym"],"reviewed_at":"2026-10-20","scope":{"en":"The published 1,507-task dataset, its information levels, and current submission rules.","zh":"公开的 1,507 任务数据集、信息等级及当前提交规则。"},"sections":[{"id":"overview","source_ids":["paper"],"text":{"en":"CyberGym focuses on vulnerability reproduction across 1,507 real vulnerabilities from 188 software projects. Its central question is whether an agent can understand an existing bug report, investigate the relevant code, and produce a test input that demonstrates the reported failure.","zh":"CyberGym 聚焦漏洞复现，收录来自 188 个软件项目的 1,507 个真实漏洞。它主要考察智能体能否读懂已有漏洞报告、分析相关代码，并生成能够呈现该故障的测试输入。"},"title":{"en":"What it measures","zh":"测什么"}},{"id":"tasks","source_ids":["paper"],"text":{"en":"The main Level 1 setting provides source code and a textual vulnerability description. Other levels change the available evidence: Level 0 removes the description, Level 2 adds a crash trace, and Level 3 adds patch information. These are different information settings, so the level belongs beside every result.","zh":"主要的 Level 1 设置提供源代码和文字漏洞描述。其他等级改变可用线索：Level 0 不提供描述，Level 2 增加崩溃堆栈，Level 3 增加补丁信息。各等级的信息条件不同，展示结果时应同时标注等级。"},"title":{"en":"What the agent receives","zh":"任务如何设置"}},{"id":"evaluation","source_ids":["faq"],"text":{"en":"Verification checks that the submitted input triggers the vulnerability in the unpatched program and does not trigger it after the fix. Current submission rules require the agent to select one final proof of concept; success from any intermediate attempt is a different statistic and should be identified separately.","zh":"验证器检查提交的输入是否在未修复程序中触发漏洞、在修复后不再触发。当前规则要求智能体选定一个最终概念验证用例；“过程中任意一次成功”属于另一种统计，不能直接替代最终提交成功率。"},"title":{"en":"How success is checked","zh":"如何判分"}},{"id":"use_cases","source_ids":["repo","submission"],"text":{"en":"Use it to compare how agents read unfamiliar code and turn written reports into executable evidence. For an engineering comparison, consider success together with token use, elapsed time, and tool requests, which the submission guidance asks evaluators to report.","zh":"适合比较智能体阅读陌生代码、把文字报告转化为可执行证据的能力。工程选型时，还应结合 token 消耗、耗时和工具请求量判断效果；官方提交指南要求报告这些成本信息。"},"title":{"en":"When it is useful","zh":"适合用来判断什么"}},{"id":"limitations","source_ids":["faq","paper"],"text":{"en":"A successful reproduction does not establish complete system compromise. Agent scaffolding, tool budgets, network access, and access to runnable target environments affect comparability. The current default setup withholds patched runtime images from the agent; do not confuse that rule with the paper’s separate patch-informed information level.","zh":"成功复现漏洞不等于获得完整系统控制。智能体框架、工具预算、网络访问及目标程序运行环境都会影响可比性。当前默认设置不向智能体开放修复后的运行镜像，这与论文中另设的补丁信息等级需要区分。"},"title":{"en":"What a score leaves out","zh":"读分时的限制"}}],"sources":[{"id":"paper","title":"CyberGym paper, version 3","url":"https://arxiv.org/html/2506.02548v3"},{"id":"repo","title":"CyberGym official repository","url":"https://github.com/sunblaze-ucb/cybergym"},{"id":"faq","title":"CyberGym evaluation FAQ","url":"https://github.com/sunblaze-ucb/cybergym/blob/main/FAQ.md"},{"id":"submission","title":"CyberGym submission guidance","url":"https://github.com/sunblaze-ucb/cybergym/blob/main/SUBMISSION.md"}],"summary":{"en":"CyberGym tests whether an agent can turn a real software vulnerability report into a reproducible test case.","zh":"CyberGym 测试智能体能否把真实软件的漏洞报告转化为可验证的复现用例。"},"title":"CyberGym"},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cybergym:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybergym?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cybergym","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"llm_stats_id","primary":"cybersecurity","source_record":"catalog_d269420d62ea46a3"},"modality":"text","name":"CyberGym","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":84.5,"display_multiplier":100,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":0.845,"raw_min":0.413,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cybergym:glm-5.3","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybergym?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0731"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash Cyber"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-cybergym","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybergym?top_n=500","topics":["coding","agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["security"],"collected_at":null,"description":"Real-world cybersecurity tasks; tool permissions and time limits are part of the result, not incidental to it.","descriptions":{"en":"Real-world cybersecurity tasks; tool permissions and time limits are part of the result, not incidental to it."},"editorial_guide":{"id":"cybergym","record_keys":["llm-stats:cybergym","model-reports:cybergym"],"reviewed_at":"2026-10-20","scope":{"en":"The published 1,507-task dataset, its information levels, and current submission rules.","zh":"公开的 1,507 任务数据集、信息等级及当前提交规则。"},"sections":[{"id":"overview","source_ids":["paper"],"text":{"en":"CyberGym focuses on vulnerability reproduction across 1,507 real vulnerabilities from 188 software projects. Its central question is whether an agent can understand an existing bug report, investigate the relevant code, and produce a test input that demonstrates the reported failure.","zh":"CyberGym 聚焦漏洞复现，收录来自 188 个软件项目的 1,507 个真实漏洞。它主要考察智能体能否读懂已有漏洞报告、分析相关代码，并生成能够呈现该故障的测试输入。"},"title":{"en":"What it measures","zh":"测什么"}},{"id":"tasks","source_ids":["paper"],"text":{"en":"The main Level 1 setting provides source code and a textual vulnerability description. Other levels change the available evidence: Level 0 removes the description, Level 2 adds a crash trace, and Level 3 adds patch information. These are different information settings, so the level belongs beside every result.","zh":"主要的 Level 1 设置提供源代码和文字漏洞描述。其他等级改变可用线索：Level 0 不提供描述，Level 2 增加崩溃堆栈，Level 3 增加补丁信息。各等级的信息条件不同，展示结果时应同时标注等级。"},"title":{"en":"What the agent receives","zh":"任务如何设置"}},{"id":"evaluation","source_ids":["faq"],"text":{"en":"Verification checks that the submitted input triggers the vulnerability in the unpatched program and does not trigger it after the fix. Current submission rules require the agent to select one final proof of concept; success from any intermediate attempt is a different statistic and should be identified separately.","zh":"验证器检查提交的输入是否在未修复程序中触发漏洞、在修复后不再触发。当前规则要求智能体选定一个最终概念验证用例；“过程中任意一次成功”属于另一种统计，不能直接替代最终提交成功率。"},"title":{"en":"How success is checked","zh":"如何判分"}},{"id":"use_cases","source_ids":["repo","submission"],"text":{"en":"Use it to compare how agents read unfamiliar code and turn written reports into executable evidence. For an engineering comparison, consider success together with token use, elapsed time, and tool requests, which the submission guidance asks evaluators to report.","zh":"适合比较智能体阅读陌生代码、把文字报告转化为可执行证据的能力。工程选型时，还应结合 token 消耗、耗时和工具请求量判断效果；官方提交指南要求报告这些成本信息。"},"title":{"en":"When it is useful","zh":"适合用来判断什么"}},{"id":"limitations","source_ids":["faq","paper"],"text":{"en":"A successful reproduction does not establish complete system compromise. Agent scaffolding, tool budgets, network access, and access to runnable target environments affect comparability. The current default setup withholds patched runtime images from the agent; do not confuse that rule with the paper’s separate patch-informed information level.","zh":"成功复现漏洞不等于获得完整系统控制。智能体框架、工具预算、网络访问及目标程序运行环境都会影响可比性。当前默认设置不向智能体开放修复后的运行镜像，这与论文中另设的补丁信息等级需要区分。"},"title":{"en":"What a score leaves out","zh":"读分时的限制"}}],"sources":[{"id":"paper","title":"CyberGym paper, version 3","url":"https://arxiv.org/html/2506.02548v3"},{"id":"repo","title":"CyberGym official repository","url":"https://github.com/sunblaze-ucb/cybergym"},{"id":"faq","title":"CyberGym evaluation FAQ","url":"https://github.com/sunblaze-ucb/cybergym/blob/main/FAQ.md"},{"id":"submission","title":"CyberGym submission guidance","url":"https://github.com/sunblaze-ucb/cybergym/blob/main/SUBMISSION.md"}],"summary":{"en":"CyberGym tests whether an agent can turn a real software vulnerability report into a reproducible test case.","zh":"CyberGym 测试智能体能否把真实软件的漏洞报告转化为可验证的复现用例。"},"title":"CyberGym"},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000cybergym\u0000tencent_hy4_preview\u0000cybergym\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000cybergym\u0000tencent_hy4_preview\u0000cybergym\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:cybergym","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["cybersecurity"],"join":"review:14H034160212:2026-10-15","primary":"cybersecurity","source_record":null},"modality":null,"name":"CyberGym","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":88.1,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":88.1,"raw_min":78.4,"source_reference":{"obs_id":"curated\u0000cybergym\u0000deepseek_v4_1_flash_technical_report\u0000cybergym\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000cybergym\u0000deepseek_v4_1_flash_technical_report\u0000cybergym\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"cybergym","source":"model_reports","source_url":"https://openreview.net/forum?id=2YvbLQEdYt","topics":["coding","agents","cybersecurity"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_cybergym-e2e_b518f5fa"},"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"CyberGym-E2E evaluates AI agents on end-to-end cybersecurity tasks, covering vulnerability discovery, proof-of-concept generation, and patch generation, using 920 real-world vulnerabilities from 139 open-source projects.","descriptions":{"en":"CyberGym-E2E evaluates AI agents on end-to-end cybersecurity tasks, covering vulnerability discovery, proof-of-concept generation, and patch generation, using 920 real-world vulnerabilities from 139 open-source projects."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04460","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_cybergym-e2e_b518f5fa"},"modality":null,"name":"CyberGym-E2E","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04460","source_url":"https://arxiv.org/abs/2606.04460"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04460","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04460","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object is a dataset for privacy-aware cybersecurity question answering, covering key security domains with private entity labels. The main capability evaluated is QA accuracy and masking performance.","descriptions":{"en":"The evaluation object is a dataset for privacy-aware cybersecurity question answering, covering key security domains with private entity labels. The main capability evaluated is QA accuracy and masking performance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24765","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","knowledge-expert-qa"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_cybermaskqa_2283c85d"},"modality":null,"name":"CyberMaskQA","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24765","source_url":"https://arxiv.org/abs/2605.24765"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24765","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24765","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CyberSecEval is a comprehensive benchmark developed to help bolster the cybersecurity of LLMs. It provides a thorough evaluation in two crucial security domains: the propensity to generate insecure code and the level of compliance when asked to assist in cyberattacks. CyberSecEval旨在评估LLM的安全性，聚焦于大模型生成不安全代码的倾向以及当被要求协助网络攻击时的合规性水平。","descriptions":{"en":"CyberSecEval is a comprehensive benchmark developed to help bolster the cybersecurity of LLMs. It provides a thorough evaluation in two crucial security domains: the propensity to generate insecure code and the level of compliance when asked to assist in cyberattacks.","zh":"CyberSecEval旨在评估LLM的安全性，聚焦于大模型生成不安全代码的倾向以及当被要求协助网络攻击时的合规性水平。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1349","languages":["English","Multilingual"],"library_direction":null,"modality":null,"name":"CyberSecEval","openness":"unknown","publisher":"Meta","released":"2023-12-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1349-cyberseceval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CyberSecEval","topics":["coding","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4432,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_09526d651b3985e7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"CyberSecEval 4 is an evaluation suite covering cybersecurity-related capabilities and risks of large language models. The insecure-code-generation tracks measure whether a model produces vulnerable code: the Instruct track presents coding requests designed to elicit known insecure patterns, while the Autocomplete track prompts the model with code context leading up to a known insecure pattern, with vulnerabilities detected via static analysis.","descriptions":{"en":"CyberSecEval 4 is an evaluation suite covering cybersecurity-related capabilities and risks of large language models. The insecure-code-generation tracks measure whether a model produces vulnerable code: the Instruct track presents coding requests designed to elicit known insecure patterns, while the Autocomplete track prompts the model with code context leading up to a known insecure pattern, with vulnerabilities detected via static analysis."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cyberseceval-4:mai-thinking-1","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cyberseceval-4?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cyberseceval-4","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","safety-alignment"],"join":"llm_stats_id","primary":"cybersecurity","source_record":"catalog_09526d651b3985e7"},"modality":"text","name":"CyberSecEval 4","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":63.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.63,"raw_min":0.63,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cyberseceval-4:mai-thinking-1","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cyberseceval-4?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"}],"slug":"llm-stats-cyberseceval-4","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cyberseceval-4?top_n=500","topics":["coding","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Cybersecurity Capture the Flag (CTF) benchmark for evaluating LLMs in offensive security challenges. Contains diverse cybersecurity tasks including cryptography, web exploitation, binary analysis, and forensics to assess AI capabilities in cybersecurity problem-solving.","descriptions":{"en":"Cybersecurity Capture the Flag (CTF) benchmark for evaluating LLMs in offensive security challenges. Contains diverse cybersecurity tasks including cryptography, web exploitation, binary analysis, and forensics to assess AI capabilities in cybersecurity problem-solving."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:cybersecurity-ctfs:o1-mini","reported_at":"2024-09-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybersecurity-ctfs?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:cybersecurity-ctfs","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"llm_stats_id","primary":"cybersecurity","source_record":"catalog_d472c7b25fe16c49"},"modality":"text","name":"Cybersecurity CTFs","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":77.60000000000001,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.776,"raw_min":0.287,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:cybersecurity-ctfs:gpt-5.3-codex","reported_date":"2026-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybersecurity-ctfs?top_n=500"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"}],"slug":"llm-stats-cybersecurity-ctfs","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybersecurity-ctfs?top_n=500","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:barryq/cypherturn","source_record":"bm_cypherturn_9823bd27"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the multi-turn sessions through which analysts actually work.","descriptions":{"en":"Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the multi-turn sessions through which analysts actually work."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.36987","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_cypherturn_9823bd27"},"modality":null,"name":"CypherTurn","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36987","source_url":"https://arxiv.org/abs/2609.36987"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36987","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36987","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"GPU kernels generated by large language model (LLM) agents can remain less efficient than expert implementations, but runtime alone does not reveal how the gap relates to design discovery and implementation.","descriptions":{"en":"GPU kernels generated by large language model (LLM) agents can remain less efficient than expert implementations, but runtime alone does not reveal how the gap relates to design discovery and implementation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.03226","languages":[],"library_direction":{"basis":"model-review-daily","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_d2k-bench_b3f88d68"},"modality":null,"name":"D2K-Bench","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03226","source_url":"https://arxiv.org/abs/2610.03226"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03226","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03226","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:tjunlp-lab/d2vbench","source_record":"bm_d2vbench_a4547106"},"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"D2VBench evaluates LLM value alignment using 10,000 daily dilemma scenarios covering 158 fine-grained value concepts, with a hybrid paradigm of multiple-choice and open-ended questions.","descriptions":{"en":"D2VBench evaluates LLM value alignment using 10,000 daily dilemma scenarios covering 158 fine-grained value concepts, with a hybrid paradigm of multiple-choice and open-ended questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.19834","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_d2vbench_a4547106"},"modality":null,"name":"D2VBench","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19834","source_url":"https://arxiv.org/abs/2607.19834"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19834","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19834","topics":["safety"],"unit":null},{"aliases":["OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"D3-Omni diagnoses multimodal judge models across T2I, T2V, and TTS tasks with 53 balanced orthogonal binary dimensions and 10,671 samples that isolate specific capability failures.","descriptions":{"en":"D3-Omni diagnoses multimodal judge models across T2I, T2V, and TTS tasks with 53 balanced orthogonal binary dimensions and 10,671 samples that isolate specific capability failures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24160","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio","health-medicine"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_omnijudge-or-omnibias-diagnosing-multimoda_74cb9dd9"},"modality":null,"name":"D3-Omni","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24160","source_url":"https://arxiv.org/abs/2608.24160"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24160","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24160","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2506.23719","repository":null,"source_record":"catalog_8c5fe50f950d000a"},"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DABstep is a benchmark for evaluating AI agents’ multi-step reasoning and planning abilities in realistic data analysis tasks, targeting code-executing language model agents. DABstep 是一个用于评估 AI 智能体在现实多步骤数据分析任务中推理与规划能力的基准，面向具备代码执行能力的语言模型代理。该基准包含450多个任务，源自金融分析平台，涵盖结构化数据处理、非结构化文档理解和跨源信息整合。","descriptions":{"en":"DABstep is a benchmark for evaluating AI agents’ multi-step reasoning and planning abilities in realistic data analysis tasks, targeting code-executing language model agents.","zh":"DABstep 是一个用于评估 AI 智能体在现实多步骤数据分析任务中推理与规划能力的基准，面向具备代码执行能力的语言模型代理。该基准包含450多个任务，源自金融分析平台，涵盖结构化数据处理、非结构化文档理解和跨源信息整合。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:2030","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"artifact:arxiv:2506.23719","primary":"professional-work","source_record":"catalog_8c5fe50f950d000a"},"modality":null,"name":"DABstep","openness":"unknown","publisher":"Adyen , HuggingFace","released":"2025-06-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2030-dabstep","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DABstep","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_dailybench_da2d9ccf"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DailyBench is a unified benchmark for evaluating AI-generated image detectors on modern full-image synthesis and object-level manipulation. It comprises FakeBench, with images from recent open-source and commercial generative models, and ManipulationBench, with subtle edits to real images. Detectors are scored by balanced accuracy.","descriptions":{"en":"DailyBench is a unified benchmark for evaluating AI-generated image detectors on modern full-image synthesis and object-level manipulation. It comprises FakeBench, with images from recent open-source and commercial generative models, and ManipulationBench, with subtle edits to real images. Detectors are scored by balanced accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24016","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_dailybench_da2d9ccf"},"modality":null,"name":"DailyBench","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24016","source_url":"https://arxiv.org/abs/2607.24016"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24016","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24016","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":48,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:lliar-liar/daily-omni","source_record":"catalog_a10b3e867f137732"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","video"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DailyOmni evaluates multimodal models on daily-life video understanding tasks.","descriptions":{"en":"DailyOmni evaluates multimodal models on daily-life video understanding tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:dailyomni:mimo-v2.5","reported_at":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dailyomni?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:dailyomni","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_a10b3e867f137732"},"modality":"multimodal","name":"DailyOmni","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":83.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.835,"raw_min":0.835,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:dailyomni:mimo-v2.5","reported_date":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dailyomni?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"}],"slug":"llm-stats-dailyomni","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dailyomni?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":6,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"claire_record_id","repository":"gh:agi-eval-official/dailyreport","source_record":"bm_dailyreport_bfd6c726"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DailyReport is a benchmark with 150 open-ended daily search tasks and 3,546 rubrics, evaluating search agents on information-seeking tasks. Tasks are decomposed into subtasks with cascade rubrics across dimensions, yielding interpretable scores and a user preference score.","descriptions":{"en":"DailyReport is a benchmark with 150 open-ended daily search tasks and 3,546 rubrics, evaluating search agents on information-seeking tasks. Tasks are decomposed into subtasks with cascade rubrics across dimensions, yielding interpretable scores and a user preference score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.12871","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_dailyreport_bfd6c726"},"modality":null,"name":"DailyReport","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12871","source_url":"https://arxiv.org/abs/2606.12871"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12871","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12871","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":44,"github_stars_30d":28,"hf_dataset_downloads":581,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:zhikaixu24/das","source_record":"bm_das-bench_4090fd3a"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"DAS-Bench provides 30 topics for evaluating academic survey generation, with DAS-Eval scoring 16 criteria across citation quality, taxonomic synthesis, hierarchical discourse, and manuscript reliability.","descriptions":{"en":"DAS-Bench provides 30 topics for evaluating academic survey generation, with DAS-Eval scoring 16 criteria across citation quality, taxonomic synthesis, hierarchical discourse, and manuscript reliability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.18034","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","chat-writing"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_das-bench_4090fd3a"},"modality":null,"name":"DAS-Bench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.18034","source_url":"https://arxiv.org/abs/2608.18034"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-18034","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18034","topics":["language"],"unit":null},{"aliases":["Leveraging Fine-grained Error Correction in Korean Speech Recognition for Consultation Services"],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Dialogue-level ASR error correction benchmark dataset with 1,974 dialogues and 115,460 utterances from Korean call center interactions.","descriptions":{"en":"Dialogue-level ASR error correction benchmark dataset with 1,974 dialogues and 115,460 utterances from Korean call center interactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09889","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","speech-audio"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_leveraging-fine-grained-error-correction-i_1f6405bc"},"modality":null,"name":"DasanCallDial","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09889","source_url":"https://arxiv.org/abs/2609.09889"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09889","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09889","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_dataclaweval_e7e41e17"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DataClawEval evaluates autonomous data-engineering agents across 100 end-to-end tasks spanning PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL, with deterministic rule-based grading in isolated sandboxes.","descriptions":{"en":"DataClawEval evaluates autonomous data-engineering agents across 100 end-to-end tasks spanning PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL, with deterministic rule-based grading in isolated sandboxes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.28033","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","code-generation"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_dataclaweval_e7e41e17"},"modality":null,"name":"DataClawEval","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28033","source_url":"https://arxiv.org/abs/2607.28033"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28033","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28033","topics":["coding","agents"],"unit":null},{"aliases":["DataComp-VLM: Improved Open Datasets for Vision-Language Models"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"A controlled benchmark for comparing vision-language training-data curation strategies across specified model scales and token budgets.","descriptions":{"en":"A controlled benchmark for comparing vision-language training-data curation strategies across specified model scales and token budgets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.28551","languages":[],"library_direction":null,"modality":null,"name":"DataComp for VLMs (DCVLM)","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.28551","source_url":"https://arxiv.org/abs/2606.28551"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-28551","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28551","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:sohasegawa/datagovbench","source_record":"bm_datagovbench_274a71be"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"DataGovBench evaluates LLMs on real-world data analysis using government open data. It comprises Table QA (complex decomposable questions with textual or visual answers) and Table Insight (exploratory data analysis producing expert-level findings). Scoring uses unambiguous reference-based metrics.","descriptions":{"en":"DataGovBench evaluates LLMs on real-world data analysis using government open data. It comprises Table QA (complex decomposable questions with textual or visual answers) and Table Insight (exploratory data analysis producing expert-level findings). Scoring uses unambiguous reference-based metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.06482","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_datagovbench_274a71be"},"modality":null,"name":"DataGovBench","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06482","source_url":"https://arxiv.org/abs/2607.06482"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06482","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06482","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_datakernelbench_0cd9eeb8"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM-generated CUDA or Triton kernels for database query operations on GPUs, measuring execution speedup over torch.compile on TPC-H workloads.","descriptions":{"en":"Evaluates LLM-generated CUDA or Triton kernels for database query operations on GPUs, measuring execution speedup over torch.compile on TPC-H workloads."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25061","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_datakernelbench_0cd9eeb8"},"modality":null,"name":"DataKernelBench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25061","source_url":"https://arxiv.org/abs/2608.25061"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25061","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25061","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.PL"],"collected_at":"2026-10-14T09:44:51Z","description":"Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write.","descriptions":{"en":"Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37233","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_datalogbench_2e960bdb"},"modality":null,"name":"DatalogBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37233","source_url":"https://arxiv.org/abs/2609.37233"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37233","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37233","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:datasense-bench/datasense-bench","source_record":"bm_datasense-bench_ae716f95"},"categories":["AI Scientist","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"As claims about recursive self-improvement (RSI) and artificial general intelligence (AGI) proliferate, we ask a simple question: do frontier AI models have a sense of data, i.e., can they reliably select the right data for training?","descriptions":{"en":"As claims about recursive self-improvement (RSI) and artificial general intelligence (AGI) proliferate, we ask a simple question: do frontier AI models have a sense of data, i.e., can they reliably select the right data for training?"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.12190","languages":[],"library_direction":{"basis":"model-review-daily","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_datasense-bench_ae716f95"},"modality":null,"name":"DataSense-Bench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12190","source_url":"https://arxiv.org/abs/2610.12190"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12190","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12190","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":34,"github_stars_30d":8,"hf_dataset_downloads":null,"hf_paper_upvotes":33,"join":"claire_record_id","repository":"gh:hkustdial/dataspace","source_record":"bm_dataspace_1846702d"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DataSpace evaluates data agents on analytical questions over heterogeneous workspaces containing CSV, JSON, SQLite, Markdown, PDF, and video artifacts. Agents must discover and integrate relevant evidence and return complete tabular results, scored by a deterministic evaluator using header-invariant column alignment and type-aware comparison.","descriptions":{"en":"DataSpace evaluates data agents on analytical questions over heterogeneous workspaces containing CSV, JSON, SQLite, Markdown, PDF, and video artifacts. Agents must discover and integrate relevant evidence and return complete tabular results, scored by a deterministic evaluator using header-invariant column alignment and type-aware comparison."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.03451","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","deep-research"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_dataspace_1846702d"},"modality":null,"name":"DataSpace","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03451","source_url":"https://arxiv.org/abs/2608.03451"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03451","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03451","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:surge-ai/dayjob","source_record":"bm_dayjob_acfaf282"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Professional work often starts with a brief request that leaves the professional to work out what is needed, which documents matter, and whether the request's premise holds.","descriptions":{"en":"Professional work often starts with a brief request that leaves the professional to work out what is needed, which documents matter, and whether the request's premise holds."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.01306","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_dayjob_acfaf282"},"modality":null,"name":"DAYJOB","openness":"unknown","publisher":null,"released":"2026-10-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.01306","source_url":"https://arxiv.org/abs/2610.01306"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-01306","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01306","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DB"],"collected_at":"2026-10-14T09:44:51Z","description":"A production-fidelity benchmark for LLM-based database operations agents, using instrumented PostgreSQL environments with active workloads and outcome-first evaluation across 106 scenarios.","descriptions":{"en":"A production-fidelity benchmark for LLM-based database operations agents, using instrumented PostgreSQL environments with active workloads and outcome-first evaluation across 106 scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.22165","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","agentic-coding"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_dba-bench_29cc8ff8"},"modality":null,"name":"DBA-Bench","openness":"unknown","publisher":null,"released":"2026-07-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22165","source_url":"https://arxiv.org/abs/2607.22165"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22165","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22165","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DB"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs across five database lifecycle phases using tasks like schema design, SQL implementation, and debugging, with a focus on holistic database management capabilities.","descriptions":{"en":"Evaluates LLMs across five database lifecycle phases using tasks like schema design, SQL implementation, and debugging, with a focus on holistic database management capabilities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03794","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","professional-work"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_dblifebench_c72360a2"},"modality":null,"name":"DBLifeBench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03794","source_url":"https://arxiv.org/abs/2608.03794"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03794","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03794","topics":["coding"],"unit":null},{"aliases":["Evaluating Agentic Code Repair Capabilities in Distributed Systems"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_evaluating-agentic-code-repair-capabilitie_2495b9d3"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"DDBench is a code-repair benchmark with 60 historical bugs from 13 open-source distributed systems, evaluated under symptom-only and context-augmented conditions.","descriptions":{"en":"DDBench is a code-repair benchmark with 60 historical bugs from 13 open-source distributed systems, evaluated under symptom-only and context-augmented conditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14863","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_evaluating-agentic-code-repair-capabilitie_2495b9d3"},"modality":null,"name":"DDBench","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14863","source_url":"https://arxiv.org/abs/2608.14863"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14863","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14863","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"DDX-TRACE evaluates medical diagnostic trajectories in multimodal neuroradiology over 211 physician-adjudicated cases, where models request imaging studies, update differential diagnoses, and stop with a localized final diagnosis under hidden evidence.","descriptions":{"en":"DDX-TRACE evaluates medical diagnostic trajectories in multimodal neuroradiology over 211 physician-adjudicated cases, where models request imaging studies, update differential diagnoses, and stop with a localized final diagnosis under hidden evidence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.23629","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_ddx-trace_5ca2d5dc"},"modality":null,"name":"DDX-TRACE","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23629","source_url":"https://arxiv.org/abs/2605.23629"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23629","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23629","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yyuhhhh/debtnegotiation","source_record":"bm_debtbench_7013b5c5"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DebtBench evaluates negotiation dialogue systems in debt collection with persona-enriched user profiles to reflect behavioral heterogeneity. It uses a dataset of negotiation scenarios with multiple user personas, scoring financial recovery and interaction experience metrics.","descriptions":{"en":"DebtBench evaluates negotiation dialogue systems in debt collection with persona-enriched user profiles to reflect behavioral heterogeneity. It uses a dataset of negotiation scenarios with multiple user personas, scoring financial recovery and interaction experience metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.25218","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","chat-writing"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_debtbench_7013b5c5"},"modality":null,"name":"DebtBench","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25218","source_url":"https://arxiv.org/abs/2607.25218"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25218","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25218","topics":["agents","language","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","ACL 2024","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DebugBench is an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python. DebugBench 是一个包含 4,253 个实例的 LLM 调试基准。它涵盖了 C++、Java 和 Python 中的四个主要错误类别和 18 种次要类型。","descriptions":{"en":"DebugBench is an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python.","zh":"DebugBench 是一个包含 4,253 个实例的 LLM 调试基准。它涵盖了 C++、Java 和 Python 中的四个主要错误类别和 18 种次要类型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1078","languages":[],"library_direction":null,"modality":null,"name":"DebugBench","openness":"open","publisher":"THUNLP","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1078-debugbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DebugBench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":5,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":84,"join":"claire_record_id","repository":null,"source_record":"bm_decepeval_12c54228"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their reliable deployment.","descriptions":{"en":"As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their reliable deployment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.07967","languages":[],"library_direction":{"basis":"model-review-daily","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_decepeval_12c54228"},"modality":null,"name":"DecepEval","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.07967","source_url":"https://arxiv.org/abs/2610.07967"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-07967","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07967","topics":["safety"],"unit":null},{"aliases":["decision-layer-benchmark"],"attention_signals":null,"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A token-cost model for an AI agent's decision layer: estimating monthly spend for five decision questions across naive, cached, and tuned OpenAI, Anthropic, and Google baselines versus Jev.","descriptions":{"en":"A token-cost model for an AI agent's decision layer: estimating monthly spend for five decision questions across naive, cached, and tuned OpenAI, Anthropic, and Google baselines versus Jev."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:gohaike/decision-layer-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Decision Layer Benchmark","openness":"unknown","publisher":null,"released":"2026-10-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:gohaike/decision-layer-benchmark","source_url":"https://github.com/GOhaike/decision-layer-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-gohaike-decision-layer-benchmark","source":"claire_radar","source_url":"https://github.com/GOhaike/decision-layer-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:supreethbare/decision-model-bench","source_record":"bm_decision-model-bench_23575868"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The repository compares typed decision models on several enterprise tasks, but it is a one-off harness without a formal benchmark name or clear external submission mechanism.","descriptions":{"en":"The repository compares typed decision models on several enterprise tasks, but it is a one-off harness without a formal benchmark name or clear external submission mechanism."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:supreethbare/decision-model-bench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding","professional-work","agent-skills-evolution"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_decision-model-bench_23575868"},"modality":null,"name":"decision-model-bench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:supreethbare/decision-model-bench","source_url":"https://github.com/supreethbare/decision-model-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-supreethbare-decision-model-bench","source":"claire_radar","source_url":"https://github.com/supreethbare/decision-model-bench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["productivity","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DECK-Bench is Moonshot AI's internal evaluation of agents on creating and reasoning over presentation-style knowledge-work artifacts.","descriptions":{"en":"DECK-Bench is Moonshot AI's internal evaluation of agents on creating and reasoning over presentation-style knowledge-work artifacts."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:deck-bench:kimi-k3","reported_at":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deck-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:deck-bench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_a6a008c7ba1cc43a"},"modality":"text","name":"DECK-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":73.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.735,"raw_min":0.735,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:deck-bench:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deck-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"}],"slug":"llm-stats-deck-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deck-bench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"DECODEM provides benchmark datasets for evaluating automated extraction of corporate governance variables from organizational documents, with human-annotated charters and bylaws.","descriptions":{"en":"DECODEM provides benchmark datasets for evaluating automated extraction of corporate governance variables from organizational documents, with human-annotated charters and bylaws."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.15879","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","document-ocr"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_decodem_118f6692"},"modality":null,"name":"DECODEM","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15879","source_url":"https://arxiv.org/abs/2607.15879"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15879","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15879","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":30,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_decompbench_f0f37f4f"},"categories":["Agents","General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"DeCompBench evaluates agent safety against decomposition attacks, where harmful tasks are broken into benign subtasks; it measures refusal rates and objective fulfillment on decomposed variants.","descriptions":{"en":"DeCompBench evaluates agent safety against decomposition attacks, where harmful tasks are broken into benign subtasks; it measures refusal rates and objective fulfillment on decomposed variants."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13994","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","safety-alignment"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_decompbench_f0f37f4f"},"modality":null,"name":"DeCompBench","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13994","source_url":"https://arxiv.org/abs/2606.13994"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13994","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13994","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_deep4ge_710e720a"},"categories":["Software & AI Compute","Vision & 3D","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Deep4ge is a dataset of 14,227 training runs from 59 DNN programs with documented faults, providing per-epoch features for fault detection and diagnosis tasks, including binary detection, multi-class diagnosis, and early prediction.","descriptions":{"en":"Deep4ge is a dataset of 14,227 training runs from 59 DNN programs with documented faults, providing per-epoch features for fault detection and diagnosis tasks, including binary detection, multi-class diagnosis, and early prediction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.12868","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_deep4ge_710e720a"},"modality":null,"name":"Deep4ge","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.12868","source_url":"https://arxiv.org/abs/2607.12868"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-12868","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12868","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CY"],"collected_at":"2026-10-14T09:44:51Z","description":"DeepBiasBench is a benchmark for in-depth probing of social biases in LVLMs, built using an adaptive framework with dynamic test data generation and iterative rewriting.","descriptions":{"en":"DeepBiasBench is a benchmark for in-depth probing of social biases in LVLMs, built using an adaptive framework with dynamic test data generation and iterative rewriting."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.11228","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_deepbias_70d900bc"},"modality":null,"name":"DeepBias","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11228","source_url":"https://arxiv.org/abs/2607.11228"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11228","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11228","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tangdouer1005/deepchart","source_record":"bm_deepchart_94c0e719"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Faithful chart generation in real-world data-science workflows requires grounding visualizations in scattered evidence, computing chart-ready quantities, and rendering them accurately.","descriptions":{"en":"Faithful chart generation in real-world data-science workflows requires grounding visualizations in scattered evidence, computing chart-ready quantities, and rendering them accurately."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.26757","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","visual-reasoning"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_deepchart_94c0e719"},"modality":null,"name":"DEEPCHART","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26757","source_url":"https://arxiv.org/abs/2608.26757"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26757","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26757","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","多模态模型","VLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Deepfake-Eval-2024 is an in-the-wild deepfake dataset. Deepfake-Eval-2024 contains 44 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing contemporary manipulation technologies, diverse media content, 88 different website sources, and 52 different languages. Deepfake-Eval-2024 是一个真实场景的深度伪造数据集。该数据集包含44小时的视频、56.5小时的音频和1,975张图像，涵盖当代篡改技术、多样化的媒体内容、来自88个不同网站来源的素材以及52种不同语言。","descriptions":{"en":"Deepfake-Eval-2024 is an in-the-wild deepfake dataset. Deepfake-Eval-2024 contains 44 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing contemporary manipulation technologies, diverse media content, 88 different website sources, and 52 different languages.","zh":"Deepfake-Eval-2024 是一个真实场景的深度伪造数据集。该数据集包含44小时的视频、56.5小时的音频和1,975张图像，涵盖当代篡改技术、多样化的媒体内容、来自88个不同网站来源的素材以及52种不同语言。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1605","languages":[],"library_direction":null,"modality":null,"name":"Deepfake-Eval-2024","openness":"unknown","publisher":"TrueMedia.org, etc.","released":"2025-03-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1605-deepfake-eval-2024","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Deepfake-Eval-2024","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":17164,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":37,"join":"llm_stats_id","repository":null,"source_record":"catalog_bebcb6daaaabeab9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DeepPlanning evaluates LLMs on complex multi-step planning tasks requiring long-horizon reasoning, goal decomposition, and strategic decision-making.","descriptions":{"en":"DeepPlanning evaluates LLMs on complex multi-step planning tasks requiring long-horizon reasoning, goal decomposition, and strategic decision-making."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:deep-planning:qwen3.5-397b-a17b","reported_at":"2026-02-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deep-planning?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:deep-planning","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","tool-use"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_bebcb6daaaabeab9"},"modality":"text","name":"DeepPlanning","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":62.3,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.623,"raw_min":0.176,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:deep-planning:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deep-planning?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-deep-planning","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deep-planning?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DeepResearch Bench is a comprehensive benchmark designed to evaluate large language model (LLM) agents on complex research tasks. DeepResearch Bench 是一个面向大型语言模型（LLM）智能体的综合性评测基准，专为评估其在复杂研究任务中的表现而设计。 该基准包含 100 个由 22 个领域的专家精心设计的博士级研究任务，涵盖多领域的深度研究需求。","descriptions":{"en":"DeepResearch Bench is a comprehensive benchmark designed to evaluate large language model (LLM) agents on complex research tasks.","zh":"DeepResearch Bench 是一个面向大型语言模型（LLM）智能体的综合性评测基准，专为评估其在复杂研究任务中的表现而设计。 该基准包含 100 个由 22 个领域的专家精心设计的博士级研究任务，涵盖多领域的深度研究需求。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1987","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"DeepResearchBench","openness":"unknown","publisher":"University of Science and Technology of China , Metastone Technology, Beijing","released":"2025-06-13","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1987-deepresearchbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DeepResearchBench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"llm_stats_id","repository":null,"source_record":"catalog_b710395859fc9fe8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DeepSearchQA is a benchmark for evaluating deep search and question-answering capabilities, testing models' ability to perform multi-hop reasoning and information retrieval across complex knowledge domains.","descriptions":{"en":"DeepSearchQA is a benchmark for evaluating deep search and question-answering capabilities, testing models' ability to perform multi-hop reasoning and information retrieval across complex knowledge domains."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:deepsearchqa:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepsearchqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:deepsearchqa","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","factuality"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_b710395859fc9fe8"},"modality":"text","name":"DeepSearchQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":95.0,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.95,"raw_min":0.746,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:deepsearchqa:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepsearchqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"}],"slug":"llm-stats-deepsearchqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepsearchqa?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["DeepSearchQA"],"attention_signals":null,"categories":["agent"],"collected_at":null,"description":"Live-web deep research; F1 grading depends on the reference answer set.","descriptions":{"en":"Live-web deep research; F1 grading depends on the reference answer set."},"evidence_summary":{"document_count":2,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000deepsearchqa\u0000moonshot_kimi_k3_model_card\u0000deepsearchqa\u0000F1, reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"first_score_reported_at":"2026-06-13","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000deepsearchqa\u0000moonshot_kimi_k3_model_card\u0000deepsearchqa\u0000F1, reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:deepsearchqa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["deep-research"],"join":"review:14H034160212:2026-10-15","primary":"deep-research","source_record":null},"modality":null,"name":"DeepSearchQA","openness":"unknown","publisher":null,"released":"2026-02-10","released_reference":{"basis":"benchmark_release","source_key":"model-reports:deepsearchqa","source_url":"https://huggingface.co/datasets/PokeeAI/DeepSearchQA"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":95.0,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":95.0,"raw_min":95.0,"source_reference":{"obs_id":"curated\u0000deepsearchqa\u0000moonshot_kimi_k3_model_card\u0000deepsearchqa\u0000F1, reasoning=max\u0000Kimi K3","observation_id":"curated\u0000deepsearchqa\u0000moonshot_kimi_k3_model_card\u0000deepsearchqa\u0000F1, reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"}],"slug":"deepsearchqa","source":"model_reports","source_url":"https://huggingface.co/datasets/PokeeAI/DeepSearchQA","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1821,"github_stars_30d":157,"hf_dataset_downloads":1224,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:datacurve-ai/deep-swe","source_record":"bm_deepswe_ebdcd38e"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on 113 original, long-horizon software engineering tasks across 91 open-source repositories in five languages. Tasks are written from scratch, with hand-written verifiers that check requested functionality and accept any correct implementation.","descriptions":{"en":"Evaluates coding agents on 113 original, long-horizon software engineering tasks across 91 open-source repositories in five languages. Tasks are written from scratch, with hand-written verifiers that check requested functionality and accept any correct implementation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.07946","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_deepswe_ebdcd38e"},"modality":null,"name":"DeepSWE","openness":"unknown","publisher":null,"released":"2026-05-18","released_reference":{"basis":"first_public","source_key":"claire-radar:2607.07946","source_url":"https://datacurve.ai/research"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-07946","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07946","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1821,"github_stars_30d":157,"hf_dataset_downloads":1224,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:datacurve-ai/deep-swe","source_record":"bm_deepswe_ebdcd38e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DeepSWE is a software engineering agent benchmark evaluated with the mini-swe-agent harness, where each task is solved in an isolated container with no internet access. It measures an agent's ability to autonomously resolve real-world coding issues end to end.","descriptions":{"en":"DeepSWE is a software engineering agent benchmark evaluated with the mini-swe-agent harness, where each task is solved in an isolated container with no internet access. It measures an agent's ability to autonomously resolve real-world coding issues end to end."},"editorial_guide":{"id":"deepswe","record_keys":["model-reports:deepswe","llm-stats:deepswe","llm-stats:deepswe-1.0","llm-stats:deepswe-1.1"],"reviewed_at":"2026-10-20","scope":{"en":"Datacurve's DeepSWE v1 / v1.1. The task collection is shared, but verification changed in v1.1; check the version on each score.","zh":"Datacurve 的 DeepSWE v1 / v1.1。两版使用同一批任务，但 v1.1 调整了验证流程；比较成绩时须确认版本。"},"sections":[{"id":"tasks","source_ids":["introduction","httpx"],"text":{"en":"Datacurve's collection contains 113 tasks across 91 repositories in five programming languages. An agent explores a pinned codebase and implements a requested change. For example, the HTTPX task asks for incremental JSON streaming with both synchronous and asynchronous interfaces.","zh":"Datacurve 的题集包含 113 道任务，覆盖 91 个仓库和五种编程语言。Agent 从固定版本的代码出发，自行探索并实现需求。例如，HTTPX 题目要求增量解析 JSON 流，同时支持同步与异步接口。"},"title":{"en":"What the agent actually does","zh":"Agent 实际要做什么"}},{"id":"grading","source_ids":["revision","grader"],"text":{"en":"In v1.1, only the committed patch is applied in a fresh verifier container. Behavioral and designated regression tests decide success, rather than similarity to a reference implementation. Required tests must pass; partial test progress is diagnostic information, not partial task credit.","zh":"v1.1 只提取已提交的补丁，在全新的验证容器中执行测试。判分依据是功能行为和指定的回归测试，而非代码与参考解的相似度。要求的测试必须通过；部分测试通过可用于分析失败，但不等于任务获得部分分数。"},"title":{"en":"How a solution passes","zh":"怎样才算通过"}},{"id":"metrics","source_ids":["paper"],"text":{"en":"The paper's pass@1 averages each task's success fraction across independent attempts. Its pass@4 asks whether any of up to four attempts succeeds. Compare the same benchmark version, agent framework and budget, and read uncertainty alongside small score differences.","zh":"论文中的 pass@1 先计算每道题多次独立尝试的通过比例，再对所有题取平均；pass@4 则看每题最多四次尝试中是否至少成功一次。比较时应对齐版本、Agent 框架与预算，并结合误差区间理解小幅分差。"},"title":{"en":"Read pass@1 and pass@4 separately","zh":"区分 pass@1 与 pass@4"}},{"id":"fit","source_ids":["introduction"],"text":{"en":"Use the task definitions and execution traces to examine repository exploration, feature integration and missed requirements. The published mini-swe-agent setup gives models a shared bash interface; it does not reproduce every model's native coding product.","zh":"结合题面与执行轨迹，可以观察仓库探索、功能集成和需求遗漏。官方采用 mini-swe-agent，为模型提供共同的 bash 接口；这并不能完整代表各模型在原生编程产品中的体验。"},"title":{"en":"When it is useful","zh":"适合用来观察什么"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Binary tests cannot measure all aspects of maintainability or correctness. The paper treats its LLM-assisted verifier audit as a second opinion, not ground truth. Original tasks reduce exposure to existing public fixes, but publication can create future contamination risk.","zh":"二元测试不能覆盖可维护性与正确性的所有方面。论文将 LLM 辅助复审视为补充判断，而非真值。原创任务减少了对既有公开补丁的依赖，但题目公开后仍可能进入后续训练资料。"},"title":{"en":"What a pass cannot establish","zh":"通过评测仍不能说明什么"}},{"id":"review","source_ids":["audit","review"],"text":{"en":"Independent reviewers have reported remaining verifier problems in v1.1, including conflicts between hidden tests and agent-authored tests. These are version-specific audit findings, not a recalculated model ranking. A low score warrants inspection of the failed tests as well as the proposed code.","zh":"独立评审指出 v1.1 仍有验证器问题，包括隐藏测试与 Agent 自写测试发生冲突。这些属于特定版本的审计发现，不能直接换算成新的模型排名。遇到低分时，应同时检查失败的测试和提交的代码。"},"title":{"en":"Read verifier audits alongside the leaderboard","zh":"结合验证器审计阅读榜单"}}],"sources":[{"id":"introduction","title":"Datacurve · DeepSWE introduction and methodology","url":"https://deepswe.datacurve.ai/blog/deepswe"},{"id":"revision","title":"Datacurve · DeepSWE v1.1 verification changes","url":"https://deepswe.datacurve.ai/blog/deepswe-v1-1"},{"id":"paper","title":"DeepSWE paper · Evaluation and limitations","url":"https://arxiv.org/html/2607.07946v1"},{"id":"httpx","title":"Official task · HTTPX streaming JSON","url":"https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/httpx-streaming-json-iteration/instruction.md"},{"id":"grader","title":"Official verifier · Required behavior and regression tests","url":"https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/cliffy-config-file-parsing/tests/grader.py"},{"id":"audit","title":"Independent audit · DeepSWE v1.1 issue #52","url":"https://github.com/datacurve-ai/deep-swe/issues/52"},{"id":"review","title":"Samaritan Research · DeepSWE v1.1 benchmark review","url":"https://samaritan-research.org/benchmarks/deepswe/review"}],"summary":{"en":"Can an agent understand an unfamiliar repository, build a feature across several files, and keep existing behavior working? DeepSWE tests that workflow with original engineering tasks.","zh":"Agent 能否读懂陌生仓库、跨文件实现新功能，同时保持已有功能正常？DeepSWE 用原创工程任务考察这一完整过程。"},"title":"DeepSWE"},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:deepswe:glm-5.2","reported_at":"2026-06-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:deepswe","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"bm_deepswe_ebdcd38e"},"modality":"text","name":"DeepSWE","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":72.7,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.727,"raw_min":0.23,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:deepswe:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0731"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-deepswe","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["DeepSWE"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Original repository-level engineering tasks from Datacurve. Version 1.1 grades committed patches in a separate verifier container. Version, agent framework, sampling policy and budget must accompany pass@1 or pass@4 results; a passing test suite is not a complete code-quality review.","descriptions":{"en":"Original repository-level engineering tasks from Datacurve. Version 1.1 grades committed patches in a separate verifier container. Version, agent framework, sampling policy and budget must accompany pass@1 or pass@4 results; a passing test suite is not a complete code-quality review."},"editorial_guide":{"id":"deepswe","record_keys":["model-reports:deepswe","llm-stats:deepswe","llm-stats:deepswe-1.0","llm-stats:deepswe-1.1"],"reviewed_at":"2026-10-20","scope":{"en":"Datacurve's DeepSWE v1 / v1.1. The task collection is shared, but verification changed in v1.1; check the version on each score.","zh":"Datacurve 的 DeepSWE v1 / v1.1。两版使用同一批任务，但 v1.1 调整了验证流程；比较成绩时须确认版本。"},"sections":[{"id":"tasks","source_ids":["introduction","httpx"],"text":{"en":"Datacurve's collection contains 113 tasks across 91 repositories in five programming languages. An agent explores a pinned codebase and implements a requested change. For example, the HTTPX task asks for incremental JSON streaming with both synchronous and asynchronous interfaces.","zh":"Datacurve 的题集包含 113 道任务，覆盖 91 个仓库和五种编程语言。Agent 从固定版本的代码出发，自行探索并实现需求。例如，HTTPX 题目要求增量解析 JSON 流，同时支持同步与异步接口。"},"title":{"en":"What the agent actually does","zh":"Agent 实际要做什么"}},{"id":"grading","source_ids":["revision","grader"],"text":{"en":"In v1.1, only the committed patch is applied in a fresh verifier container. Behavioral and designated regression tests decide success, rather than similarity to a reference implementation. Required tests must pass; partial test progress is diagnostic information, not partial task credit.","zh":"v1.1 只提取已提交的补丁，在全新的验证容器中执行测试。判分依据是功能行为和指定的回归测试，而非代码与参考解的相似度。要求的测试必须通过；部分测试通过可用于分析失败，但不等于任务获得部分分数。"},"title":{"en":"How a solution passes","zh":"怎样才算通过"}},{"id":"metrics","source_ids":["paper"],"text":{"en":"The paper's pass@1 averages each task's success fraction across independent attempts. Its pass@4 asks whether any of up to four attempts succeeds. Compare the same benchmark version, agent framework and budget, and read uncertainty alongside small score differences.","zh":"论文中的 pass@1 先计算每道题多次独立尝试的通过比例，再对所有题取平均；pass@4 则看每题最多四次尝试中是否至少成功一次。比较时应对齐版本、Agent 框架与预算，并结合误差区间理解小幅分差。"},"title":{"en":"Read pass@1 and pass@4 separately","zh":"区分 pass@1 与 pass@4"}},{"id":"fit","source_ids":["introduction"],"text":{"en":"Use the task definitions and execution traces to examine repository exploration, feature integration and missed requirements. The published mini-swe-agent setup gives models a shared bash interface; it does not reproduce every model's native coding product.","zh":"结合题面与执行轨迹，可以观察仓库探索、功能集成和需求遗漏。官方采用 mini-swe-agent，为模型提供共同的 bash 接口；这并不能完整代表各模型在原生编程产品中的体验。"},"title":{"en":"When it is useful","zh":"适合用来观察什么"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Binary tests cannot measure all aspects of maintainability or correctness. The paper treats its LLM-assisted verifier audit as a second opinion, not ground truth. Original tasks reduce exposure to existing public fixes, but publication can create future contamination risk.","zh":"二元测试不能覆盖可维护性与正确性的所有方面。论文将 LLM 辅助复审视为补充判断，而非真值。原创任务减少了对既有公开补丁的依赖，但题目公开后仍可能进入后续训练资料。"},"title":{"en":"What a pass cannot establish","zh":"通过评测仍不能说明什么"}},{"id":"review","source_ids":["audit","review"],"text":{"en":"Independent reviewers have reported remaining verifier problems in v1.1, including conflicts between hidden tests and agent-authored tests. These are version-specific audit findings, not a recalculated model ranking. A low score warrants inspection of the failed tests as well as the proposed code.","zh":"独立评审指出 v1.1 仍有验证器问题，包括隐藏测试与 Agent 自写测试发生冲突。这些属于特定版本的审计发现，不能直接换算成新的模型排名。遇到低分时，应同时检查失败的测试和提交的代码。"},"title":{"en":"Read verifier audits alongside the leaderboard","zh":"结合验证器审计阅读榜单"}}],"sources":[{"id":"introduction","title":"Datacurve · DeepSWE introduction and methodology","url":"https://deepswe.datacurve.ai/blog/deepswe"},{"id":"revision","title":"Datacurve · DeepSWE v1.1 verification changes","url":"https://deepswe.datacurve.ai/blog/deepswe-v1-1"},{"id":"paper","title":"DeepSWE paper · Evaluation and limitations","url":"https://arxiv.org/html/2607.07946v1"},{"id":"httpx","title":"Official task · HTTPX streaming JSON","url":"https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/httpx-streaming-json-iteration/instruction.md"},{"id":"grader","title":"Official verifier · Required behavior and regression tests","url":"https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/cliffy-config-file-parsing/tests/grader.py"},{"id":"audit","title":"Independent audit · DeepSWE v1.1 issue #52","url":"https://github.com/datacurve-ai/deep-swe/issues/52"},{"id":"review","title":"Samaritan Research · DeepSWE v1.1 benchmark review","url":"https://samaritan-research.org/benchmarks/deepswe/review"}],"summary":{"en":"Can an agent understand an unfamiliar repository, build a feature across several files, and keep existing behavior working? DeepSWE tests that workflow with original engineering tasks.","zh":"Agent 能否读懂陌生仓库、跨文件实现新功能，同时保持已有功能正常？DeepSWE 用原创工程任务考察这一完整过程。"},"title":"DeepSWE"},"evidence_summary":{"document_count":3,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000deepswe\u0000zai_glm_5_3_flash_model_card\u0000deepswe_v1.1\u0000DeepSWE v1.1, mini-swe-agent harness, temp 0.95, top_p 1.0, timeout 6h, 400K context\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000deepswe\u0000zai_glm_5_3_flash_model_card\u0000deepswe_v1.1\u0000DeepSWE v1.1, mini-swe-agent harness, temp 0.95, top_p 1.0, timeout 6h, 400K context\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:deepswe","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"DeepSWE","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"model-reports:deepswe","source_url":"https://deepswe.datacurve.ai/blog/deepswe"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":74.2,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":74.2,"raw_min":63.4,"source_reference":{"obs_id":"curated\u0000deepswe\u0000deepseek_v4_1_flash_technical_report\u0000deepswe_v1_1\u0000think max, Resolved\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000deepswe\u0000deepseek_v4_1_flash_technical_report\u0000deepswe_v1_1\u0000think max, Resolved\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"deepswe","source":"model_reports","source_url":"https://deepswe.datacurve.ai/blog/deepswe","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1821,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:datacurve-ai/deep-swe","source_record":"catalog_eca7687d567a1c3a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DeepSWE 1.0 pass@1 benchmark as reported by Artificial Analysis using provider-specific harness runs.","descriptions":{"en":"DeepSWE 1.0 pass@1 benchmark as reported by Artificial Analysis using provider-specific harness runs."},"editorial_guide":{"id":"deepswe","record_keys":["model-reports:deepswe","llm-stats:deepswe","llm-stats:deepswe-1.0","llm-stats:deepswe-1.1"],"reviewed_at":"2026-10-20","scope":{"en":"Datacurve's DeepSWE v1 / v1.1. The task collection is shared, but verification changed in v1.1; check the version on each score.","zh":"Datacurve 的 DeepSWE v1 / v1.1。两版使用同一批任务，但 v1.1 调整了验证流程；比较成绩时须确认版本。"},"sections":[{"id":"tasks","source_ids":["introduction","httpx"],"text":{"en":"Datacurve's collection contains 113 tasks across 91 repositories in five programming languages. An agent explores a pinned codebase and implements a requested change. For example, the HTTPX task asks for incremental JSON streaming with both synchronous and asynchronous interfaces.","zh":"Datacurve 的题集包含 113 道任务，覆盖 91 个仓库和五种编程语言。Agent 从固定版本的代码出发，自行探索并实现需求。例如，HTTPX 题目要求增量解析 JSON 流，同时支持同步与异步接口。"},"title":{"en":"What the agent actually does","zh":"Agent 实际要做什么"}},{"id":"grading","source_ids":["revision","grader"],"text":{"en":"In v1.1, only the committed patch is applied in a fresh verifier container. Behavioral and designated regression tests decide success, rather than similarity to a reference implementation. Required tests must pass; partial test progress is diagnostic information, not partial task credit.","zh":"v1.1 只提取已提交的补丁，在全新的验证容器中执行测试。判分依据是功能行为和指定的回归测试，而非代码与参考解的相似度。要求的测试必须通过；部分测试通过可用于分析失败，但不等于任务获得部分分数。"},"title":{"en":"How a solution passes","zh":"怎样才算通过"}},{"id":"metrics","source_ids":["paper"],"text":{"en":"The paper's pass@1 averages each task's success fraction across independent attempts. Its pass@4 asks whether any of up to four attempts succeeds. Compare the same benchmark version, agent framework and budget, and read uncertainty alongside small score differences.","zh":"论文中的 pass@1 先计算每道题多次独立尝试的通过比例，再对所有题取平均；pass@4 则看每题最多四次尝试中是否至少成功一次。比较时应对齐版本、Agent 框架与预算，并结合误差区间理解小幅分差。"},"title":{"en":"Read pass@1 and pass@4 separately","zh":"区分 pass@1 与 pass@4"}},{"id":"fit","source_ids":["introduction"],"text":{"en":"Use the task definitions and execution traces to examine repository exploration, feature integration and missed requirements. The published mini-swe-agent setup gives models a shared bash interface; it does not reproduce every model's native coding product.","zh":"结合题面与执行轨迹，可以观察仓库探索、功能集成和需求遗漏。官方采用 mini-swe-agent，为模型提供共同的 bash 接口；这并不能完整代表各模型在原生编程产品中的体验。"},"title":{"en":"When it is useful","zh":"适合用来观察什么"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Binary tests cannot measure all aspects of maintainability or correctness. The paper treats its LLM-assisted verifier audit as a second opinion, not ground truth. Original tasks reduce exposure to existing public fixes, but publication can create future contamination risk.","zh":"二元测试不能覆盖可维护性与正确性的所有方面。论文将 LLM 辅助复审视为补充判断，而非真值。原创任务减少了对既有公开补丁的依赖，但题目公开后仍可能进入后续训练资料。"},"title":{"en":"What a pass cannot establish","zh":"通过评测仍不能说明什么"}},{"id":"review","source_ids":["audit","review"],"text":{"en":"Independent reviewers have reported remaining verifier problems in v1.1, including conflicts between hidden tests and agent-authored tests. These are version-specific audit findings, not a recalculated model ranking. A low score warrants inspection of the failed tests as well as the proposed code.","zh":"独立评审指出 v1.1 仍有验证器问题，包括隐藏测试与 Agent 自写测试发生冲突。这些属于特定版本的审计发现，不能直接换算成新的模型排名。遇到低分时，应同时检查失败的测试和提交的代码。"},"title":{"en":"Read verifier audits alongside the leaderboard","zh":"结合验证器审计阅读榜单"}}],"sources":[{"id":"introduction","title":"Datacurve · DeepSWE introduction and methodology","url":"https://deepswe.datacurve.ai/blog/deepswe"},{"id":"revision","title":"Datacurve · DeepSWE v1.1 verification changes","url":"https://deepswe.datacurve.ai/blog/deepswe-v1-1"},{"id":"paper","title":"DeepSWE paper · Evaluation and limitations","url":"https://arxiv.org/html/2607.07946v1"},{"id":"httpx","title":"Official task · HTTPX streaming JSON","url":"https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/httpx-streaming-json-iteration/instruction.md"},{"id":"grader","title":"Official verifier · Required behavior and regression tests","url":"https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/cliffy-config-file-parsing/tests/grader.py"},{"id":"audit","title":"Independent audit · DeepSWE v1.1 issue #52","url":"https://github.com/datacurve-ai/deep-swe/issues/52"},{"id":"review","title":"Samaritan Research · DeepSWE v1.1 benchmark review","url":"https://samaritan-research.org/benchmarks/deepswe/review"}],"summary":{"en":"Can an agent understand an unfamiliar repository, build a feature across several files, and keep existing behavior working? DeepSWE tests that workflow with original engineering tasks.","zh":"Agent 能否读懂陌生仓库、跨文件实现新功能，同时保持已有功能正常？DeepSWE 用原创工程任务考察这一完整过程。"},"title":"DeepSWE"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:deepswe-1.0:grok-4.5","reported_at":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.0?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:deepswe-1.0","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_eca7687d567a1c3a"},"modality":"text","name":"DeepSWE 1.0","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":62.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.62,"raw_min":0.62,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:deepswe-1.0:grok-4.5","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.0?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"}],"slug":"llm-stats-deepswe-1-0","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.0?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1821,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:datacurve-ai/deep-swe","source_record":"catalog_8498c3e07ce29fb5"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DeepSWE 1.1 evaluates software engineering agents using the mini-swe-agent harness.","descriptions":{"en":"DeepSWE 1.1 evaluates software engineering agents using the mini-swe-agent harness."},"editorial_guide":{"id":"deepswe","record_keys":["model-reports:deepswe","llm-stats:deepswe","llm-stats:deepswe-1.0","llm-stats:deepswe-1.1"],"reviewed_at":"2026-10-20","scope":{"en":"Datacurve's DeepSWE v1 / v1.1. The task collection is shared, but verification changed in v1.1; check the version on each score.","zh":"Datacurve 的 DeepSWE v1 / v1.1。两版使用同一批任务，但 v1.1 调整了验证流程；比较成绩时须确认版本。"},"sections":[{"id":"tasks","source_ids":["introduction","httpx"],"text":{"en":"Datacurve's collection contains 113 tasks across 91 repositories in five programming languages. An agent explores a pinned codebase and implements a requested change. For example, the HTTPX task asks for incremental JSON streaming with both synchronous and asynchronous interfaces.","zh":"Datacurve 的题集包含 113 道任务，覆盖 91 个仓库和五种编程语言。Agent 从固定版本的代码出发，自行探索并实现需求。例如，HTTPX 题目要求增量解析 JSON 流，同时支持同步与异步接口。"},"title":{"en":"What the agent actually does","zh":"Agent 实际要做什么"}},{"id":"grading","source_ids":["revision","grader"],"text":{"en":"In v1.1, only the committed patch is applied in a fresh verifier container. Behavioral and designated regression tests decide success, rather than similarity to a reference implementation. Required tests must pass; partial test progress is diagnostic information, not partial task credit.","zh":"v1.1 只提取已提交的补丁，在全新的验证容器中执行测试。判分依据是功能行为和指定的回归测试，而非代码与参考解的相似度。要求的测试必须通过；部分测试通过可用于分析失败，但不等于任务获得部分分数。"},"title":{"en":"How a solution passes","zh":"怎样才算通过"}},{"id":"metrics","source_ids":["paper"],"text":{"en":"The paper's pass@1 averages each task's success fraction across independent attempts. Its pass@4 asks whether any of up to four attempts succeeds. Compare the same benchmark version, agent framework and budget, and read uncertainty alongside small score differences.","zh":"论文中的 pass@1 先计算每道题多次独立尝试的通过比例，再对所有题取平均；pass@4 则看每题最多四次尝试中是否至少成功一次。比较时应对齐版本、Agent 框架与预算，并结合误差区间理解小幅分差。"},"title":{"en":"Read pass@1 and pass@4 separately","zh":"区分 pass@1 与 pass@4"}},{"id":"fit","source_ids":["introduction"],"text":{"en":"Use the task definitions and execution traces to examine repository exploration, feature integration and missed requirements. The published mini-swe-agent setup gives models a shared bash interface; it does not reproduce every model's native coding product.","zh":"结合题面与执行轨迹，可以观察仓库探索、功能集成和需求遗漏。官方采用 mini-swe-agent，为模型提供共同的 bash 接口；这并不能完整代表各模型在原生编程产品中的体验。"},"title":{"en":"When it is useful","zh":"适合用来观察什么"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Binary tests cannot measure all aspects of maintainability or correctness. The paper treats its LLM-assisted verifier audit as a second opinion, not ground truth. Original tasks reduce exposure to existing public fixes, but publication can create future contamination risk.","zh":"二元测试不能覆盖可维护性与正确性的所有方面。论文将 LLM 辅助复审视为补充判断，而非真值。原创任务减少了对既有公开补丁的依赖，但题目公开后仍可能进入后续训练资料。"},"title":{"en":"What a pass cannot establish","zh":"通过评测仍不能说明什么"}},{"id":"review","source_ids":["audit","review"],"text":{"en":"Independent reviewers have reported remaining verifier problems in v1.1, including conflicts between hidden tests and agent-authored tests. These are version-specific audit findings, not a recalculated model ranking. A low score warrants inspection of the failed tests as well as the proposed code.","zh":"独立评审指出 v1.1 仍有验证器问题，包括隐藏测试与 Agent 自写测试发生冲突。这些属于特定版本的审计发现，不能直接换算成新的模型排名。遇到低分时，应同时检查失败的测试和提交的代码。"},"title":{"en":"Read verifier audits alongside the leaderboard","zh":"结合验证器审计阅读榜单"}}],"sources":[{"id":"introduction","title":"Datacurve · DeepSWE introduction and methodology","url":"https://deepswe.datacurve.ai/blog/deepswe"},{"id":"revision","title":"Datacurve · DeepSWE v1.1 verification changes","url":"https://deepswe.datacurve.ai/blog/deepswe-v1-1"},{"id":"paper","title":"DeepSWE paper · Evaluation and limitations","url":"https://arxiv.org/html/2607.07946v1"},{"id":"httpx","title":"Official task · HTTPX streaming JSON","url":"https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/httpx-streaming-json-iteration/instruction.md"},{"id":"grader","title":"Official verifier · Required behavior and regression tests","url":"https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/cliffy-config-file-parsing/tests/grader.py"},{"id":"audit","title":"Independent audit · DeepSWE v1.1 issue #52","url":"https://github.com/datacurve-ai/deep-swe/issues/52"},{"id":"review","title":"Samaritan Research · DeepSWE v1.1 benchmark review","url":"https://samaritan-research.org/benchmarks/deepswe/review"}],"summary":{"en":"Can an agent understand an unfamiliar repository, build a feature across several files, and keep existing behavior working? DeepSWE tests that workflow with original engineering tasks.","zh":"Agent 能否读懂陌生仓库、跨文件实现新功能，同时保持已有功能正常？DeepSWE 用原创工程任务考察这一完整过程。"},"title":"DeepSWE"},"evidence_summary":{"document_count":1,"model_count":25,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:deepswe-1.1:claude-sonnet-4-6","reported_at":"2026-02-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.1?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:deepswe-1.1","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_8498c3e07ce29fb5"},"modality":"text","name":"DeepSWE 1.1","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":25,"score_direction":"higher_is_better","score_summary":{"display_max":73.0,"display_multiplier":100,"model_count":25,"model_count_basis":"source_model_id","numeric_count":25,"raw_max":0.73,"raw_min":0.12,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:deepswe-1.1:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.1?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Laguna S 2.1"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-deepswe-1-1","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.1?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models"],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A formally grounded benchmark and generation pipeline for evaluating defeasible abduction through automatically verifiable hypothesis construction.","descriptions":{"en":"A formally grounded benchmark and generation pipeline for evaluating defeasible abduction through automatically verifiable hypothesis construction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.18557","languages":[],"library_direction":null,"modality":null,"name":"DeFAb","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18557","source_url":"https://arxiv.org/abs/2606.18557"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18557","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18557","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:varun-2538/koan","source_record":"bm_defiflowbench_75ae0752"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark of 207 prompts for synthesizing DeFi workflows from natural language.","descriptions":{"en":"Benchmark of 207 prompts for synthesizing DeFi workflows from natural language."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.11504","languages":[],"library_direction":null,"modality":null,"name":"DeFiFlowBench","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11504","source_url":"https://arxiv.org/abs/2609.11504"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11504","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11504","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"DelegateCI-Bench evaluates privacy-conscious query rewriting for LLM delegation, with 3,167 samples combining synthetic data across 20 task types, real user queries from WildChat, and a medical challenge set. Systems rewrite queries to suppress non-essential sensitive information.","descriptions":{"en":"DelegateCI-Bench evaluates privacy-conscious query rewriting for LLM delegation, with 3,167 samples combining synthetic data across 20 task types, real user queries from WildChat, and a medical challenge set. Systems rewrite queries to suppress non-essential sensitive information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04067","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_delegateci-bench_0d61203c"},"modality":null,"name":"DelegateCI-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04067","source_url":"https://arxiv.org/abs/2606.04067"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04067","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04067","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:pielord757/delegation-bench","source_record":"bm_delegationbench_13990237"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AI agents that send emails, edit files, and make purchases must decide when to act on their own and when to check with the user first.","descriptions":{"en":"AI agents that send emails, edit files, and make purchases must decide when to act on their own and when to check with the user first."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.05532","languages":[],"library_direction":{"basis":"model-review-daily","categories":["tool-use","agent-security"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_delegationbench_13990237"},"modality":null,"name":"DelegationBench","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05532","source_url":"https://arxiv.org/abs/2610.05532"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05532","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05532","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates search-enabled LLMs on reconstructing security-level delisting events from public sources, using a 1,200-record benchmark with joint accuracy metrics.","descriptions":{"en":"Evaluates search-enabled LLMs on reconstructing security-level delisting events from public sources, using a 1,200-record benchmark with joint accuracy metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22770","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","finance-legal"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_delistbench_3f7c82bb"},"modality":null,"name":"DelistBench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22770","source_url":"http://arxiv.org/abs/2608.22770v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22770","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22770v1","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:algorithmicresearchgroup/deltaml-bench-public","source_record":"bm_deltaml-bench_b6518e0f"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"DeltaML-Bench contains 48 tasks from research papers requiring agents to improve published baselines within open-source repositories, evaluated under compute constraints with fixed grading and integrity checks.","descriptions":{"en":"DeltaML-Bench contains 48 tasks from research papers requiring agents to improve published baselines within open-source repositories, evaluated under compute constraints with fixed grading and integrity checks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.19653","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_deltaml-bench_b6518e0f"},"modality":null,"name":"DeltaML-Bench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.19653","source_url":"https://arxiv.org/abs/2608.19653"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-19653","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19653","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:snyderconsulting/delverl","source_record":"bm_delverl_572c8749"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates local game-playing agents on a deterministic, procedurally generated, partially observed roguelike where agents must secure a key and reach the exit on each floor, scored by floor depth under a 1,500-turn cap.","descriptions":{"en":"Evaluates local game-playing agents on a deterministic, procedurally generated, partially observed roguelike where agents must secure a key and reach the exit on each floor, scored by floor depth under a 1,500-turn cap."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:snyderconsulting/delverl","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multi-agent-games","spatial-3d"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_delverl_572c8749"},"modality":null,"name":"DelveRL","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:snyderconsulting/delverl","source_url":"https://github.com/SnyderConsulting/DelveRL"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-snyderconsulting-delverl","source":"claire_radar","source_url":"https://github.com/SnyderConsulting/DelveRL","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"A cross-regime benchmark for testing whether agent-runtime records are sufficient to reconstruct decision-level governance properties under deterministic degradation conditions.","descriptions":{"en":"A cross-regime benchmark for testing whether agent-runtime records are sufficient to reconstruct decision-level governance properties under deterministic degradation conditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.20634","languages":[],"library_direction":null,"modality":null,"name":"DEMM-Bench","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20634","source_url":"https://arxiv.org/abs/2606.20634"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20634","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20634","topics":["agents"],"unit":null},{"aliases":["Genesis: A Generative Engine for Hierarchical Satellite Image Synthesis","Genesis"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Provides a multi-scale pyramid dataset with diverse geographic regions for evaluating hierarchical satellite image synthesis, along with pyramid-level metrics.","descriptions":{"en":"Provides a multi-scale pyramid dataset with diverse geographic regions for evaluating hierarchical satellite image synthesis, along with pyramid-level metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.02683","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","image-video-generation"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_genesis-a-generative-engine-for-hierarchic_beb0a71b"},"modality":null,"name":"dense500","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02683","source_url":"https://arxiv.org/abs/2609.02683"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02683","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02683","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on 203 real-world dependency-upgrade tasks requiring source adaptation, scored by task completion rate.","descriptions":{"en":"Evaluates coding agents on 203 real-world dependency-upgrade tasks requiring source adaptation, scored by task completion rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30300","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_depbench_c7c815d1"},"modality":null,"name":"DEPBENCH","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30300","source_url":"https://arxiv.org/abs/2608.30300"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30300","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30300","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"DeployBench evaluates LLM agents on 51 research-artifact deployment tasks across AI/ML, systems, and scientific computing. Tasks require setting up environments with multi-language toolchains and system-level dependencies, verified by hidden pipelines that execute the paper's experiments and check outputs.","descriptions":{"en":"DeployBench evaluates LLM agents on 51 research-artifact deployment tasks across AI/ML, systems, and scientific computing. Tasks require setting up environments with multi-language toolchains and system-level dependencies, verified by hidden pipelines that execute the paper's experiments and check outputs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05238","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","ai-rnd"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_deploybench_34fae562"},"modality":null,"name":"DeployBench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05238","source_url":"https://arxiv.org/abs/2606.05238"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05238","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05238","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_00f9bfbf6e386381"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Dermatology multiple choice question assessment benchmark for evaluating medical knowledge and diagnostic reasoning in dermatological conditions and treatments.","descriptions":{"en":"Dermatology multiple choice question assessment benchmark for evaluating medical knowledge and diagnostic reasoning in dermatological conditions and treatments."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:dermmcqa:medgemma-4b-it","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dermmcqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:dermmcqa","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_00f9bfbf6e386381"},"modality":"text","name":"DermMCQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":71.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.718,"raw_min":0.718,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:dermmcqa:medgemma-4b-it","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dermmcqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-20","date_precision":"model_announcement","name":"MedGemma 4B IT"}],"slug":"llm-stats-dermmcqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dermmcqa?top_n=500","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":602,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":98,"join":"llm_stats_id","repository":null,"source_record":"catalog_6acfcea734e55ac9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","code","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Design2Code evaluates the ability to generate code (HTML/CSS/JS) from visual designs.","descriptions":{"en":"Design2Code evaluates the ability to generate code (HTML/CSS/JS) from visual designs."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:design2code:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/design2code?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:design2code","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_6acfcea734e55ac9"},"modality":"image","name":"Design2Code","openness":"unknown","publisher":null,"released":"2024-03-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:design2code","source_url":"https://arxiv.org/abs/2403.03163"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":0.948,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.948,"raw_min":0.934,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:design2code:glm-5v-turbo","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/design2code?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-design2code","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/design2code?top_n=500","topics":["coding","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":90,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:mrwwk/deskcraft","source_record":"bm_deskcraft_5fe792c2"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"DeskCraft evaluates desktop GUI agents on long-horizon professional workflows and human-in-the-loop collaboration across 538 executable tasks in live Ubuntu desktop environments. Covers design, video, audio, and 3D creation software, with execution-based verification.","descriptions":{"en":"DeskCraft evaluates desktop GUI agents on long-horizon professional workflows and human-in-the-loop collaboration across 538 executable tasks in live Ubuntu desktop environments. Covers design, video, audio, and 3D creation software, with execution-based verification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.03103","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","professional-work"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_deskcraft_5fe792c2"},"modality":null,"name":"DeskCraft","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03103","source_url":"https://arxiv.org/abs/2606.03103"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03103","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03103","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Desktop-Delta Bench (DDB) evaluates computer-use models on step-level GUI transition understanding through two tasks: temporal ordering of 3-frame observations and before-after pair classification with five action types, covering 2,013 human-verified instances across ~15 applications.","descriptions":{"en":"Desktop-Delta Bench (DDB) evaluates computer-use models on step-level GUI transition understanding through two tasks: temporal ordering of 3-frame observations and before-after pair classification with five action types, covering 2,013 human-verified instances across ~15 applications."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.26041","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_desktop-delta-bench_ac655444"},"modality":null,"name":"Desktop-Delta Bench","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26041","source_url":"https://arxiv.org/abs/2607.26041"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26041","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26041","topics":["computer-use","agents"],"unit":null},{"aliases":["Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval"],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"DevDataBench evaluates structured-metadata retrieval using grounded, facet-targeted queries across 15 languages.","descriptions":{"en":"DevDataBench evaluates structured-metadata retrieval using grounded, facet-targeted queries across 15 languages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.30473","languages":[],"library_direction":null,"modality":null,"name":"DevDataBench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30473","source_url":"https://arxiv.org/abs/2606.30473"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30473","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30473","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:agenticorglab/devicesworld","source_record":"bm_devicesworld_1a9e0be1"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"DevicesWorld is an executable benchmark for cross-device agent evaluation, comprising 6,140 tasks across Android, Linux, and SmartHome environments, with rule-based verifiers for automatic scoring.","descriptions":{"en":"DevicesWorld is an executable benchmark for cross-device agent evaluation, comprising 6,140 tasks across Android, Linux, and SmartHome environments, with rule-based verifiers for automatic scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.13465","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","tool-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_devicesworld_1a9e0be1"},"modality":null,"name":"DevicesWorld","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.13465","source_url":"https://arxiv.org/abs/2607.13465"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-13465","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13465","topics":["computer-use","agents"],"unit":null},{"aliases":["devin-agent-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:majorminorstudio/devin-agent-benchmark","source_record":"bm_devin-agent-benchmark_d7459cef"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"A paired software-repair evaluation comparing Devin's SWE-2 Medium and Max modes on 30 unique bugs across three difficulty tiers, with hidden tests and a clean regression suite defining success.","descriptions":{"en":"A paired software-repair evaluation comparing Devin's SWE-2 Medium and Max modes on 30 unique bugs across three difficulty tiers, with hidden tests and a clean regression suite defining success."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:majorminorstudio/devin-agent-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_devin-agent-benchmark_d7459cef"},"modality":null,"name":"Devin + SWE-2 Reasoning Effort Benchmark","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:majorminorstudio/devin-agent-benchmark","source_url":"https://github.com/MAJORminorStudio/devin-agent-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-majorminorstudio-devin-agent-benchmark","source":"claire_radar","source_url":"https://github.com/MAJORminorStudio/devin-agent-benchmark","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_dexjoco-x_6aeabd55"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"As dexterous hands proliferate, collecting data and training policies separately for every morphology becomes increasingly impractical.","descriptions":{"en":"As dexterous hands proliferate, collecting data and training policies separately for every morphology becomes increasingly impractical."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.03278","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_dexjoco-x_6aeabd55"},"modality":null,"name":"DexJoCo-X","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03278","source_url":"https://arxiv.org/abs/2610.03278"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03278","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03278","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A six-axis evaluation harness for regulator-aligned assessment of supervisory tickets produced in decentralized-finance risk monitoring.","descriptions":{"en":"A six-axis evaluation harness for regulator-aligned assessment of supervisory tickets produced in decentralized-finance risk monitoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.19501","languages":[],"library_direction":null,"modality":null,"name":"DeXposure-Bench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19501","source_url":"https://arxiv.org/abs/2606.19501"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19501","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19501","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":117,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:ycyao216/dexverse","source_record":"bm_dexverse_baeb3cfa"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates dexterous manipulation across 100 tasks, multiple embodiments, and visual variations, with 3,180 demonstrations and a VR teleoperation interface.","descriptions":{"en":"Evaluates dexterous manipulation across 100 tasks, multiple embodiments, and visual variations, with 3,180 demonstrations and a VR teleoperation interface."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.08751","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_dexverse_baeb3cfa"},"modality":null,"name":"DexVerse","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08751","source_url":"https://arxiv.org/abs/2607.08751"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08751","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08751","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Finance","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"DFAH-Bench measures whether financial agents reproduce decisions and observable execution paths across qualified replay groups.","descriptions":{"en":"DFAH-Bench measures whether financial agents reproduce decisions and observable execution paths across qualified replay groups."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.20491","languages":[],"library_direction":null,"modality":null,"name":"DFAH-Bench","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20491","source_url":"https://arxiv.org/abs/2607.20491"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20491","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20491","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jeremy1392/dgf-bench","source_record":"bm_dgf-bench_f72216ca"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates tool-using LLM governance agents reviewing evidence, focusing on resistance to planted deception.","descriptions":{"en":"Evaluates tool-using LLM governance agents reviewing evidence, focusing on resistance to planted deception."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.34913","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","agent-security"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_dgf-bench_f72216ca"},"modality":null,"name":"DGF-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34913","source_url":"https://arxiv.org/abs/2609.34913"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34913","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34913","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:luongnv89/dgx-spark-llm-lab","source_record":"bm_dgx-spark-llm-lab_ca498b49"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"This repository contains tools for benchmarking LLM configurations (model, quantization, serving flags, thinking mode) on local hardware, with hidden executable tests and leaderboard-like reports.","descriptions":{"en":"This repository contains tools for benchmarking LLM configurations (model, quantization, serving flags, thinking mode) on local hardware, with hidden executable tests and leaderboard-like reports."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:luongnv89/dgx-spark-llm-lab","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_dgx-spark-llm-lab_ca498b49"},"modality":null,"name":"dgx-spark-llm-lab","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:luongnv89/dgx-spark-llm-lab","source_url":"https://github.com/luongnv89/dgx-spark-llm-lab"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-luongnv89-dgx-spark-llm-lab","source":"claire_radar","source_url":"https://github.com/luongnv89/dgx-spark-llm-lab","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"DiagChain evaluates LLM agents on evidence-grounded attack chain reconstruction from heterogeneous telemetry. It provides 69 scenarios and five metrics for stage-wise assessment of reasoning steps.","descriptions":{"en":"DiagChain evaluates LLM agents on evidence-grounded attack chain reconstruction from heterogeneous telemetry. It provides 69 scenarios and five metrics for stage-wise assessment of reasoning steps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03591","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_diagchain_064ef8b0"},"modality":null,"name":"DiagChain","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03591","source_url":"https://arxiv.org/abs/2608.03591"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03591","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03591","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aigrounding/diagram-mmu","source_record":"bm_diagram-mmu_a41bffa4"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Diagram-MMU evaluates multimodal language models on parsing scientific diagrams into LaTeX TikZ code, editing diagram code, and answering questions about diagrams, using 3.7k diagrams and 18.3k questions across six domains.","descriptions":{"en":"Diagram-MMU evaluates multimodal language models on parsing scientific diagrams into LaTeX TikZ code, editing diagram code, and answering questions about diagrams, using 3.7k diagrams and 18.3k questions across six domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.12262","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","code-generation"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_diagram-mmu_a41bffa4"},"modality":null,"name":"Diagram-MMU","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12262","source_url":"https://arxiv.org/abs/2608.12262"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12262","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12262","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DICE-BENCH is a benchmark for evaluating large language models’ tool-use capabilities in multi-round, multi-party dialogues, focusing on function selection, parameter filling, and dialogue context comprehension. DICE-BENCH 是一个评估大型语言模型在多轮、多方对话中工具调用能力的基准，涵盖函数选择、参数填充和对话上下文理解等维度。","descriptions":{"en":"DICE-BENCH is a benchmark for evaluating large language models’ tool-use capabilities in multi-round, multi-party dialogues, focusing on function selection, parameter filling, and dialogue context comprehension.","zh":"DICE-BENCH 是一个评估大型语言模型在多轮、多方对话中工具调用能力的基准，涵盖函数选择、参数填充和对话上下文理解等维度。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2032","languages":[],"library_direction":null,"modality":null,"name":"DICE-BENCH","openness":"open","publisher":"IPAI,Seoul National University ,etc","released":"2025-06-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2032-dice-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DICE-BENCH","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:pku-icst-mipl/dico_bench_eccv2026","source_record":"bm_dicobench_558e401f"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"DiCoBench evaluates multimodal LLMs on multi-image fine-grained perception using high-resolution (up to 2K) image pairs. It includes 765 multiple-choice questions across two tracks: differential and commonality visual cues, covering 8 perception tasks, with exact-match scoring.","descriptions":{"en":"DiCoBench evaluates multimodal LLMs on multi-image fine-grained perception using high-resolution (up to 2K) image pairs. It includes 765 multiple-choice questions across two tracks: differential and commonality visual cues, covering 8 perception tasks, with exact-match scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.26602","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_dicobench_558e401f"},"modality":null,"name":"DiCoBench","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26602","source_url":"https://arxiv.org/abs/2606.26602"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26602","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26602","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents on discovery of unknown transformation rules across 70 independent games with seven difficulty tiers, requiring experimentation and rule generalization.","descriptions":{"en":"Evaluates AI agents on discovery of unknown transformation rules across 70 independent games with seven difficulty tiers, requiring experimentation and rule generalization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12593","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","multi-agent-games"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_dig-bench_37529e7b"},"modality":null,"name":"DiG-bench","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12593","source_url":"https://arxiv.org/abs/2608.12593"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12593","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12593","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"DirectorBench evaluates long-form video generation across 5 dimensions (script, visual, audio, cross-modal, stability) using 80 metadata entries, 7 user profiles, and 40 checkpoint criteria.","descriptions":{"en":"DirectorBench evaluates long-form video generation across 5 dimensions (script, visual, audio, cross-modal, stability) using 80 metadata entries, 7 user profiles, and 40 checkpoint criteria."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30090","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_directorbench_ecf5066a"},"modality":null,"name":"DirectorBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30090","source_url":"https://arxiv.org/abs/2605.30090"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30090","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30090","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tamuchen18/disasterbench_open","source_record":"bm_disasterbench_d2342204"},"categories":["Agents","General AI","Language & Knowledge","Planning"],"collected_at":"2026-10-14T09:44:51Z","description":"DisasterBench is a benchmark for evaluating structured multi-agent planning over disaster-response tools. It includes 233 expert-verified tasks, 26 agents, 81 typed compatibility edges, and 5 planning paradigms. It uses First-Point-of-Failure (FPoF) for step-level failure attribution.","descriptions":{"en":"DisasterBench is a benchmark for evaluating structured multi-agent planning over disaster-response tools. It includes 233 expert-verified tasks, 26 agents, 81 typed compatibility edges, and 5 planning paradigms. It uses First-Point-of-Failure (FPoF) for step-level failure attribution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.27957","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","multi-agent-games"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_disasterbench_d2342204"},"modality":null,"name":"DisasterBench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27957","source_url":"https://arxiv.org/abs/2605.27957"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27957","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27957","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":9,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tanmoutt/disasterbench","source_record":"bm_disasterbench_325517be"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal reasoning for UAV-based disaster response across 14 scene types and 9 tasks spanning pre-, during-, and post-disaster stages.","descriptions":{"en":"Evaluates multimodal reasoning for UAV-based disaster response across 14 scene types and 9 tasks spanning pre-, during-, and post-disaster stages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.06217","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","embodied-robotics"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_disasterbench_325517be"},"modality":null,"name":"DisasterBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06217","source_url":"https://arxiv.org/abs/2606.06217"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06217","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06217","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":1347,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_discern-can-ai-agents-work-like-scientists_283453cc"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents on evidence-grounded scientific reasoning across 203 tasks in eight life-science tracks, covering data integrity, analysis verification, and hypothesis generation under adversarial review.","descriptions":{"en":"Evaluates AI agents on evidence-grounded scientific reasoning across 203 tasks in eight life-science tracks, covering data integrity, analysis verification, and hypothesis generation under adversarial review."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33357","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_discern-can-ai-agents-work-like-scientists_283453cc"},"modality":null,"name":"DISCERN","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33357","source_url":"https://arxiv.org/abs/2609.33357"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33357","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33357","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":null,"source_record":"bm_discobench_0216743e"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"DiscoBench evaluates search agents on clarification-aware deep search, covering 211 samples and 463 ambiguity instances across 11 domains, with four ambiguity types, measuring task utility, ambiguity detection, interaction strategy, and cost efficiency.","descriptions":{"en":"DiscoBench evaluates search agents on clarification-aware deep search, covering 211 samples and 463 ambiguity instances across 11 domains, with four ambiguity types, measuring task utility, ambiguity detection, interaction strategy, and cost efficiency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.27669","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_discobench_0216743e"},"modality":null,"name":"DiscoBench","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27669","source_url":"https://arxiv.org/abs/2606.27669"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27669","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27669","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_discoverphysics_9ae7e548"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DiscoverPhysics is an interactive benchmark for LLM agents to discover laws of motion in simulated worlds with altered physics, scoring trajectory MSE and explanation quality.","descriptions":{"en":"DiscoverPhysics is an interactive benchmark for LLM agents to discover laws of motion in simulated worlds with altered physics, scoring trajectory MSE and explanation quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26087","languages":[],"library_direction":{"basis":"model-review","categories":["science","world-models"],"join":"claire_record_id","primary":"science","source_record":"bm_discoverphysics_9ae7e548"},"modality":null,"name":"DiscoverPhysics","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26087","source_url":"https://arxiv.org/abs/2605.26087"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26087","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26087","topics":["agents","science"],"unit":null},{"aliases":["distilbert-onnx-quantization-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:parth-udawant/distilbert-onnx-quantization-benchmark","source_record":"bm_distilbert-onnx-quantization-benchmark_2d54c553"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"An end-to-end ONNX quantization benchmark for a DistilBERT sentiment classifier, comparing PyTorch FP32, ONNX FP32, dynamic INT8, and static INT8 variants on accuracy, latency, and model size using a fixed 200-sample SST-2 validation set.","descriptions":{"en":"An end-to-end ONNX quantization benchmark for a DistilBERT sentiment classifier, comparing PyTorch FP32, ONNX FP32, dynamic INT8, and static INT8 variants on accuracy, latency, and model size using a fixed 200-sample SST-2 validation set."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:parth-udawant/distilbert-onnx-quantization-benchmark","languages":[],"library_direction":null,"modality":null,"name":"DistilBERT ONNX Quantization Benchmark","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:parth-udawant/distilbert-onnx-quantization-benchmark","source_url":"https://github.com/Parth-Udawant/distilbert-onnx-quantization-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-parth-udawant-distilbert-onnx-quantization-benchmark","source":"claire_radar","source_url":"https://github.com/Parth-Udawant/distilbert-onnx-quantization-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":10,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yizheng-sun/distract-bench","source_record":"bm_distract-bench_3873c48a"},"categories":["General AI","Multimodal","Reasoning","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Distract-Bench evaluates robustness of vision-language models to semantic visual distractions, which are meaningful but task-irrelevant cues that preserve the ground-truth answer.","descriptions":{"en":"Distract-Bench evaluates robustness of vision-language models to semantic visual distractions, which are meaningful but task-irrelevant cues that preserve the ground-truth answer."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.08894","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_distract-bench_3873c48a"},"modality":null,"name":"Distract-Bench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08894","source_url":"https://arxiv.org/abs/2606.08894"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08894","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08894","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:skylenage-ai/dlawbench","source_record":"bm_dlawbench_4866c9b4"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"DLawBench evaluates multi-turn legal consultation in Chinese and U.S. law across four client personas. It scores LLMs on information gathering, legal reasoning, and claim support using 461 cases, 5,532 paired fact entries, inquiry and issue rubrics, and a public leaderboard.","descriptions":{"en":"DLawBench evaluates multi-turn legal consultation in Chinese and U.S. law across four client personas. It scores LLMs on information gathering, legal reasoning, and claim support using 461 cases, 5,532 paired fact entries, inquiry and issue rubrics, and a public leaderboard."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.13931","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_dlawbench_4866c9b4"},"modality":null,"name":"DLawBench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13931","source_url":"https://arxiv.org/abs/2606.13931"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13931","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13931","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:2404.16006","repository":null,"source_record":"catalog_3b039d0c4f726cef"},"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VLB provides a robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. Based on LlavaBench and MMvet, we have curated two more challenging versions of the datasets: LlavaBench_hard and MMvet_hard. These are the hardest multimodal combinations. VLB 为 LVLMs 提供了一种稳健且全面的评估，降低了数据污染并具有灵活的复杂性。基于 LlavaBench 和 MMvet，我们精心制作了两个更具挑战性的数据集版本：LlavaBench_hard 和 MMvet_hard。这些是我们动态策略中最具挑战性的多模态组合（V1+L4）。","descriptions":{"en":"VLB provides a robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. Based on LlavaBench and MMvet, we have curated two more challenging versions of the datasets: LlavaBench_hard and MMvet_hard. These are the hardest multimodal combinations.","zh":"VLB 为 LVLMs 提供了一种稳健且全面的评估，降低了数据污染并具有灵活的复杂性。基于 LlavaBench 和 MMvet，我们精心制作了两个更具挑战性的数据集版本：LlavaBench_hard 和 MMvet_hard。这些是我们动态策略中最具挑战性的多模态组合（V1+L4）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1648","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2404.16006","primary":"visual-reasoning","source_record":"catalog_3b039d0c4f726cef"},"modality":"multimodal","name":"DME","openness":"unknown","publisher":"SJTU, Shanghai AI Laboratory, etc.","released":"2024-10-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1648-dme","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DME","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:yyyujintang/dmv-bench","source_record":"bm_dmv-bench_d16bc24a"},"categories":["Agents","Agents & Tool Use","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"DMV-Bench evaluates visual memory of multimodal agents in an interactive shopping environment with 1,000 product variants. Agents run autonomous sessions and must recall cued product images via exact URL match, with text leakage controlled.","descriptions":{"en":"DMV-Bench evaluates visual memory of multimodal agents in an interactive shopping environment with 1,000 product variants. Agents run autonomous sessions and must recall cued product images via exact URL match, with text leakage controlled."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.27499","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","visual-reasoning"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_dmv-bench_d16bc24a"},"modality":null,"name":"DMV-Bench","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27499","source_url":"https://arxiv.org/abs/2606.27499"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27499","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27499","topics":["agents","multimodal"],"unit":null},{"aliases":["Can LLMs Normalize Databases? A Benchmark and Multi-Agent Framework for Schema Normalization"],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of 3,275 samples for evaluating LLM-driven database normalization from 1NF to BCNF.","descriptions":{"en":"A benchmark of 3,275 samples for evaluating LLM-driven database normalization from 1NF to BCNF."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11141","languages":[],"library_direction":null,"modality":null,"name":"DNBENCH","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11141","source_url":"https://arxiv.org/abs/2609.11141"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11141","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11141","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A belief-shift evaluation in Werewolf measures whether models update beliefs in response to accusers or accusations, using belief updating as a signal of communication skills.","descriptions":{"en":"A belief-shift evaluation in Werewolf measures whether models update beliefs in response to accusers or accusations, using belief updating as a signal of communication skills."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.12446","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_do-llms-trust-the-accuser-or-the-accusatio_020c1e79"},"modality":null,"name":"Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12446","source_url":"https://arxiv.org/abs/2609.12446"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12446","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12446","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"DOC2CI evaluates LLM-generated CI/CD configuration YAML against reference configurations from four CI services, measuring exact match and schema validity.","descriptions":{"en":"DOC2CI evaluates LLM-generated CI/CD configuration YAML against reference configurations from four CI services, measuring exact match and schema validity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01451","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_doc2ci_60d5a789"},"modality":null,"name":"Doc2CI","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01451","source_url":"https://arxiv.org/abs/2608.01451"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01451","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01451","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:setonliang/doc2db-bench","source_record":"bm_doc2db-bench_2f955ada"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Doc2DB-Bench evaluates document-to-database construction, converting long heterogeneous documents into normalized relational databases with entity identities, keys, cross-table links, and integrity constraints. It includes 203 document instances across 42 schemas and 7 domains, with fine-grained capability annotations for intra-table extraction and inter-table reasoning.","descriptions":{"en":"Doc2DB-Bench evaluates document-to-database construction, converting long heterogeneous documents into normalized relational databases with entity identities, keys, cross-table links, and integrity constraints. It includes 203 document instances across 42 schemas and 7 domains, with fine-grained capability annotations for intra-table extraction and inter-table reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.08459","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","code-generation"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_doc2db-bench_2f955ada"},"modality":null,"name":"Doc2DB-Bench","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08459","source_url":"https://arxiv.org/abs/2608.08459"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08459","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08459","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates source attribution in Document VQA by grounding answers to layout elements such as text blocks, tables, and images.","descriptions":{"en":"Evaluates source attribution in Document VQA by grounding answers to layout elements such as text blocks, tables, and images."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.20574","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_docattribench_19cad9ee"},"modality":null,"name":"DocAttriBench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.20574","source_url":"https://arxiv.org/abs/2609.20574"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-20574","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20574","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"DocFormBench evaluates content-aware document formatting for LLMs and multimodal models, using accuracy and efficiency metrics, with a workflow method DocFormFlow for target localization and modification.","descriptions":{"en":"DocFormBench evaluates content-aware document formatting for LLMs and multimodal models, using accuracy and efficiency metrics, with a workflow method DocFormFlow for target localization and modification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01936","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","document-ocr"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_docformbench_158426e5"},"modality":null,"name":"DocFormBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01936","source_url":"https://arxiv.org/abs/2606.01936"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01936","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01936","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates integrated chart-context reasoning in document-style images.","descriptions":{"en":"Evaluates integrated chart-context reasoning in document-style images."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02059","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","document-ocr"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_dochop_e279a131"},"modality":null,"name":"DocHop","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02059","source_url":"https://arxiv.org/abs/2609.02059"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02059","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02059","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_docprivacybench_1e095c41"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"DocPrivacyBench evaluates susceptibility of document understanding MLLMs to relational privacy leakage when visual evidence is absent or minimal, using KIE tasks on identity documents.","descriptions":{"en":"DocPrivacyBench evaluates susceptibility of document understanding MLLMs to relational privacy leakage when visual evidence is absent or minimal, using KIE tasks on identity documents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12911","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","document-ocr"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_docprivacybench_1e095c41"},"modality":null,"name":"DocPrivacyBench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12911","source_url":"https://arxiv.org/abs/2608.12911"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12911","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12911","topics":["safety"],"unit":null},{"aliases":["documentvision-multimodal-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:master72o/documentvision-multimodal-benchmark","source_record":"bm_documentvision-multimodal-benchmark_ca85dc49"},"categories":["Biology & Drug Discovery","Finance","Multimodal","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates OCR pipelines and vision-language models on dense tables, key-value extraction, charts, and layout grounding across high-resolution enterprise document images.","descriptions":{"en":"Evaluates OCR pipelines and vision-language models on dense tables, key-value extraction, charts, and layout grounding across high-resolution enterprise document images."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:master72o/documentvision-multimodal-benchmark","languages":[],"library_direction":{"basis":"model-review-daily","categories":["document-ocr","visual-reasoning"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_documentvision-multimodal-benchmark_ca85dc49"},"modality":null,"name":"DocumentVision","openness":"unknown","publisher":null,"released":"2026-10-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:master72o/documentvision-multimodal-benchmark","source_url":"https://github.com/master72o/documentvision-multimodal-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-master72o-documentvision-multimodal-benchmark","source":"claire_radar","source_url":"https://github.com/master72o/documentvision-multimodal-benchmark","topics":["multimodal","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_1df91e499cc9d53e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","image_to_text","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A dataset for Visual Question Answering on document images containing 50,000 questions defined on 12,000+ document images. The benchmark tests AI's ability to understand document structure and content, requiring models to comprehend document layout and perform information retrieval to answer questions about document images.","descriptions":{"en":"A dataset for Visual Question Answering on document images containing 50,000 questions defined on 12,000+ document images. The benchmark tests AI's ability to understand document structure and content, requiring models to comprehend document layout and perform information retrieval to answer questions about document images."},"evidence_summary":{"document_count":1,"model_count":28,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:docvqa:grok-1.5","reported_at":"2024-03-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:docvqa","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_1df91e499cc9d53e"},"modality":"multimodal","name":"DocVQA","openness":"unknown","publication":{"evidence":[{"note":"The project identifies this as its first dataset and the 2020 challenge task 1, and names the Mathew, Karatzas and Jawahar paper. Its task and corpus description match the source record.","url":"https://site.docvqa.org/datasets/docvqa"}],"id":"docvqa-original","links":[{"kind":"paper","title":"DocVQA: A Dataset for VQA on Document Images","url":"https://arxiv.org/abs/2007.00398"},{"kind":"website","title":"DocVQA original dataset page","url":"https://site.docvqa.org/datasets/docvqa"}],"record_keys":["llm-stats:docvqa"],"reviewed_at":"2026-10-26","scope":{"en":"The original single-document DocVQA dataset, introduced for the 2020 challenge; collection and multi-page variants are separate tasks.","zh":"2020 年挑战赛引入的原始单文档 DocVQA 数据集；文档集问答及多页变体属于其他任务。"}},"publisher":null,"released":"2020-07-01","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: DocVQA: A Dataset for VQA on Document Images. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:docvqa","source_url":"https://arxiv.org/abs/2007.00398"},"repo_kind":"not_found","repo_resolution_status":"not_found","score_count":28,"score_direction":"higher_is_better","score_summary":{"display_max":96.39999999999999,"display_multiplier":100,"model_count":28,"model_count_basis":"source_model_id","numeric_count":28,"raw_max":0.964,"raw_min":0.758,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:docvqa:qwen2.5-vl-72b","reported_date":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-03-28","date_precision":"model_announcement","name":"Grok-1.5"},{"date":"2024-04-12","date_precision":"model_announcement","name":"Grok-1.5V"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2 mini"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-docvqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqa?top_n=500","topics":["multimodal"],"unit":null},{"aliases":["DocVQA"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Saturated at the frontier; ANLS scoring is lenient to OCR near-misses.","descriptions":{"en":"Saturated at the frontier; ANLS scoring is lenient to OCR near-misses."},"evidence_summary":{"document_count":2,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:docvqa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["document-ocr"],"join":"review:14H034160212:2026-10-15","primary":"document-ocr","source_record":null},"modality":null,"name":"DocVQA","openness":"unknown","publisher":null,"released":"2020-07-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:docvqa","source_url":"https://www.docvqa.org/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"docvqa","source":"model_reports","source_url":"https://www.docvqa.org/","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_42d39fa20c2f1413"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DocVQA is a Visual Question Answering benchmark on document images containing 50,000 questions defined on 12,000+ document images. The benchmark focuses on understanding document structure and content to answer questions about various document types including letters, memos, notes, and reports from the UCSF Industry Documents Library.","descriptions":{"en":"DocVQA is a Visual Question Answering benchmark on document images containing 50,000 questions defined on 12,000+ document images. The benchmark focuses on understanding document structure and content to answer questions about various document types including letters, memos, notes, and reports from the UCSF Industry Documents Library."},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:docvqatest:qwen2-vl-72b","reported_at":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqatest?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:docvqatest","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_42d39fa20c2f1413"},"modality":"multimodal","name":"DocVQAtest","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":97.1,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.971,"raw_min":0.942,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:docvqatest:qwen3-vl-235b-a22b-instruct","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqatest?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-docvqatest","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqatest?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robustness","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A controlled evaluation of 13 attribution methods across CNN and Vision Transformer architectures using five quality and cost dimensions.","descriptions":{"en":"A controlled evaluation of 13 attribution methods across CNN and Vision Transformer architectures using five quality and cost dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.02396","languages":[],"library_direction":null,"modality":null,"name":"Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02396","source_url":"https://arxiv.org/abs/2608.02396"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02396","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02396","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_dogbench_143af879"},"categories":["Code & Software","Factuality","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"We introduce DoGBENCH (Documentation Generation Benchmark), to our knowledge, the first benchmark for generating and maintaining real user-facing software documentation.","descriptions":{"en":"We introduce DoGBENCH (Documentation Generation Benchmark), to our knowledge, the first benchmark for generating and maintaining real user-facing software documentation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.39909","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","professional-work"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_dogbench_143af879"},"modality":null,"name":"DoGBench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39909","source_url":"https://arxiv.org/abs/2609.39909"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39909","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39909","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates agent memory through task completion.","descriptions":{"en":"Evaluates agent memory through task completion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24971","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_dolphinbench_8f9fb14c"},"modality":null,"name":"DolphinBench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24971","source_url":"https://arxiv.org/abs/2609.24971"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24971","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24971","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"DOSEBENCH evaluates LLM decision-making on over-the-counter dosing questions, with 81 curated scenarios for adult acetaminophen and ibuprofen use. Correct answers require tracking dose timing, computing rolling 24-hour intake, and following product-label constraints.","descriptions":{"en":"DOSEBENCH evaluates LLM decision-making on over-the-counter dosing questions, with 81 curated scenarios for adult acetaminophen and ibuprofen use. Correct answers require tracking dose timing, computing rolling 24-hour intake, and following product-label constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04262","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_dosebench_cf8cdec2"},"modality":null,"name":"DOSEBENCH","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04262","source_url":"https://arxiv.org/abs/2606.04262"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04262","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04262","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["instruction_following","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Doubao Multi-Turn Bench evaluates models on multi-turn conversational tasks, measuring context retention, instruction following, and coherent reasoning across extended dialogues.","descriptions":{"en":"Doubao Multi-Turn Bench evaluates models on multi-turn conversational tasks, measuring context retention, instruction following, and coherent reasoning across extended dialogues."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:doubao-multi-turn-bench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/doubao-multi-turn-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:doubao-multi-turn-bench","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","chat-writing"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_755debb970fbb325"},"modality":"text","name":"Doubao Multi-Turn Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":52.5,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.525,"raw_min":0.49,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:doubao-multi-turn-bench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/doubao-multi-turn-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-doubao-multi-turn-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/doubao-multi-turn-bench?top_n=500","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","LLM敏感性评估","大语言模型","LLM","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DOVE (Dataset Of Variation Evaluation) is a large-scale dataset containing prompt perturbations of various evaluation benchmarks. DOVE（变异评估数据集），这是一个大规模数据集，包含了各种评估基准的提示扰动。","descriptions":{"en":"DOVE (Dataset Of Variation Evaluation) is a large-scale dataset containing prompt perturbations of various evaluation benchmarks.","zh":"DOVE（变异评估数据集），这是一个大规模数据集，包含了各种评估基准的提示扰动。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1731","languages":[],"library_direction":null,"modality":null,"name":"DOVE","openness":"unknown","publisher":"The Hebrew University of Jerusalem, IBM Research AI, MIT，etc.","released":"2025-03-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1731-dove","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DOVE","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_dr-cik_017f835a"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Dr-CiK is a benchmark for evaluating whether agents can retrieve forecasting-relevant supporting context from a document corpus, filter distractors, distill evidence, and generate forecasts. It provides context ablations and evaluates deep research and forecasting methods.","descriptions":{"en":"Dr-CiK is a benchmark for evaluating whether agents can retrieve forecasting-relevant supporting context from a document corpus, filter distractors, distill evidence, and generate forecasts. It provides context ablations and evaluates deep research and forecasting methods."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27904","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_dr-cik_017f835a"},"modality":null,"name":"Dr-CiK","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27904","source_url":"https://arxiv.org/abs/2605.27904"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27904","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27904","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_c8c3d2e4ed91e9d5"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DRACO is a deep research benchmark that evaluates an agent's ability to gather, synthesize, and reason over information to answer complex research questions. Scores are based on official rubrics per question, with the final score being the average across all questions.","descriptions":{"en":"DRACO is a deep research benchmark that evaluates an agent's ability to gather, synthesize, and reason over information to answer complex research questions. Scores are based on official rubrics per question, with the final score being the average across all questions."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:draco:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/draco?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:draco","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_c8c3d2e4ed91e9d5"},"modality":"text","name":"DRACO","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":73.22999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.7323,"raw_min":0.7323,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:draco:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/draco?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-draco","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/draco?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["long_context"],"collected_at":null,"description":"Deep-research style long-context agent tasks; the retrieval and tool stack is part of the score.","descriptions":{"en":"Deep-research style long-context agent tasks; the retrieval and tool stack is part of the score."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000draco\u0000tencent_hy4_preview\u0000draco\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000draco\u0000tencent_hy4_preview\u0000draco\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:draco","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["deep-research"],"join":"review:14H034160212:2026-10-15","primary":"deep-research","source_record":null},"modality":null,"name":"DRACO","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":77.2,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":77.2,"raw_min":77.2,"source_reference":{"obs_id":"curated\u0000draco\u0000tencent_hy4_preview\u0000draco\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000draco\u0000tencent_hy4_preview\u0000draco\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"draco","source":"model_reports","source_url":"https://huggingface.co/datasets/perplexity-ai/draco","topics":["long-context"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DragOn is a drag grounding benchmark and training dataset for GUI agents across four interaction types: text highlighting, cell selection, element resizing, and slider manipulation, with 3.5M training tasks and 2,000 evaluation examples.","descriptions":{"en":"DragOn is a drag grounding benchmark and training dataset for GUI agents across four interaction types: text highlighting, cell selection, element resizing, and slider manipulation, with 3.5M training tasks and 2,000 evaluation examples."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06322","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_dragon_22a47a0d"},"modality":null,"name":"DragOn","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06322","source_url":"https://arxiv.org/abs/2606.06322"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06322","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06322","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","RAG","大语言模型","LLM","知识储备","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DRAGON 是一个用于评估检索增强生成（RAG）系统在俄语新闻语境中事实性与检索能力的动态基准，支持对检索器与生成器组件的全面评估。 DRAGON is a dynamic benchmark for evaluating Retrieval-Augmented Generation (RAG) systems in Russian news contexts, supporting comprehensive assessment of both retriever and generator components.","descriptions":{"en":"DRAGON 是一个用于评估检索增强生成（RAG）系统在俄语新闻语境中事实性与检索能力的动态基准，支持对检索器与生成器组件的全面评估。","zh":"DRAGON is a dynamic benchmark for evaluating Retrieval-Augmented Generation (RAG) systems in Russian news contexts, supporting comprehensive assessment of both retriever and generator components."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2045","languages":[],"library_direction":null,"modality":null,"name":"DRAGON","openness":"unknown","publisher":"SberAI , ITMO , MISIS , HSE , MWSAI","released":"2025-07-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2045-dragon","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DRAGON","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates every stage of short-drama production, from script to finished drama, across five axes and 63 leaf dimensions using professional annotations.","descriptions":{"en":"Evaluates every stage of short-drama production, from script to finished drama, across five axes and 63 leaf dimensions using professional annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.00646","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_dramachain-bench_c37136e7"},"modality":null,"name":"DramaChain Bench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00646","source_url":"https://arxiv.org/abs/2609.00646"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00646","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00646","topics":[],"unit":null},{"aliases":["DrawAI: Agentic Benchmark and Workflow for Making Raster Images Editable"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"DrawAI-Bench evaluates systems that reconstruct editable, structured artifacts from raster images across figures, slides, posters, and diagrams.","descriptions":{"en":"DrawAI-Bench evaluates systems that reconstruct editable, structured artifacts from raster images across figures, slides, posters, and diagrams."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00548","languages":[],"library_direction":null,"modality":null,"name":"DrawAI-Bench","openness":"unknown","publisher":null,"released":"2026-08-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00548","source_url":"https://arxiv.org/abs/2608.00548"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00548","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00548","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object is unclear from the provided information.","descriptions":{"en":"The evaluation object is unclear from the provided information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.15418","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_drawingvqa_70f98873"},"modality":null,"name":"DrawingVQA","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15418","source_url":"https://arxiv.org/abs/2607.15418"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15418","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15418","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_drflow_2f7bee63"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DRFLOW evaluates an agent's ability to predict personalized workflows, sequences of action-steps, from heterogeneous sources across five domains. It includes 100 tasks with reference workflow steps and multiple diagnostic metrics.","descriptions":{"en":"DRFLOW evaluates an agent's ability to predict personalized workflows, sequences of action-steps, from heterogeneous sources across five domains. It includes 100 tasks with reference workflow steps and multiple diagnostic metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18191","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_drflow_2f7bee63"},"modality":null,"name":"DRFLOW","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18191","source_url":"https://arxiv.org/abs/2606.18191"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18191","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18191","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"DRGBT-1K is a large-scale benchmark for dynamic RGBT tracking with 1,045 real-world sequences, 795K frame pairs, dense annotations, and a unified evaluation protocol across 20 trackers, plus an online leaderboard.","descriptions":{"en":"DRGBT-1K is a large-scale benchmark for dynamic RGBT tracking with 1,045 real-world sequences, 795K frame pairs, dense annotations, and a unified evaluation protocol across 20 trackers, plus an online leaderboard."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19772","languages":[],"library_direction":null,"modality":null,"name":"DRGBT-1K","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19772","source_url":"https://arxiv.org/abs/2607.19772"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19772","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19772","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"Modern LLM agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing benchmarks generally assume that users always accurately and sufficiently communicate a fixed intent.","descriptions":{"en":"Modern LLM agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing benchmarks generally assume that users always accurately and sufficiently communicate a fixed intent."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38604","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_drift-bench_7ce665c0"},"modality":null,"name":"Drift-Bench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38604","source_url":"https://arxiv.org/abs/2609.38604"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38604","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38604","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yihaoding/drillbench","source_record":"bm_drillbench_73c407fb"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Autoregressive modelling has achieved remarkable success in language and sequence tasks by learning to predict future states from previous observation.","descriptions":{"en":"Autoregressive modelling has achieved remarkable success in language and sequence tasks by learning to predict future states from previous observation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.01204","languages":[],"library_direction":{"basis":"model-review","categories":["world-models"],"join":"claire_record_id","primary":"world-models","source_record":"bm_drillbench_73c407fb"},"modality":null,"name":"DrillBench","openness":"unknown","publisher":null,"released":"2026-10-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.01204","source_url":"https://arxiv.org/abs/2610.01204"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-01204","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01204","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation targets conversational implicature in question utterances, using a semi-automated pipeline to generate question-context-interpretation instances with controlled variation.","descriptions":{"en":"The evaluation targets conversational implicature in question utterances, using a semi-automated pipeline to generate question-context-interpretation instances with controlled variation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24267","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_drinq_46ba6b80"},"modality":null,"name":"DRInQ","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24267","source_url":"https://arxiv.org/abs/2605.24267"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24267","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24267","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":87,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_drivemotion_aff5c9d5"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"DriveMotion provides 393 hours of driver motion sequences with synchronized driving context for continuous driver motion forecasting.","descriptions":{"en":"DriveMotion provides 393 hours of driver motion sequences with synchronized driving context for continuous driver motion forecasting."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08117","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_drivemotion_aff5c9d5"},"modality":null,"name":"DriveMotion","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08117","source_url":"https://arxiv.org/abs/2609.08117"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08117","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08117","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Autonomous Driving","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"DriveSpatial is a benchmark of 15.6K human-verified QA pairs across 20 tasks from five AD datasets, evaluating VLMs on cognitive scene construction, multi-view relational understanding, temporal reasoning, and generalization.","descriptions":{"en":"DriveSpatial is a benchmark of 15.6K human-verified QA pairs across 20 tasks from five AD datasets, evaluating VLMs on cognitive scene construction, multi-view relational understanding, temporal reasoning, and generalization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.23176","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","spatial-3d"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_drivespatial_f218db55"},"modality":null,"name":"DRIVESPATIAL","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23176","source_url":"https://arxiv.org/abs/2605.23176"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23176","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23176","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested.","descriptions":{"en":"Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38948","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","visual-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_drivingbench_2def078a"},"modality":null,"name":"DrivingBench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38948","source_url":"https://arxiv.org/abs/2609.38948"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38948","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38948","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_d90ee9ccf6bea1d2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.","descriptions":{"en":"DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts."},"evidence_summary":{"document_count":1,"model_count":30,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:drop:gpt-3.5-turbo-0125","reported_at":"2023-03-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/drop?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:drop","languages":[],"library_direction":{"basis":"model-review-2","categories":["abstract-reasoning","math"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_d90ee9ccf6bea1d2"},"modality":"text","name":"DROP","openness":"unknown","publisher":null,"released":"2019-03-01","released_reference":{"basis":"paper_first_version","note":"First version introducing the DROP reading-comprehension benchmark.","source_key":"opencompass:536","source_url":"https://arxiv.org/abs/1903.00161"},"repo_kind":null,"repo_resolution_status":null,"score_count":30,"score_direction":"higher_is_better","score_summary":{"display_max":91.60000000000001,"display_multiplier":100,"model_count":30,"model_count_basis":"source_model_id","numeric_count":30,"raw_max":0.916,"raw_min":0.286,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:drop:deepseek-v3","reported_date":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/drop?top_n=500"},"unit":null},"scored_models":[{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2023-03-21","date_precision":"model_announcement","name":"GPT-3.5 Turbo"},{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"},{"date":"2024-04-09","date_precision":"model_announcement","name":"GPT-4 Turbo"},{"date":"2024-05-13","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"}],"slug":"llm-stats-drop","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/drop?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["DROP"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Legacy reading-comprehension baseline.","descriptions":{"en":"Legacy reading-comprehension baseline."},"evidence_summary":{"document_count":6,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000drop\u0000google_gemini_1_5_report\u0000drop\u0000variable shots, F1\u0000Gemini 1.5 Pro","reported_at":"2024-03-08","source_url":"https://arxiv.org/abs/2403.05530"},"first_score_reported_at":"2024-03-08","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000drop\u0000google_gemini_1_5_report\u0000drop\u0000variable shots, F1\u0000Gemini 1.5 Pro","reported_at":"2024-03-08","source_url":"https://arxiv.org/abs/2403.05530"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:drop","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["knowledge-expert-qa"],"join":"review:14H034160212:2026-10-15","primary":"knowledge-expert-qa","source_record":null},"modality":null,"name":"DROP","openness":"unknown","publisher":null,"released":"2019-03-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:drop","source_url":"https://allenai.org/data/drop"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":92.2,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":92.2,"raw_min":74.9,"source_reference":{"obs_id":"curated\u0000drop\u0000deepseek_r1_report\u0000drop\u00003-shot F1\u0000DeepSeek-R1","observation_id":"curated\u0000drop\u0000deepseek_r1_report\u0000drop\u00003-shot F1\u0000DeepSeek-R1","reported_at":"2025-01-22","reported_date":"2025-01-22","source_id":"deepseek_r1_report","source_url":"https://arxiv.org/abs/2501.12948"},"unit":"percent"},"scored_models":[{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2024-03-08","date_precision":"document_publication","name":"Gemini 1.5 Pro"}],"slug":"drop","source":"model_reports","source_url":"https://allenai.org/data/drop","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:1903.00161","repository":null,"source_record":"catalog_d90ee9ccf6bea1d2"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DROP is a QA dataset which tests comprehensive understanding of paragraphs. In this crowdsourced, adversarially-created, 96k question-answering benchmark, a system must resolve multiple references in a question, map them onto a paragraph, and perform discrete operations over them (such as addition, counting, or sorting). DROP 是一个测试段落综合理解能力的 QA 数据集。在这个众包、对抗性创建的 96K 问题解答基准中，系统必须解析问题中的多个引用，将它们映射到段落中，并对它们执行离散操作（如加法、计数或排序）。","descriptions":{"en":"DROP is a QA dataset which tests comprehensive understanding of paragraphs. In this crowdsourced, adversarially-created, 96k question-answering benchmark, a system must resolve multiple references in a question, map them onto a paragraph, and perform discrete operations over them (such as addition, counting, or sorting).","zh":"DROP 是一个测试段落综合理解能力的 QA 数据集。在这个众包、对抗性创建的 96K 问题解答基准中，系统必须解析问题中的多个引用，将它们映射到段落中，并对它们执行离散操作（如加法、计数或排序）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:536","languages":[],"library_direction":{"basis":"model-review-2","categories":["abstract-reasoning","math"],"join":"artifact:arxiv:1903.00161","primary":"abstract-reasoning","source_record":"catalog_d90ee9ccf6bea1d2"},"modality":null,"name":"DROP","openness":"unknown","publisher":null,"released":"2019-03-01","released_reference":{"basis":"paper_first_version","note":"First version introducing the DROP reading-comprehension benchmark.","source_key":"opencompass:536","source_url":"https://arxiv.org/abs/1903.00161"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-536-drop","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DROP","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DrugBench evaluates AI control protocols for mitigating medication-related harm using 3,671 medical conversations and FDA drug labels, covering drug interactions, contraindications, dosing constraints, and patient action restrictions. Introduces severity-based monitoring.","descriptions":{"en":"DrugBench evaluates AI control protocols for mitigating medication-related harm using 3,671 medical conversations and FDA drug labels, covering drug interactions, contraindications, dosing constraints, and patient action restrictions. Introduces severity-based monitoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.20663","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","safety-alignment"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_drugbench_dc75a792"},"modality":null,"name":"DrugBench","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20663","source_url":"https://arxiv.org/abs/2606.20663"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20663","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20663","topics":["safety","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:drylab-bio/drylab-bench","source_record":"bm_drylab-bench_c3c6c8a0"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"As large language models (LLMs) acquire advanced biological design capabilities, they offer substantial research benefits but simultaneously introduce severe dual-use risks, such as engineering enhanced pathogens or drug-resistant bacteria.","descriptions":{"en":"As large language models (LLMs) acquire advanced biological design capabilities, they offer substantial research benefits but simultaneously introduce severe dual-use risks, such as engineering enhanced pathogens or drug-resistant bacteria."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.13116","languages":[],"library_direction":{"basis":"model-review-daily","categories":["safety-alignment","science"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_drylab-bench_c3c6c8a0"},"modality":null,"name":"DRYLAB-Bench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13116","source_url":"https://arxiv.org/abs/2610.13116"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13116","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13116","topics":["agents","safety","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2211.11501","repository":null,"source_record":"catalog_c2f4e4d2c39668a6"},"categories":["代码","code","大语言模型","LLM","代码工程","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DS-1000 is a code generation benchmark with a thousand data science questions spanning seven Python libraries that (1) reflects diverse, realistic, and practical use cases, (2) has a reliable metric, (3) defends against memorization by perturbing questions. DS-1000 是一个代码生成基准测试，包含一千个数据科学问题，涵盖七个Python库，其特点是（1）反映多样化、现实且实用的用例，（2）具有可靠的度量标准，（3）通过扰乱问题来防止记忆化。","descriptions":{"en":"DS-1000 is a code generation benchmark with a thousand data science questions spanning seven Python libraries that (1) reflects diverse, realistic, and practical use cases, (2) has a reliable metric, (3) defends against memorization by perturbing questions.","zh":"DS-1000 是一个代码生成基准测试，包含一千个数据科学问题，涵盖七个Python库，其特点是（1）反映多样化、现实且实用的用例，（2）具有可靠的度量标准，（3）通过扰乱问题来防止记忆化。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:544","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","professional-work"],"join":"artifact:arxiv:2211.11501","primary":"code-generation","source_record":"catalog_c2f4e4d2c39668a6"},"modality":null,"name":"DS-1000","openness":"unknown","publisher":null,"released":"2022-11-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-544-ds-1000","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DS-1000","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Data Science Arena Code benchmark for evaluating LLMs on realistic data science code generation tasks. Tests capabilities in complex data processing, analysis, and programming across popular Python libraries used in data science workflows.","descriptions":{"en":"Data Science Arena Code benchmark for evaluating LLMs on realistic data science code generation tasks. Tests capabilities in complex data processing, analysis, and programming across popular Python libraries used in data science workflows."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ds-arena-code:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-arena-code?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ds-arena-code","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","chat-writing"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_bb19a3a75070caab"},"modality":"text","name":"DS-Arena-Code","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":63.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.631,"raw_min":0.631,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ds-arena-code:deepseek-v2.5","reported_date":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-arena-code?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"}],"slug":"llm-stats-ds-arena-code","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-arena-code?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DeepSeek's internal Fill-in-the-Middle evaluation dataset for measuring code completion performance improvements in data science contexts","descriptions":{"en":"DeepSeek's internal Fill-in-the-Middle evaluation dataset for measuring code completion performance improvements in data science contexts"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ds-fim-eval:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-fim-eval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ds-fim-eval","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_7296791033c44d09"},"modality":"text","name":"DS-FIM-Eval","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":78.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.783,"raw_min":0.783,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ds-fim-eval:deepseek-v2.5","reported_date":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-fim-eval?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"}],"slug":"llm-stats-ds-fim-eval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-fim-eval?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:vis-nlp/dsagentbench","source_record":"bm_dsagentbench_96674f61"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DSAgentBench evaluates language agents on end-to-end data-science workflows in real computer environments. It comprises 275 tasks spanning data wrangling, exploration, modeling, visualization, and validation, with deterministic evaluators verifying analytical correctness, visual outputs, and model performance.","descriptions":{"en":"DSAgentBench evaluates language agents on end-to-end data-science workflows in real computer environments. It comprises 275 tasks spanning data wrangling, exploration, modeling, visualization, and validation, with deterministic evaluators verifying analytical correctness, visual outputs, and model performance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.10366","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_dsagentbench_96674f61"},"modality":null,"name":"DSAgentBench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10366","source_url":"https://arxiv.org/abs/2608.10366"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10366","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10366","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DSBench-FullStack is DeepSeek's internal full-stack development test set for evaluating coding agents on end-to-end software engineering tasks.","descriptions":{"en":"DSBench-FullStack is DeepSeek's internal full-stack development test set for evaluating coding agents on end-to-end software engineering tasks."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:dsbench-fullstack:deepseek-v4-flash-0731","reported_at":"2026-07-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-fullstack?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:dsbench-fullstack","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_ef6f49085728d89e"},"modality":"text","name":"DSBench-FullStack","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":71.1,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.711,"raw_min":0.687,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:dsbench-fullstack:deepseek-v4-pro-0813","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-fullstack?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0731"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"}],"slug":"llm-stats-dsbench-fullstack","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-fullstack?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DSBench-Hard is DeepSeek's internal test set of difficult coding-agent problems.","descriptions":{"en":"DSBench-Hard is DeepSeek's internal test set of difficult coding-agent problems."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:dsbench-hard:deepseek-v4-flash-0731","reported_at":"2026-07-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-hard?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:dsbench-hard","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_a72275bd6c7025a9"},"modality":"text","name":"DSBench-Hard","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":67.2,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.672,"raw_min":0.596,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:dsbench-hard:deepseek-v4-pro-0813","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-hard?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0731"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"}],"slug":"llm-stats-dsbench-hard","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-hard?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates execution efficiency of LLM-generated data science code across 1,000 tasks using correctness and efficiency scores from stress-testing harnesses.","descriptions":{"en":"Evaluates execution efficiency of LLM-generated data science code across 1,000 tasks using correctness and efficiency scores from stress-testing harnesses."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30248","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-skills-evolution","science"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_dseffi-bench_7ed7558d"},"modality":null,"name":"DSEffi-Bench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30248","source_url":"https://arxiv.org/abs/2608.30248"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30248","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30248","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Conversational MLLM agents are increasingly expected to assist in professional workflows, from AI research and engineering design to product management and business operations.","descriptions":{"en":"Conversational MLLM agents are increasingly expected to assist in professional workflows, from AI research and engineering design to product management and business operations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.08102","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_dsv-mem_a76aa04f"},"modality":null,"name":"DSV-Mem","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.08102","source_url":"https://arxiv.org/abs/2610.08102"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-08102","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08102","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Urban diagnosis integrates heterogeneous observations to identify urban problems, localize affected areas, and investigate contributing factors, informing evidence-based urban planning and management.","descriptions":{"en":"Urban diagnosis integrates heterogeneous observations to identify urban problems, localize affected areas, and investigate contributing factors, informing evidence-based urban planning and management."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.09374","languages":[],"library_direction":{"basis":"model-review-daily","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_duda-bench_6ceccd78"},"modality":null,"name":"DUDA-Bench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.09374","source_url":"https://arxiv.org/abs/2610.09374"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-09374","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09374","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_60e77101e76950a9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"DUDE (Document Understanding Dataset and Evaluation) tests multi-page, multi-domain document understanding and reasoning.","descriptions":{"en":"DUDE (Document Understanding Dataset and Evaluation) tests multi-page, multi-domain document understanding and reasoning."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:dude:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dude?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:dude","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","long-context"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_60e77101e76950a9"},"modality":"multimodal","name":"DUDE","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":83.1,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.831,"raw_min":0.828,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:dude:seed-2.1-turbo","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dude?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-dude","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dude?top_n=500","topics":["multimodal","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agent security under dual-control interaction where both agent and user influence shared environment state.","descriptions":{"en":"Evaluates LLM agent security under dual-control interaction where both agent and user influence shared environment state."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24662","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_duma-bench_3a5b0f14"},"modality":null,"name":"DUMA-Bench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24662","source_url":"https://arxiv.org/abs/2609.24662"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24662","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24662","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_dumatebench_7a085138"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Autonomous agents are increasingly adopted to complete complex, multi-tool workflows in real-world settings.","descriptions":{"en":"Autonomous agents are increasingly adopted to complete complex, multi-tool workflows in real-world settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.26546","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_dumatebench_7a085138"},"modality":null,"name":"DuMateBench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26546","source_url":"https://arxiv.org/abs/2608.26546"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26546","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26546","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"DungeonBench evaluates tactical reasoning in Dungeons & Dragons combat with two tracks: Encounter for single fights and Day for linked encounters with persistent resources, using a shared decision stream of complete tactical observations and legal options.","descriptions":{"en":"DungeonBench evaluates tactical reasoning in Dungeons & Dragons combat with two tracks: Encounter for single fights and Day for linked encounters with persistent resources, using a shared decision stream of complete tactical observations and legal options."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.29577","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_dungeonbench_a497b825"},"modality":null,"name":"DungeonBench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29577","source_url":"https://arxiv.org/abs/2607.29577"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29577","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29577","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DunphyBench evaluates long-horizon embodied decision-making in housing environments, requiring agents to navigate and choose options aligned with multi-dimensional human preferences under partial observations.","descriptions":{"en":"DunphyBench evaluates long-horizon embodied decision-making in housing environments, requiring agents to navigate and choose options aligned with multi-dimensional human preferences under partial observations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01456","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_dunphybench_f4ebbce5"},"modality":null,"name":"DunphyBench","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01456","source_url":"https://arxiv.org/abs/2608.01456"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01456","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01456","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":21,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:robotcontrolstack/duobench","source_record":"bm_duobench_6c56fbaf"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"DuoBench is a benchmarking framework for bimanual manipulation policies on the FR3 Duo platform, with eleven tasks across four coordination categories, simulated and partially real-world, plus human-teleoperated datasets. It proposes stage-based evaluation beyond binary success.","descriptions":{"en":"DuoBench is a benchmarking framework for bimanual manipulation policies on the FR3 Duo platform, with eleven tasks across four coordination categories, simulated and partially real-world, plus human-teleoperated datasets. It proposes stage-based evaluation beyond binary success."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.11901","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_duobench_6c56fbaf"},"modality":null,"name":"DuoBench","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11901","source_url":"https://arxiv.org/abs/2606.11901"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11901","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11901","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bomiaowang/dvbench","source_record":"bm_dvbench_8df5362a"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal large language models on data videos that combine dynamic charts and structured narratives.","descriptions":{"en":"Evaluates multimodal large language models on data videos that combine dynamic charts and structured narratives."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29711","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_dvbench_8df5362a"},"modality":null,"name":"DVBench","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29711","source_url":"https://arxiv.org/abs/2608.29711"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29711","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29711","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves.","descriptions":{"en":"Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.03020","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_dyadmem_f88ba107"},"modality":null,"name":"DyadMem","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03020","source_url":"https://arxiv.org/abs/2610.03020"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03020","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03020","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","推理","Reasoning","代码","Code","Dynamic Benchmarking","ICML 2025","大语言模型","LLM","逻辑推理","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DyCodeEval introduces methods to generate dynamic evaluation dataset and metric. Using multi-agent cooperation to rewrite benchmarks at evaluation time, it generates semantically equivalent, diverse, and non-deterministic problems—reducing data contamination and enabling more trustworthy evaluation. DyCodeEval 提出了一种动态生成评测数据集和评测指标的方法。通过多智能体协作，在评测时对基准题目进行重写，生成语义等价、多样化且非确定性的问题，从而减少数据污染，实现更可信的评测。","descriptions":{"en":"DyCodeEval introduces methods to generate dynamic evaluation dataset and metric. Using multi-agent cooperation to rewrite benchmarks at evaluation time, it generates semantically equivalent, diverse, and non-deterministic problems—reducing data contamination and enabling more trustworthy evaluation.","zh":"DyCodeEval 提出了一种动态生成评测数据集和评测指标的方法。通过多智能体协作，在评测时对基准题目进行重写，生成语义等价、多样化且非确定性的问题，从而减少数据污染，实现更可信的评测。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1971","languages":[],"library_direction":null,"modality":null,"name":"DyCodeEval","openness":"unknown","publisher":"Columbia University","released":"2025-06-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1971-dycodeeval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DyCodeEval","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Geometric reasoning","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language models on ego-motion understanding using counterfactual 3D rendering to separate visual appearance from kinematic properties.","descriptions":{"en":"Evaluates vision-language models on ego-motion understanding using counterfactual 3D rendering to separate visual appearance from kinematic properties."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01059","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","video-understanding"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_dyn-3d_1d5a4006"},"modality":null,"name":"Dyn-3D","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01059","source_url":"https://arxiv.org/abs/2609.01059"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01059","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01059","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for robot manipulation in dynamic environments, including dynamic manipulation and bimanual coordination tasks.","descriptions":{"en":"A benchmark for robot manipulation in dynamic environments, including dynamic manipulation and bimanual coordination tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.22119","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_dynabench_70c2d7da"},"modality":null,"name":"DynaBench","openness":"unknown","publisher":null,"released":"2026-07-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22119","source_url":"https://arxiv.org/abs/2607.22119"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22119","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22119","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":29,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"llm_stats_id","repository":null,"source_record":"catalog_ab9fe765bfe10432"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multimodal mathematics and reasoning benchmark focused on dynamic visual problem solving.","descriptions":{"en":"A multimodal mathematics and reasoning benchmark focused on dynamic visual problem solving."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:dynamath:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dynamath?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:dynamath","languages":[],"library_direction":{"basis":"model-review","categories":["math","visual-reasoning"],"join":"llm_stats_id","primary":"math","source_record":"catalog_ab9fe765bfe10432"},"modality":"multimodal","name":"DynaMath","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":88.0,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.88,"raw_min":0.681,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:dynamath:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dynamath?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-dynamath","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dynamath?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":29,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"artifact:arxiv:2411.00836","repository":null,"source_record":"catalog_ab9fe765bfe10432"},"categories":["多模态","Multimodal","推理","Reasoning","数学","Math","多模态模型","VLM","逻辑推理","数理能力","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"DynaMath is a dynamic visual math benchmark designed for in-depth assessment of VLMs. It includes 501 high-quality, multi-topic seed questions, each represented as a Python program enabling the automatic generation of a much larger set of concrete questions. DynaMath用于评估多模态大模型的数学能力，包括501个高质量、多主题的种子问题，每个问题都以Python程序表示，能够自动生成大量具体的多样化问题。","descriptions":{"en":"DynaMath is a dynamic visual math benchmark designed for in-depth assessment of VLMs. It includes 501 high-quality, multi-topic seed questions, each represented as a Python program enabling the automatic generation of a much larger set of concrete questions.","zh":"DynaMath用于评估多模态大模型的数学能力，包括501个高质量、多主题的种子问题，每个问题都以Python程序表示，能够自动生成大量具体的多样化问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1374","languages":[],"library_direction":{"basis":"model-review","categories":["math","visual-reasoning"],"join":"artifact:arxiv:2411.00836","primary":"math","source_record":"catalog_ab9fe765bfe10432"},"modality":"multimodal","name":"DynaMath","openness":"open","publisher":"University of Illinois at Urbana-Champaign","released":"2024-10-29","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1374-dynamath","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DynaMath","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":19,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:wenyaxie023/dynamicmem","source_record":"bm_dynamicmem_faf51af5"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"DynamicMem evaluates long-horizon memory in LLM agents through a synthetic benchmark with 15 months of multi-app activity per user across 16 applications, including attributes, habits, and preferences that evolve and must be inferred from scattered evidence. Scoring occurs at quarterly checkpoints, tracking performance as history grows.","descriptions":{"en":"DynamicMem evaluates long-horizon memory in LLM agents through a synthetic benchmark with 15 months of multi-app activity per user across 16 applications, including attributes, habits, and preferences that evolve and must be inferred from scattered evidence. Scoring occurs at quarterly checkpoints, tracking performance as history grows."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22877","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_dynamicmem_faf51af5"},"modality":null,"name":"DynamicMem","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22877","source_url":"https://arxiv.org/abs/2606.22877"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22877","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22877","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"While Vision-Language-Action (VLA) models excel in static tasks, they struggle in dynamic environments where objects are in motion (e.g., conveyor belt manipulation).","descriptions":{"en":"While Vision-Language-Action (VLA) models excel in static tasks, they struggle in dynamic environments where objects are in motion (e.g., conveyor belt manipulation)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.39198","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_dynbench_a85a28b7"},"modality":null,"name":"DynBench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39198","source_url":"https://arxiv.org/abs/2609.39198"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39198","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39198","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Dynamic graph anomaly detection requires repeated decisions as graph structure and class prevalence drift, yet detector benchmarks usually score a fixed pipeline after current labels are known.","descriptions":{"en":"Dynamic graph anomaly detection requires repeated decisions as graph structure and class prevalence drift, yet detector benchmarks usually score a fixed pipeline after current labels are known."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33980","languages":[],"library_direction":null,"modality":null,"name":"DynGraphAgentBench","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33980","source_url":"https://arxiv.org/abs/2609.33980"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33980","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33980","topics":[],"unit":null},{"aliases":["DynoFluxBench: Benchmarking Kinodynamic Space-Time Planners in Dynamic Environments"],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark framework for comparing kinodynamic planners in known dynamic environments with unbounded arrival time, facilitating systematic evaluation of planning algorithms.","descriptions":{"en":"A benchmark framework for comparing kinodynamic planners in known dynamic environments with unbounded arrival time, facilitating systematic evaluation of planning algorithms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.18549","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_dynofluxbench-benchmarking-kinodynamic-spa_a7d07b0f"},"modality":null,"name":"DynoFluxBench","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18549","source_url":"https://arxiv.org/abs/2609.18549"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18549","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18549","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_e-bench_c1793d08"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"E-Bench evaluates multi-step tool-use agents in synthetic state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. It requires agents to discover hidden information and compose multiple tool calls before changing state, with deterministic grading by database-state diffs.","descriptions":{"en":"E-Bench evaluates multi-step tool-use agents in synthetic state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. It requires agents to discover hidden information and compose multiple tool calls before changing state, with deterministic grading by database-state diffs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.23722","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_e-bench_c1793d08"},"modality":null,"name":"E-Bench","openness":"unknown","publisher":null,"released":"2026-07-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23722","source_url":"https://arxiv.org/abs/2607.23722"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23722","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23722","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":109,"github_stars_30d":23,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"claire_record_id","repository":"gh:qwenlm/e-commercebench","source_record":"bm_e-commerce-bench_400b2b29"},"categories":["Agents & Tool Use","Logistics & Operations","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agents over a deterministic 365-day simulated e-commerce operation, scoring asset growth, negotiation, fraud avoidance, efficiency, and learning.","descriptions":{"en":"Evaluates LLM agents over a deterministic 365-day simulated e-commerce operation, scoring asset growth, negotiation, fraud avoidance, efficiency, and learning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.30730","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","multi-agent-games"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_e-commerce-bench_400b2b29"},"modality":null,"name":"E-Commerce Bench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30730","source_url":"https://arxiv.org/abs/2608.30730"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30730","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30730","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","ACL 2024","大语言模型","LLM","知识储备","Knowledge"],"collected_at":"2026-08-18T00:00:00+00:00","description":"E-EVAL is the first comprehensive evaluation benchmark specifically tailored for Chinese K-12 education. E-EVAL comprises 4,351 multiple-choice questions spanning primary, middle, and high school levels, covering a diverse array of subjects. E-EVAL 是首个专门针对中国 K-12 教育的综合评估基准。E-EVAL 包含 4,351 道选择题，涵盖小学、初中和高中各个年级，涉及多种学科。","descriptions":{"en":"E-EVAL is the first comprehensive evaluation benchmark specifically tailored for Chinese K-12 education. E-EVAL comprises 4,351 multiple-choice questions spanning primary, middle, and high school levels, covering a diverse array of subjects.","zh":"E-EVAL 是首个专门针对中国 K-12 教育的综合评估基准。E-EVAL 包含 4,351 道选择题，涵盖小学、初中和高中各个年级，涉及多种学科。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1080","languages":["Chinese"],"library_direction":null,"modality":null,"name":"E-EVAL","openness":"open","publisher":"University of Science and Technology of China","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1080-e-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/E-EVAL","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:wanng-ide/e2a-bench","source_record":"bm_e2a-bench_904f36b9"},"categories":["Finance","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates financial vision-language models on evidence-to-action reliability across 969 queries using deterministic evidence anchors and metrics for grounding, consistency, calibration, and coverage.","descriptions":{"en":"Evaluates financial vision-language models on evidence-to-action reliability across 969 queries using deterministic evidence anchors and metrics for grounding, consistency, calibration, and coverage."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.14302","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","visual-reasoning"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_e2a-bench_904f36b9"},"modality":null,"name":"E2A-Bench","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.14302","source_url":"https://arxiv.org/abs/2609.14302"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-14302","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14302","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories.","descriptions":{"en":"Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38335","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_e2e-swe_542b27dd"},"modality":null,"name":"E2E-SWE","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38335","source_url":"https://arxiv.org/abs/2609.38335"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38335","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38335","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"An advanced evaluation benchmark for LLMs on AI compliance using a knowledge graph and legal experts.","descriptions":{"en":"An advanced evaluation benchmark for LLMs on AI compliance using a knowledge graph and legal experts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.26175","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal","knowledge-expert-qa"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_eadc_e4faf393"},"modality":null,"name":"EADC","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.26175","source_url":"https://arxiv.org/abs/2609.26175"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-26175","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26175","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_earlydx_668e5f94"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"EarlyDx is a benchmark for open-ended early diagnosis from emergency department encounters in MIMIC-IV, using admission-time records and LLM-auditor-supervised free-text labels.","descriptions":{"en":"EarlyDx is a benchmark for open-ended early diagnosis from emergency department encounters in MIMIC-IV, using admission-time records and LLM-auditor-supervised free-text labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.28788","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_earlydx_668e5f94"},"modality":null,"name":"EarlyDx","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28788","source_url":"https://arxiv.org/abs/2607.28788"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28788","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28788","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_earnings25_937fc49b"},"categories":["Finance","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Earnings25 is a finance-domain benchmark for evaluating automatic speech recognition (ASR) on English-language earnings calls. It includes testset-full (498 hours of S&P 500 earnings calls from Q4 2025) and testset-segmented (46 hours of 290 segments from 2025 U.S. earnings calls), with aligned transcripts and metadata like speaker roles and industry labels, enabling speaker- and industry-aware evaluation beyond WER.","descriptions":{"en":"Earnings25 is a finance-domain benchmark for evaluating automatic speech recognition (ASR) on English-language earnings calls. It includes testset-full (498 hours of S&P 500 earnings calls from Q4 2025) and testset-segmented (46 hours of 290 segments from 2025 U.S. earnings calls), with aligned transcripts and metadata like speaker roles and industry labels, enabling speaker- and industry-aware evaluation beyond WER."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.23813","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","finance-legal"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_earnings25_937fc49b"},"modality":null,"name":"Earnings25","openness":"unknown","publisher":null,"released":"2026-07-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23813","source_url":"https://arxiv.org/abs/2607.23813"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23813","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23813","topics":["finance"],"unit":null},{"aliases":["EarthShift: a benchmark for measuring robustness to real-world distribution shifts in Earth observation"],"attention_signals":null,"categories":["General AI","Robustness","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A public testbed that measures how Earth observation models perform in- versus out-of-distribution across shifts in source, time, geography, and sensor, covering 11 tasks and 5 shift types.","descriptions":{"en":"A public testbed that measures how Earth observation models perform in- versus out-of-distribution across shifts in source, time, geography, and sensor, covering 11 tasks and 5 shift types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29330","languages":[],"library_direction":null,"modality":null,"name":"EarthShift","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29330","source_url":"https://arxiv.org/abs/2605.29330"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29330","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29330","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_earthverse_1bd65be4"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates scientific agents on 405 reproducible tasks across 199 documented natural hazard events, scoring fine-grained answer units and task-specific rubrics.","descriptions":{"en":"Evaluates scientific agents on 405 reproducible tasks across 199 documented natural hazard events, scoring fine-grained answer units and task-specific rubrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23525","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_earthverse_1bd65be4"},"modality":null,"name":"EarthVerse","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23525","source_url":"http://arxiv.org/abs/2608.23525v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23525","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23525v1","topics":["science"],"unit":null},{"aliases":["Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":"gh:ooohs/easel","source_record":"bm_paint-what-you-see-benchmarking-dexterous-_f2ccb455"},"categories":["Agents","Agents & Tool Use","Multimodal","Software & AI Compute","Tool use"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates dexterous visual tool use through reference-guided painting, semantic annotation, handwriting, and path planning tasks.","descriptions":{"en":"Evaluates dexterous visual tool use through reference-guided painting, semantic annotation, handwriting, and path planning tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25417","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_paint-what-you-see-benchmarking-dexterous-_f2ccb455"},"modality":null,"name":"EASEL","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25417","source_url":"https://arxiv.org/abs/2608.25417"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25417","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25417","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":147,"github_stars_30d":11,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"claire_record_id","repository":"gh:internrobotics/ebench","source_record":"bm_ebench_35561bef"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"EBench is a simulation benchmark for diagnosing generalist mobile manipulation policies. It comprises 26 manipulation tasks annotated along five capability dimensions (scene, atomic skill, horizon, precision, mobility) and four generalization dimensions (object, background, instruction, mixed), with strict train/test splits and held-out online evaluation.","descriptions":{"en":"EBench is a simulation benchmark for diagnosing generalist mobile manipulation policies. It comprises 26 manipulation tasks annotated along five capability dimensions (scene, atomic skill, horizon, precision, mobility) and four generalization dimensions (object, background, instruction, mixed), with strict train/test splits and held-out online evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.18239","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_ebench_35561bef"},"modality":null,"name":"EBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18239","source_url":"https://arxiv.org/abs/2606.18239"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18239","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18239","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Factuality","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"EC-Reason-Bench is a training-free diagnostic protocol for LLM enzyme classification, isolating four reasoning levers: output structure, external knowledge, reasoning structure, and robustness, to measure why LLMs fail on EC number prediction.","descriptions":{"en":"EC-Reason-Bench is a training-free diagnostic protocol for LLM enzyme classification, isolating four reasoning levers: output structure, external knowledge, reasoning structure, and robustness, to measure why LLMs fail on EC number prediction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.26397","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_ec-reason-bench_6994eec4"},"modality":null,"name":"EC-Reason-Bench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26397","source_url":"https://arxiv.org/abs/2607.26397"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26397","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26397","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language model memory along efficiency, compression, and calibration axes beyond raw accuracy.","descriptions":{"en":"Evaluates vision-language model memory along efficiency, compression, and calibration axes beyond raw accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.00103","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["factuality","agent-memory"],"join":"claire_record_id","primary":"factuality","source_record":"bm_eccbench_68383aba"},"modality":null,"name":"ECCBench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00103","source_url":"https://arxiv.org/abs/2609.00103"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00103","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00103","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ECG-InterpBench evaluates interpretability of ECG foundation models using sparse autoencoders with matched capacity, measuring reconstruction fidelity, clinical feature accessibility, and cross-seed reproducibility across 450 cells.","descriptions":{"en":"ECG-InterpBench evaluates interpretability of ECG foundation models using sparse autoencoders with matched capacity, measuring reconstruction fidelity, clinical feature accessibility, and cross-seed reproducibility across 450 cells."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27404","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_ecg-interpbench_df0017be"},"modality":null,"name":"ECG-InterpBench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27404","source_url":"https://arxiv.org/abs/2607.27404"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27404","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27404","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Factuality","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates contextual ECG knowledge via 10,904 True/False questions from ECG references, scored by zero-shot and fine-tuned accuracy on a held-out test set.","descriptions":{"en":"Evaluates contextual ECG knowledge via 10,904 True/False questions from ECG references, scored by zero-shot and fine-tuned accuracy on a held-out test set."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30893","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_ecgquest_55522c43"},"modality":null,"name":"ECGQuest","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30893","source_url":"https://arxiv.org/abs/2608.30893"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30893","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30893","topics":[],"unit":null},{"aliases":["Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"Echoverse releases stateful application environments for evaluating computer-use agents against application-grounded task outcomes.","descriptions":{"en":"Echoverse releases stateful application environments for evaluating computer-use agents against application-grounded task outcomes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.28074","languages":[],"library_direction":null,"modality":null,"name":"Echoverse","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28074","source_url":"https://arxiv.org/abs/2607.28074"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28074","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28074","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":null,"source_record":"catalog_b17fe74e48c6cabd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multilingual closed-book question answering dataset that evaluates cross-lingual knowledge transfer in large language models across 12 languages, using knowledge-seeking questions based on Wikipedia articles that exist only in one language","descriptions":{"en":"A multilingual closed-book question answering dataset that evaluates cross-lingual knowledge transfer in large language models across 12 languages, using knowledge-seeking questions based on Wikipedia articles that exist only in one language"},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:eclektic:gemma-3-12b-it","reported_at":"2025-03-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/eclektic?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:eclektic","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","factuality"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_b17fe74e48c6cabd"},"modality":"text","name":"ECLeKTic","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":19.0,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.19,"raw_min":0.014,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:eclektic:gemma-3n-e4b-it","reported_date":"2025-06-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/eclektic?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"}],"slug":"llm-stats-eclektic","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/eclektic?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"EcoAgent-Bench evaluates LLM agents' economic decision-making under budget constraints across 304 tasks in five families, with priced actions and metrics for micro accuracy and economic consistency.","descriptions":{"en":"EcoAgent-Bench evaluates LLM agents' economic decision-making under budget constraints across 304 tasks in five families, with priced actions and metrics for micro accuracy and economic consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05519","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","tool-use"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_ecoagent-bench_99f855a3"},"modality":null,"name":"EcoAgent-Bench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05519","source_url":"https://arxiv.org/abs/2608.05519"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05519","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05519","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Advertising & Pricing","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark covering 11 commercial editing tasks with task-specific visual evaluation and CTR-based ranking.","descriptions":{"en":"A benchmark covering 11 commercial editing tasks with task-specific visual evaluation and CTR-based ranking."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.26375","languages":[],"library_direction":null,"modality":null,"name":"Ecom-Bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.26375","source_url":"https://arxiv.org/abs/2609.26375"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-26375","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26375","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_ecomagentbench_d7694649"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"EComAgentBench evaluates LLM-based shopping agents on 662 long-horizon product selection tasks built from real Amazon data. Each task requires uncovering hidden requirements spread across an explicit query, tool-gated profile, and scripted clarification, verifying candidates, and committing to one product within 100 tool calls. Scoring uses typed, source-tagged rubrics for each requirement.","descriptions":{"en":"EComAgentBench evaluates LLM-based shopping agents on 662 long-horizon product selection tasks built from real Amazon data. Each task requires uncovering hidden requirements spread across an explicit query, tool-gated profile, and scripted clarification, verifying candidates, and committing to one product within 100 tool calls. Scoring uses typed, source-tagged rubrics for each requirement."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17698","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","professional-work"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_ecomagentbench_d7694649"},"modality":null,"name":"EComAgentBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17698","source_url":"https://arxiv.org/abs/2606.17698"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17698","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17698","topics":["agents"],"unit":null},{"aliases":["ESAC"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ecowestern/esac","source_record":"bm_esac_a48c7a20"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Compact benchmark program with ESAC-GI released (42 items, 75 points) measuring general intelligence across logic, math, factual recall, reading comprehension, abstraction, instruction-following, writing quality, and response-depth calibration; ESAC-AG partially implemented.","descriptions":{"en":"Compact benchmark program with ESAC-GI released (42 items, 75 points) measuring general intelligence across logic, math, factual recall, reading comprehension, abstraction, instruction-following, writing quality, and response-depth calibration; ESAC-AG partially implemented."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:ecowestern/esac","languages":[],"library_direction":{"basis":"model-review","categories":["composite-index"],"join":"claire_record_id","primary":"composite-index","source_record":"bm_esac_a48c7a20"},"modality":null,"name":"EcoWestern Short and Cheap Benchmark","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:ecowestern/esac","source_url":"https://github.com/EcoWestern/ESAC"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-ecowestern-esac","source":"claire_radar","source_url":"https://github.com/EcoWestern/ESAC","topics":[],"unit":null},{"aliases":["How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards"],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Predicting non-pecuniary damage awards at the European Court of Human Rights from case information.","descriptions":{"en":"Predicting non-pecuniary damage awards at the European Court of Human Rights from case information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.18908","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_how-much-is-a-human-right-worth-ecthr-npd-_d1b9f09a"},"modality":null,"name":"ECtHR-NPD","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18908","source_url":"https://arxiv.org/abs/2609.18908"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18908","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18908","topics":[],"unit":null},{"aliases":["edge-model-compression-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:suresh4330/edge-model-compression-benchmark","source_record":"bm_edge-model-compression-benchmark_42f099b1"},"categories":["Factuality","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"EDGE-BENCH evaluates four ResNet-18 compression approaches—baseline, INT8 quantization, pruning, and knowledge distillation—on CIFAR-100 via accuracy, size, parameters, and CPU latency.","descriptions":{"en":"EDGE-BENCH evaluates four ResNet-18 compression approaches—baseline, INT8 quantization, pruning, and knowledge distillation—on CIFAR-100 via accuracy, size, parameters, and CPU latency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:suresh4330/edge-model-compression-benchmark","languages":[],"library_direction":{"basis":"model-review-2","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_edge-model-compression-benchmark_42f099b1"},"modality":null,"name":"EDGE-BENCH","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:suresh4330/edge-model-compression-benchmark","source_url":"https://github.com/suresh4330/edge-model-compression-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-suresh4330-edge-model-compression-benchmark","source":"claire_radar","source_url":"https://github.com/suresh4330/edge-model-compression-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":466,"github_stars_30d":25,"hf_dataset_downloads":4705,"hf_paper_upvotes":15,"join":"claire_record_id","repository":"gh:bytedance-seed/edgebench","source_record":"bm_edgebench_88dc3c92"},"categories":["Agents","Agents & Tool Use","Environment Learning","Environment learning","General AI","Learning from feedback","Long-horizon task execution","Mathematics & Formal Science","Scaling Laws","Scientific Research & AI for Science","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"EdgeBench evaluates autonomous agents on 134 real-world tasks across scientific discovery, software engineering, optimization, knowledge work, formal mathematics, and games. Each task requires 12+ hours of continuous interaction with multi-level feedback. Scoring tracks agent performance over time (at 2,4,6,8,10,12 hours). A public leaderboard is maintained; 51 tasks and evaluation framework are open-sourced.","descriptions":{"en":"EdgeBench evaluates autonomous agents on 134 real-world tasks across scientific discovery, software engineering, optimization, knowledge work, formal mathematics, and games. Each task requires 12+ hours of continuous interaction with multi-level feedback. Scoring tracks agent performance over time (at 2,4,6,8,10,12 hours). A public leaderboard is maintained; 51 tasks and evaluation framework are open-sourced."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.05155","languages":[],"library_direction":{"basis":"model-review-2","categories":["agentic-coding","ai-rnd","science"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_edgebench_88dc3c92"},"modality":null,"name":"EdgeBench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"first_public","source_key":"claire-radar:2607.05155","source_url":"https://edge-bench.org/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05155","topics":["coding","agents","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":27,"github_stars_30d":-1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:michailfragkiskos/edgemosaic","source_record":"bm_edgemosaic_837772bc"},"categories":["Finance","Language & Knowledge","Mobile & Personal Computing","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Defines a manifest-driven on-device AI evaluation lab that aggregates latency, memory, and thermal metrics across model, delegate, and configuration combinations.","descriptions":{"en":"Defines a manifest-driven on-device AI evaluation lab that aggregates latency, memory, and thermal metrics across model, delegate, and configuration combinations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:michailfragkiskos/edgemosaic","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_edgemosaic_837772bc"},"modality":null,"name":"EdgeMosaic","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:michailfragkiskos/edgemosaic","source_url":"https://github.com/michailfragkiskos/EdgeMosaic"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-michailfragkiskos-edgemosaic","source":"claire_radar","source_url":"https://github.com/michailfragkiskos/EdgeMosaic","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:solunny/edit2tikz","source_record":"bm_edit2tikz_4730ba41"},"categories":["Software & AI Compute","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Edit2TikZ evaluates instruction-guided scientific figure editing with TikZ code, featuring 1,548 samples with textual or visual localization requests and multi-step edits, using a human-aligned evaluation framework to measure edit completion and content preservation.","descriptions":{"en":"Edit2TikZ evaluates instruction-guided scientific figure editing with TikZ code, featuring 1,548 samples with textual or visual localization requests and multi-step edits, using a human-aligned evaluation framework to measure edit completion and content preservation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.13441","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_edit2tikz_4730ba41"},"modality":null,"name":"Edit2TikZ","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13441","source_url":"https://arxiv.org/abs/2608.13441"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13441","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13441","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:torux-bughunter/editclevr","source_record":"bm_editclevr_a686a8c4"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"EditCLEVR is a paired-scene intervention benchmark for object-centric representations, with before/after CLEVR renders and a known attribute change. Includes metrics like SGIA and Delta-SGIA for semantic faithfulness, with probe-free diagnostics.","descriptions":{"en":"EditCLEVR is a paired-scene intervention benchmark for object-centric representations, with before/after CLEVR renders and a known attribute change. Includes metrics like SGIA and Delta-SGIA for semantic faithfulness, with probe-free diagnostics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.22705","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_editclevr_a686a8c4"},"modality":null,"name":"EditCLEVR","openness":"unknown","publisher":null,"released":"2026-07-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22705","source_url":"https://arxiv.org/abs/2607.22705"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22705","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22705","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["创作","Creation","指令跟随","Instruct","多模态模型","VLM","视觉生成","Visual Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"EditInspector is a novel benchmark for evaluation of text-guided image edits, based on human annotations collected using an extensive template for edit verification. We leverage EditInspector to evaluate the performance of state-of-the-art (SoTA) vision and language models. EditInspector 评估最先进（SoTA）视觉和语言模型在多个维度上评估编辑的性能，包括准确性、瑕疵检测、视觉质量、与图像场景的无缝融合、遵循常识以及描述编辑引起变化的能力。","descriptions":{"en":"EditInspector is a novel benchmark for evaluation of text-guided image edits, based on human annotations collected using an extensive template for edit verification. We leverage EditInspector to evaluate the performance of state-of-the-art (SoTA) vision and language models.","zh":"EditInspector 评估最先进（SoTA）视觉和语言模型在多个维度上评估编辑的性能，包括准确性、瑕疵检测、视觉质量、与图像场景的无缝融合、遵循常识以及描述编辑引起变化的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1970","languages":[],"library_direction":null,"modality":null,"name":"EditInspector","openness":"unknown","publisher":"The Hebrew University of Jerusalem , TelAviv University , Google Research","released":"2025-06-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1970-editinspector","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EditInspector","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"EduArt evaluates art-historical knowledge and visual reasoning in multimodal LLMs using 871 human-authored questions in Italian and English, covering multiple formats and languages. Scoring is based on accuracy and psychometric properties.","descriptions":{"en":"EduArt evaluates art-historical knowledge and visual reasoning in multimodal LLMs using 871 human-authored questions in Italian and English, covering multiple formats and languages. Scoring is based on accuracy and psychometric properties."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02007","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","knowledge-expert-qa","multilingual"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_eduart_6e0cd321"},"modality":null,"name":"EduArt","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02007","source_url":"https://arxiv.org/abs/2607.02007"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02007","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02007","topics":["multimodal","reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Factuality","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"EduClaw-Bench evaluates pedagogical LLM agents in a simulated 30-day tutoring relationship with a knowledge-tracing-based simulated learner, scoring learning gain, responsiveness, helpfulness, and curriculum design across 55 scenarios.","descriptions":{"en":"EduClaw-Bench evaluates pedagogical LLM agents in a simulated 30-day tutoring relationship with a knowledge-tracing-based simulated learner, scoring learning gain, responsiveness, helpfulness, and curriculum design across 55 scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03206","languages":[],"library_direction":null,"modality":null,"name":"EduClaw-Bench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03206","source_url":"https://arxiv.org/abs/2608.03206"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03206","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03206","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"EduPluginBench is an executable benchmark for evaluating code-generation models on producing plugins that meet governed ecosystem requirements including least privilege, telemetry consent, provenance, and bounded failure. It uses 1,440 mutants and 120 clean references with staged admission levels P0-P4.","descriptions":{"en":"EduPluginBench is an executable benchmark for evaluating code-generation models on producing plugins that meet governed ecosystem requirements including least privilege, telemetry consent, provenance, and bounded failure. It uses 1,440 mutants and 120 clean references with staged admission levels P0-P4."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00739","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_edupluginbench_5240df80"},"modality":null,"name":"EduPluginBench","openness":"unknown","publisher":null,"released":"2026-08-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00739","source_url":"https://arxiv.org/abs/2608.00739"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00739","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00739","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for educational validity of video generation models, grounded in the Knowledge-Skills-Attitude framework.","descriptions":{"en":"Benchmark for educational validity of video generation models, grounded in the Knowledge-Skills-Attitude framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26918","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_eduvideobench_97f1a2a9"},"modality":null,"name":"EduVideoBench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26918","source_url":"https://arxiv.org/abs/2605.26918"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26918","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26918","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":42,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:xiaozhangyes/eeg-editbench","source_record":"bm_eeg-editbench_8f55ec4b"},"categories":["General AI","Information retrieval","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"EEG-EditBench evaluates EEG-to-image retrieval models using 2,137 controlled edits of 200 THINGS-EEG2 test images, covering object identity, attributes, background, and presence, with metrics like 200-way accuracy and 2AFC accuracy.","descriptions":{"en":"EEG-EditBench evaluates EEG-to-image retrieval models using 2,137 controlled edits of 200 THINGS-EEG2 test images, covering object identity, attributes, background, and presence, with metrics like 200-way accuracy and 2AFC accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27857","languages":[],"library_direction":null,"modality":null,"name":"EEG-EditBench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27857","source_url":"https://arxiv.org/abs/2607.27857"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27857","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27857","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Ego-MC-Bench is a benchmark for evaluating reactive step-by-step task guidance in cooking scenarios, focusing on timely interventions when mistakes occur.","descriptions":{"en":"Ego-MC-Bench is a benchmark for evaluating reactive step-by-step task guidance in cooking scenarios, focusing on timely interventions when mistakes occur."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09547","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","embodied-robotics"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_ego-mc-bench_91c3532a"},"modality":null,"name":"Ego-MC-Bench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09547","source_url":"https://arxiv.org/abs/2606.09547"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09547","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09547","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Ego-METAS evaluates online temporal action segmentation in egocentric video, where models must select sensor modalities (RGB, audio, gaze, IMU, monochrome) per timestep to maximize accuracy under hardware-representative energy budgets. It includes 100+ hours of untrimmed video from multiple datasets.","descriptions":{"en":"Ego-METAS evaluates online temporal action segmentation in egocentric video, where models must select sensor modalities (RGB, audio, gaze, IMU, monochrome) per timestep to maximize accuracy under hardware-representative energy budgets. It includes 100+ hours of untrimmed video from multiple datasets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02246","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_ego-metas_beb4f202"},"modality":null,"name":"Ego-METAS","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02246","source_url":"https://arxiv.org/abs/2606.02246"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02246","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02246","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":34,"join":"claire_record_id","repository":"gh:ego2act/ego2act","source_record":"bm_ego2act_ce647b42"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Video generation models are increasingly being explored as world simulators for embodied planning and learning.","descriptions":{"en":"Video generation models are increasingly being explored as world simulators for embodied planning and learning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.01092","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_ego2act_ce647b42"},"modality":null,"name":"Ego2Act","openness":"unknown","publisher":null,"released":"2026-10-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.01092","source_url":"https://arxiv.org/abs/2610.01092"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-01092","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01092","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Planning","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Egocentric videos capture how people carry out everyday activities, yet testing an agent requires evaluating the consequences of actions it chooses itself.","descriptions":{"en":"Egocentric videos capture how people carry out everyday activities, yet testing an agent requires evaluating the consequences of actions it chooses itself."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.02715","languages":[],"library_direction":{"basis":"model-review-daily","categories":["world-models","video-understanding"],"join":"claire_record_id","primary":"world-models","source_record":"bm_ego2world_d76ce5f6"},"modality":null,"name":"Ego2World","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.02715","source_url":"https://arxiv.org/abs/2610.02715"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-02715","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02715","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoAfford is a benchmark for egocentric referring segmentation with task-oriented affordance grounding, comprising 15.5k images and 102 real images. It includes EgoLens, a 3B MLLM reference model.","descriptions":{"en":"EgoAfford is a benchmark for egocentric referring segmentation with task-oriented affordance grounding, comprising 15.5k images and 102 real images. It includes EgoLens, a 3B MLLM reference model."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04533","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","embodied-robotics"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_egoafford_be5c2b17"},"modality":null,"name":"EgoAfford","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04533","source_url":"https://arxiv.org/abs/2608.04533"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04533","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04533","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_egobench_d0f8580c"},"categories":["Agents & Tool Use","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoBench is an interactive multimodal benchmark for tool-using agents. It comprises 1,045 egocentric-video-grounded tasks across four daily scenarios, with a user-agent-tool interactive environment. It assesses multimodal perception, tool invocation with multi-hop reasoning, and dynamic user interaction.","descriptions":{"en":"EgoBench is an interactive multimodal benchmark for tool-using agents. It comprises 1,045 egocentric-video-grounded tasks across four daily scenarios, with a user-agent-tool interactive environment. It assesses multimodal perception, tool invocation with multi-hop reasoning, and dynamic user interaction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27820","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","video-understanding"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_egobench_d0f8580c"},"modality":null,"name":"EgoBench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27820","source_url":"https://arxiv.org/abs/2605.27820"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27820","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27820","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":1109,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_egoexo-next_501db957"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Vision-language models (VLMs) are increasingly evaluated for egocentric and cross-view video reasoning, yet existing benchmarks largely focus on semantic event understanding, temporal relations, or correspondence between already observed v…","descriptions":{"en":"Vision-language models (VLMs) are increasingly evaluated for egocentric and cross-view video reasoning, yet existing benchmarks largely focus on semantic event understanding, temporal relations, or correspondence between already observed v…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.04506","languages":[],"library_direction":{"basis":"model-review-daily","categories":["video-understanding","spatial-3d"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_egoexo-next_501db957"},"modality":null,"name":"EgoExo-Next","openness":"unknown","publisher":null,"released":"2026-10-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.04506","source_url":"https://arxiv.org/abs/2610.04506"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-04506","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04506","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jhcor/egogapbench","source_record":"bm_egogapbench_95dd2183"},"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoGapBench evaluates egocentric action selection in multi-agent scenes, isolating the ability to choose actions from the agent's perspective when other agents are present. The benchmark includes training and test splits with human performance as reference.","descriptions":{"en":"EgoGapBench evaluates egocentric action selection in multi-agent scenes, isolating the ability to choose actions from the agent's perspective when other agents are present. The benchmark includes training and test splits with human performance as reference."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.00547","languages":[],"library_direction":{"basis":"model-review-2","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_egogapbench_95dd2183"},"modality":null,"name":"EgoGapBench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00547","source_url":"https://arxiv.org/abs/2607.00547"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00547","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00547","topics":["agents"],"unit":null},{"aliases":["Beyond Visual Quality: Evaluating Physical Consistency under Ego-Motion with EgoGenEval"],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:internrobotics/egogeneval","source_record":"bm_beyond-visual-quality-evaluating-physical-_fca65bd7"},"categories":["Geometric reasoning","Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoGenEval offers 1,400 cases and 2,360 target views to assess physical consistency of visual generators under ego-motion, measuring Camera Motion Grounding and Scene State Preservation.","descriptions":{"en":"EgoGenEval offers 1,400 cases and 2,360 target views to assess physical consistency of visual generators under ego-motion, measuring Camera Motion Grounding and Scene State Preservation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.11172","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","world-models"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_beyond-visual-quality-evaluating-physical-_fca65bd7"},"modality":null,"name":"EgoGenEval","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11172","source_url":"https://arxiv.org/abs/2609.11172"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11172","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11172","topics":[],"unit":null},{"aliases":["EgoInertia-MI: A Multimodal Egocentric Vision and IMU Benchmark for Motor Impairment Assessment"],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"A multimodal benchmark of synchronized egocentric video and wearable IMU signals covering 19 activities, with action recognition and motor impairment severity estimation tasks scored by Macro-F1.","descriptions":{"en":"A multimodal benchmark of synchronized egocentric video and wearable IMU signals covering 19 activities, with action recognition and motor impairment severity estimation tasks scored by Macro-F1."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.03934","languages":[],"library_direction":null,"modality":null,"name":"EgoInertia-MI","openness":"unknown","publisher":null,"released":"2026-07-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03934","source_url":"https://arxiv.org/abs/2607.03934"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03934","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03934","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoMaize evaluates first-person maize instance segmentation, requiring ownership-consistent plant masks and plant-owned stem and tassel cues from close-range field images with severe same-class overlap.","descriptions":{"en":"EgoMaize evaluates first-person maize instance segmentation, requiring ownership-consistent plant masks and plant-owned stem and tassel cues from close-range field images with severe same-class overlap."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.12350","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_egomaize_0a02296c"},"modality":null,"name":"EgoMaize","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12350","source_url":"https://arxiv.org/abs/2609.12350"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12350","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12350","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_egomonth_17f19a53"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoMonth benchmarks month-level egocentric video understanding with 300+ hours from 20 participants over 20-120 days, 1,443 QA pairs, and a 14-task framework across schema consolidation, episodic indexing, and cascading reasoning.","descriptions":{"en":"EgoMonth benchmarks month-level egocentric video understanding with 300+ hours from 20 participants over 20-120 days, 1,443 QA pairs, and a 14-task framework across schema consolidation, episodic indexing, and cascading reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.13113","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_egomonth_17f19a53"},"modality":null,"name":"EgoMonth","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13113","source_url":"https://arxiv.org/abs/2608.13113"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13113","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13113","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","推理","Reasoning","多模态模型","VLM","视频理解","Video Understanding","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"EgoNormia is a challenging QA benchmark that tests VLMs' ability to reason over norms in context. The datset consists of 1,853 physically grounded egocentric interaction clips from Ego4D and corresponding five-way multiple-choice questions tasks for each. EgoNormia 是一个具有挑战性的问答基准，用于测试 VLMs 在上下文中推理规范的能力。该数据集包含来自 Ego4D 的 1,853 个物理基础化的以自我为中心的交互剪辑，以及每个剪辑对应的五选一多项选择题任务。","descriptions":{"en":"EgoNormia is a challenging QA benchmark that tests VLMs' ability to reason over norms in context. The datset consists of 1,853 physically grounded egocentric interaction clips from Ego4D and corresponding five-way multiple-choice questions tasks for each.","zh":"EgoNormia 是一个具有挑战性的问答基准，用于测试 VLMs 在上下文中推理规范的能力。该数据集包含来自 Ego4D 的 1,853 个物理基础化的以自我为中心的交互剪辑，以及每个剪辑对应的五选一多项选择题任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1604","languages":[],"library_direction":null,"modality":null,"name":"EgoNormia","openness":"unknown","publisher":"University of Arizona, Stanford University, etc.","released":"2025-02-27","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1604-egonormia","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EgoNormia","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_egoprox_628390c5"},"categories":["General AI","Geometric reasoning","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoProx evaluates multimodal large language models on egocentric 3D proximity reasoning, with tasks organized along a cognitive hierarchy covering intention, exploration, exploitation, and chain-of-actions reasoning.","descriptions":{"en":"EgoProx evaluates multimodal large language models on egocentric 3D proximity reasoning, with tasks organized along a cognitive hierarchy covering intention, exploration, exploitation, and chain-of-actions reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24456","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","video-understanding"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_egoprox_628390c5"},"modality":null,"name":"EgoProx","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24456","source_url":"https://arxiv.org/abs/2605.24456"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24456","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24456","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_egosafe-bench_9e855ed0"},"categories":["General AI","Reasoning","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoSafe-Bench evaluates visual safety understanding in first-person video, using 12,000 QA samples from 3,000 clips under the Hierarchical Reasoning Evaluation (HRE) protocol, which requires reasoning from feature anchoring to intent inference.","descriptions":{"en":"EgoSafe-Bench evaluates visual safety understanding in first-person video, using 12,000 QA samples from 3,000 clips under the Hierarchical Reasoning Evaluation (HRE) protocol, which requires reasoning from feature anchoring to intent inference."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.26518","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","safety-alignment"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_egosafe-bench_9e855ed0"},"modality":null,"name":"EgoSafe-Bench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26518","source_url":"https://arxiv.org/abs/2607.26518"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26518","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26518","topics":["reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_egosafetybench_4d21ada3"},"categories":["Robotics","Robotics & Embodied AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoSafetyBench is a diagnostic egocentric video benchmark of 1,200 robot-view scenarios to evaluate vision-language models as runtime safety guards. It assesses situational awareness across routine, suspicious, obvious, and contextual hazards, and visual-channel robustness against misleading in-scene text.","descriptions":{"en":"EgoSafetyBench is a diagnostic egocentric video benchmark of 1,200 robot-view scenarios to evaluate vision-language models as runtime safety guards. It assesses situational awareness across routine, suspicious, obvious, and contextual hazards, and visual-channel robustness against misleading in-scene text."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.00218","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","embodied-robotics","video-understanding"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_egosafetybench_4d21ada3"},"modality":null,"name":"EgoSafetyBench","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00218","source_url":"https://arxiv.org/abs/2607.00218"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00218","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00218","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_egosat_1b88fd52"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoSAT evaluates vision-language models on egocentric video reasoning in streaming settings. It contains 1,997 videos (165 hours) and about 4,800 QA pairs covering retrospective, online, and prospective reasoning tasks.","descriptions":{"en":"EgoSAT evaluates vision-language models on egocentric video reasoning in streaming settings. It contains 1,997 videos (165 hours) and about 4,800 QA pairs covering retrospective, online, and prospective reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.24422","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_egosat_1b88fd52"},"modality":null,"name":"EgoSAT","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24422","source_url":"https://arxiv.org/abs/2606.24422"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24422","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24422","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":122,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_65396ab97dcda4af"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A diagnostic benchmark for very long-form video language understanding consisting of over 5000 human curated multiple choice questions based on 3-minute video clips from Ego4D, covering a broad range of natural human activities and behaviors","descriptions":{"en":"A diagnostic benchmark for very long-form video language understanding consisting of over 5000 human curated multiple choice questions based on 3-minute video clips from Ego4D, covering a broad range of natural human activities and behaviors"},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:egoschema:gemini-1.0-pro","reported_at":"2024-02-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/egoschema?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:egoschema","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_65396ab97dcda4af"},"modality":"video","name":"EgoSchema","openness":"unknown","publisher":null,"released":"2023-08-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:egoschema","source_url":"https://arxiv.org/abs/2308.09126"},"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":77.9,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.779,"raw_min":0.557,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:egoschema:qwen2-vl-72b","reported_date":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/egoschema?top_n=500"},"unit":null},"scored_models":[{"date":"2024-02-15","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-egoschema","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/egoschema?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_egostream_a12d21a7"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Egostream evaluates streaming episodic memory in egocentric vision through 2,250 curated questions across seven cognitive dimensions, expanded into 8,528 recall-conditioned evaluations using the Answer Validity Window to test recall from instant to ultra-long-term.","descriptions":{"en":"Egostream evaluates streaming episodic memory in egocentric vision through 2,250 curated questions across seven cognitive dimensions, expanded into 8,528 recall-conditioned evaluations using the Answer Validity Window to test recall from instant to ultra-long-term."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.31557","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","agent-memory"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_egostream_a12d21a7"},"modality":null,"name":"EGOSTREAM","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31557","source_url":"https://arxiv.org/abs/2605.31557"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31557","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31557","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":15,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:russell-zeng/egotactile","source_record":"bm_egotactile_e93c424a"},"categories":["Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"EgoTactile is a dataset pairing egocentric video with full-hand pressure supervision for everyday objects, including a bare-hand transfer subset.","descriptions":{"en":"EgoTactile is a dataset pairing egocentric video with full-hand pressure supervision for everyday objects, including a bare-hand transfer subset."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09243","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_egotactile_e93c424a"},"modality":null,"name":"EgoTactile","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09243","source_url":"https://arxiv.org/abs/2606.09243"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09243","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09243","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Biology & Drug Discovery","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"EHR-Complex evaluates clinical agent performance on interactive reasoning over MIMIC-IV electronic health records. It consists of about 52K tasks across six clinical intents, requiring agents to execute SQL or Python in a sandboxed environment to answer patient- and population-level queries. Scoring is based on exact-match accuracy against expected outcomes.","descriptions":{"en":"EHR-Complex evaluates clinical agent performance on interactive reasoning over MIMIC-IV electronic health records. It consists of about 52K tasks across six clinical intents, requiring agents to execute SQL or Python in a sandboxed environment to answer patient- and population-level queries. Scoring is based on exact-match accuracy against expected outcomes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.23301","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","tool-use"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_ehr-complex_08f2ee7c"},"modality":null,"name":"EHR-Complex","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23301","source_url":"https://arxiv.org/abs/2606.23301"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23301","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23301","topics":["agents","reasoning-math","medical"],"unit":null},{"aliases":["ELICITED: EHR-grounded Longitudinal Interactive Conversations for Information-seeking Triage Evaluation and Decision-making"],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"EHR2Dial-Triage evaluates triage conversations grounded in EHR data across information elicitation, evidence use, five-level Emergency Severity Index prediction, and patient-facing communication.","descriptions":{"en":"EHR2Dial-Triage evaluates triage conversations grounded in EHR data across information elicitation, evidence use, five-level Emergency Severity Index prediction, and patient-facing communication."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09024","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_elicited-ehr-grounded-longitudinal-interac_ec19f117"},"modality":null,"name":"EHR2Dial-Triage","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09024","source_url":"https://arxiv.org/abs/2608.09024"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09024","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09024","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"EHRBench evaluates LLM-based clinical decision-making using nearly 1M QA items generated from real EHR trajectories, covering diagnosis, treatment, and prognosis tasks.","descriptions":{"en":"EHRBench evaluates LLM-based clinical decision-making using nearly 1M QA items generated from real EHR trajectories, covering diagnosis, treatment, and prognosis tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30637","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_ehrbench_146544b7"},"modality":null,"name":"EHRBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30637","source_url":"https://arxiv.org/abs/2605.30637"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30637","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30637","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"EHRNote-ChatQA is a benchmark for evidence-grounded multi-turn clinical QA over longitudinal discharge summaries. Built from MIMIC-IV, it includes 967 patient-level samples and 16,072 expert-verified QA pairs across eight clinical categories, with evidence-grounding QA pairs.","descriptions":{"en":"EHRNote-ChatQA is a benchmark for evidence-grounded multi-turn clinical QA over longitudinal discharge summaries. Built from MIMIC-IV, it includes 967 patient-level samples and 16,072 expert-verified QA pairs across eight clinical categories, with evidence-grounding QA pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15735","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","long-context"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_ehrnote-chatqa_748f904b"},"modality":null,"name":"EHRNote-ChatQA","openness":"unknown","publisher":null,"released":"2026-06-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15735","source_url":"https://arxiv.org/abs/2606.15735"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15735","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15735","topics":["long-context","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","NeurIPS 2024","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"EHRNoteQA evaluates LLM's ability to assist clinical decision-making based on electronic health records, comprising 962 different QA pairs each linked to distinct patients' discharge summaries. EHRNoteQA用于评估LLM基于电子健康记录辅助临床决策的能力，由962个问答对组成，每个问答对都与不同患者的出院总结相关联。","descriptions":{"en":"EHRNoteQA evaluates LLM's ability to assist clinical decision-making based on electronic health records, comprising 962 different QA pairs each linked to distinct patients' discharge summaries.","zh":"EHRNoteQA用于评估LLM基于电子健康记录辅助临床决策的能力，由962个问答对组成，每个问答对都与不同患者的出院总结相关联。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1327","languages":[],"library_direction":null,"modality":null,"name":"EHRNoteQA","openness":"unknown","publisher":"KAIST","released":"2024-02-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1327-ehrnoteqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EHRNoteQA","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_eibench_cb3bf853"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"EIBench is a simulator-based benchmark for interactive emotion management, containing 2,222 scenarios across a 2x2 taxonomy of Support, Defense, Repair, and Charm. It evaluates LLM agents on multi-turn dialogue where a user simulator updates emotion-relation states and provides anchor-based scoring.","descriptions":{"en":"EIBench is a simulator-based benchmark for interactive emotion management, containing 2,222 scenarios across a 2x2 taxonomy of Support, Defense, Repair, and Charm. It evaluates LLM agents on multi-turn dialogue where a user simulator updates emotion-relation states and provides anchor-based scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15532","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_eibench_cb3bf853"},"modality":null,"name":"EIBench","openness":"unknown","publisher":null,"released":"2026-06-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15532","source_url":"https://arxiv.org/abs/2606.15532"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15532","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15532","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":37,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:zeroloss-lab/elbench","source_record":"bm_elbench_6c33e0fa"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"ELBench is a benchmark for education-facing large language models, evaluating General Capability, Safety and Trustworthiness, Basic Education, and High-Level Cultivation under a common protocol with 2,939 items.","descriptions":{"en":"ELBench is a benchmark for education-facing large language models, evaluating General Capability, Safety and Trustworthiness, Basic Education, and High-Level Cultivation under a common protocol with 2,939 items."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.09548","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","safety-alignment"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_elbench_6c33e0fa"},"modality":null,"name":"ELBench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09548","source_url":"https://arxiv.org/abs/2608.09548"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09548","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09548","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":37,"hf_paper_upvotes":0,"join":"artifact:gh:zeroloss-lab/elbench","repository":"gh:zeroloss-lab/elbench","source_record":"bm_elbench_6c33e0fa"},"categories":["推理","Reasoning","学科","Examination","安全","Safety","教育","多模态模型","开源收录"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ELBench is a multidimensional benchmark for education-facing large language models. It covers safety and trustworthiness, high-level educational cultivation, and  general capabilities, combining objective evaluation with LLM-as-a-Judge. ELBench 是面向教育场景大语言模型的多维评测集，覆盖安全可信、高阶育人和通用能力。公开任务包含安全回答、教育开放任务、选择题、数学与推理任务，并结合客观规则和大模型判决进行评测。","descriptions":{"en":"ELBench is a multidimensional benchmark for education-facing large language models. It covers safety and trustworthiness, high-level educational cultivation, and  general capabilities, combining objective evaluation with LLM-as-a-Judge.","zh":"ELBench 是面向教育场景大语言模型的多维评测集，覆盖安全可信、高阶育人和通用能力。公开任务包含安全回答、教育开放任务、选择题、数学与推理任务，并结合客观规则和大模型判决进行评测。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:2571","languages":["English","Chinese"],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","safety-alignment"],"join":"artifact:gh:zeroloss-lab/elbench","primary":"knowledge-expert-qa","source_record":"bm_elbench_6c33e0fa"},"modality":null,"name":"ELBench","openness":"open","publisher":"华东师范大学","released":"2026-06-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2571-elbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ELBench","topics":["multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Mobile & Personal Computing","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ElderBench is a benchmark for evaluating mobile GUI agents on tasks elicited from older adults, focusing on natural language patterns like indirect speech and ambiguity. It assesses agent performance on 249 tasks across 20 apps.","descriptions":{"en":"ElderBench is a benchmark for evaluating mobile GUI agents on tasks elicited from older adults, focusing on natural language patterns like indirect speech and ambiguity. It assesses agent performance on 249 tasks across 20 apps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.04850","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["computer-use","speech-audio"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_elderbench_e3027052"},"modality":null,"name":"ElderBench","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04850","source_url":"https://arxiv.org/abs/2609.04850"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04850","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04850","topics":["agents"],"unit":null},{"aliases":["election-bias-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kilojoules/election-bias-benchmark","source_record":"bm_election-bias-benchmark_5e0c5cb6"},"categories":["General AI","Safety & Trustworthiness","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Investigates whether a candidate's AGI-regulation stance influences LLM outputs on unrelated property tax positions, using factorial candidate set and two readouts (blind rubric judge and logit forced-choice).","descriptions":{"en":"Investigates whether a candidate's AGI-regulation stance influences LLM outputs on unrelated property tax positions, using factorial candidate set and two readouts (blind rubric judge and logit forced-choice)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:kilojoules/election-bias-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal","safety-alignment"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_election-bias-benchmark_5e0c5cb6"},"modality":null,"name":"Election: issue-contingent bias benchmark","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:kilojoules/election-bias-benchmark","source_url":"https://github.com/kilojoules/election-bias-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-kilojoules-election-bias-benchmark","source":"claire_radar","source_url":"https://github.com/kilojoules/election-bias-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":7,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"claire_record_id","repository":"gh:tencent/elephantbench","source_record":"bm_elephantbench_79ce069e"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Closed-book knowledge probe with 1,094 source-traceable questions that measure whether language models recall multiple verified accounts of long-tail facts, scored via complete, partial, or failed recall rates.","descriptions":{"en":"Closed-book knowledge probe with 1,094 source-traceable questions that measure whether language models recall multiple verified accounts of long-tail facts, scored via complete, partial, or failed recall rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.28478","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_elephantbench_79ce069e"},"modality":null,"name":"ElephantBench","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28478","source_url":"https://arxiv.org/abs/2608.28478"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28478","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28478","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Lifelong physical agents must reason over extended interactions where past events continue to shape the world long after they disappear from view.","descriptions":{"en":"Lifelong physical agents must reason over extended interactions where past events continue to shape the world long after they disappear from view."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.05013","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics","world-models"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_ember-bench_4c6f7148"},"modality":null,"name":"EMBER-Bench","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05013","source_url":"https://arxiv.org/abs/2610.05013"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05013","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05013","topics":["reasoning-math"],"unit":null},{"aliases":["Embodied-Bench"],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Embodied-BenchForge is an agentic workflow that transforms user-specified evaluation intents into embodied benchmark artifacts.","descriptions":{"en":"Embodied-BenchForge is an agentic workflow that transforms user-specified evaluation intents into embodied benchmark artifacts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.13082","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics","document-ocr"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_embodied-bench_5c4c02cd"},"modality":null,"name":"Embodied-BenchForge","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.13082","source_url":"https://arxiv.org/abs/2609.13082"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-13082","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.13082","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Geometric reasoning","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Embodied3DBench is a robot-centric benchmark for low-level spatial intelligence in embodied 3D environments. It includes 6 task categories: Grounding, Spatial Relation Prediction, Multi-view Correspondence, Affordance Prediction, Grasp Point Prediction, and Trajectory Prediction, with 21k QA pairs across 12 subcategories.","descriptions":{"en":"Embodied3DBench is a robot-centric benchmark for low-level spatial intelligence in embodied 3D environments. It includes 6 task categories: Grounding, Spatial Relation Prediction, Multi-view Correspondence, Affordance Prediction, Grasp Point Prediction, and Trajectory Prediction, with 21k QA pairs across 12 subcategories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29074","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_embodied3dbench_d183e439"},"modality":null,"name":"Embodied3DBench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29074","source_url":"https://arxiv.org/abs/2605.29074"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29074","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29074","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","Embodied Decision Making","NeurIPS 2024","物理智能","Embodied AI","逻辑推理","具身交互","Embodied Interaction"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Embodied Agent Interface supports the formalization of various types of tasks and input-output specifications of LLM-based modules, offering a comprehensive assessment of LLMs' performance for different subtasks and pinpointing the strengths and weaknesses in LLM-powered embodied AI systems. Embodied Agent Interface支持各种类型的任务和基于LLM模块的输入输出规范的形式化，对LLM在不同子任务中的性能进行了全面评估，指出了基于LLM的具身AI系统的优势和劣势。","descriptions":{"en":"Embodied Agent Interface supports the formalization of various types of tasks and input-output specifications of LLM-based modules, offering a comprehensive assessment of LLMs' performance for different subtasks and pinpointing the strengths and weaknesses in LLM-powered embodied AI systems.","zh":"Embodied Agent Interface支持各种类型的任务和基于LLM模块的输入输出规范的形式化，对LLM在不同子任务中的性能进行了全面评估，指出了基于LLM的具身AI系统的优势和劣势。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1243","languages":[],"library_direction":null,"modality":null,"name":"EmbodiedAgentInterface","openness":"open","publisher":"Stanford University","released":"2024-06-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1243-embodiedagentinterface","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EmbodiedAgentInterface","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":348,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":35,"join":"artifact:arxiv:2502.09560","repository":null,"source_record":"catalog_494a81e5eb85c5f2"},"categories":["多模态","Multimodal","智能体","Agent","物理智能","Embodied AI","跨模态推理","Cross-modal Reasoning","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"EmbodiedBench is a comprehensive benchmark designed to evaluate Multi-modal Large Language Models (MLLMs) as embodied agents. EmbodiedBench，这是一个旨在评估多模态大型语言模型作为具身智能体的全面基准。","descriptions":{"en":"EmbodiedBench is a comprehensive benchmark designed to evaluate Multi-modal Large Language Models (MLLMs) as embodied agents.","zh":"EmbodiedBench，这是一个旨在评估多模态大型语言模型作为具身智能体的全面基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1564","languages":[],"library_direction":{"basis":"anchor-review","categories":["embodied-robotics","abstract-reasoning"],"join":"artifact:arxiv:2502.09560","primary":"embodied-robotics","source_record":"catalog_494a81e5eb85c5f2"},"modality":"multimodal","name":"EmbodiedBench","openness":"unknown","publisher":"University of Illinois Urbana-Champaign, etc.","released":"2025-02-13","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1564-embodiedbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EmbodiedBench","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":8,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":30,"join":"claire_record_id","repository":null,"source_record":"bm_embodiedclbench_a90bee59"},"categories":["Robotics & Embodied AI","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"Continual learning requires models to accumulate past experience and transfer it to new tasks.","descriptions":{"en":"Continual learning requires models to accumulate past experience and transfer it to new tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13021","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_embodiedclbench_a90bee59"},"modality":null,"name":"EmbodiedCLBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13021","source_url":"https://arxiv.org/abs/2610.13021"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13021","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13021","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for embodied memory in long-horizon embodied tasks with 2,554 interactive episodes across four task families.","descriptions":{"en":"Benchmark for embodied memory in long-horizon embodied tasks with 2,554 interactive episodes across four task families."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.28236","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics","agent-memory"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_embodiedmemory-bench_ec684251"},"modality":null,"name":"EmbodiedMemory-Bench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.28236","source_url":"https://arxiv.org/abs/2609.28236"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-28236","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28236","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Simulation benchmark for coding agents spanning contact-rich manipulation, deformable objects, and long-horizon tasks up to half an hour.","descriptions":{"en":"Simulation benchmark for coding agents spanning contact-rich manipulation, deformable objects, and long-horizon tasks up to half an hour."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27308","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding","embodied-robotics"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_embodiedswe-bench_64b838a4"},"modality":null,"name":"EMBODIEDSWE-BENCH","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27308","source_url":"https://arxiv.org/abs/2609.27308"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27308","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27308","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:guopj1/embpass","source_record":"bm_embpass_bb24b231"},"categories":["Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Panoramic images provide a complete 360-degree field of view, enabling comprehensive scene understanding for embodied perception.","descriptions":{"en":"Panoramic images provide a complete 360-degree field of view, enabling comprehensive scene understanding for embodied perception."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.03248","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics","spatial-3d"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_embpass_bb24b231"},"modality":null,"name":"EmbPASS","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03248","source_url":"https://arxiv.org/abs/2610.03248"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03248","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03248","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":32,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:mengfeidu/embspatial-bench","source_record":"catalog_fd305ae2bc69d610"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["spatial_reasoning","embodied","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"EmbSpatialBench evaluates embodied spatial understanding and reasoning capabilities.","descriptions":{"en":"EmbSpatialBench evaluates embodied spatial understanding and reasoning capabilities."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:embspatialbench:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/embspatialbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:embspatialbench","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_fd305ae2bc69d610"},"modality":"image","name":"EmbSpatialBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":0.846,"display_multiplier":1,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.846,"raw_min":0.825,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:embspatialbench:qwen3.6-27b","reported_date":"2026-04-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/embspatialbench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-embspatialbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/embspatialbench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"eMEM-Bench v1 evaluates embodied memory systems through 988 probes across eight cognitive-psychology paradigms (DRM lures, pattern separation, pattern completion, source monitoring, context-dependent retrieval, long-horizon interference, serial position, foil retention curve) in ProcTHOR-10K scenes.","descriptions":{"en":"eMEM-Bench v1 evaluates embodied memory systems through 988 probes across eight cognitive-psychology paradigms (DRM lures, pattern separation, pattern completion, source monitoring, context-dependent retrieval, long-horizon interference, serial position, foil retention curve) in ProcTHOR-10K scenes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03374","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","embodied-robotics"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_emem-bench_594a2f57"},"modality":null,"name":"eMEM-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03374","source_url":"https://arxiv.org/abs/2606.03374"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03374","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03374","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Foundation models (FMs) are increasingly being developed for general time series and physiological signals, yet their transferability to downstream physiological tasks remains poorly understood.","descriptions":{"en":"Foundation models (FMs) are increasingly being developed for general time series and physiological signals, yet their transferability to downstream physiological tasks remains poorly understood."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.06450","languages":[],"library_direction":{"basis":"model-review-daily","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_emg-fm-bench_4b604513"},"modality":null,"name":"EMG-FM-Bench","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.06450","source_url":"https://arxiv.org/abs/2610.06450"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-06450","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06450","topics":["medical"],"unit":null},{"aliases":["greencalculus-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:greencalculus/greencalculus-benchmark","source_record":"bm_greencalculus-benchmark_23a8b636"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark of 467 emission-factor questions scored for numeric accuracy and citation correctness across five models without tools.","descriptions":{"en":"Benchmark of 467 emission-factor questions scored for numeric accuracy and citation correctness across five models without tools."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:greencalculus/greencalculus-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","factuality"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_greencalculus-benchmark_23a8b636"},"modality":null,"name":"Emission-factor accuracy benchmark","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:greencalculus/greencalculus-benchmark","source_url":"https://github.com/greencalculus/greencalculus-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-greencalculus-greencalculus-benchmark","source":"claire_radar","source_url":"https://github.com/greencalculus/greencalculus-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":69,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:emma-bench/emma","source_record":"catalog_f4aa0655cdb8d4fc"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"EMMA (Enhanced MultiModal reAsoning) is a benchmark for organic multimodal reasoning across mathematics, physics, chemistry, and coding.","descriptions":{"en":"EMMA (Enhanced MultiModal reAsoning) is a benchmark for organic multimodal reasoning across mathematics, physics, chemistry, and coding."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:emma:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/emma?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:emma","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math","science"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_f4aa0655cdb8d4fc"},"modality":"multimodal","name":"EMMA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":79.3,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.793,"raw_min":0.784,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:emma:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/emma?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-emma","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/emma?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":69,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"artifact:arxiv:2501.05444","repository":"gh:emma-bench/emma","source_record":"catalog_f4aa0655cdb8d4fc"},"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"EMMA is composed of 2,788 problems, of which 1,796 are newly constructed, across four domains. Within each subject, we further provide fine-grained labels for each question based on the specific skills it measures. EMMA 由 2,788 个问题组成，其中 1,796 个是新构建的，涵盖四个领域。在每个主题中，我们根据所测量的具体技能为每个问题提供细粒度标签。","descriptions":{"en":"EMMA is composed of 2,788 problems, of which 1,796 are newly constructed, across four domains. Within each subject, we further provide fine-grained labels for each question based on the specific skills it measures.","zh":"EMMA 由 2,788 个问题组成，其中 1,796 个是新构建的，涵盖四个领域。在每个主题中，我们根据所测量的具体技能为每个问题提供细粒度标签。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1640","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math","science"],"join":"artifact:arxiv:2501.05444","primary":"visual-reasoning","source_record":"catalog_f4aa0655cdb8d4fc"},"modality":"multimodal","name":"EMMA","openness":"restricted","publisher":"UESTC, SYSU, etc.","released":"2025-01-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1640-emma","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EMMA","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"EmoSBench evaluates emotional intelligence in spoken language models across four theoretical branches: perceiving, understanding, using, and managing emotion, with ten subtasks.","descriptions":{"en":"EmoSBench evaluates emotional intelligence in spoken language models across four theoretical branches: perceiving, understanding, using, and managing emotion, with ten subtasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09189","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","chat-writing"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_emosbench_d26ab0ed"},"modality":null,"name":"EmoSBench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09189","source_url":"https://arxiv.org/abs/2608.09189"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09189","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09189","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tonyheng6/emotrace-bench","source_record":"bm_emotrace-bench_6b16f14a"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"EmoTrace-Bench evaluates text-to-video generation on emotional expression by controlling prompt cues and intensity, and scoring videos on target emotion, perceived intensity, and support across ten audiovisual cue channels.","descriptions":{"en":"EmoTrace-Bench evaluates text-to-video generation on emotional expression by controlling prompt cues and intensity, and scoring videos on target emotion, perceived intensity, and support across ten audiovisual cue channels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.15452","languages":[],"library_direction":{"basis":"model-review-daily","categories":["image-video-generation","chat-writing"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_emotrace-bench_6b16f14a"},"modality":null,"name":"EmoTrace-Bench","openness":"unknown","publisher":null,"released":"2026-10-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.15452","source_url":"https://arxiv.org/abs/2610.15452"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-15452","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15452","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"EMPATH evaluates safety of emotional-support chatbots via auditor-generated multi-turn conversations scored on 19 metrics across crisis handling, therapeutic quality, conversational integrity, emotional safety, and cultural adaptation.","descriptions":{"en":"EMPATH evaluates safety of emotional-support chatbots via auditor-generated multi-turn conversations scored on 19 metrics across crisis handling, therapeutic quality, conversational integrity, emotional safety, and cultural adaptation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.30256","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","chat-writing","health-medicine"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_empath_eb9c1b58"},"modality":null,"name":"EMPATH","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30256","source_url":"https://arxiv.org/abs/2606.30256"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30256","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30256","topics":["language","safety","medical"],"unit":null},{"aliases":["On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos"],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates temporal vision-language models on surgical endoscopy video understanding under six realistic corruptions, using public videos and standardized prompts.","descriptions":{"en":"Evaluates temporal vision-language models on surgical endoscopy video understanding under six realistic corruptions, using public videos and standardized prompts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14262","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","video-understanding"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_on-the-robustness-of-temporal-vision-langu_c61c2c1d"},"modality":null,"name":"Endo-C6","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14262","source_url":"https://arxiv.org/abs/2608.14262"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14262","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14262","topics":["multimodal","medical"],"unit":null},{"aliases":["EnergyBridge: Benchmarking Household Energy Management, User Participation, and Grid Flexibility"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:agentic-intelligence-lab/energybridge","source_record":"bm_energybridge-benchmarking-household-energy_942a2142"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"EnergyBridge evaluates household energy management by connecting capacity reporting, event-specific household authorization simulation, and physical execution in EnergyPlus environments for Tianjin and Berlin, scoring authorization, energy use, and capacity commitment.","descriptions":{"en":"EnergyBridge evaluates household energy management by connecting capacity reporting, event-specific household authorization simulation, and physical execution in EnergyPlus environments for Tianjin and Berlin, scoring authorization, energy use, and capacity commitment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.08691","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_energybridge-benchmarking-household-energy_942a2142"},"modality":null,"name":"EnergyBridge","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08691","source_url":"https://arxiv.org/abs/2608.08691"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08691","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08691","topics":["agents"],"unit":null},{"aliases":["EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:changcv2021/engintervene","source_record":"bm_engintervene-benchmarking-multimodal-engin_ffd7ddfb"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal engineering state understanding and design intervention reasoning through 3,229 questions across seven domains, spanning state grounding, relational reasoning, diagnosis, and intervention tasks.","descriptions":{"en":"Evaluates multimodal engineering state understanding and design intervention reasoning through 3,229 questions across seven domains, spanning state grounding, relational reasoning, diagnosis, and intervention tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.33261","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_engintervene-benchmarking-multimodal-engin_ffd7ddfb"},"modality":null,"name":"EngIntervene","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33261","source_url":"https://arxiv.org/abs/2609.33261"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33261","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33261","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":["Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"Enginuity evaluates vision-language models on structured parts-table extraction and visual question answering over complex engineering diagrams.","descriptions":{"en":"Enginuity evaluates vision-language models on structured parts-table extraction and visual question answering over complex engineering diagrams."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03410","languages":[],"library_direction":null,"modality":null,"name":"Enginuity","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03410","source_url":"https://arxiv.org/abs/2606.03410"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03410","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03410","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":43,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":72,"join":"claire_record_id","repository":"gh:hongcheng-gao/engiworld","source_record":"bm_engiworld_98536ba1"},"categories":["CAD","Chip Design & EDA","General AI","Geometric reasoning","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"Autonomous agents have made rapid progress in general-purpose computer use, but reliable automation of professional industrial engineering remains out of reach, as engineering workflows demand reasoning over geometric and physical constrai…","descriptions":{"en":"Autonomous agents have made rapid progress in general-purpose computer use, but reliable automation of professional industrial engineering remains out of reach, as engineering workflows demand reasoning over geometric and physical constrai…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.37686","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","professional-work"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_engiworld_98536ba1"},"modality":null,"name":"EngiWorld","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37686","source_url":"https://arxiv.org/abs/2609.37686"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37686","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37686","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"While large language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained reasoning, complex state management, and continuous cross-domain abstraction.","descriptions":{"en":"While large language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained reasoning, complex state management, and continuous cross-domain abstraction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.39284","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_engrambench_11c5e611"},"modality":null,"name":"EngramBench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39284","source_url":"https://arxiv.org/abs/2609.39284"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39284","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39284","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for emotional need-aware proactive memory retrieval in support agents, with 1,800+ dialogues mapped to Maslow's hierarchy.","descriptions":{"en":"Benchmark for emotional need-aware proactive memory retrieval in support agents, with 1,800+ dialogues mapped to Maslow's hierarchy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27240","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_enpmr-bench_0b8d24ff"},"modality":null,"name":"ENPMR-Bench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27240","source_url":"https://arxiv.org/abs/2605.27240"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27240","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27240","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","Logistics & Operations","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-term trade-offs.","descriptions":{"en":"LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-term trade-offs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37658","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_enterprisebench_4bef0fca"},"modality":null,"name":"EnterpriseBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37658","source_url":"https://arxiv.org/abs/2609.37658"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37658","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37658","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":49,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":81,"join":"claire_record_id","repository":"gh:frontisai/enterpriseclawbench","source_record":"bm_enterpriseclawbench_252ea1ca"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on realistic enterprise workflows using 852 reproducible tasks with recovered fixtures, prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Scoring covers artifact delivery, visual quality, cost, runtime, and skill transfer.","descriptions":{"en":"Evaluates coding agents on realistic enterprise workflows using 852 reproducible tasks with recovered fixtures, prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Scoring covers artifact delivery, visual quality, cost, runtime, and skill transfer."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.23654","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","agentic-coding"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_enterpriseclawbench_252ea1ca"},"modality":null,"name":"EnterpriseClawBench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23654","source_url":"https://arxiv.org/abs/2606.23654"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23654","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23654","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"EnterpriseMem-Bench is a multi-turn Text-to-SQL benchmark with 300 sessions and 1,400 turns across three enterprise domains, featuring deterministic ground truth and per-turn memory-critical annotations.","descriptions":{"en":"EnterpriseMem-Bench is a multi-turn Text-to-SQL benchmark with 300 sessions and 1,400 turns across three enterprise domains, featuring deterministic ground truth and per-turn memory-critical annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26394","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","agent-memory"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_enterprisemem-bench_96086215"},"modality":null,"name":"EnterpriseMem-Bench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26394","source_url":"https://arxiv.org/abs/2605.26394"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26394","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26394","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["agentic","business"],"collected_at":"2026-08-25T10:41:06Z","description":"Agentic business operations","descriptions":{"en":"Agentic business operations"},"evidence_summary":{"document_count":1,"model_count":33,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:enterpriseops-gym-aa:f0083258-8646-45b8-8082-7aaf6c2ea82a","reported_at":"2025-08-05","source_url":"https://artificialanalysis.ai/evaluations/enterprise-ops-gym-aa"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:enterpriseops-gym-aa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"EnterpriseOps-Gym-AA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":33,"score_direction":"higher_is_better","score_summary":{"display_max":51.11906893464637,"display_multiplier":100,"model_count":33,"model_count_basis":"source_model_id","numeric_count":33,"raw_max":0.5111906893464637,"raw_min":0.25544613548194567,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:enterpriseops-gym-aa:cd55210d-358e-4df1-ba9c-9acb5f186cc9","reported_date":"2026-06-09","source_url":"https://artificialanalysis.ai/evaluations/enterprise-ops-gym-aa"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"}],"slug":"artificial-analysis-enterpriseops-gym-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/enterprise-ops-gym-aa","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Information retrieval","Robustness","Safety & Trustworthiness","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The benchmark evaluates LLM instruction adherence and robustness in enterprise retrieval scenarios, using 983 expert-validated samples across six domains, simulating retrieval noise, knowledge gaps, and factual conflicts.","descriptions":{"en":"The benchmark evaluates LLM instruction adherence and robustness in enterprise retrieval scenarios, using 983 expert-validated samples across six domains, simulating retrieval noise, knowledge gaps, and factual conflicts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.11584","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","instruction-following"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_enterpriserag_49e3b609"},"modality":null,"name":"EnterpriseRAG","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11584","source_url":"https://arxiv.org/abs/2608.11584"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11584","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11584","topics":["language"],"unit":null},{"aliases":["ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering"],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"claire_record_id","repository":"gh:scitix/entlore","source_record":"bm_entlore-a-graph-grounded-benchmark-for-lat_335f471f"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates enterprise question answering over 2,341 documents and 907 questions spanning explicit lookup, cross-source composition, and latent organizational reasoning.","descriptions":{"en":"Evaluates enterprise question answering over 2,341 documents and 907 questions spanning explicit lookup, cross-source composition, and latent organizational reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.10679","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","professional-work"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_entlore-a-graph-grounded-benchmark-for-lat_335f471f"},"modality":null,"name":"ENTLORE","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10679","source_url":"https://arxiv.org/abs/2608.10679"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10679","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10679","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","Long Context"],"collected_at":"2026-10-14T09:44:51Z","description":"EntSQL evaluates text-to-SQL systems on enterprise knowledge grounding, with 1,066 Chinese-English examples across five business domains requiring private business knowledge. Systems generate SQL from questions and schema, with some provided long-form documents.","descriptions":{"en":"EntSQL evaluates text-to-SQL systems on enterprise knowledge grounding, with 1,066 Chinese-English examples across five business domains requiring private business knowledge. Systems generate SQL from questions and schema, with some provided long-form documents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03363","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","professional-work"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_entsql_088384b0"},"modality":null,"name":"EntSQL","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03363","source_url":"https://arxiv.org/abs/2606.03363"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03363","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03363","topics":["coding","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":1558,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_envship_5107b438"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"EnvShip is a unified multi-region framework for context-aware and cross-region vessel trajectory forecasting, with standardized tracks and evaluation protocols using public AIS data.","descriptions":{"en":"EnvShip is a unified multi-region framework for context-aware and cross-region vessel trajectory forecasting, with standardized tracks and evaluation protocols using public AIS data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15240","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_envship_5107b438"},"modality":null,"name":"EnvShip","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15240","source_url":"https://arxiv.org/abs/2606.15240"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15240","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15240","topics":[],"unit":null},{"aliases":["Benchmarking_MLIPs"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:soham0605/benchmarking_mlips","source_record":"bm_benchmarking-mlips_97111d88"},"categories":["Language & Knowledge","Materials & Chemistry","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates MLIP equation-of-state predictions against DFT for volume-scaled structures, reporting percent errors in equilibrium volume and bulk modulus.","descriptions":{"en":"Evaluates MLIP equation-of-state predictions against DFT for volume-scaled structures, reporting percent errors in equilibrium volume and bulk modulus."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:soham0605/benchmarking_mlips","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_benchmarking-mlips_97111d88"},"modality":null,"name":"eos-bench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:soham0605/benchmarking_mlips","source_url":"https://github.com/Soham0605/Benchmarking_MLIPs"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-soham0605-benchmarking-mlips","source":"claire_radar","source_url":"https://github.com/Soham0605/Benchmarking_MLIPs","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"EpiBench evaluates AI agents on short-horizon epigenomics analysis tasks across CUT&Tag/CUT&RUN, ATAC-seq, ChIP-seq, and DNA methylation workflows, with deterministically gradable answers.","descriptions":{"en":"EpiBench evaluates AI agents on short-horizon epigenomics analysis tasks across CUT&Tag/CUT&RUN, ATAC-seq, ChIP-seq, and DNA methylation workflows, with deterministically gradable answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13602","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_epibench_be78266a"},"modality":null,"name":"EpiBench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13602","source_url":"https://arxiv.org/abs/2606.13602"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13602","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13602","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"EpiBench evaluates LLMs' epitope reasoning from antibody and antigen sequences across five tasks: targetable region discovery, epitope identification, binning, functional assessment, and escape assessment.","descriptions":{"en":"EpiBench evaluates LLMs' epitope reasoning from antibody and antigen sequences across five tasks: targetable region discovery, epitope identification, binning, functional assessment, and escape assessment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06022","languages":[],"library_direction":{"basis":"model-review","categories":["science","health-medicine"],"join":"claire_record_id","primary":"science","source_record":"bm_epibench_00d62089"},"modality":null,"name":"EpiBench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06022","source_url":"https://arxiv.org/abs/2608.06022"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06022","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06022","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"(E-commerce Product Review Dataset for Sentiment Analysis), also known as EPRSTMT, is a binary sentiment analysis dataset based on product reviews on e-commerce platform. Each sample is labelled as Positive or Negative. It collect by ICIP Lab of Beijing Normal University. EPRSTMT，也称作电子商务产品评论情感分析数据集，是一个基于电子商务平台上的产品评论的二元情感分析数据集。每个样本都被标记为积极或消极。该数据集由北京师范大学 ICIP 实验室收集。","descriptions":{"en":"(E-commerce Product Review Dataset for Sentiment Analysis), also known as EPRSTMT, is a binary sentiment analysis dataset based on product reviews on e-commerce platform. Each sample is labelled as Positive or Negative. It collect by ICIP Lab of Beijing Normal University.","zh":"EPRSTMT，也称作电子商务产品评论情感分析数据集，是一个基于电子商务平台上的产品评论的二元情感分析数据集。每个样本都被标记为积极或消极。该数据集由北京师范大学 ICIP 实验室收集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:522","languages":[],"library_direction":null,"modality":null,"name":"EPRSTMT","openness":"unknown","publisher":null,"released":"2021-07-15","released_reference":{"basis":"paper_first_version","note":"The FewCLUE introduction includes its new EPRSTMT product-review sentiment task.","source_key":"opencompass:522","source_url":"https://arxiv.org/abs/2107.07498"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-522-eprstmt","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EPRSTMT","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":70,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_ab06169b3c2aeee2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","roleplay","general","creativity","writing"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"EQ-Bench is an LLM-judged test evaluating active emotional intelligence abilities, understanding, insight, empathy, and interpersonal skills. The test set contains 45 challenging roleplay scenarios, most of which constitute pre-written prompts spanning 3 turns. The benchmark evaluates the performance of models by validating responses against several criteria and conducts pairwise comparisons to report a normalized Elo computation for each model.","descriptions":{"en":"EQ-Bench is an LLM-judged test evaluating active emotional intelligence abilities, understanding, insight, empathy, and interpersonal skills. The test set contains 45 challenging roleplay scenarios, most of which constitute pre-written prompts spanning 3 turns. The benchmark evaluates the performance of models by validating responses against several criteria and conducts pairwise comparisons to report a normalized Elo computation for each model."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:eq-bench:grok-4.1-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/eq-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:eq-bench","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_ab06169b3c2aeee2"},"modality":"text","name":"EQ-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":1586.0,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":1586.0,"raw_min":1585.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:eq-bench:grok-4.1-thinking-2025-11-17","reported_date":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/eq-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1"},{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-eq-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/eq-bench?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","Medical","科学智能","AI for Science","逻辑推理","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ER-Reason is a large-scale benchmark suite for evaluating the clinical reasoning capabilities of large language models (LLMs) in the emergency room (ER) — a high-stakes environment where clinicians make rapid, life-critical decisions. ER-Reason 是一个大规模基准套件，用于评估大语言模型（LLMs）在急诊室（ER）中的临床推理能力。急诊室是一个高风险环境，临床医生需要快速做出关乎生命的关键决策。","descriptions":{"en":"ER-Reason is a large-scale benchmark suite for evaluating the clinical reasoning capabilities of large language models (LLMs) in the emergency room (ER) — a high-stakes environment where clinicians make rapid, life-critical decisions.","zh":"ER-Reason 是一个大规模基准套件，用于评估大语言模型（LLMs）在急诊室（ER）中的临床推理能力。急诊室是一个高风险环境，临床医生需要快速做出关乎生命的关键决策。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1892","languages":[],"library_direction":null,"modality":null,"name":"ER-Reason","openness":"unknown","publisher":"University of California, Berkeley,etc.","released":"2025-05-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1892-er-reason","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ER-Reason","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_erbench_e22588c9"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ERBench is an evaluation framework for equation discovery algorithms, assessing recovery of groundtruth formulas under varying dimensionality, sampling size, distribution, and domain.","descriptions":{"en":"ERBench is an evaluation framework for equation discovery algorithms, assessing recovery of groundtruth formulas under varying dimensionality, sampling size, distribution, and domain."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09276","languages":[],"library_direction":{"basis":"model-review","categories":["science","math"],"join":"claire_record_id","primary":"science","source_record":"bm_erbench_e22588c9"},"modality":null,"name":"ERBench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09276","source_url":"https://arxiv.org/abs/2606.09276"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09276","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09276","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":12,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kaixuewen/ergeobench","source_record":"bm_ergeobench_37de7936"},"categories":["Multimodal","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ERGeoBench evaluates vision-driven embodied geo-localization in MLLMs with 2,207 globally distributed street-view panoramas under single-view, panorama-view, and embodied-view settings. It measures foundational perception, spatial awareness, common sense reasoning, and geo-localization reasoning.","descriptions":{"en":"ERGeoBench evaluates vision-driven embodied geo-localization in MLLMs with 2,207 globally distributed street-view panoramas under single-view, panorama-view, and embodied-view settings. It measures foundational perception, spatial awareness, common sense reasoning, and geo-localization reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.31251","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","embodied-robotics"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_ergeobench_37de7936"},"modality":null,"name":"ERGeoBench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31251","source_url":"https://arxiv.org/abs/2605.31251"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31251","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31251","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["Anchor: Mitigating Artifact Drift in Agent Benchmark Generation"],"attention_signals":null,"categories":["Agents","Language & Knowledge","Manufacturing & Process Control"],"collected_at":"2026-09-25T04:40:22Z","description":"ERP-Bench evaluates long-horizon procurement and manufacturing tasks in a production-grade ERP system using state-based verification and solver-certified solutions.","descriptions":{"en":"ERP-Bench evaluates long-horizon procurement and manufacturing tasks in a production-grade ERP system using state-based verification and solver-certified solutions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26321","languages":[],"library_direction":null,"modality":null,"name":"ERP-Bench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26321","source_url":"https://arxiv.org/abs/2605.26321"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26321","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26321","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:gair-nlp/erp-bench","source_record":"bm_erpbench_cad5b1ca"},"categories":["Finance","Language & Knowledge","Logistics & Operations","Planning","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ERPBench evaluates LLM agents in a six-round ERP simulation with pricing, production, procurement, inventory, finance, and shared-market competition. It includes 100 fixed problems tested in Solo (vs rule-based opponents) and Arena (six agents competing) ecologies, scoring based on firm valuation.","descriptions":{"en":"ERPBench evaluates LLM agents in a six-round ERP simulation with pricing, production, procurement, inventory, finance, and shared-market competition. It includes 100 fixed problems tested in Solo (vs rule-based opponents) and Arena (six agents competing) ecologies, scoring based on firm valuation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.04667","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","multi-agent-games"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_erpbench_cad5b1ca"},"modality":null,"name":"ERPBench","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04667","source_url":"https://arxiv.org/abs/2609.04667"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04667","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04667","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","Logistics & Operations","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ERPBench evaluates screenshot-only agents on a live ERP system with scoring against database ground-truth values.","descriptions":{"en":"ERPBench evaluates screenshot-only agents on a live ERP system with scoring against database ground-truth values."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17885","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_erpbench_1215d137"},"modality":null,"name":"ERPBench","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17885","source_url":"https://arxiv.org/abs/2609.17885"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17885","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17885","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":307,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"llm_stats_id","repository":"gh:embodiedreasoning/erqa","source_record":"catalog_a5789abbeab1284a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","spatial_reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Embodied Reasoning Question Answering benchmark consisting of 400 multiple-choice visual questions across spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning for evaluating AI capabilities in physical world interactions","descriptions":{"en":"Embodied Reasoning Question Answering benchmark consisting of 400 multiple-choice visual questions across spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning for evaluating AI capabilities in physical world interactions"},"evidence_summary":{"document_count":1,"model_count":24,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:erqa:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/erqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:erqa","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","spatial-3d","visual-reasoning"],"join":"llm_stats_id","primary":"embodied-robotics","source_record":"catalog_a5789abbeab1284a"},"modality":"multimodal","name":"ERQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":24,"score_direction":"higher_is_better","score_summary":{"display_max":77.8,"display_multiplier":100,"model_count":24,"model_count_basis":"source_model_id","numeric_count":24,"raw_max":0.778,"raw_min":0.352,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:erqa:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/erqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-erqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/erqa?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":382,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:lunaproject22/erqa-plus","source_record":"bm_erqa-plus_148e4067"},"categories":["Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ERQA-Plus evaluates embodied reasoning in AI systems with 1,766 question-answer instances grounded in 711 robot-centric images, organized by a taxonomy covering perceptual, action-centric, social-interaction, navigation-environmental, and commonsense reasoning. Scoring uses overall accuracy and SBERT similarity.","descriptions":{"en":"ERQA-Plus evaluates embodied reasoning in AI systems with 1,766 question-answer instances grounded in 711 robot-centric images, organized by a taxonomy covering perceptual, action-centric, social-interaction, navigation-environmental, and commonsense reasoning. Scoring uses overall accuracy and SBERT similarity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.17639","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","visual-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_erqa-plus_148e4067"},"modality":null,"name":"ERQA-Plus","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17639","source_url":"https://arxiv.org/abs/2606.17639"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17639","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17639","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":"gh:salinaria/erunderstand","source_record":"bm_erunderstand_657f59fd"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"ERUnderstand evaluates vision-language models on structured understanding of entity-relationship diagrams. The benchmark contains 2,960 diagrams across curated educational sources, real-world schemas, and synthetic data, with standardized machine-readable JSON annotations. Scoring uses F1, BLEU, and graph edit distance to measure how accurately models recover schema elements such as entities, relationships, attributes, and extended ER constructs.","descriptions":{"en":"ERUnderstand evaluates vision-language models on structured understanding of entity-relationship diagrams. The benchmark contains 2,960 diagrams across curated educational sources, real-world schemas, and synthetic data, with standardized machine-readable JSON annotations. Scoring uses F1, BLEU, and graph edit distance to measure how accurately models recover schema elements such as entities, relationships, attributes, and extended ER constructs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.24707","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","document-ocr"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_erunderstand_657f59fd"},"modality":null,"name":"ERUnderstand","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24707","source_url":"https://arxiv.org/abs/2607.24707"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24707","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24707","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Speech & Audio"],"collected_at":"2026-10-14T09:44:51Z","description":"ESCUCHA is a Spanish speech understanding benchmark evaluating LALMs across heterogeneous acoustic conditions and reasoning abilities. It includes 1,000 human-curated audio-question pairs spanning perceptual and reasoning categories, with multiple accents and non-normative speech.","descriptions":{"en":"ESCUCHA is a Spanish speech understanding benchmark evaluating LALMs across heterogeneous acoustic conditions and reasoning abilities. It includes 1,000 human-curated audio-question pairs spanning perceptual and reasoning categories, with multiple accents and non-normative speech."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.17812","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_escucha_bdd66fa8"},"modality":null,"name":"ESCUCHA","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.17812","source_url":"https://arxiv.org/abs/2607.17812"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-17812","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17812","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ESF-Bench evaluates slot filling in enterprise contexts, covering 810 multi-turn dialogues and 6,530 slots across 8 domains, with a taxonomy of 57 challenging scenarios.","descriptions":{"en":"ESF-Bench evaluates slot filling in enterprise contexts, covering 810 multi-turn dialogues and 6,530 slots across 8 domains, with a taxonomy of 57 challenging scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.23326","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_esf-bench_d9226708"},"modality":null,"name":"ESF-Bench","openness":"unknown","publisher":null,"released":"2026-07-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23326","source_url":"https://arxiv.org/abs/2607.23326"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23326","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23326","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":3,"hf_dataset_downloads":704,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bit-pie/esrp","source_record":"bm_esrp-bench_4847c977"},"categories":["Planning","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout.","descriptions":{"en":"This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.27371","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_esrp-bench_4847c977"},"modality":null,"name":"ESRP-Bench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27371","source_url":"https://arxiv.org/abs/2608.27371"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27371","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27371","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating Ethereum DeFi representations under various distribution shifts.","descriptions":{"en":"A benchmark for evaluating Ethereum DeFi representations under various distribution shifts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23659","languages":[],"library_direction":null,"modality":null,"name":"ETH-TraceBench","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23659","source_url":"https://arxiv.org/abs/2609.23659"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23659","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23659","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_euroexec_53125f62"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"EuroExec is an expert-authored benchmark of 413 open-ended European executive tasks evaluated by human experts. It uses a multi-attribute rubric and preference ranking to compute a Solve Rate.","descriptions":{"en":"EuroExec is an expert-authored benchmark of 413 open-ended European executive tasks evaluated by human experts. It uses a multi-attribute rubric and preference ranking to compute a Solve Rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04549","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_euroexec_53125f62"},"modality":null,"name":"EuroExec","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04549","source_url":"https://arxiv.org/abs/2608.04549"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04549","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04549","topics":[],"unit":null},{"aliases":["EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A multilingual benchmark for verifiable instruction following, derived from IFEval and localized to 50 European languages and regional varieties via task-preserving translation.","descriptions":{"en":"A multilingual benchmark for verifiable instruction following, derived from IFEval and localized to 50 European languages and regional varieties via task-preserving translation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.05043","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","instruction-following"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_euroalpaca-task-preserving-localisation-of_6eb5fc44"},"modality":null,"name":"European-IFEval","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05043","source_url":"https://arxiv.org/abs/2609.05043"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05043","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05043","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:gorkafs13/euskal-agent-bench","source_record":"bm_euskal-agent-bench_06279148"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Euskal-agent-bench is a project evaluating LLM agents in Basque, Spanish, and English across 30 tasks with direct prompting and tool-using agent architectures, measuring capability and agentic-stack degradation.","descriptions":{"en":"Euskal-agent-bench is a project evaluating LLM agents in Basque, Spanish, and English across 30 tasks with direct prompting and tool-using agent architectures, measuring capability and agentic-stack degradation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:gorkafs13/euskal-agent-bench","languages":[],"library_direction":null,"modality":null,"name":"euskal-agent-bench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:gorkafs13/euskal-agent-bench","source_url":"https://github.com/GorkaFS13/euskal-agent-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-gorkafs13-euskal-agent-bench","source":"claire_radar","source_url":"https://github.com/GorkaFS13/euskal-agent-bench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_evalawarebench_6e7fbcdf"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"EvalAwareBench is a factor-controlled benchmark of 100 paired safety-capability tasks, each toggling one of eight evaluation-awareness triggers while holding the underlying request fixed. It measures recognition and behavioral change in language models under evaluation.","descriptions":{"en":"EvalAwareBench is a factor-controlled benchmark of 100 paired safety-capability tasks, each toggling one of eight evaluation-awareness triggers while holding the underlying request fixed. It measures recognition and behavioral change in language models under evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.23055","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_evalawarebench_6e7fbcdf"},"modality":null,"name":"EvalAwareBench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23055","source_url":"https://arxiv.org/abs/2605.23055"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23055","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23055","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bagheri365/evallab","source_record":"bm_evallab_50a8738a"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Provides a frozen 120-task Python coding benchmark with paired comparisons and uncertainty analysis focused on ranking stability.","descriptions":{"en":"Provides a frozen 120-task Python coding benchmark with paired comparisons and uncertainty analysis focused on ranking stability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:bagheri365/evallab","languages":[],"library_direction":null,"modality":null,"name":"EvalLab","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:bagheri365/evallab","source_url":"https://github.com/bagheri365/evallab"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-bagheri365-evallab","source":"claire_radar","source_url":"https://github.com/bagheri365/evallab","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1826,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_dfdf41ea3061f83c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A rigorous code synthesis evaluation framework that augments existing datasets with extensive test cases generated by LLM and mutation-based strategies to better assess functional correctness of generated code, including HumanEval+ with 80x more test cases","descriptions":{"en":"A rigorous code synthesis evaluation framework that augments existing datasets with extensive test cases generated by LLM and mutation-based strategies to better assess functional correctness of generated code, including HumanEval+ with 80x more test cases"},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:evalplus:qwen2-72b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/evalplus?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:evalplus","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_dfdf41ea3061f83c"},"modality":"text","name":"EvalPlus","openness":"unknown","publisher":null,"released":"2023-05-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:evalplus","source_url":"https://arxiv.org/abs/2305.01210"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":0.803,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.803,"raw_min":0.703,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:evalplus:kimi-k2-base","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/evalplus?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"}],"slug":"llm-stats-evalplus","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/evalplus?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":["Evaluation Agent","Evaluation-Agent"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"A promptable evaluation framework rather than a fixed test set: it plans its own queries per model, so two runs need not ask the same questions. It does report numbers -- an average score per round and the per-prompt scores behind it -- but each is measured over the handful of prompts that run chose for that model, so they describe the run rather than a scale models can be ranked on.","descriptions":{"en":"A promptable evaluation framework rather than a fixed test set: it plans its own queries per model, so two runs need not ask the same questions. It does report numbers -- an average score per round and the per-prompt scores behind it -- but each is measured over the handful of prompts that run chose for that model, so they describe the run rather than a scale models can be ranked on."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:evaluation_agent","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["image-video-generation"],"join":"review:14H034160212:2026-10-15","primary":"image-video-generation","source_record":null},"modality":null,"name":"Evaluation Agent","openness":"unknown","publisher":null,"released":"2024-12-10","released_reference":{"basis":"paper_first_version","note":"First version of the paper introducing the Evaluation Agent framework, the identifier its repository cites. The repository was created two months earlier, which dates the code rather than the benchmark.","source_key":"model-reports:evaluation_agent","source_url":"https://arxiv.org/abs/2412.09645"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"evaluation_agent","source":"model_reports","source_url":"https://github.com/Vchitect/Evaluation-Agent","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of 50 diverse task-policy pairs where task completion requires an operation prohibited by a runtime monitor.","descriptions":{"en":"A benchmark of 50 diverse task-policy pairs where task completion requires an operation prohibited by a runtime monitor."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.30217","languages":[],"library_direction":null,"modality":null,"name":"EvasionBench","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30217","source_url":"https://arxiv.org/abs/2609.30217"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30217","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30217","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:evemtbench/evemtbench-benchmark","source_record":"bm_evemtbench_b0ef8e9f"},"categories":["General AI","Language & Knowledge","eess.SY"],"collected_at":"2026-10-14T09:44:51Z","description":"Power-system protection benchmark on EMT waveform data with multiple grids, protocols, and tasks, including fault detection, classification, and location.","descriptions":{"en":"Power-system protection benchmark on EMT waveform data with multiple grids, protocols, and tasks, including fault detection, classification, and location."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.28149","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_evemtbench_b0ef8e9f"},"modality":null,"name":"EvEMTBench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.28149","source_url":"https://arxiv.org/abs/2609.28149"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-28149","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28149","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Event ActivityNet is a large-scale simulated-event benchmark for untrimmed action understanding, derived from ActivityNet videos. It includes 3,263 videos, 200 action classes, event-voxel representations, temporal annotations, and supports recognition, event-language alignment, and online temporal localization.","descriptions":{"en":"Event ActivityNet is a large-scale simulated-event benchmark for untrimmed action understanding, derived from ActivityNet videos. It includes 3,263 videos, 200 action classes, event-voxel representations, temporal annotations, and supports recognition, event-language alignment, and online temporal localization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01948","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_event-activitynet_532df77c"},"modality":null,"name":"Event ActivityNet","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01948","source_url":"https://arxiv.org/abs/2608.01948"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01948","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01948","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_evid-bench_e9fc3b86"},"categories":["Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"EVID-Bench is a benchmark for search-grounded video misinformation detection, covering 222 videos and multiple manipulation types.","descriptions":{"en":"EVID-Bench is a benchmark for search-grounded video misinformation detection, covering 222 videos and multiple manipulation types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04098","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","factuality","deep-research"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_evid-bench_e9fc3b86"},"modality":null,"name":"EVID-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04098","source_url":"https://arxiv.org/abs/2606.04098"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04098","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04098","topics":[],"unit":null},{"aliases":["Evidence-Bounded Mental Health Reasoning from Heterogeneous Speech Protocols"],"attention_signals":null,"categories":["Biology & Drug Discovery","Reasoning","Speech & Audio"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates evidence-bounded mental health reasoning over 1,870 packages from six heterogeneous speech-text sources, scoring diagnosis accuracy and claim violations under modality masks.","descriptions":{"en":"Evaluates evidence-bounded mental health reasoning over 1,870 packages from six heterogeneous speech-text sources, scoring diagnosis accuracy and claim violations under modality masks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.31014","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["health-medicine","speech-audio"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_evidence-bounded-mental-health-reasoning-f_e3e77fe7"},"modality":null,"name":"Evidence Package Benchmark","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.31014","source_url":"https://arxiv.org/abs/2608.31014"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-31014","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31014","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":19,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":"gh:rucaibox/evo-bench","source_record":"bm_evo-bench_a13a17e9"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evo-Bench is a benchmark for evaluating whether language models can autonomously improve their agent harness, with 608 harness-sensitive tasks across Search, Office, and General domains, fixed policy model, and resource budget.","descriptions":{"en":"Evo-Bench is a benchmark for evaluating whether language models can autonomously improve their agent harness, with 608 harness-sensitive tasks across Search, Office, and General domains, fixed policy model, and resource budget."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.09096","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_evo-bench_a13a17e9"},"modality":null,"name":"Evo-Bench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09096","source_url":"https://arxiv.org/abs/2608.09096"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09096","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09096","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":301,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_evoagentbench_cbe93d38"},"categories":["Agents","Code","Code & Software","Factuality","General AI","Reasoning","Self-Evolution","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"EvoAgentBench evaluates agent self-evolution via ability transfer across web research, algorithmic reasoning, software engineering, and knowledge work, using trace-grounded abilities and domain graphs with train/test splits.","descriptions":{"en":"EvoAgentBench evaluates agent self-evolution via ability transfer across web research, algorithmic reasoning, software engineering, and knowledge work, using trace-grounded abilities and domain graphs with train/test splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.05202","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_evoagentbench_cbe93d38"},"modality":null,"name":"EvoAgentBench","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.05202","source_url":"https://arxiv.org/abs/2607.05202"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05202","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05202","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":26,"github_stars_30d":4,"hf_dataset_downloads":null,"hf_paper_upvotes":146,"join":"claire_record_id","repository":"gh:aiden0526/evoarena","source_record":"bm_evoarena_70b33a0d"},"categories":["Agents & Tool Use","General AI","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"EvoArena is a benchmark suite for LLM agents in dynamic environments, covering terminal workflows, software repositories, and social preferences. It evaluates step and chain accuracy under progressive environment updates.","descriptions":{"en":"EvoArena is a benchmark suite for LLM agents in dynamic environments, covering terminal workflows, software repositories, and social preferences. It evaluates step and chain accuracy under progressive environment updates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.13681","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_evoarena_70b33a0d"},"modality":null,"name":"EvoArena","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13681","source_url":"https://arxiv.org/abs/2606.13681"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13681","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13681","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":null,"source_record":"bm_evobrowsecomp_ef2e05ce"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"EvoBrowseComp evaluates search agents on evolving knowledge with 800 contamination-free questions synthesized from live web traversal. The benchmark is designed to be regularly updated to prevent contamination.","descriptions":{"en":"EvoBrowseComp evaluates search agents on evolving knowledge with 800 contamination-free questions synthesized from live web traversal. The benchmark is designed to be regularly updated to prevent contamination."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13120","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_evobrowsecomp_ef2e05ce"},"modality":null,"name":"EvoBrowseComp","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13120","source_url":"https://arxiv.org/abs/2606.13120"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13120","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13120","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_evoclawbench_03620568"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"EvoClawBench evaluates whether an agent runtime can convert evidence from its own runs into reusable skills that improve fresh executions. It covers 100 tasks and 502 sub-problems across coding, data, office, security, operations, and domain-document workflows, comparing direct execution, pre-authored skills, and post-run skill summarization.","descriptions":{"en":"EvoClawBench evaluates whether an agent runtime can convert evidence from its own runs into reusable skills that improve fresh executions. It covers 100 tasks and 502 sub-problems across coding, data, office, security, operations, and domain-document workflows, comparing direct execution, pre-authored skills, and post-run skill summarization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09711","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_evoclawbench_03620568"},"modality":null,"name":"EvoClawBench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09711","source_url":"https://arxiv.org/abs/2607.09711"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09711","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09711","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_evocode-bench_60a115e8"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"EvoCode-Bench evaluates coding agents in multi-turn iterative interactions, with 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, specifies requirements via observable behavior, and uses cumulative executable tests to verify new and prior requirements. Scoring uses MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state.","descriptions":{"en":"EvoCode-Bench evaluates coding agents in multi-turn iterative interactions, with 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, specifies requirements via observable behavior, and uses cumulative executable tests to verify new and prior requirements. Scoring uses MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24110","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_evocode-bench_60a115e8"},"modality":null,"name":"EvoCode-Bench","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24110","source_url":"https://arxiv.org/abs/2605.24110"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24110","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24110","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:maps-research/evogenui-bench","source_record":"bm_evogenui-bench_3b239061"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates large language models as multi-turn generative UI assistants on 150 five-turn tasks with browser-executed artifacts, using turn-level, episode-level, and Adjacent Pass Retention metrics.","descriptions":{"en":"Evaluates large language models as multi-turn generative UI assistants on 150 five-turn tasks with browser-executed artifacts, using turn-level, episode-level, and Adjacent Pass Retention metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29387","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","chat-writing"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_evogenui-bench_3b239061"},"modality":null,"name":"EvoGenUI-Bench","openness":"unknown","publisher":null,"released":"2026-08-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29387","source_url":"https://arxiv.org/abs/2608.29387"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29387","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29387","topics":["coding","language"],"unit":null},{"aliases":["EvoHarmBench: Breaking Content Moderation with Iterative Human-Like Evasion"],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates content moderation systems against iteratively evolved adversarial samples across 229 semantic sub-clusters and five violation categories, with scoring based on attack success rate under readability constraints.","descriptions":{"en":"Evaluates content moderation systems against iteratively evolved adversarial samples across 229 semantic sub-clusters and five violation categories, with scoring based on attack success rate under readability constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27844","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_evoharmbench-breaking-content-moderation-w_fad69c38"},"modality":null,"name":"EvoHarmBench","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27844","source_url":"https://arxiv.org/abs/2608.27844"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27844","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27844","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"EvoNav-Bench evaluates lifelong navigation agents in evolving environments built on ProcTHOR, with environment modifications between tasks.","descriptions":{"en":"EvoNav-Bench evaluates lifelong navigation agents in evolving environments built on ProcTHOR, with environment modifications between tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08292","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_evonav-bench_6370db2f"},"modality":null,"name":"EvoNav-Bench","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08292","source_url":"https://arxiv.org/abs/2609.08292"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08292","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08292","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark proposed to track individual capabilities during artifact-level self-evolution.","descriptions":{"en":"A benchmark proposed to track individual capabilities during artifact-level self-evolution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24663","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_evopathbench_dd352c31"},"modality":null,"name":"EvoPathBench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24663","source_url":"https://arxiv.org/abs/2609.24663"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24663","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24663","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":174,"github_stars_30d":-1,"hf_dataset_downloads":null,"hf_paper_upvotes":48,"join":"claire_record_id","repository":"gh:linzwcs/evopolicygym","source_record":"bm_evopolicygym_6d1d11a9"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"EvoPolicyGym evaluates autonomous policy evolution in interactive RL environments. A harness-model agent iteratively edits an executable policy under a fixed interaction budget. The benchmark records trajectories of programs, submissions, feedback, and selection, and scores agents on held-out episodes.","descriptions":{"en":"EvoPolicyGym evaluates autonomous policy evolution in interactive RL environments. A harness-model agent iteratively edits an executable policy under a fixed interaction budget. The benchmark records trajectories of programs, submissions, feedback, and selection, and scores agents on held-out episodes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.02440","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","agent-skills-evolution"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_evopolicygym_6d1d11a9"},"modality":null,"name":"EvoPolicyGym","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02440","source_url":"https://arxiv.org/abs/2607.02440"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02440","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02440","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources.","descriptions":{"en":"Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.03153","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-security","professional-work"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_evoriskbench_77efcd34"},"modality":null,"name":"EvoRiskBench","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03153","source_url":"https://arxiv.org/abs/2610.03153"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03153","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03153","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates continual visual learning under evolving semantic concept shift using ImageNet, iNaturalist, CUB-200-2011, and DomainNet with semantic transitions and metrics such as Rewrite Accuracy, Preservation Accuracy, Obsolete Retention, and Selective Revision Score.","descriptions":{"en":"Evaluates continual visual learning under evolving semantic concept shift using ImageNet, iNaturalist, CUB-200-2011, and DomainNet with semantic transitions and metrics such as Rewrite Accuracy, Preservation Accuracy, Obsolete Retention, and Selective Revision Score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23903","languages":[],"library_direction":null,"modality":null,"name":"EvoShift-Bench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23903","source_url":"https://arxiv.org/abs/2608.23903"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23903","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23903","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","智能体","Agent","指令跟随","Instruct","worldmodel","embodied AI","manipulate","物理智能","指令遵循","Instruction Following","具身交互","Embodied Interaction","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"EWMBench is a benchmark for evaluating Embodied World Models, covering aspects such as scene consistency, motion correctness, and semantic alignment. It includes a diverse dataset and multi-dimensional metrics tailored for embodied manipulation tasks. EWMBench 是一个用于评估具身世界模型的基准，涵盖“场景一致性”、“动作正确性”和“语义对齐”等方面。它包含多样化的数据集和面向具身任务的多维评估指标，能够揭示现有模型的局限，并为生成具物理基础、任务导向的视频提供评价标准。","descriptions":{"en":"EWMBench is a benchmark for evaluating Embodied World Models, covering aspects such as scene consistency, motion correctness, and semantic alignment. It includes a diverse dataset and multi-dimensional metrics tailored for embodied manipulation tasks.","zh":"EWMBench 是一个用于评估具身世界模型的基准，涵盖“场景一致性”、“动作正确性”和“语义对齐”等方面。它包含多样化的数据集和面向具身任务的多维评估指标，能够揭示现有模型的局限，并为生成具物理基础、任务导向的视频提供评价标准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1843","languages":[],"library_direction":null,"modality":"multimodal","name":"EWMBench","openness":"restricted","publisher":"Agibot","released":"2025-05-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1843-ewmbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EWMBench","topics":["agents","multimodal","reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Autonomous Driving","Planning","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards.","descriptions":{"en":"Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37871","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_exceptiondrive_1841c2ae"},"modality":null,"name":"ExceptionDrive","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37871","source_url":"https://arxiv.org/abs/2609.37871"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37871","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37871","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"EXE-Bench evaluates AI-based Windows malware detectors on performance, temporal robustness, adversarial robustness, and computational overhead, aggregated into a single score.","descriptions":{"en":"EXE-Bench evaluates AI-based Windows malware detectors on performance, temporal robustness, adversarial robustness, and computational overhead, aggregated into a single score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24177","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_exe-bench_d4e86d18"},"modality":null,"name":"EXE-Bench","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24177","source_url":"https://arxiv.org/abs/2607.24177"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24177","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24177","topics":["cybersecurity"],"unit":null},{"aliases":["WaveTLM: Reliable Time-Series Language Modeling through Task Compilation"],"attention_signals":null,"categories":["Software & AI Compute","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark spanning time-series forecasting, imputation, classification, anomaly detection, and waveform analysis across contract-grounded tasks.","descriptions":{"en":"A benchmark spanning time-series forecasting, imputation, classification, anomaly detection, and waveform analysis across contract-grounded tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.18812","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["forecasting","document-ocr","finance-legal"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_wavetlm-reliable-time-series-language-mode_043e031a"},"modality":null,"name":"ExecTS-QA","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18812","source_url":"https://arxiv.org/abs/2609.18812"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18812","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18812","topics":[],"unit":null},{"aliases":["ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecasting"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zest86/exphy","source_record":"bm_exphy-a-benchmark-for-explicit-physical-pr_ac5ba09f"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"24,000 simulated scenes for multi-object trajectory forecasting with explicit labels for mass, friction, and restitution, including ID and OOD splits over physical parameters and initial states.","descriptions":{"en":"24,000 simulated scenes for multi-object trajectory forecasting with explicit labels for mass, friction, and restitution, including ID and OOD splits over physical parameters and initial states."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.20009","languages":[],"library_direction":{"basis":"model-review","categories":["world-models"],"join":"claire_record_id","primary":"world-models","source_record":"bm_exphy-a-benchmark-for-explicit-physical-pr_ac5ba09f"},"modality":null,"name":"ExPhy","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.20009","source_url":"https://arxiv.org/abs/2608.20009"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-20009","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20009","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":14,"join":"claire_record_id","repository":"gh:explainbench/explainbench-cli","source_record":"bm_explainbench_0a1efae5"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates code explanations from agents by checking whether explanations enable an LLM to correctly answer questions about intended behavior and patch effects, using a question-based suite derived from agent patches.","descriptions":{"en":"Evaluates code explanations from agents by checking whether explanations enable an LLM to correctly answer questions about intended behavior and patch effects, using a question-based suite derived from agent patches."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.26451","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","agentic-coding"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_explainbench_0a1efae5"},"modality":null,"name":"ExplainBench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26451","source_url":"https://arxiv.org/abs/2607.26451"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26451","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26451","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bahumanyarg11/tta-explanation-stability-pipeline","source_record":"bm_explanation-stability-of-test-time-adaptat_4ac1ea1b"},"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates explanation stability of test-time adaptation methods across histopathology datasets and architectures, measuring drift in attribution maps along with accuracy and calibration.","descriptions":{"en":"Evaluates explanation stability of test-time adaptation methods across histopathology datasets and architectures, measuring drift in attribution maps along with accuracy and calibration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.07062","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_explanation-stability-of-test-time-adaptat_4ac1ea1b"},"modality":null,"name":"Explanation Stability of Test-Time Adaptation in Computational Pathology: A Large-Scale Benchmark","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07062","source_url":"https://arxiv.org/abs/2608.07062"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07062","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07062","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":429,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:exploitbench/exploitbench","source_record":"catalog_19fc6e81d91e8949"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ExploitBench is a cybersecurity benchmark that evaluates a model's ability to discover and exploit software vulnerabilities, reported as the fraction of challenges where the model captures the target (Cap%).","descriptions":{"en":"ExploitBench is a cybersecurity benchmark that evaluates a model's ability to discover and exploit software vulnerabilities, reported as the fraction of challenges where the model captures the target (Cap%)."},"editorial_guide":{"id":"exploitbench","record_keys":["llm-stats:exploitbench","model-reports:exploitbench"],"reviewed_at":"2026-10-20","scope":{"en":"The initial v8-bench release: 41 patched V8 vulnerabilities and 16 capability flags.","zh":"首个 v8-bench 版本：41 个已修复的 V8 漏洞、16 项能力标记。"},"sections":[{"id":"overview","source_ids":["paper","spec"],"text":{"en":"ExploitBench’s first benchmark, v8-bench, uses 41 patched vulnerabilities in the V8 JavaScript and WebAssembly engine. It records 16 capability flags grouped into five tiers, ranging from reaching affected code and reproducing a bug to stronger control capabilities. The flags preserve useful evidence of partial progress.","zh":"ExploitBench 的首个评测 v8-bench 使用 V8 JavaScript 与 WebAssembly 引擎中 41 个已修复漏洞，记录分属五个层级的 16 项能力标记，从到达相关代码、复现故障，到更强的控制能力。这些标记保留了阶段性进展的信息。"},"title":{"en":"What it measures","zh":"测什么"}},{"id":"tasks","source_ids":["repo","spec"],"text":{"en":"The agent receives vulnerability context, including a patch diff, source code, and relevant program builds, and produces a test artifact for the benchmark environment. This is evaluation on already identified, patched bugs. It should be distinguished from discovering an unknown vulnerability without such starting information.","zh":"智能体获得包括补丁差异、源代码和相关程序构建在内的漏洞材料，并为评测环境生成测试产物。任务基于已经发现并修复的漏洞；它与没有这些先验线索的未知漏洞发现属于不同设置。"},"title":{"en":"A known-vulnerability setting","zh":"任务如何设置"}},{"id":"evaluation","source_ids":["home","spec"],"text":{"en":"Deterministic verifiers check the capability flags; an LLM judge does not assign the result. Capability coverage measures achieved flags, not the percentage of vulnerabilities fully exploited. The public results distinguish fixed experiments from a maximum-achievement view with varying seeds and budgets, so preserve the reported aggregation and regime.","zh":"结果由确定性验证器检查能力标记，不由大模型评委打分。能力覆盖率衡量获得的标记，不能解释为完全利用成功的漏洞比例。公开结果区分固定实验与使用不同随机种子、预算的最大表现视图，阅读时应保留原始汇总方式和实验设置。"},"title":{"en":"Read capability coverage carefully","zh":"如何判分"}},{"id":"use_cases","source_ids":["paper","repo"],"text":{"en":"Use it to investigate where progress stops within a difficult software target and to compare agent configurations under controlled conditions. A tier profile can distinguish reliable bug reproduction from later capabilities that a single success percentage would hide, while still crediting intermediate work.","zh":"适合研究智能体在复杂软件目标上推进到哪个阶段，以及在受控条件下比较不同配置。分层结果可以区分稳定的故障复现与更后续的能力，同时记录中间工作，补充单一成功率难以表达的信息。"},"title":{"en":"When it is useful","zh":"适合用来判断什么"}},{"id":"limitations","source_ids":["home","paper","repo"],"text":{"en":"The initial suite is specific to V8 and public patched bugs; it does not establish general zero-day discovery ability. Baseline runs, automated coaching, and native CLI agents use different protocols. Public prior information and runtime conditions can affect results, making matched settings essential for a fair comparison.","zh":"初始任务集限定于 V8 和已公开修复的漏洞，不能据此认定通用零日漏洞发现能力。基线运行、自动引导与原生 CLI 智能体采用不同协议；公开的既有信息和运行环境也可能影响结果，公平比较需要匹配这些条件。"},"title":{"en":"Version and protocol boundaries","zh":"读分时的限制"}}],"sources":[{"id":"home","title":"ExploitBench official methodology and result views","url":"https://exploitbench.ai/"},{"id":"paper","title":"ExploitBench paper","url":"https://arxiv.org/abs/2605.14153"},{"id":"repo","title":"ExploitBench official repository","url":"https://github.com/exploitbench/exploitbench/blob/main/README.md"},{"id":"spec","title":"Official V8 benchmark specification","url":"https://github.com/exploitbench/exploitbench/blob/main/benchmarks/bench-v8/SPEC.md"}],"summary":{"en":"ExploitBench measures stages of progress on known V8 vulnerabilities, using separate capability checks rather than one binary outcome.","zh":"ExploitBench 对已知 V8 漏洞上的不同能力分别验证，呈现阶段性进展，而非仅给出单一成败结果。"},"title":"ExploitBench"},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:exploitbench:claude-fable-5","reported_at":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:exploitbench","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"llm_stats_id","primary":"cybersecurity","source_record":"catalog_19fc6e81d91e8949"},"modality":"text","name":"ExploitBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":78.0,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.78,"raw_min":0.332,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:exploitbench:claude-fable-5","reported_date":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-exploitbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitbench?top_n=500","topics":["coding","agents","safety","cybersecurity"],"unit":null},{"aliases":["ExploitBench","ExploitBench (Cap%)"],"attention_signals":null,"categories":["security"],"collected_at":null,"description":"Offensive-security capability measured as a risk indicator against a preparedness threshold, not a leaderboard to top. Reported as a capability percentage. Published by Seunghyun Lee and David Brumley (Carnegie Mellon; Brumley also lists Bugcrowd) as arXiv:2605.14153 with code at github.com/exploitbench/exploitbench. 41 V8 N-day vulnerabilities scored by 16 grader-verified capability flags; vendor runs report the capability percentage without publishing per-flag results, so a reported number is not reproducible from the paper alone.","descriptions":{"en":"Offensive-security capability measured as a risk indicator against a preparedness threshold, not a leaderboard to top. Reported as a capability percentage. Published by Seunghyun Lee and David Brumley (Carnegie Mellon; Brumley also lists Bugcrowd) as arXiv:2605.14153 with code at github.com/exploitbench/exploitbench. 41 V8 N-day vulnerabilities scored by 16 grader-verified capability flags; vendor runs report the capability percentage without publishing per-flag results, so a reported number is not reproducible from the paper alone."},"editorial_guide":{"id":"exploitbench","record_keys":["llm-stats:exploitbench","model-reports:exploitbench"],"reviewed_at":"2026-10-20","scope":{"en":"The initial v8-bench release: 41 patched V8 vulnerabilities and 16 capability flags.","zh":"首个 v8-bench 版本：41 个已修复的 V8 漏洞、16 项能力标记。"},"sections":[{"id":"overview","source_ids":["paper","spec"],"text":{"en":"ExploitBench’s first benchmark, v8-bench, uses 41 patched vulnerabilities in the V8 JavaScript and WebAssembly engine. It records 16 capability flags grouped into five tiers, ranging from reaching affected code and reproducing a bug to stronger control capabilities. The flags preserve useful evidence of partial progress.","zh":"ExploitBench 的首个评测 v8-bench 使用 V8 JavaScript 与 WebAssembly 引擎中 41 个已修复漏洞，记录分属五个层级的 16 项能力标记，从到达相关代码、复现故障，到更强的控制能力。这些标记保留了阶段性进展的信息。"},"title":{"en":"What it measures","zh":"测什么"}},{"id":"tasks","source_ids":["repo","spec"],"text":{"en":"The agent receives vulnerability context, including a patch diff, source code, and relevant program builds, and produces a test artifact for the benchmark environment. This is evaluation on already identified, patched bugs. It should be distinguished from discovering an unknown vulnerability without such starting information.","zh":"智能体获得包括补丁差异、源代码和相关程序构建在内的漏洞材料，并为评测环境生成测试产物。任务基于已经发现并修复的漏洞；它与没有这些先验线索的未知漏洞发现属于不同设置。"},"title":{"en":"A known-vulnerability setting","zh":"任务如何设置"}},{"id":"evaluation","source_ids":["home","spec"],"text":{"en":"Deterministic verifiers check the capability flags; an LLM judge does not assign the result. Capability coverage measures achieved flags, not the percentage of vulnerabilities fully exploited. The public results distinguish fixed experiments from a maximum-achievement view with varying seeds and budgets, so preserve the reported aggregation and regime.","zh":"结果由确定性验证器检查能力标记，不由大模型评委打分。能力覆盖率衡量获得的标记，不能解释为完全利用成功的漏洞比例。公开结果区分固定实验与使用不同随机种子、预算的最大表现视图，阅读时应保留原始汇总方式和实验设置。"},"title":{"en":"Read capability coverage carefully","zh":"如何判分"}},{"id":"use_cases","source_ids":["paper","repo"],"text":{"en":"Use it to investigate where progress stops within a difficult software target and to compare agent configurations under controlled conditions. A tier profile can distinguish reliable bug reproduction from later capabilities that a single success percentage would hide, while still crediting intermediate work.","zh":"适合研究智能体在复杂软件目标上推进到哪个阶段，以及在受控条件下比较不同配置。分层结果可以区分稳定的故障复现与更后续的能力，同时记录中间工作，补充单一成功率难以表达的信息。"},"title":{"en":"When it is useful","zh":"适合用来判断什么"}},{"id":"limitations","source_ids":["home","paper","repo"],"text":{"en":"The initial suite is specific to V8 and public patched bugs; it does not establish general zero-day discovery ability. Baseline runs, automated coaching, and native CLI agents use different protocols. Public prior information and runtime conditions can affect results, making matched settings essential for a fair comparison.","zh":"初始任务集限定于 V8 和已公开修复的漏洞，不能据此认定通用零日漏洞发现能力。基线运行、自动引导与原生 CLI 智能体采用不同协议；公开的既有信息和运行环境也可能影响结果，公平比较需要匹配这些条件。"},"title":{"en":"Version and protocol boundaries","zh":"读分时的限制"}}],"sources":[{"id":"home","title":"ExploitBench official methodology and result views","url":"https://exploitbench.ai/"},{"id":"paper","title":"ExploitBench paper","url":"https://arxiv.org/abs/2605.14153"},{"id":"repo","title":"ExploitBench official repository","url":"https://github.com/exploitbench/exploitbench/blob/main/README.md"},{"id":"spec","title":"Official V8 benchmark specification","url":"https://github.com/exploitbench/exploitbench/blob/main/benchmarks/bench-v8/SPEC.md"}],"summary":{"en":"ExploitBench measures stages of progress on known V8 vulnerabilities, using separate capability checks rather than one binary outcome.","zh":"ExploitBench 对已知 V8 漏洞上的不同能力分别验证，呈现阶段性进展，而非仅给出单一成败结果。"},"title":"ExploitBench"},"evidence_summary":{"document_count":2,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:exploitbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["cybersecurity"],"join":"review:14H034160212:2026-10-15","primary":"cybersecurity","source_record":null},"modality":null,"name":"ExploitBench","openness":"unknown","publisher":null,"released":"2026-05-13","released_reference":{"basis":"benchmark_release","source_key":"model-reports:exploitbench","source_url":"https://exploitbench.ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"exploitbench","source":"model_reports","source_url":"https://exploitbench.ai/","topics":["coding","agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_f0c046f65c15f6b8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel. Given a vulnerability and a proof-of-vulnerability input, agents must craft a working end-to-end exploit that achieves unauthorized code execution.","descriptions":{"en":"ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel. Given a vulnerability and a proof-of-vulnerability input, agents must craft a working end-to-end exploit that achieves unauthorized code execution."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:exploitgym:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitgym?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:exploitgym","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"llm_stats_id","primary":"cybersecurity","source_record":"catalog_f0c046f65c15f6b8"},"modality":"text","name":"ExploitGym","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":33.7,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.337,"raw_min":0.124,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:exploitgym:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitgym?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-exploitgym","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitgym?top_n=500","topics":["coding","agents","safety"],"unit":null},{"aliases":["ExploitGym"],"attention_signals":null,"categories":["security"],"collected_at":null,"description":"Exploitation capability gym; different vulnerability classes have very different difficulty profiles.","descriptions":{"en":"Exploitation capability gym; different vulnerability classes have very different difficulty profiles."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000exploitgym\u0000deepseek_v4_1_flash_technical_report\u0000exploitgym\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"first_score_reported_at":"2026-09-10","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000exploitgym\u0000deepseek_v4_1_flash_technical_report\u0000exploitgym\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:exploitgym","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["cybersecurity"],"join":"review:14H034160212:2026-10-15","primary":"cybersecurity","source_record":null},"modality":null,"name":"ExploitGym","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":15.3,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":15.3,"raw_min":15.3,"source_reference":{"obs_id":"curated\u0000exploitgym\u0000deepseek_v4_1_flash_technical_report\u0000exploitgym\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000exploitgym\u0000deepseek_v4_1_flash_technical_report\u0000exploitgym\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"}],"slug":"exploitgym","source":"model_reports","source_url":"https://github.com/sunblaze-ucb/exploitgym","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":30,"join":"artifact:arxiv:2609.30199","repository":null,"source_record":"catalog_2124d88ab5565a6c"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A framework built on verifiable Alien Worlds where rules are executable and every answer can be checked exactly.","descriptions":{"en":"A framework built on verifiable Alien Worlds where rules are executable and every answer can be checked exactly."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.30199","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"artifact:arxiv:2609.30199","primary":"abstract-reasoning","source_record":"catalog_2124d88ab5565a6c"},"modality":null,"name":"ExplorationBench","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30199","source_url":"https://arxiv.org/abs/2609.30199"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30199","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30199","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:leo-liuyao/expomotion","source_record":"bm_expomotion_345f5881"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ExpoMotion is a large-scale benchmark for multi-exposure fusion with dynamic scenes, containing 1,738 sequences and 10,909 images across diverse environments. It provides high-fidelity ground truth for reference-based evaluation and a separate set for no-reference evaluation. The benchmark includes training and testing splits with controlled and real-world motions.","descriptions":{"en":"ExpoMotion is a large-scale benchmark for multi-exposure fusion with dynamic scenes, containing 1,738 sequences and 10,909 images across diverse environments. It provides high-fidelity ground truth for reference-based evaluation and a separate set for no-reference evaluation. The benchmark includes training and testing splits with controlled and real-world motions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.03110","languages":[],"library_direction":null,"modality":null,"name":"ExpoMotion","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03110","source_url":"https://arxiv.org/abs/2607.03110"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03110","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03110","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":107,"github_stars_30d":21,"hf_dataset_downloads":15709,"hf_paper_upvotes":25,"join":"claire_record_id","repository":"gh:run-llama/extractbench","source_record":"bm_extractbench_2ecdf546"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ExtractBench evaluates schema-guided extraction from enterprise documents. Given a document and a user-defined JSON schema, systems must return schema-valid JSON with correct values, include every record of repeated structures, mark missing fields as null, and provide source evidence. The benchmark includes 4,869 pages across 370 documents, 8 business domains, and 67 document types, with tags for challenge, perception, table structure, length, and domain. Scoring uses unified value F1 for value accuracy and word- and page-level F1 for grounding.","descriptions":{"en":"ExtractBench evaluates schema-guided extraction from enterprise documents. Given a document and a user-defined JSON schema, systems must return schema-valid JSON with correct values, include every record of repeated structures, mark missing fields as null, and provide source evidence. The benchmark includes 4,869 pages across 370 documents, 8 business domains, and 67 document types, with tags for challenge, perception, table structure, length, and domain. Scoring uses unified value F1 for value accuracy and word- and page-level F1 for grounding."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.29677","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","professional-work"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_extractbench_2ecdf546"},"modality":null,"name":"ExtractBench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29677","source_url":"https://arxiv.org/abs/2607.29677"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29677","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29677","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":8,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:naver-ai/extremewhenbench","source_record":"bm_extremewhenbench_bffe4798"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ExtremeWhenBench evaluates natural-language temporal grounding in hour-long videos, with 2,273 open-form queries over 194 videos. Metrics include mIoU and R@k for predicted time intervals, with strict parse-failure handling.","descriptions":{"en":"ExtremeWhenBench evaluates natural-language temporal grounding in hour-long videos, with 2,273 open-form queries over 194 videos. Metrics include mIoU and R@k for predicted time intervals, with strict parse-failure handling."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.12300","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_extremewhenbench_bffe4798"},"modality":null,"name":"ExtremeWhenBench","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12300","source_url":"https://arxiv.org/abs/2606.12300"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12300","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12300","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:pkuthm/eyevqa","source_record":"bm_eyevqa_fa1e4100"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insufficiently characterized.","descriptions":{"en":"Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insufficiently characterized."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.32352","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_eyevqa_fa1e4100"},"modality":null,"name":"EyeVQA","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32352","source_url":"https://arxiv.org/abs/2609.32352"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32352","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32352","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_eyt-bench_32cb0b9c"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"EYT-Bench evaluates multi-turn dialogue capabilities of LLMs using a decoupled three-party setup with user simulation, target modeling, and judging.","descriptions":{"en":"EYT-Bench evaluates multi-turn dialogue capabilities of LLMs using a decoupled three-party setup with user simulation, target modeling, and judging."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.10428","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","instruction-following"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_eyt-bench_32cb0b9c"},"modality":null,"name":"EYT-Bench","openness":"unknown","publisher":null,"released":"2026-07-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10428","source_url":"https://arxiv.org/abs/2607.10428"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10428","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10428","topics":["language"],"unit":null},{"aliases":["Measuring in-context algorithmic reasoning in language models against an exact Bayes-optimal standard"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"F-ICL evaluates in-context algorithmic inference against an exactly computed Bayes-optimal posterior under a bounded universal prior.","descriptions":{"en":"F-ICL evaluates in-context algorithmic inference against an exactly computed Bayes-optimal posterior under a bounded universal prior."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01575","languages":[],"library_direction":null,"modality":null,"name":"F-ICL","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01575","source_url":"https://arxiv.org/abs/2608.01575"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01575","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01575","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"lib_facts_grounding"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","factuality","grounding"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A benchmark evaluating language models' ability to generate factually accurate and well-grounded responses based on long-form input context, comprising 1,719 examples with documents up to 32k tokens requiring detailed responses that are fully grounded in provided documents","descriptions":{"en":"A benchmark evaluating language models' ability to generate factually accurate and well-grounded responses based on long-form input context, comprising 1,719 examples with documents up to 32k tokens requiring detailed responses that are fully grounded in provided documents"},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:facts-grounding:gemini-2.0-flash","reported_at":"2024-12-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/facts-grounding?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:facts-grounding","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"lib_facts_grounding"},"modality":"text","name":"FACTS Grounding","openness":"unknown","publication":{"evidence":[{"note":"The original FACTS team announcement identifies Google DeepMind and Google Research as collaborators, links the paper and Kaggle release, and describes 1,719 document-grounded examples with contexts up to 32,000 tokens. These match the source record. It separately documents 860 public and 859 held-out examples.","url":"https://deepmind.google/blog/facts-grounding-a-new-benchmark-for-evaluating-the-factuality-of-large-language-models/"}],"id":"facts-grounding-original","links":[{"kind":"paper","title":"The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input","url":"https://arxiv.org/abs/2501.03200"},{"kind":"website","title":"FACTS Grounding original announcement","url":"https://deepmind.google/blog/facts-grounding-a-new-benchmark-for-evaluating-the-factuality-of-large-language-models/"}],"record_keys":["llm-stats:facts-grounding"],"reviewed_at":"2026-10-26","scope":{"en":"The original FACTS Grounding benchmark for long responses grounded in supplied documents. Its public and held-out sets have different access conditions; other benchmarks in the later FACTS suite are separate tasks.","zh":"原始 FACTS Grounding，评估长篇回答是否忠实依据给定文档。公开集与保留评测集的访问条件不同，后续 FACTS 套件中的其他评测属于不同任务。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":87.8,"display_multiplier":100,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":0.878,"raw_min":0.364,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:facts-grounding:gemini-2.5-pro-preview-06-05","reported_date":"2025-06-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/facts-grounding?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"}],"slug":"llm-stats-facts-grounding","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/facts-grounding?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":460,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:shmsw25/factscore","source_record":"catalog_64e8f9d40676caec"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A fine-grained atomic evaluation metric for factual precision in long-form text generation that breaks generated text into atomic facts and computes the percentage supported by reliable knowledge sources, with automated assessment using retrieval and language models","descriptions":{"en":"A fine-grained atomic evaluation metric for factual precision in long-form text generation that breaks generated text into atomic facts and computes the percentage supported by reliable knowledge sources, with automated assessment using retrieval and language models"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:factscore:gpt-5-2025-08-07","reported_at":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/factscore?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:factscore","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_64e8f9d40676caec"},"modality":"text","name":"FActScore","openness":"unknown","publisher":null,"released":"2023-05-23","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:factscore","source_url":"https://arxiv.org/abs/2305.14251"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":97.03,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.9703,"raw_min":0.01,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:factscore:grok-4.1-2025-11-17","reported_date":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/factscore?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1"}],"slug":"llm-stats-factscore","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/factscore?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_failbench_15b91781"},"categories":["General AI","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A robot failure detection benchmark with 2,197 manipulation attempts from real and simulated sources.","descriptions":{"en":"A robot failure detection benchmark with 2,197 manipulation attempts from real and simulated sources."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.03611","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_failbench_15b91781"},"modality":null,"name":"FailBench","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03611","source_url":"https://arxiv.org/abs/2609.03611"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03611","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03611","topics":[],"unit":null},{"aliases":["failure-transparent-agents"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:junru-zhu/failure-transparent-agents","source_record":"bm_failure-transparent-agents_1c58abe3"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Measures whether LLM agents admit tool failures instead of claiming success across 100 tasks covering web, attachment, execution, permission, and stale-data failures under varying pressure and instruction conditions.","descriptions":{"en":"Measures whether LLM agents admit tool failures instead of claiming success across 100 tasks covering web, attachment, execution, permission, and stale-data failures under varying pressure and instruction conditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:junru-zhu/failure-transparent-agents","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","tool-use"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_failure-transparent-agents_1c58abe3"},"modality":null,"name":"Failure-Transparent Agents","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:junru-zhu/failure-transparent-agents","source_url":"https://github.com/junru-zhu/failure-transparent-agents"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-junru-zhu-failure-transparent-agents","source":"claire_radar","source_url":"https://github.com/junru-zhu/failure-transparent-agents","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark that audits distributive bias in LLM resource allocation, varying task type, comparison context, and audit transparency across 600 financial-assistance requests.","descriptions":{"en":"A benchmark that audits distributive bias in LLM resource allocation, varying task type, comparison context, and audit transparency across 600 financial-assistance requests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.28934","languages":[],"library_direction":null,"modality":null,"name":"FairFund-Bench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28934","source_url":"https://arxiv.org/abs/2607.28934"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28934","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28934","topics":[],"unit":null},{"aliases":["FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:vectorinstitute/fairlens","source_record":"bm_fairlens-benchmarking-fairness-in-vision-l_bcc6b156"},"categories":["General AI","Multimodal","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark and evaluation framework measuring fairness and validity of VLM responses in hiring, legal, and healthcare using over 100K image-question pairs per model across demographic groups.","descriptions":{"en":"A benchmark and evaluation framework measuring fairness and validity of VLM responses in hiring, legal, and healthcare using over 100K image-question pairs per model across demographic groups."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01691","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","safety-alignment","health-medicine"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_fairlens-benchmarking-fairness-in-vision-l_bcc6b156"},"modality":null,"name":"FAIRLENS","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01691","source_url":"https://arxiv.org/abs/2609.01691"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01691","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01691","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:aminurhossain/fairrsfm","source_record":"bm_fairrsfm_e283aef0"},"categories":["General AI","Robustness","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Remote sensing foundation models (RSFMs) are commonly evaluated using aggregate metrics, which can hide systematic performance disparities across ecological regions.","descriptions":{"en":"Remote sensing foundation models (RSFMs) are commonly evaluated using aggregate metrics, which can hide systematic performance disparities across ecological regions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.05790","languages":[],"library_direction":{"basis":"model-review-daily","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_fairrsfm_e283aef0"},"modality":null,"name":"FairRSFM","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05790","source_url":"https://arxiv.org/abs/2610.05790"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05790","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05790","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_faithformbench_8ab938df"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating faithfulness of mathematical chain-of-thought autoformalisation, using perturbed reasoning steps to test validity and invalidity preservation.","descriptions":{"en":"A benchmark for evaluating faithfulness of mathematical chain-of-thought autoformalisation, using perturbed reasoning steps to test validity and invalidity preservation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10916","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_faithformbench_8ab938df"},"modality":null,"name":"FaithformBench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10916","source_url":"https://arxiv.org/abs/2608.10916"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10916","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10916","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:peilin717/fakecontext-bench","source_record":"bm_fakecontextbench_b8700db6"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark with 3,500 cases of pseudoscientific claims across seven domains, scored by a deterministic whole-answer judge.","descriptions":{"en":"Benchmark with 3,500 cases of pseudoscientific claims across seven domains, scored by a deterministic whole-answer judge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.27603","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","factuality"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_fakecontextbench_b8700db6"},"modality":null,"name":"FakeContextBench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27603","source_url":"https://arxiv.org/abs/2609.27603"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27603","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27603","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:cryptoailab/fakei2v-bench","source_record":"bm_fakei2v-bench_02c67b9d"},"categories":["General AI","Vision & 3D","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"FakeI2V-Bench is a benchmark for evaluating image-level and video-level deepfake detectors in video detection scenarios. It comprises 97,548 videos generated by recent generation models and covers multiple content categories, with detectors scored by AUC on detection tasks.","descriptions":{"en":"FakeI2V-Bench is a benchmark for evaluating image-level and video-level deepfake detectors in video detection scenarios. It comprises 97,548 videos generated by recent generation models and covers multiple content categories, with detectors scored by AUC on detection tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.03096","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_fakei2v-bench_02c67b9d"},"modality":null,"name":"FakeI2V-Bench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03096","source_url":"https://arxiv.org/abs/2608.03096"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03096","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03096","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","安全","Safety","大语言模型","LLM","逻辑推理","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Safety alignment approaches in large language models (LLMs) often lead\nto the over-refusal of benign queries, significantly diminishing their utility\nin sensitive scenarios. To address this challenge, we introduce FalseReject,\na comprehensive resource containing 16k seemingly toxic queries accompani FalseReject 构建了包含 16 000 条表面“有毒”但实为良性的查询样本，覆盖 44 个安全相关类别，并提出一种基于图信息的对抗多智能体交互框架，用以生成多样且复杂的提示–响应对，并在响应中引入显式推理链，帮助模型更准确地区分安全与不安全上下文；该工作还为标准指令调优模型和推理导向模型分别准备了专项训练集，并附带人工标注的基准测试集，针对 29 款最先进 LLM 进行了大规模评估，结果表明经 FalseReject 监督微调后，模型在显著减少对良性查询的过度拒绝的同时，不仅未损失整体安全性，也保持了语言生成能力，为敏感场景下提升 LLM 可用性提供了首个系统化、可复现的资源与方法框","descriptions":{"en":"Safety alignment approaches in large language models (LLMs) often lead\nto the over-refusal of benign queries, significantly diminishing their utility\nin sensitive scenarios. To address this challenge, we introduce FalseReject,\na comprehensive resource containing 16k seemingly toxic queries accompani","zh":"FalseReject 构建了包含 16 000 条表面“有毒”但实为良性的查询样本，覆盖 44 个安全相关类别，并提出一种基于图信息的对抗多智能体交互框架，用以生成多样且复杂的提示–响应对，并在响应中引入显式推理链，帮助模型更准确地区分安全与不安全上下文；该工作还为标准指令调优模型和推理导向模型分别准备了专项训练集，并附带人工标注的基准测试集，针对 29 款最先进 LLM 进行了大规模评估，结果表明经 FalseReject 监督微调后，模型在显著减少对良性查询的过度拒绝的同时，不仅未损失整体安全性，也保持了语言生成能力，为敏感场景下提升 LLM 可用性提供了首个系统化、可复现的资源与方法框"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1980","languages":["English"],"library_direction":null,"modality":null,"name":"FalseReject","openness":"unknown","publisher":"Amazon","released":"2025-05-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1980-falsereject","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FalseReject","topics":["reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_fam-bench_1946f5cd"},"categories":["Biology & Drug Discovery","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"FAM-Bench evaluates multimodal language and vision-language models on Food-as-Medicine reasoning. It includes 2500 expert-verified instances across 13 health conditions, with two tasks: dish-level suitability assessment (judging if a dish is suitable for a condition) and comparative dish analysis (ranking four dishes by suitability).","descriptions":{"en":"FAM-Bench evaluates multimodal language and vision-language models on Food-as-Medicine reasoning. It includes 2500 expert-verified instances across 13 health conditions, with two tasks: dish-level suitability assessment (judging if a dish is suitable for a condition) and comparative dish analysis (ranking four dishes by suitability)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.31410","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_fam-bench_1946f5cd"},"modality":null,"name":"FAM-Bench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31410","source_url":"https://arxiv.org/abs/2605.31410"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31410","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31410","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Video super-resolution (VSR) is normally judged by PSNR and SSIM on clips that were downsampled bicubically, although in surveillance its purpose is to make faces and licence plates \\emph{recognisable}.","descriptions":{"en":"Video super-resolution (VSR) is normally judged by PSNR and SSIM on clips that were downsampled bicubically, although in surveillance its purpose is to make faces and licence plates \\emph{recognisable}."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.39649","languages":[],"library_direction":null,"modality":null,"name":"FANVIDv2","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39649","source_url":"https://arxiv.org/abs/2609.39649"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39649","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39649","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":7,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:snowflake-ai-research/fastkernels","source_record":"bm_fastkernels_60e561f1"},"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"FastKernels evaluates GPU kernel generation for production inference across 46 representative architectures spanning 8 categories. It measures correctness and speedup of candidate kernels against reference implementations at multiple abstraction levels, from single-kernel to end-to-end serving.","descriptions":{"en":"FastKernels evaluates GPU kernel generation for production inference across 46 representative architectures spanning 8 categories. It measures correctness and speedup of candidate kernels against reference implementations at multiple abstraction levels, from single-kernel to end-to-end serving."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.23215","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_fastkernels_60e561f1"},"modality":null,"name":"FastKernels","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23215","source_url":"https://arxiv.org/abs/2605.23215"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23215","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23215","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.NI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of 200 troubleshooting scenarios across eight network topologies evaluating network troubleshooting LLM agents under unreliable user tickets, including false fault reports and incorrect device attribution.","descriptions":{"en":"A benchmark of 200 troubleshooting scenarios across eight network topologies evaluating network troubleshooting LLM agents under unreliable user tickets, including false fault reports and incorrect device attribution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27021","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_fault-bench_59abb402"},"modality":null,"name":"FaulT-Bench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27021","source_url":"https://arxiv.org/abs/2608.27021"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27021","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27021","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_fbhm_44ffb389"},"categories":["Cybersecurity","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"FBHM evaluates vision-language models on hateful meme detection across 25 rhetorical functionalities and 10 target communities, with 5,000 memes. Performance is measured by Macro-F1 score on this curated dataset.","descriptions":{"en":"FBHM evaluates vision-language models on hateful meme detection across 25 rhetorical functionalities and 10 target communities, with 5,000 memes. Performance is measured by Macro-F1 score on this curated dataset."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.31349","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_fbhm_44ffb389"},"modality":null,"name":"FBHM","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31349","source_url":"https://arxiv.org/abs/2605.31349"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31349","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31349","topics":["safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes.","descriptions":{"en":"136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27571","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","health-medicine"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_fde-bench_e84f759d"},"modality":null,"name":"FDE-Bench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27571","source_url":"https://arxiv.org/abs/2609.27571"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27571","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27571","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","FEA基准测试","科学智能","AI for Science","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"FEABench is a benchmark to evaluate the ability of large language models (LLMs) and LLM agents to simulate and solve physics, mathematics and engineering problems using finite element analysis (FEA). FEABench，一个用于评估大型语言模型和LLM代理使用有限元分析（FEA）模拟和解决物理、数学及工程问题能力的基准测试。","descriptions":{"en":"FEABench is a benchmark to evaluate the ability of large language models (LLMs) and LLM agents to simulate and solve physics, mathematics and engineering problems using finite element analysis (FEA).","zh":"FEABench，一个用于评估大型语言模型和LLM代理使用有限元分析（FEA）模拟和解决物理、数学及工程问题能力的基准测试。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1733","languages":[],"library_direction":null,"modality":null,"name":"FEABench","openness":"unknown","publisher":"Google Research，Harvard University","released":"2025-04-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1733-feabench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FEABench","topics":["agents","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Data-driven prognostics and health management has emerged as a key enabler for Industry 4.0, yet the development of robust remaining useful life (RUL) estimation models is often limited by the scarcity of run-to-failure data.","descriptions":{"en":"Data-driven prognostics and health management has emerged as a key enabler for Industry 4.0, yet the development of robust remaining useful life (RUL) estimation models is often limited by the scarcity of run-to-failure data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.26433","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_fedcmapss_9057f5e4"},"modality":null,"name":"FedCMAPSS","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26433","source_url":"https://arxiv.org/abs/2608.26433"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26433","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26433","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mobile & Personal Computing","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"This entry describes a benchmark for federated long-tail learning, but no task dataset, scoring protocol, or public code release is evidenced.","descriptions":{"en":"This entry describes a benchmark for federated long-tail learning, but no task dataset, scoring protocol, or public code release is evidenced."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15625","languages":[],"library_direction":null,"modality":null,"name":"FedLTLib","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15625","source_url":"https://arxiv.org/abs/2609.15625"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15625","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15625","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"FedMABench is an open-source benchmark for federated training and evaluation of mobile agents, specifically designed for heterogeneous scenarios. FedMABench 是一个开源的联邦训练和评估移动代理的基准，特别为异构场景设计。","descriptions":{"en":"FedMABench is an open-source benchmark for federated training and evaluation of mobile agents, specifically designed for heterogeneous scenarios.","zh":"FedMABench 是一个开源的联邦训练和评估移动代理的基准，特别为异构场景设计。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1618","languages":[],"library_direction":null,"modality":null,"name":"FedMABench","openness":"unknown","publisher":"ZJU, SJTU, etc.","released":"2025-03-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1618-fedmabench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FedMABench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_fepbench_bbac7f51"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"FEPBench evaluates text-to-image models on natural-science illustration generation using fine-grained atom set annotations, assessing instruction faithfulness, reasoning enrichment, and semantic precision across disciplines.","descriptions":{"en":"FEPBench evaluates text-to-image models on natural-science illustration generation using fine-grained atom set annotations, assessing instruction faithfulness, reasoning enrichment, and semantic precision across disciplines."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05949","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","science"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_fepbench_bbac7f51"},"modality":null,"name":"FEPBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05949","source_url":"https://arxiv.org/abs/2606.05949"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05949","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05949","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_fiducia-bench_925d991c"},"categories":["Agents","Finance","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates governability of financial agents across 626 episodes and 100 KYC/AML task variants, measuring escalation, abstention, and audit trail completeness under different agent architectures.","descriptions":{"en":"Evaluates governability of financial agents across 626 episodes and 100 KYC/AML task variants, measuring escalation, abstention, and audit trail completeness under different agent architectures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.16055","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","safety-alignment"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_fiducia-bench_925d991c"},"modality":null,"name":"Fiducia-bench","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16055","source_url":"https://arxiv.org/abs/2608.16055"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16055","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16055","topics":["agents","safety","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":133,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":"gh:future-house/lab-bench","source_record":"catalog_5db7d2d8065ea818"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FigQA is a multiple-choice benchmark on interpreting scientific figures from biology papers. It evaluates dual-use biological knowledge and multimodal reasoning relevant to bioweapons development.","descriptions":{"en":"FigQA is a multiple-choice benchmark on interpreting scientific figures from biology papers. It evaluates dual-use biological knowledge and multimodal reasoning relevant to bioweapons development."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:figqa:grok-4.1-thinking-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/figqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:figqa","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"llm_stats_id","primary":"science","source_record":"catalog_5db7d2d8065ea818"},"modality":"multimodal","name":"FigQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":89.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.89,"raw_min":0.34,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:figqa:claude-mythos-preview","reported_date":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/figqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-figqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/figqa?top_n=500","topics":["multimodal","safety","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":37,"github_stars_30d":6,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:neo-yk/filmops","source_record":"bm_filmbench_2191702d"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"FilmBench evaluates text-to-video and reference-to-video generation using a Cinematic Language taxonomy with 1,169 prompts and 35 sub-metrics.","descriptions":{"en":"FilmBench evaluates text-to-video and reference-to-video generation using a Cinematic Language taxonomy with 1,169 prompts and 35 sub-metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.24241","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_filmbench_2191702d"},"modality":null,"name":"FilmBench","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24241","source_url":"https://arxiv.org/abs/2607.24241"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24241","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24241","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","大语言模型","LLM","知识储备","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Ant Group and Shanghai University of Finance and Economics jointly launched the financial benchmark，Fin-Eva Version 1.0, covering multiple financial scenarios and subjects such as wealth management, insurance, investment research. The number of this benchmark's questions reaches 13,000+. 蚂蚁集团、上海财经大学联合推出金融评测集Fin-Eva Version 1.0，覆盖财富管理、保险、投资研究等多个金融场景以及金融专业主题学科，总评测题数目达到13,000+。","descriptions":{"en":"Ant Group and Shanghai University of Finance and Economics jointly launched the financial benchmark，Fin-Eva Version 1.0, covering multiple financial scenarios and subjects such as wealth management, insurance, investment research. The number of this benchmark's questions reaches 13,000+.","zh":"蚂蚁集团、上海财经大学联合推出金融评测集Fin-Eva Version 1.0，覆盖财富管理、保险、投资研究等多个金融场景以及金融专业主题学科，总评测题数目达到13,000+。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:895","languages":[],"library_direction":null,"modality":null,"name":"Fin-Eva","openness":"unknown","publisher":null,"released":"2023-12-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-895-fin-eva","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Fin-Eva","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:abhisheksharma2411/finalitybench","source_record":"bm_finalitybench_36a71d92"},"categories":["Agents","Finance","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"FinalityBench evaluates agent decisions in financial exception scenarios where payment systems have inconsistent views due to message faults. It uses a hidden canonical event log and derives system states via faulted delivery. Scoring measures executed monetary effects against a reference outcome.","descriptions":{"en":"FinalityBench evaluates agent decisions in financial exception scenarios where payment systems have inconsistent views due to message faults. It uses a hidden canonical event log and derives system states via faulted delivery. Scoring measures executed monetary effects against a reference outcome."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.04706","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","tool-use"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finalitybench_36a71d92"},"modality":null,"name":"FinalityBench","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04706","source_url":"https://arxiv.org/abs/2609.04706"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04706","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04706","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_180c3adb07bcb406"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","finance","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Finance Agent is a benchmark for evaluating AI models on agentic financial analysis tasks, testing their ability to process financial data, perform calculations, and generate accurate analyses across various financial domains.","descriptions":{"en":"Finance Agent is a benchmark for evaluating AI models on agentic financial analysis tasks, testing their ability to process financial data, perform calculations, and generate accurate analyses across various financial domains."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:finance-agent:claude-opus-4-6","reported_at":"2026-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:finance-agent","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","tool-use"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_180c3adb07bcb406"},"modality":"text","name":"Finance Agent","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":64.4,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.644,"raw_min":0.537,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:finance-agent:claude-opus-4-7","reported_date":"2026-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"}],"slug":"llm-stats-finance-agent","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent?top_n=500","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":164,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_ffa6855012d81d65"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","finance","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Finance Agent v1.1 is an agentic financial-analysis benchmark that evaluates models on real-world finance workflows, including retrieving and reasoning over financial documents and performing multi-step calculations.","descriptions":{"en":"Finance Agent v1.1 is an agentic financial-analysis benchmark that evaluates models on real-world finance workflows, including retrieving and reasoning over financial documents and performing multi-step calculations."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:finance-agent-v1.1:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v1.1?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:finance-agent-v1.1","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","deep-research"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_ffa6855012d81d65"},"modality":"text","name":"Finance Agent v1.1","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":60.699999999999996,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.607,"raw_min":0.56,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:finance-agent-v1.1:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v1.1?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-finance-agent-v1-1","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v1.1?top_n=500","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":70,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:vals-ai/finance-agent-v2","source_record":"catalog_2cbc1aa62e647719"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","finance","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Finance Agent v2 is an agentic financial-analysis benchmark from Vals that evaluates models on real-world finance workflows, measuring their ability to retrieve and reason over financial documents, perform multi-step calculations, and produce accurate analyses.","descriptions":{"en":"Finance Agent v2 is an agentic financial-analysis benchmark from Vals that evaluates models on real-world finance workflows, measuring their ability to retrieve and reason over financial documents, perform multi-step calculations, and produce accurate analyses."},"evidence_summary":{"document_count":1,"model_count":26,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:finance-agent-v2:claude-haiku-4-5-20251001","reported_at":"2025-10-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:finance-agent-v2","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","tool-use"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_2cbc1aa62e647719"},"modality":"text","name":"Finance Agent v2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":26,"score_direction":"higher_is_better","score_summary":{"display_max":57.86,"display_multiplier":100,"model_count":26,"model_count_basis":"source_model_id","numeric_count":26,"raw_max":0.5786,"raw_min":0.2789,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:finance-agent-v2:gemini-3.5-flash","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-05-06","date_precision":"model_announcement","name":"Grok 4.3"},{"date":"2026-03-09","date_precision":"model_announcement","name":"Grok-4.20 Beta Reasoning"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-finance-agent-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v2?top_n=500","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:gauthamrohan/financial-sentiment-benchmark","source_record":"bm_financial-sentiment-benchmark_ea047ce6"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates financial news sentiment classification comparing TF-IDF, DistilBERT fine-tuned, QLoRA, and prompted GPT-5-nano on a 300-sentence test subset using macro-F1.","descriptions":{"en":"Evaluates financial news sentiment classification comparing TF-IDF, DistilBERT fine-tuned, QLoRA, and prompted GPT-5-nano on a 300-sentence test subset using macro-F1."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:gauthamrohan/financial-sentiment-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_financial-sentiment-benchmark_ea047ce6"},"modality":null,"name":"financial-sentiment-benchmark","openness":"unknown","publisher":null,"released":"2026-08-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:gauthamrohan/financial-sentiment-benchmark","source_url":"https://github.com/gauthamRohan/financial-sentiment-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-gauthamrohan-financial-sentiment-benchmark","source":"claire_radar","source_url":"https://github.com/gauthamRohan/financial-sentiment-benchmark","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_financialauditbench_936e0cf0"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"As AI agents are becoming widely adopted in the financial services industry, careful measurement is essential to understand where they can be reliably deployed and where oversight and professional review remain necessary.","descriptions":{"en":"As AI agents are becoming widely adopted in the financial services industry, careful measurement is essential to understand where they can be reliably deployed and where oversight and professional review remain necessary."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32835","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_financialauditbench_936e0cf0"},"modality":null,"name":"FinancialAuditBench","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32835","source_url":"https://arxiv.org/abs/2609.32835"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32835","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32835","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:devansh1105/finbalance","source_record":"bm_finbalance_0aab2d5b"},"categories":["Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"FinBalance is a multi-document accounting reconciliation benchmark built from 710 source-document bundles across eight industries, three period types, and five difficulty levels. It evaluates models on producing journal entries, balance sheets, and inconsistency labels, with a deterministic ledger for ground truth.","descriptions":{"en":"FinBalance is a multi-document accounting reconciliation benchmark built from 710 source-document bundles across eight industries, three period types, and five difficulty levels. It evaluates models on producing journal entries, balance sheets, and inconsistency labels, with a deterministic ledger for ground truth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.15949","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finbalance_0aab2d5b"},"modality":null,"name":"FinBalance","openness":"unknown","publisher":null,"released":"2026-06-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15949","source_url":"https://arxiv.org/abs/2606.15949"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15949","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15949","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Safety & Trustworthiness","stat.AP"],"collected_at":"2026-10-14T09:44:51Z","description":"FinBench is a benchmark for evaluating calibration and uncertainty in financial forecasting with time-gated tasks, requiring probability of positive return and 80% prediction interval, scored with Brier and Winkler scores.","descriptions":{"en":"FinBench is a benchmark for evaluating calibration and uncertainty in financial forecasting with time-gated tasks, requiring probability of positive return and 80% prediction interval, scored with Brier and Winkler scores."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.16229","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finbench_34ca17f0"},"modality":null,"name":"FinBench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.16229","source_url":"https://arxiv.org/abs/2607.16229"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-16229","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16229","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"FinBoardBench is an evaluation suite based on three financial board games: Cashflow, Acquire, and Monopoly. It assesses financial skills including personal cash flow management, corporate investment and acquisition forecasting, and competitive trade negotiations with asset auctions, using game simulations as the environment.","descriptions":{"en":"FinBoardBench is an evaluation suite based on three financial board games: Cashflow, Acquire, and Monopoly. It assesses financial skills including personal cash flow management, corporate investment and acquisition forecasting, and competitive trade negotiations with asset auctions, using game simulations as the environment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27896","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","finance-legal"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_finboardbench_0e7661a6"},"modality":null,"name":"FinBoardBench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27896","source_url":"https://arxiv.org/abs/2605.27896"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27896","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27896","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:fengxianji/fincuabuild","source_record":"bm_fincuabuildbench_238a4701"},"categories":["Agents & Tool Use","Finance","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates construction of dynamic financial computer-using agent (CUA) tasks across 576 requests spanning 24 workflows and runtime variations, with qualification based on execution tests and quality checks.","descriptions":{"en":"Evaluates construction of dynamic financial computer-using agent (CUA) tasks across 576 requests spanning 24 workflows and runtime variations, with qualification based on execution tests and quality checks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.07603","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","finance-legal"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_fincuabuildbench_238a4701"},"modality":null,"name":"FinCUABuildBench","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07603","source_url":"https://arxiv.org/abs/2609.07603"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07603","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07603","topics":["computer-use","agents","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_findeepindicator_6effb730"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FinDeepIndicator evaluates deep research agents on end-to-end financial indicator construction across four stages: formula specification, data collection, indicator calculation, and answer generation. It includes 3,350 QA pairs from U.S. and Chinese markets, 10 years of historical data, and 800 listed companies.","descriptions":{"en":"FinDeepIndicator evaluates deep research agents on end-to-end financial indicator construction across four stages: formula specification, data collection, indicator calculation, and answer generation. It includes 3,350 QA pairs from U.S. and Chinese markets, 10 years of historical data, and 800 listed companies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00764","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","deep-research"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_findeepindicator_6effb730"},"modality":null,"name":"FinDeepIndicator","openness":"unknown","publisher":null,"released":"2026-08-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00764","source_url":"https://arxiv.org/abs/2608.00764"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00764","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00764","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:hedyhe/fined-bench","source_record":"bm_fined-bench_3f021fe1"},"categories":["Finance","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"FinED-Bench evaluates error detection in financial documents across nine real-world financial scenarios, covering three cognitive complexity levels. It includes over 900 documents from 2025, with data and code publicly available.","descriptions":{"en":"FinED-Bench evaluates error detection in financial documents across nine real-world financial scenarios, covering three cognitive complexity levels. It includes over 900 documents from 2025, with data and code publicly available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.12342","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","document-ocr"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_fined-bench_3f021fe1"},"modality":null,"name":"FinED-Bench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12342","source_url":"https://arxiv.org/abs/2608.12342"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12342","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12342","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"FineSightBench probes fine-scale visual perception in VLMs across scales from 4 to 48 pixels, separating perception and reasoning tasks.","descriptions":{"en":"FineSightBench probes fine-scale visual perception in VLMs across scales from 4 to 48 pixels, separating perception and reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07861","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_finesightbench_86226917"},"modality":null,"name":"FineSightBench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07861","source_url":"https://arxiv.org/abs/2606.07861"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07861","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07861","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Finance","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"FinEvo-Bench evaluates self-evolving agents on 120 real-case-grounded tasks across 20 business scenes in six financial domains, with institution-provided procedures and rubrics for quality and compliance.","descriptions":{"en":"FinEvo-Bench evaluates self-evolving agents on 120 real-case-grounded tasks across 20 business scenes in six financial domains, with institution-provided procedures and rubrics for quality and compliance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06144","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","finance-legal"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_finevo-bench_09755690"},"modality":null,"name":"FinEvo-Bench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06144","source_url":"https://arxiv.org/abs/2608.06144"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06144","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06144","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Finance","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"FinEvolveBench evaluates self-evolving agents on low-repetition financial prediction tasks with implicit rewards, measuring utility updates over delayed returns.","descriptions":{"en":"FinEvolveBench evaluates self-evolving agents on low-repetition financial prediction tasks with implicit rewards, measuring utility updates over delayed returns."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06960","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","finance-legal"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_finevolvebench_f7d5f81e"},"modality":null,"name":"FinEvolveBench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06960","source_url":"https://arxiv.org/abs/2606.06960"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06960","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06960","topics":["agents","finance"],"unit":null},{"aliases":["FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"A large-scale annotated dataset adding culture-language-region labels to existing web corpora.","descriptions":{"en":"A large-scale annotated dataset adding culture-language-region labels to existing web corpora."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.25298","languages":[],"library_direction":null,"modality":null,"name":"FineWeb-CLaR","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25298","source_url":"https://arxiv.org/abs/2609.25298"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25298","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25298","topics":["safety"],"unit":null},{"aliases":["FinExam-10K: When Retrieval Helps Financial Reasoning?"],"attention_signals":null,"categories":["Factuality","Finance","Information retrieval","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates financial reasoning with 10,198 expert-reannotated questions across CFA and FRM exams, including a public set of 5,110 questions and a sequestered set of 5,088 for a maintained leaderboard, with tracks for full coverage and context-complete reasoning.","descriptions":{"en":"Evaluates financial reasoning with 10,198 expert-reannotated questions across CFA and FRM exams, including a public set of 5,110 questions and a sequestered set of 5,088 for a maintained leaderboard, with tracks for full coverage and context-complete reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.28155","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","finance-legal"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_finexam-10k-when-retrieval-helps-financial_fa6458fb"},"modality":null,"name":"FinExam-10K","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28155","source_url":"https://arxiv.org/abs/2608.28155"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28155","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28155","topics":["reasoning-math","finance"],"unit":null},{"aliases":["FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":3022,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_finfirst-benchmarking-search-agents-for-fi_24b653bb"},"categories":["Finance","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"An open benchmark of 123 financial research tasks evaluated with atomic rubrics across information acquisition, source verification, and computation.","descriptions":{"en":"An open benchmark of 123 financial research tasks evaluated with atomic rubrics across information acquisition, source verification, and computation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.25192","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","deep-research"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finfirst-benchmarking-search-agents-for-fi_24b653bb"},"modality":null,"name":"FinFIRST","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25192","source_url":"https://arxiv.org/abs/2609.25192"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25192","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25192","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"FinFraudBench is a heterogeneous graph benchmark for financial fraud detection. It contains two datasets (CreditCard-Fraud and BankTrans-Fraud) with up to 8.99M nodes and 89.23M directed typed edges, preserving six financial entity types and fourteen edge types. The evaluation protocol covers ranking and imbalance-sensitive classification metrics.","descriptions":{"en":"FinFraudBench is a heterogeneous graph benchmark for financial fraud detection. It contains two datasets (CreditCard-Fraud and BankTrans-Fraud) with up to 8.99M nodes and 89.23M directed typed edges, preserving six financial entity types and fourteen edge types. The evaluation protocol covers ranking and imbalance-sensitive classification metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.15177","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finfraudbench_fc16410f"},"modality":null,"name":"FinFraudBench","openness":"unknown","publisher":null,"released":"2026-08-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15177","source_url":"https://arxiv.org/abs/2608.15177"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15177","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15177","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","Finance","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation focuses on financial regulatory non-compliance detection in LLM interactions. It involves a benchmark with expert-annotated labels at query and response levels, but the exact task, environment, or scoring setup is not detailed.","descriptions":{"en":"The evaluation focuses on financial regulatory non-compliance detection in LLM interactions. It involves a benchmark with expert-annotated labels at query and response levels, but the exact task, environment, or scoring setup is not detailed."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29427","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","safety-alignment"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finguard-bench_2ed68349"},"modality":null,"name":"FinGuard-Bench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29427","source_url":"https://arxiv.org/abs/2605.29427"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29427","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29427","topics":["safety","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark of 33 financial FPGA computing tasks that tests LLM hardware generation, system-level configuration tuning, and adaptation to specification changes, scoring both functional correctness and latency.","descriptions":{"en":"A benchmark of 33 financial FPGA computing tasks that tests LLM hardware generation, system-level configuration tuning, and adaptation to specification changes, scoring both functional correctness and latency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.00909","languages":[],"library_direction":null,"modality":null,"name":"FinHardBench","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00909","source_url":"https://arxiv.org/abs/2608.00909"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00909","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00909","topics":["finance"],"unit":null},{"aliases":["FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering"],"attention_signals":null,"categories":["Agents","Finance","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"The benchmark evaluates ambiguous financial question answering with clarification and intent integration across a five-category ambiguity taxonomy.","descriptions":{"en":"The benchmark evaluates ambiguous financial question answering with clarification and intent integration across a five-category ambiguity taxonomy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24002","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal","knowledge-expert-qa","computer-use"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_fininteract-benchmarking-clarification-and_19b337d0"},"modality":null,"name":"FinInteract","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24002","source_url":"https://arxiv.org/abs/2609.24002"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24002","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24002","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A financial document parsing evaluation suite covering diverse document scenarios, but release details and public access are unclear.","descriptions":{"en":"A financial document parsing evaluation suite covering diverse document scenarios, but release details and public access are unclear."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22842","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["document-ocr","finance-legal"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_finixdocbench_1d95d2be"},"modality":null,"name":"FinixDocBench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22842","source_url":"http://arxiv.org/abs/2608.22842v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22842","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22842v1","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Two tasks over cumulative longitudinal banking dialogue: reconstructing every life-event instance with its first-establishing session and reconstructing a complete 34-path financial state at consecutive checkpoints.","descriptions":{"en":"Two tasks over cumulative longitudinal banking dialogue: reconstructing every life-event instance with its first-establishing session and reconstructing a complete 34-path financial state at consecutive checkpoints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01198","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finlifebench_adb959e3"},"modality":null,"name":"FinLifeBench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01198","source_url":"https://arxiv.org/abs/2609.01198"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01198","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01198","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FinPerMA evaluates personalized memory in LLM agents using frozen longitudinal investor trajectories with theory-informed impact rules. It includes a Post-Shock checkpoint to assess integration of material events into persistent user models, with 2,994 questions from 276 personas.","descriptions":{"en":"FinPerMA evaluates personalized memory in LLM agents using frozen longitudinal investor trajectories with theory-informed impact rules. It includes a Post-Shock checkpoint to assess integration of material events into persistent user models, with 2,994 questions from 276 personas."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04095","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","finance-legal"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_finperma_25dab076"},"modality":null,"name":"FinPerMA","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04095","source_url":"https://arxiv.org/abs/2608.04095"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04095","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04095","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Finance","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates the longitudinal stability of behavioral mandates in LLM-based financial agents using a synthetic market simulation that decouples observable price from hidden fundamental value, scoring mandate adherence across calm, crash, and bubble market regimes.","descriptions":{"en":"Evaluates the longitudinal stability of behavioral mandates in LLM-based financial agents using a synthetic market simulation that decouples observable price from hidden fundamental value, scoring mandate adherence across calm, crash, and bubble market regimes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.31522","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","safety-alignment"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finpersona-bench_6543120b"},"modality":null,"name":"FinPersona-Bench","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31522","source_url":"https://arxiv.org/abs/2606.31522"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31522","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31522","topics":["agents","safety","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FinProBench evaluates financial AI agents on professional tasks using rubrics derived from practitioner deliverables. It includes 1,723 curated deliverables across 57 occupations and an evaluation set of 20 tasks.","descriptions":{"en":"FinProBench evaluates financial AI agents on professional tasks using rubrics derived from practitioner deliverables. It includes 1,723 curated deliverables across 57 occupations and an evaluation set of 20 tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04077","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finprobench_b6fc7343"},"modality":null,"name":"FinProBench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04077","source_url":"https://arxiv.org/abs/2608.04077"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04077","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04077","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":402,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_16edc7ea458bb8f7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","finance","economics"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A large-scale dataset for numerical reasoning over financial data with question-answering pairs written by financial experts, featuring complex numerical reasoning and understanding of heterogeneous representations with annotated gold reasoning programs for full explainability","descriptions":{"en":"A large-scale dataset for numerical reasoning over financial data with question-answering pairs written by financial experts, featuring complex numerical reasoning and understanding of heterogeneous representations with annotated gold reasoning programs for full explainability"},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:finqa:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:finqa","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","math"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_16edc7ea458bb8f7"},"modality":"text","name":"FinQA","openness":"unknown","publisher":null,"released":"2021-09-01","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: FinQA: A Dataset of Numerical Reasoning over Financial Data. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:finqa","source_url":"https://arxiv.org/abs/2109.00122"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":77.2,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.772,"raw_min":0.652,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:finqa:nova-pro","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"}],"slug":"llm-stats-finqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finqa?top_n=500","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:datanxt/finrank","source_record":"bm_finrank_f391afeb"},"categories":["Finance","Information retrieval","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FinRank evaluates passage retrieval, reranking, and hard-negative discrimination over SEC filings, with 1185 question-answer records and curated hard negatives, scoring recall and accuracy.","descriptions":{"en":"FinRank evaluates passage retrieval, reranking, and hard-negative discrimination over SEC filings, with 1185 question-answer records and curated hard negatives, scoring recall and accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.07400","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","finance-legal"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_finrank_f391afeb"},"modality":null,"name":"FinRank","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07400","source_url":"https://arxiv.org/abs/2608.07400"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07400","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07400","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:pratikghawate/finrca-ai-bench","source_record":"bm_finrca-bench_2d78ed7b"},"categories":["Finance","Information retrieval","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"FinRCA-Bench is a synthetic benchmark for financial reconciliation root-cause analysis, with 2,250 cases across 14 tables and 15 failure categories, providing evidence retrieval and reasoning evaluation.","descriptions":{"en":"FinRCA-Bench is a synthetic benchmark for financial reconciliation root-cause analysis, with 2,250 cases across 14 tables and 15 failure categories, providing evidence retrieval and reasoning evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.18534","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finrca-bench_2d78ed7b"},"modality":null,"name":"FinRCA-Bench","openness":"unknown","publisher":null,"released":"2026-08-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.18534","source_url":"https://arxiv.org/abs/2608.18534"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-18534","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18534","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:misterbrookt/finreportbench","source_record":"bm_finreportbench_5d3a5df7"},"categories":["Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"FinReportBench evaluates institution-grade financial report generation using 244 bilingual tasks sourced from 10,000 financial research records. It uses a 35-item rubric covering deliverability, report identity, and institutional completeness, with three judge families.","descriptions":{"en":"FinReportBench evaluates institution-grade financial report generation using 244 bilingual tasks sourced from 10,000 financial research records. It uses a 35-item rubric covering deliverability, report identity, and institutional completeness, with three judge families."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04374","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","deep-research"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finreportbench_5d3a5df7"},"modality":null,"name":"FinReportBench","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04374","source_url":"https://arxiv.org/abs/2608.04374"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04374","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04374","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates Chinese financial LLMs on static operation execution across 9,742 instances and evidence-state control via FinRisk-Ask using pre-action trajectory states.","descriptions":{"en":"Evaluates Chinese financial LLMs on static operation execution across 9,742 instances and evidence-state control via FinRisk-Ask using pre-action trajectory states."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25325","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finriskatlas_4424cb99"},"modality":null,"name":"FinRiskAtlas","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25325","source_url":"https://arxiv.org/abs/2608.25325"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25325","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25325","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":20,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"llm_stats_id","repository":"gh:randomtutu/finsearchcomp","source_record":"catalog_d0a57b04e138eb38"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search","finance","economics"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FinSearchComp T2&T3 is a combined benchmark for evaluating financial search and reasoning capabilities on Tier 2 and Tier 3 tasks, testing models' ability to retrieve and analyze complex financial information using tools.","descriptions":{"en":"FinSearchComp T2&T3 is a combined benchmark for evaluating financial search and reasoning capabilities on Tier 2 and Tier 3 tasks, testing models' ability to retrieve and analyze complex financial information using tools."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:finsearchcomp-t2-t3:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t2-t3?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:finsearchcomp-t2-t3","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","finance-legal"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_d0a57b04e138eb38"},"modality":"text","name":"FinSearchComp T2&T3","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":67.80000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.678,"raw_min":0.678,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:finsearchcomp-t2-t3:kimi-k2.5","reported_date":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t2-t3?top_n=500"},"unit":null},"scored_models":[{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"}],"slug":"llm-stats-finsearchcomp-t2-t3","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t2-t3?top_n=500","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":20,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"llm_stats_id","repository":"gh:randomtutu/finsearchcomp","source_record":"catalog_38473fc736dd8308"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search","finance","economics"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FinSearchComp-T3 is a benchmark for evaluating financial search and reasoning capabilities, testing models' ability to retrieve and analyze financial information using tools.","descriptions":{"en":"FinSearchComp-T3 is a benchmark for evaluating financial search and reasoning capabilities, testing models' ability to retrieve and analyze financial information using tools."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:finsearchcomp-t3:kimi-k2-thinking-0905","reported_at":"2025-09-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t3?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:finsearchcomp-t3","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","finance-legal"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_38473fc736dd8308"},"modality":"text","name":"FinSearchComp-T3","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":47.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.474,"raw_min":0.474,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:finsearchcomp-t3:kimi-k2-thinking-0905","reported_date":"2025-09-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t3?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"}],"slug":"llm-stats-finsearchcomp-t3","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t3?top_n=500","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Factuality","Finance","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Financial AI agents must do more than retrieve facts: investment workflows require correct quantitative execution, reliable use of procedural resources, and auditable structured outputs.","descriptions":{"en":"Financial AI agents must do more than retrieve facts: investment workflows require correct quantitative execution, reliable use of procedural resources, and auditable structured outputs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.03564","languages":[],"library_direction":{"basis":"model-review-daily","categories":["finance-legal","professional-work"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finskillbench_5d6f36ff"},"modality":null,"name":"FinSkillBench","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03564","source_url":"https://arxiv.org/abs/2610.03564"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03564","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03564","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","q-fin.CP"],"collected_at":"2026-10-14T09:44:51Z","description":"FinStressTS is a synthetic benchmark for financial time-series forecasting with controllable mechanisms and diagnostic environments.","descriptions":{"en":"FinStressTS is a synthetic benchmark for financial time-series forecasting with controllable mechanisms and diagnostic environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03184","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finstressts_aacbcf42"},"modality":null,"name":"FinStressTS","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03184","source_url":"https://arxiv.org/abs/2606.03184"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03184","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03184","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FinVerBench evaluates financial statement verification in LLMs, using SEC 10-K XBRL filings from 43 S&P 500 companies with a four-category error taxonomy (arithmetic, cross-statement linkage, year-over-year, magnitude perturbations).","descriptions":{"en":"FinVerBench evaluates financial statement verification in LLMs, using SEC 10-K XBRL filings from 43 S&P 500 companies with a four-category error taxonomy (arithmetic, cross-statement linkage, year-over-year, magnitude perturbations)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29586","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finverbench_8fc73dd6"},"modality":null,"name":"FinVerBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29586","source_url":"https://arxiv.org/abs/2605.29586"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29586","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29586","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"FinVerse is a finance-domain time-series forecasting benchmark with 116,897 series (171.1M observations), selecting 60,232 as evaluation targets. It defines 11 metric families (78 metrics) tailored to each series' economic meaning.","descriptions":{"en":"FinVerse is a finance-domain time-series forecasting benchmark with 116,897 series (171.1M observations), selecting 60,232 as evaluation targets. It defines 11 metric families (78 metrics) tailored to each series' economic meaning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03259","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_finverse_651ab4a4"},"modality":null,"name":"FinVerse","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03259","source_url":"https://arxiv.org/abs/2608.03259"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03259","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03259","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates complex physical world intelligence in multimodal large language models and agents through coupled-field fire dynamics.","descriptions":{"en":"Evaluates complex physical world intelligence in multimodal large language models and agents through coupled-field fire dynamics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23064","languages":[],"library_direction":null,"modality":null,"name":"FireWorldBench","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23064","source_url":"https://arxiv.org/abs/2609.23064"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23064","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23064","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":null,"source_record":"bm_firm-video-bench_74b2abde"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"FIRM-Video-Bench is a benchmark of 250 text-to-video generations with 750 point-wise human annotations covering Instruction Following, World Coherence, and Perceptual Quality.","descriptions":{"en":"FIRM-Video-Bench is a benchmark of 250 text-to-video generations with 750 point-wise human annotations covering Instruction Following, World Coherence, and Perceptual Quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.21839","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_firm-video-bench_74b2abde"},"modality":null,"name":"FIRM-Video-Bench","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.21839","source_url":"http://arxiv.org/abs/2608.21839v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-21839","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21839v1","topics":[],"unit":null},{"aliases":["FirmCORe: A Benchmark for Structured Reasoning about Inter-Firm Collaboration Opportunities"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"FirmCORe is a human-annotated benchmark of 2,805 firm pairs for reasoning about inter-firm collaboration opportunities.","descriptions":{"en":"FirmCORe is a human-annotated benchmark of 2,805 firm pairs for reasoning about inter-firm collaboration opportunities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17128","languages":[],"library_direction":null,"modality":null,"name":"FirmCORe","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17128","source_url":"https://arxiv.org/abs/2609.17128"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17128","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17128","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_fitaqa_e0a97bca"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"FitAQA evaluates fitness action quality assessment in MLLMs across perception, judgement, and temporal grounding tasks, using a unified taxonomy of 38 form errors in six dimensions over 30 exercises.","descriptions":{"en":"FitAQA evaluates fitness action quality assessment in MLLMs across perception, judgement, and temporal grounding tasks, using a unified taxonomy of 38 form errors in six dimensions over 30 exercises."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08736","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","health-medicine"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_fitaqa_e0a97bca"},"modality":null,"name":"FitAQA","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08736","source_url":"https://arxiv.org/abs/2608.08736"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08736","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08736","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":562,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_52d35bcd0e93306f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","code","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Flame-VLM-Code evaluates multimodal models on visual code generation tasks, measuring ability to generate code from visual inputs such as UI mockups and design specifications.","descriptions":{"en":"Flame-VLM-Code evaluates multimodal models on visual code generation tasks, measuring ability to generate code from visual inputs such as UI mockups and design specifications."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:flame-vlm-code:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/flame-vlm-code?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:flame-vlm-code","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_52d35bcd0e93306f"},"modality":"multimodal","name":"Flame-VLM-Code","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":93.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.938,"raw_min":0.938,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:flame-vlm-code:glm-5v-turbo","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/flame-vlm-code?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"}],"slug":"llm-stats-flame-vlm-code","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/flame-vlm-code?top_n=500","topics":["coding","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","NAACL 2024","大语言模型","LLM","安全对齐","Safety Alignment","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Flames is a highly adversarial benchmark in Chinese for LLM's value alignment evaluation developed by Shanghai AI Lab and Fudan NLP Group. Flames meticulously designs a dataset of 2,251 highly adversarial, manually crafted prompts, each tailored to probe a specific value dimension (i.e., Fairness, Safety, Morality, Legality, Data protection). Currently,  Flames releases 1,000 prompts for public use (Flames_1k_Chinese). Flames 是上海人工智能实验室和复旦大学 NLP团队开发的 LLM 价值对齐方向的中文高度对抗性基准。Flames 精心设计了一个由 2,251 个高度对抗性、人工创建的提示词成的评测集，每个提示词都经过精心设计，以探究特定的价值维度（即公平、安全、道德、合法、数据保护）。目前，Flames 发布了 1,000 个提示词供公众使用（Flames_1k_Chinese）。","descriptions":{"en":"Flames is a highly adversarial benchmark in Chinese for LLM's value alignment evaluation developed by Shanghai AI Lab and Fudan NLP Group. Flames meticulously designs a dataset of 2,251 highly adversarial, manually crafted prompts, each tailored to probe a specific value dimension (i.e., Fairness, Safety, Morality, Legality, Data protection). Currently,  Flames releases 1,000 prompts for public use (Flames_1k_Chinese).","zh":"Flames 是上海人工智能实验室和复旦大学 NLP团队开发的 LLM 价值对齐方向的中文高度对抗性基准。Flames 精心设计了一个由 2,251 个高度对抗性、人工创建的提示词成的评测集，每个提示词都经过精心设计，以探究特定的价值维度（即公平、安全、道德、合法、数据保护）。目前，Flames 发布了 1,000 个提示词供公众使用（Flames_1k_Chinese）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:945","languages":["Chinese"],"library_direction":null,"modality":null,"name":"Flames","openness":"restricted","publisher":null,"released":"2024-03-13","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-945-flames","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Flames","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":8,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mobiiin/wildfire_vqa","source_record":"bm_flamevqa_fd5c1a64"},"categories":["General AI","Reasoning","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"FlameVQA is a multiple-choice visual question answering benchmark for UAV-based wildfire monitoring, built on FLAME 3 with paired RGB and radiometric thermal images. It includes 34 questions per image across six capability groups, covering detection, localization, coverage estimation, cross-modal reasoning, and flight planning. Labels are generated via MLLM assistance, deterministic thermal rules, and human auditing. The dataset and code are open-source.","descriptions":{"en":"FlameVQA is a multiple-choice visual question answering benchmark for UAV-based wildfire monitoring, built on FLAME 3 with paired RGB and radiometric thermal images. It includes 34 questions per image across six capability groups, covering detection, localization, coverage estimation, cross-modal reasoning, and flight planning. Labels are generated via MLLM assistance, deterministic thermal rules, and human auditing. The dataset and code are open-source."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.27128","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_flamevqa_fd5c1a64"},"modality":null,"name":"FlameVQA","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27128","source_url":"https://arxiv.org/abs/2606.27128"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27128","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27128","topics":["reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Experimental investigation and modeling of flapping-wing aerial vehicles are limited by the scarcity of large-scale records that temporally align wing kinematics, aerodynamic responses, and flight states.","descriptions":{"en":"Experimental investigation and modeling of flapping-wing aerial vehicles are limited by the scarcity of large-scale records that temporally align wing kinematics, aerodynamic responses, and flight states."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38719","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_flapkad_0375251a"},"modality":null,"name":"FlapKAD","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38719","source_url":"https://arxiv.org/abs/2609.38719"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38719","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38719","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FlatLab is a simulation-based benchmark for robotic manipulation of flat objects, providing physical simulation, data collection, and standardized task definitions and evaluation protocols.","descriptions":{"en":"FlatLab is a simulation-based benchmark for robotic manipulation of flat objects, providing physical simulation, data collection, and standardized task definitions and evaluation protocols."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14049","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_flatlab_52af74dd"},"modality":null,"name":"FlatLab","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14049","source_url":"https://arxiv.org/abs/2608.14049"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14049","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14049","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":57,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"llm_stats_id","repository":null,"source_record":"catalog_3628c3b003f98722"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Flexible Length Question Answering dataset for evaluating the impact of input length on reasoning performance of language models, featuring True/False questions embedded in contexts of varying lengths (250-3000 tokens) across three reasoning tasks: Monotone Relations, People In Rooms, and simplified Ruletaker","descriptions":{"en":"Flexible Length Question Answering dataset for evaluating the impact of input length on reasoning performance of language models, featuring True/False questions embedded in contexts of varying lengths (250-3000 tokens) across three reasoning tasks: Monotone Relations, People In Rooms, and simplified Ruletaker"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:flenqa:phi-4-reasoning","reported_at":"2025-04-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/flenqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:flenqa","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","abstract-reasoning"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_3628c3b003f98722"},"modality":"text","name":"FlenQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":97.89999999999999,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.979,"raw_min":0.977,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:flenqa:phi-4-reasoning-plus","reported_date":"2025-04-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/flenqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"}],"slug":"llm-stats-flenqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/flenqa?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_a23faa7dfe648a78"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["speech_to_text","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Few-shot Learning Evaluation of Universal Representations of Speech - a parallel speech dataset in 102 languages built on FLoRes-101 with approximately 12 hours of speech supervision per language for tasks including ASR, speech language identification, translation and retrieval. Scores are shown as speech recognition accuracy (1 - word error rate), so higher is better.","descriptions":{"en":"Few-shot Learning Evaluation of Universal Representations of Speech - a parallel speech dataset in 102 languages built on FLoRes-101 with approximately 12 hours of speech supervision per language for tasks including ASR, speech language identification, translation and retrieval. Scores are shown as speech recognition accuracy (1 - word error rate), so higher is better."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:fleurs:gemini-1.0-pro","reported_at":"2024-02-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fleurs?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:fleurs","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_a23faa7dfe648a78"},"modality":"audio","name":"FLEURS","openness":"unknown","publisher":null,"released":"2022-05-25","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:fleurs","source_url":"https://arxiv.org/abs/2205.12446"},"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":95.89999999999999,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.959,"raw_min":0.864,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:fleurs:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fleurs?top_n=500"},"unit":null},"scored_models":[{"date":"2024-02-15","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-fleurs","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fleurs?top_n=500","topics":["multimodal","language"],"unit":null},{"aliases":["Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"FLEX evaluates expert routing across a 34-task long-horizon multimodal continual instruction-tuning sequence with reduced task-identifying textual fingerprints.","descriptions":{"en":"FLEX evaluates expert routing across a 34-task long-horizon multimodal continual instruction-tuning sequence with reduced task-identifying textual fingerprints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.01437","languages":[],"library_direction":null,"modality":null,"name":"FLEX","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01437","source_url":"https://arxiv.org/abs/2608.01437"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01437","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01437","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:joao97ribeiro/floatbench","source_record":"bm_floatbench_4e9fbef2"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FLOATBench is a tabular benchmark for surrogate modeling of tower fatigue in 22 MW floating offshore wind turbines. It provides 582,120 per-section fatigue damage labels from OpenFAST simulations across three tower geometries, with regime-aware splits and three evaluation protocols (random, within-tower regime-aware, cross-tower transfer).","descriptions":{"en":"FLOATBench is a tabular benchmark for surrogate modeling of tower fatigue in 22 MW floating offshore wind turbines. It provides 582,120 per-section fatigue damage labels from OpenFAST simulations across three tower geometries, with regime-aware splits and three evaluation protocols (random, within-tower regime-aware, cross-tower transfer)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.25717","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_floatbench_4e9fbef2"},"modality":null,"name":"FLOATBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25717","source_url":"https://arxiv.org/abs/2605.25717"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25717","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25717","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FloodReasonBench is a benchmark for vision-language model reasoning segmentation in flood response scenarios. It introduces FloodResponseSeg, a flood-specific dataset, and evaluates pipelines under lightweight visual encoding, split inference, and compressed representations. It also measures accuracy, latency, energy, and communication tradeoffs on an embedded platform.","descriptions":{"en":"FloodReasonBench is a benchmark for vision-language model reasoning segmentation in flood response scenarios. It introduces FloodResponseSeg, a flood-specific dataset, and evaluates pipelines under lightweight visual encoding, split inference, and compressed representations. It also measures accuracy, latency, energy, and communication tradeoffs on an embedded platform."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.15410","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_floodreasonbench_bd08edc8"},"modality":null,"name":"FloodReasonBench","openness":"unknown","publisher":null,"released":"2026-08-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15410","source_url":"https://arxiv.org/abs/2608.15410"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15410","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15410","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Flores is a benchmark dataset for machine translation between English and low-resource languages, which consists of sentences translated from Wikipedia, involving English and four low-resource languages, namely Nepali, Sinhala, Khmer and Pashto. Flores has two versions, we use the Flores-101 version here, which is the second version including 101 languages besides english. Flores是一个用于评估低资源语言机器翻译的基准数据集，它包含了从维基百科翻译的句子，涉及英语和四种低资源语言，分别是尼泊尔语、僧伽罗语、高棉语和普什图语。Flores有两个版本，我们这里使用的是第一个版本Flores-101，它包含有除英语外的101种语言。","descriptions":{"en":"Flores is a benchmark dataset for machine translation between English and low-resource languages, which consists of sentences translated from Wikipedia, involving English and four low-resource languages, namely Nepali, Sinhala, Khmer and Pashto. Flores has two versions, we use the Flores-101 version here, which is the second version including 101 languages besides english.","zh":"Flores是一个用于评估低资源语言机器翻译的基准数据集，它包含了从维基百科翻译的句子，涉及英语和四种低资源语言，分别是尼泊尔语、僧伽罗语、高棉语和普什图语。Flores有两个版本，我们这里使用的是第一个版本Flores-101，它包含有除英语外的101种语言。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:509","languages":["English"],"library_direction":null,"modality":null,"name":"Flores","openness":"unknown","publisher":null,"released":"2021-06-06","released_reference":{"basis":"paper_first_version","note":"The Hub record explicitly links the FLORES-101 introduction; its selected languages are contained in that benchmark.","source_key":"opencompass:509","source_url":"https://arxiv.org/abs/2106.03193"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-509-flores","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Flores","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"FLUB evaluates the reasoning and understanding abilities of LLMs. It includes three tasks with increasing difficulty, consisting of the tricky, humorous, and misleading texts collected from the real internet environment. FLUB用于评估LLM的推理和理解能力，其中包含3个难度递进的任务，由从真实互联网环境中收集的狡猾、幽默和误导性的文本构成。","descriptions":{"en":"FLUB evaluates the reasoning and understanding abilities of LLMs. It includes three tasks with increasing difficulty, consisting of the tricky, humorous, and misleading texts collected from the real internet environment.","zh":"FLUB用于评估LLM的推理和理解能力，其中包含3个难度递进的任务，由从真实互联网环境中收集的狡猾、幽默和误导性的文本构成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1324","languages":[],"library_direction":null,"modality":null,"name":"FLUB","openness":"unknown","publisher":"Tsinghua University","released":"2024-02-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1324-flub","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FLUB","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":21,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"claire_record_id","repository":"gh:analogy-ai/fm-bench","source_record":"bm_fm-bench_85cc4c12"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FM-Bench evaluates LLM agents on long-horizon football club management through 20 in-game years with 26 tools, measuring managerial decision quality via a deterministic engine scoring.","descriptions":{"en":"FM-Bench evaluates LLM agents on long-horizon football club management through 20 in-game years with 26 tools, measuring managerial decision quality via a deterministic engine scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.18423","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","tool-use"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_fm-bench_85cc4c12"},"modality":null,"name":"FM-Bench","openness":"unknown","publisher":null,"released":"2026-08-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.18423","source_url":"https://arxiv.org/abs/2608.18423"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-18423","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18423","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":25,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fideai/fmg-bench","source_record":"bm_fmg-bench_a89fdc7a"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CY"],"collected_at":"2026-10-14T09:44:51Z","description":"FMG-Bench evaluates English-language Christian theological triage and pastoral guidance in LLMs. It scores whether model responses match expected behaviors for triage levels: primary doctrine, secondary doctrine, prudential, and pastoral, with safety-related escalation appropriate to each scenario. The corpus includes 120 base scenarios plus 37 perturbation variants.","descriptions":{"en":"FMG-Bench evaluates English-language Christian theological triage and pastoral guidance in LLMs. It scores whether model responses match expected behaviors for triage levels: primary doctrine, secondary doctrine, prudential, and pastoral, with safety-related escalation appropriate to each scenario. The corpus includes 120 base scenarios plus 37 perturbation variants."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.12324","languages":[],"library_direction":null,"modality":null,"name":"FMG-Bench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12324","source_url":"https://arxiv.org/abs/2608.12324"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12324","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12324","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging.","descriptions":{"en":"Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33158","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_focus_f34c2d8f"},"modality":null,"name":"FOCUS","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33158","source_url":"https://arxiv.org/abs/2609.33158"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33158","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33158","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A physical-world benchmark for long-horizon robotic manipulation of deformable objects using real-robot data.","descriptions":{"en":"A physical-world benchmark for long-horizon robotic manipulation of deformable objects using real-robot data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10243","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_foldex_8952093b"},"modality":null,"name":"FolDeX","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10243","source_url":"https://arxiv.org/abs/2609.10243"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10243","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10243","topics":[],"unit":null},{"aliases":["llm-food-benchmark-2026-09"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tim2000s/llm-food-benchmark-2026-09","source_record":"bm_llm-food-benchmark-2026-09_1a38e73e"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A repository for a September 2026 rerun of an existing carbohydrate-estimation benchmark for LLMs on 13 food photographs.","descriptions":{"en":"A repository for a September 2026 rerun of an existing carbohydrate-estimation benchmark for LLMs on 13 food photographs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:tim2000s/llm-food-benchmark-2026-09","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","agentic-coding"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_llm-food-benchmark-2026-09_1a38e73e"},"modality":null,"name":"Food photograph benchmark, September 2026 rerun","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:tim2000s/llm-food-benchmark-2026-09","source_url":"https://github.com/tim2000s/llm-food-benchmark-2026-09"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-tim2000s-llm-food-benchmark-2026-09","source":"claire_radar","source_url":"https://github.com/tim2000s/llm-food-benchmark-2026-09","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal large language models on explainable compliance analysis in commercial kitchen surveillance, using video clips with dual-channel annotations and a composite metric.","descriptions":{"en":"Evaluates multimodal large language models on explainable compliance analysis in commercial kitchen surveillance, using video clips with dual-channel annotations and a composite metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24503","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_foodmonitor_339fb661"},"modality":null,"name":"FoodMonitor","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24503","source_url":"https://arxiv.org/abs/2605.24503"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24503","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24503","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FORCE-Bench evaluates agentic AI systems in enterprise finance across three task types: financial obligation research, financial entity performance research, and business brief generation. It includes 251 expert-annotated queries and a rubric-based scoring framework across eight dimensions: accuracy, citations, clarity, depth, groundedness, recency, relevance, and structure.","descriptions":{"en":"FORCE-Bench evaluates agentic AI systems in enterprise finance across three task types: financial obligation research, financial entity performance research, and business brief generation. It includes 251 expert-annotated queries and a rubric-based scoring framework across eight dimensions: accuracy, citations, clarity, depth, groundedness, recency, relevance, and structure."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19409","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_force-bench_be1ccc66"},"modality":null,"name":"FORCE-Bench","openness":"unknown","publisher":null,"released":"2026-07-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19409","source_url":"https://arxiv.org/abs/2607.19409"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19409","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19409","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"FORCEBENCH is a contrastive stress test for evidence-force calibration in cited RAG. It pairs fixed cited passages with evidence-calibrated claims and force-raised variants across five axes: relation, modality, scope, temporal validity, and numeric specificity. Evaluation is a fixed, locality-filtered set of 198 pairs.","descriptions":{"en":"FORCEBENCH is a contrastive stress test for evidence-force calibration in cited RAG. It pairs fixed cited passages with evidence-calibrated claims and force-raised variants across five axes: relation, modality, scope, temporal validity, and numeric specificity. Evaluation is a fixed, locality-filtered set of 198 pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28044","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","deep-research"],"join":"claire_record_id","primary":"factuality","source_record":"bm_forcebench_de197b73"},"modality":null,"name":"FORCEBENCH","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28044","source_url":"https://arxiv.org/abs/2605.28044"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28044","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28044","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates language-model decisions with budgeted forecast tools, but no distinct reusable evaluation protocol is introduced.","descriptions":{"en":"Evaluates language-model decisions with budgeted forecast tools, but no distinct reusable evaluation protocol is introduced."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27385","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_forecast-workflow-bench_4e0abe79"},"modality":null,"name":"Forecast Workflow Bench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27385","source_url":"https://arxiv.org/abs/2609.27385"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27385","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27385","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A simulated-world forecasting benchmark built on Freeciv game rollouts. Evaluates probabilistic reasoning by scoring forecasts about hidden future game states, with continuous or binary questions, paired intervention worlds, and artifacts for release.","descriptions":{"en":"A simulated-world forecasting benchmark built on Freeciv game rollouts. Evaluates probabilistic reasoning by scoring forecasts about hidden future game states, with continuous or binary questions, paired intervention worlds, and artifacts for release."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18686","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_forecastbench-sim_51b8b6fa"},"modality":null,"name":"ForecastBench-Sim","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18686","source_url":"https://arxiv.org/abs/2606.18686"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18686","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18686","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","安全","Safety","Forgery Detection","Large Vision Language Models","多模态模型","VLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models","descriptions":{"en":"A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models","zh":"A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1671","languages":[],"library_direction":null,"modality":"multimodal","name":"Forensics-bench","openness":"restricted","publisher":"The University of Hong Kong","released":"2025-03-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1671-forensics-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Forensics-bench","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_foresci_2aecb7a4"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Temporally controlled benchmark with 500 tasks across AI domains for forward-looking research judgment; includes offline knowledge bases and validation protocols.","descriptions":{"en":"Temporally controlled benchmark with 500 tasks across AI domains for forward-looking research judgment; includes offline knowledge bases and validation protocols."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00644","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting","science"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_foresci_2aecb7a4"},"modality":null,"name":"ForeSci","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00644","source_url":"https://arxiv.org/abs/2606.00644"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00644","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00644","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"A diagnostic safety benchmark for vision-language-action models, evaluating policies across physical interaction, instruction, and perception safety with cumulative cost and risk exposure metrics.","descriptions":{"en":"A diagnostic safety benchmark for vision-language-action models, evaluating policies across physical interaction, instruction, and perception safety with cumulative cost and risk exposure metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.27079","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","safety-alignment"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_foresightsafety-vla_a9025fe7"},"modality":null,"name":"ForesightSafety-VLA","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27079","source_url":"https://arxiv.org/abs/2606.27079"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27079","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27079","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"ForestBench evaluates multi-agent collaboration by representing MAS traces as unified collaboration graphs, comparing them against query-specific reference forests of successful graphs, and scoring structural similarity with a metric panel.","descriptions":{"en":"ForestBench evaluates multi-agent collaboration by representing MAS traces as unified collaboration graphs, comparing them against query-specific reference forests of successful graphs, and scoring structural similarity with a metric panel."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08605","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_forestbench_98bb6077"},"modality":null,"name":"ForestBench","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08605","source_url":"https://arxiv.org/abs/2608.08605"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08605","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08605","topics":["agents"],"unit":null},{"aliases":["One Polluted Page Is Enough: Evaluating Web Content Pollution in Generative Recommenders"],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"FORGE evaluates whether search-augmented language models recommend fake products introduced through controlled modifications to retrieved web pages.","descriptions":{"en":"FORGE evaluates whether search-augmented language models recommend fake products introduced through controlled modifications to retrieved web pages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.13610","languages":[],"library_direction":null,"modality":null,"name":"FORGE","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13610","source_url":"https://arxiv.org/abs/2606.13610"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13610","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13610","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Proposes a benchmark for evaluating forgetting dynamics in language models under continual knowledge editing, with concept-based and scenario-based QA, but no public artifacts are available.","descriptions":{"en":"Proposes a benchmark for evaluating forgetting dynamics in language models under continual knowledge editing, with concept-based and scenario-based QA, but no public artifacts are available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.26455","languages":[],"library_direction":null,"modality":null,"name":"ForgetBench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26455","source_url":"https://arxiv.org/abs/2607.26455"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26455","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26455","topics":[],"unit":null},{"aliases":["FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models"],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"175 expert-validated instances from STOC, FOCS, SODA, and COLT papers (2025-2026) for evaluating LLMs on end-to-end theoretical computer science research, including autoformalization and proof tasks.","descriptions":{"en":"175 expert-validated instances from STOC, FOCS, SODA, and COLT papers (2025-2026) for evaluating LLMs on end-to-end theoretical computer science research, including autoformalization and proof tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.20153","languages":[],"library_direction":{"basis":"model-review","categories":["math","science"],"join":"claire_record_id","primary":"math","source_record":"bm_formaltcs-benchmarking-end-to-end-frontier_6a36f0bb"},"modality":null,"name":"FormalTCS","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.20153","source_url":"https://arxiv.org/abs/2608.20153"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-20153","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20153","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"The paper introduces FormStruct-Bench, an evaluation dataset for table-form document structure recognition, but no official links to the benchmark artifacts are provided.","descriptions":{"en":"The paper introduces FormStruct-Bench, an evaluation dataset for table-form document structure recognition, but no official links to the benchmark artifacts are provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10396","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_formstruct-bench_6e003e69"},"modality":null,"name":"FormStruct-Bench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10396","source_url":"https://arxiv.org/abs/2608.10396"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10396","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10396","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","AVQA","多模态模型","VLM","逻辑推理","音频理解","Audio Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"FortisAVQA is the first dataset designed to assess the robustness of AVQA models. FortisAVQA，这是首个用于评估AVQA模型鲁棒性的数据集。","descriptions":{"en":"FortisAVQA is the first dataset designed to assess the robustness of AVQA models.","zh":"FortisAVQA，这是首个用于评估AVQA模型鲁棒性的数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1737","languages":[],"library_direction":null,"modality":"multimodal","name":"FortisAVQA","openness":"unknown","publisher":"Hong Kong University，etc.","released":"2025-04-02","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1737-fortisavqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FortisAVQA","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:lab-klc/fpco-dialog","source_record":"bm_fpco-dialog_09eb84bc"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark with 1,080 images and 10,800 question turns across 10-turn dialogues, evaluating correction and cooperation when false visual premises recur, scored by CorrTP@K and two detectors.","descriptions":{"en":"A benchmark with 1,080 images and 10,800 question turns across 10-turn dialogues, evaluating correction and cooperation when false visual premises recur, scored by CorrTP@K and two detectors."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.03331","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_fpco-dialog_09eb84bc"},"modality":null,"name":"FPCO-Dialog","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03331","source_url":"https://arxiv.org/abs/2609.03331"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03331","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03331","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sasanolab/framebench","source_record":"bm_framebench_0d2b1709"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A multiple-choice benchmark for English and Japanese that tests whether models distinguish semantic frames evoked by the same verb across contexts, using FrameNet resources and generation-verification.","descriptions":{"en":"A multiple-choice benchmark for English and Japanese that tests whether models distinguish semantic frames evoked by the same verb across contexts, using FrameNet resources and generation-verification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.03370","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_framebench_0d2b1709"},"modality":null,"name":"FrameBench","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03370","source_url":"https://arxiv.org/abs/2609.03370"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03370","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03370","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":25,"join":"llm_stats_id","repository":null,"source_record":"catalog_594cfd2607d4f09e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Factuality, Retrieval, And reasoning MEasurement Set - a unified evaluation dataset of 824 challenging multi-hop questions for testing retrieval-augmented generation systems across factuality, retrieval accuracy, and reasoning capabilities, requiring integration of 2-15 Wikipedia articles per question","descriptions":{"en":"Factuality, Retrieval, And reasoning MEasurement Set - a unified evaluation dataset of 824 challenging multi-hop questions for testing retrieval-augmented generation systems across factuality, retrieval accuracy, and reasoning capabilities, requiring integration of 2-15 Wikipedia articles per question"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frames:deepseek-v3","reported_at":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frames?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frames","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","factuality"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_594cfd2607d4f09e"},"modality":"text","name":"FRAMES","openness":"unknown","publisher":null,"released":"2024-09-19","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:frames","source_url":"https://arxiv.org/abs/2409.12941"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":87.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.87,"raw_min":0.733,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frames:kimi-k2-thinking-0905","reported_date":"2025-09-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frames?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"}],"slug":"llm-stats-frames","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frames?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Robustness","Safety & Trustworthiness","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A protocol-sensitive benchmark for adversarial robustness in financial fraud and credit risk, evaluating the same model-attack-defence setting under three protocols: unconstrained, post-hoc filtering, and deployment-aware constraint-integrated attacks.","descriptions":{"en":"A protocol-sensitive benchmark for adversarial robustness in financial fraud and credit risk, evaluating the same model-attack-defence setting under three protocols: unconstrained, post-hoc filtering, and deployment-aware constraint-integrated attacks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24551","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","safety-alignment"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_fraudbench_81b0f089"},"modality":null,"name":"FraudBench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24551","source_url":"https://arxiv.org/abs/2608.24551"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24551","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24551","topics":["safety","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","NAACL 2024","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"FREB-TQA is a Fine-grained Robustness Evaluation Benchmark for Table Question Answering. FREB-TQA 是一个细粒度的稳健性评估基准，专注于表格问答（TQA）。","descriptions":{"en":"FREB-TQA is a Fine-grained Robustness Evaluation Benchmark for Table Question Answering.","zh":"FREB-TQA 是一个细粒度的稳健性评估基准，专注于表格问答（TQA）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1145","languages":[],"library_direction":null,"modality":null,"name":"FREB-TQA","openness":"unknown","publisher":"Bosch Center for Artificial Intelligence","released":"2024-06-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1145-freb-tqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FREB-TQA","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aswincsekar/freightbidbench","source_record":"bm_freightbidbench_ee372bca"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"FreightBidBench is a public-calibrated, closed-loop benchmark for real-time truckload bid acceptance with explicit operational feasibility and economics, including pickup reach, appointment windows, hours-of-service, and yard delays. It provides two full-horizon hindsight ceilings for evaluation.","descriptions":{"en":"FreightBidBench is a public-calibrated, closed-loop benchmark for real-time truckload bid acceptance with explicit operational feasibility and economics, including pickup reach, appointment windows, hours-of-service, and yard delays. It provides two full-horizon hindsight ceilings for evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.07343","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_freightbidbench_ee372bca"},"modality":null,"name":"FreightBidBench","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.07343","source_url":"https://arxiv.org/abs/2607.07343"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-07343","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07343","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_1fd0ed42ac0061cc"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning","language","finance","general","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"French version of MMLU-Pro, a multilingual benchmark for evaluating language models' cross-lingual reasoning capabilities across 14 diverse domains including mathematics, physics, chemistry, law, engineering, psychology, and health.","descriptions":{"en":"French version of MMLU-Pro, a multilingual benchmark for evaluating language models' cross-lingual reasoning capabilities across 14 diverse domains including mathematics, physics, chemistry, law, engineering, psychology, and health."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:french-mmlu:ministral-8b-instruct-2410","reported_at":"2024-10-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/french-mmlu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:french-mmlu","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_1fd0ed42ac0061cc"},"modality":"text","name":"French MMLU","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":57.49999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.575,"raw_min":0.575,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:french-mmlu:ministral-8b-instruct-2410","reported_date":"2024-10-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/french-mmlu?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"}],"slug":"llm-stats-french-mmlu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/french-mmlu?top_n=500","topics":["reasoning-math","language","finance","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"FreshCache-Bench provides 8,072 base queries across five freshness classes with ground truth staleness labels from web snapshots at 1, 12, 24 hours, and 7 days, expanded to 31,201 queries via paraphrase generation, for evaluating semantic caching in retrieval-augmented LLMs.","descriptions":{"en":"FreshCache-Bench provides 8,072 base queries across five freshness classes with ground truth staleness labels from web snapshots at 1, 12, 24 hours, and 7 days, expanded to 31,201 queries via paraphrase generation, for evaluating semantic caching in retrieval-augmented LLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.04281","languages":[],"library_direction":null,"modality":null,"name":"FreshCache-Bench","openness":"unknown","publisher":null,"released":"2026-07-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04281","source_url":"https://arxiv.org/abs/2607.04281"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04281","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04281","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"FriendBench evaluates dyadic familiarity inference in humans and models using 96 dyads, comparing accuracy across modalities.","descriptions":{"en":"FriendBench evaluates dyadic familiarity inference in humans and models using 96 dyads, comparing accuracy across modalities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.29602","languages":[],"library_direction":null,"modality":null,"name":"FriendBench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29602","source_url":"https://arxiv.org/abs/2607.29602"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29602","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29602","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:chirindaopensource/from_financial_sentiment_classification_to_return_predictability","source_record":"bm_from-financial-sentiment-classification-to_65cf9b8b"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on financial sentiment classification and downstream return predictability using a reproducible QLoRA fine-tuning and evaluation pipeline.","descriptions":{"en":"Evaluates LLMs on financial sentiment classification and downstream return predictability using a reproducible QLoRA fine-tuning and evaluation pipeline."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:chirindaopensource/from_financial_sentiment_classification_to_return_predictability","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_from-financial-sentiment-classification-to_65cf9b8b"},"modality":null,"name":"From Financial Sentiment Classification to Return Predictability","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:chirindaopensource/from_financial_sentiment_classification_to_return_predictability","source_url":"https://github.com/chirindaopensource/from_financial_sentiment_classification_to_return_predictability"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-chirindaopensource-from-financial-sentiment-classification-to-return-predictability","source":"claire_radar","source_url":"https://github.com/chirindaopensource/from_financial_sentiment_classification_to_return_predictability","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:factnn/frompitch2board","source_record":"bm_frompitch2board_1fe15ef2"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A deterministic football-management benchmark evaluating autonomous agents across configurable factors using a headless simulation.","descriptions":{"en":"A deterministic football-management benchmark evaluating autonomous agents across configurable factors using a headless simulation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.34710","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_frompitch2board_1fe15ef2"},"modality":null,"name":"FromPitch2Board","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34710","source_url":"https://arxiv.org/abs/2609.34710"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34710","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34710","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:shubhraj5575/frontier-agent-benchmark","source_record":"bm_frontier-agent-benchmark_f8aadff6"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates autonomous AI engineering agents on engineering quality dimensions such as testing, architecture, and maintainability, using observed telemetry with provenance labels.","descriptions":{"en":"Evaluates autonomous AI engineering agents on engineering quality dimensions such as testing, architecture, and maintainability, using observed telemetry with provenance labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:shubhraj5575/frontier-agent-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Frontier Agent Benchmark (FAB)","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:shubhraj5575/frontier-agent-benchmark","source_url":"https://github.com/shubhraj5575/frontier-agent-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-shubhraj5575-frontier-agent-benchmark","source":"claire_radar","source_url":"https://github.com/shubhraj5575/frontier-agent-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:loopglitch26/frontier-autolab","source_record":"bm_frontier-autolab_38f68291"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-agent LLM systems are increasingly structured like organizations, with roles, critics and shared memory, yet they are evaluated on tasks that last minutes.","descriptions":{"en":"Multi-agent LLM systems are increasingly structured like organizations, with roles, critics and shared memory, yet they are evaluated on tasks that last minutes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.36739","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","multi-agent-games"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_frontier-autolab_38f68291"},"modality":null,"name":"Frontier Autolab","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36739","source_url":"https://arxiv.org/abs/2609.36739"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36739","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36739","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Frontier Financial Judgement assesses agents' ability to identify valuation-relevant financial information from 656 synthetic and real news items, matching expert labels.","descriptions":{"en":"Frontier Financial Judgement assesses agents' ability to identify valuation-relevant financial information from 656 synthetic and real news items, matching expert labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20645","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_frontier-financial-judgement_bf94dbc0"},"modality":null,"name":"Frontier Financial Judgement","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20645","source_url":"https://arxiv.org/abs/2607.20645"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20645","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20645","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":8146,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_29b51586e16c2654"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Frontier Science is a benchmark of exceptionally challenging scientific reasoning problems spanning advanced natural-science domains, designed to test expert-level scientific understanding and multi-step reasoning.","descriptions":{"en":"Frontier Science is a benchmark of exceptionally challenging scientific reasoning problems spanning advanced natural-science domains, designed to test expert-level scientific understanding and multi-step reasoning."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontier-science:mai-code-1-flash","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-science?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontier-science","languages":[],"library_direction":{"basis":"model-review","categories":["science","knowledge-expert-qa"],"join":"llm_stats_id","primary":"science","source_record":"catalog_29b51586e16c2654"},"modality":"text","name":"Frontier Science","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":58.199999999999996,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.582,"raw_min":0.582,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontier-science:mai-code-1-flash","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-science?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"}],"slug":"llm-stats-frontier-science","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-science?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":887,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:harbor-framework/frontier-bench","source_record":"catalog_eb184e86d3bc200b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Frontier-Bench v0.1 evaluates agentic terminal coding. Anthropic reports results using the mini-SWE-agent harness and a GKE backend, measured as mean reward across five attempts per task.","descriptions":{"en":"Frontier-Bench v0.1 evaluates agentic terminal coding. Anthropic reports results using the mini-SWE-agent harness and a GKE backend, measured as mean reward across five attempts per task."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontier-bench-v0.1:claude-opus-5","reported_at":"2026-07-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-bench-v0.1?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontier-bench-v0.1","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_eb184e86d3bc200b"},"modality":"text","name":"Frontier-Bench v0.1","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":43.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.433,"raw_min":0.433,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontier-bench-v0.1:claude-opus-5","reported_date":"2026-07-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-bench-v0.1?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"}],"slug":"llm-stats-frontier-bench-v0-1","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-bench-v0.1?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["Frontier-Eng","Frontier-Engineering Bench"],"attention_signals":null,"categories":["agents"],"collected_at":null,"description":"Medal Score uses peer-relative thresholds, not solve rate. September 2026 corrected results combine rescoring and shorter reruns; separate v1 (47 tasks), v1-lite (10 tasks), and historical average ranks.","descriptions":{"en":"Medal Score uses peer-relative thresholds, not solve rate. September 2026 corrected results combine rescoring and shorter reruns; separate v1 (47 tasks), v1-lite (10 tasks), and historical average ranks."},"editorial_guide":{"id":"frontier-engineering","record_keys":["model-reports:frontier_engineering"],"reviewed_at":"2026-10-27","scope":{"en":"The 47-task v1 suite and fixed 10-task v1-lite subset. September 2026 corrected Medal Scores are separate from the original paper’s average ranks.","zh":"覆盖 v1 的 47 道题及固定的 10 题 v1-lite 子集。2026 年 9 月修正后的 Medal Score 与原论文平均名次分开阅读。"},"sections":[{"id":"task","source_ids":["home","paper"],"text":{"en":"The agent starts from working code, proposes edits and receives objective values and constraint feedback from task-specific simulators or verifiers. The 47 tasks span five categories, including computing, decision science, control, communications and physical engineering. Examples include battery charging and memory allocation.","zh":"Agent 从已有可运行代码出发，修改方案，再接收专用仿真器或验证程序返回的目标值与约束反馈。47 道题覆盖计算、决策科学、控制、通信及物理工程五类领域，例如电池充电与内存分配。"},"title":{"en":"An engineering optimization loop","zh":"工程优化的迭代过程"}},{"id":"metric","source_ids":["leaderboard","medals"],"text":{"en":"Per task, meeting the available gold, silver or bronze threshold earns 1.00, 0.67 or 0.33. The mean over 47 or 10 tasks stays on a 0–1 scale. Invalid results earn zero; absent medal tiers earn no credit. This is relative to published peer thresholds, not the percentage of engineering problems solved.","zh":"每题达到可用的金、银、铜阈值分别得 1.00、0.67、0.33，再对 47 或 10 题求均值，范围为 0–1。无效结果计零，不存在的奖牌档位不计分。这是相对已发布同伴阈值的表现，不是工程问题解决率。"},"title":{"en":"What the Medal Score means","zh":"Medal Score 怎样计分"}},{"id":"version","source_ids":["home","leaderboard"],"text":{"en":"The authors strengthened evaluator isolation and corrected validation problems. The September 15 scores combine re-evaluated submissions with replacement candidates from shorter reruns; they are not a fresh equal-budget experiment. Keep v1 and v1-lite separate. The original nine-model rank table also includes gpt-oss-120b, whose raw scores were not retained for the updated eight-model podium.","zh":"作者加强了候选方案与验证器的隔离并修正验证问题。9 月 15 日成绩混合旧提交重评与更短重跑产生的替代方案，并非重新完成的一轮统一预算实验。v1 与 v1-lite 也应分开比较。历史九模型名次表还包含 gpt-oss-120b；更新奖牌榜未保留其原始分数，因此只含八模型。"},"title":{"en":"The September corrections matter","zh":"必须区分修正前后的结果"}},{"id":"medal-table","source_ids":["leaderboard"],"table":{"caption":{"en":"Corrected Medal Scores · scores dated 2026-09-15","zh":"修正后 Medal Score · 成绩日期 2026-09-15"},"columns":[{"en":"Model","zh":"模型"},{"en":"v1 · 47 tasks","zh":"v1 · 47 题"},{"en":"v1-lite · 10 tasks","zh":"v1-lite · 10 题"}],"note":{"en":"Higher is better; 0–1. Preserve the corrected-evaluator protocol above.","zh":"越高越好，范围 0–1；应结合上文的修正后评测协议阅读。"},"rows":[["Claude Opus 4.6","0.533","0.501"],["GPT-5.4","0.454","0.267"],["GLM-5","0.347","0.300"],["Gemini 3.1 Pro Preview","0.284","0.300"],["DeepSeek V3.2","0.269","0.299"],["Grok 4.20","0.220","0.167"],["Seed 2.0 Pro","0.206","0.100"],["Qwen3 Coder Next","0.170","0.066"]]},"text":{"en":"All eight published models are shown below. The two columns use different task sets.","zh":"下表保留全部八个发布模型；两列使用不同题集。"},"title":{"en":"Published corrected scores","zh":"修正后完整奖牌榜"}},{"id":"historical","source_ids":["paper","leaderboard"],"table":{"caption":{"en":"Original paper · mean within-task rank","zh":"原论文 · 题内平均名次"},"columns":[{"en":"Model","zh":"模型"},{"en":"Mean rank","zh":"平均名次"}],"note":{"en":"Original evaluators; 9 models; lower is better. Retained only as an attributed historical table.","zh":"原始验证器、九模型，越低越好；仅作为有出处的历史表保留。"},"rows":[["GPT-5.4","3.54"],["Claude Opus 4.6","3.63"],["GLM-5","4.34"],["DeepSeek V3.2","4.76"],["gpt-oss-120b","4.81"],["Gemini 3.1 Pro Preview","5.53"],["Grok 4.20","5.82"],["Seed 2.0 Pro","5.86"],["Qwen3 Coder Next","6.71"]]},"text":{"en":"The original paper uses within-task ranks averaged across 47 tasks, under OpenEvolve with 100 iterations; GPT-5.4 uses a full-suite retest. This lower-is-better metric is neither the current Medal Score nor a new measurement after the September fixes.","zh":"原论文在 OpenEvolve 的 100 轮预算下，对 47 题的题内名次取平均；GPT-5.4 使用全套复测结果。该指标越低越好，既不是当前 Medal Score，也不是 9 月修正后的重新测量。"},"title":{"en":"Historical average rank","zh":"历史平均名次"}},{"id":"limits","source_ids":["paper","readme","home"],"text":{"en":"Task objectives have different physical units, so raw per-task scores should not be averaged directly. Hardware, simulator dependencies, search method and evaluator version matter. The original paper’s historical curves do not establish that invalid-solution problems were absent; the later corrections are part of the record.","zh":"各任务原始目标具有不同物理单位，不能直接对原始分数求平均。硬件、仿真依赖、搜索方式和验证器版本都会影响复现。原论文的历史曲线不能证明不存在无效解获分问题；后续修正也是解读结果的一部分。"},"title":{"en":"Read a score with the environment","zh":"结合环境理解成绩"}}],"sources":[{"id":"home","title":"Frontier-Engineering official project and correction notice","url":"https://lab.einsia.ai/frontier-eng/"},{"id":"leaderboard","title":"Official leaderboard · corrected scores and historical ranks","url":"https://lab.einsia.ai/frontier-eng/leaderboard/"},{"id":"paper","title":"Frontier-Eng paper v2 · evaluation protocol and Table 2","url":"https://arxiv.org/html/2604.12290v2"},{"id":"medals","title":"Pinned released medal scoring documentation","url":"https://github.com/Einsia/Frontier-Engineering/blob/ccf51f7b2ae9498539e151cc7feb5b1df06543ff/leaderboard/README.md"},{"id":"readme","title":"Official repository · environments and version branches","url":"https://github.com/Einsia/Frontier-Engineering"}],"summary":{"en":"Can an agent repeatedly improve an engineering design while respecting its hard constraints? The task is to find a stronger feasible solution through executable feedback.","zh":"Agent 能否在硬约束下不断改进工程方案？这里考查根据程序反馈迭代寻找更好的可行解。"},"title":"Frontier-Engineering"},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000frontier_engineering\u0000einsia_frontier_engineering_leaderboard_2026_09_15\u0000frontier_engineering_v1_2026_09_15\u0000OpenEvolve; 47 tasks; 2026-09-15 corrected evaluators and medal thresholds; released submissions rescored plus replacement candidates from shorter reruns, not a uniform 100-iteration rerun; mean gold/silver/bronze credit 1.00/0.67/0.33; invalid or unavailable-tier credit 0\u0000Claude Opus 4.6","reported_at":"2026-09-15","source_url":"https://lab.einsia.ai/frontier-eng/leaderboard/"},"first_score_reported_at":"2026-09-15","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000frontier_engineering\u0000einsia_frontier_engineering_leaderboard_2026_09_15\u0000frontier_engineering_v1_2026_09_15\u0000OpenEvolve; 47 tasks; 2026-09-15 corrected evaluators and medal thresholds; released submissions rescored plus replacement candidates from shorter reruns, not a uniform 100-iteration rerun; mean gold/silver/bronze credit 1.00/0.67/0.33; invalid or unavailable-tier credit 0\u0000Claude Opus 4.6","reported_at":"2026-09-15","source_url":"https://lab.einsia.ai/frontier-eng/leaderboard/"},"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"model-reports:frontier_engineering","languages":[],"library_direction":null,"modality":null,"name":"Frontier-Engineering","openness":"unknown","publication":{"evidence":[{"note":"The author project names this benchmark and links its introducing paper and available release resources. This exact-record review preserves the documented task version, metrics and evaluation setup.","url":"https://lab.einsia.ai/frontier-eng/"},{"note":"Navers Lab / Einsia research index links this named project. Method papers and memory software in the same index are not added as benchmarks.","url":"https://lab.einsia.ai/research/"}],"id":"einsia-frontier-engineering-original","links":[{"kind":"website","title":"Frontier-Engineering · official project","url":"https://lab.einsia.ai/frontier-eng/"},{"kind":"paper","title":"Frontier-Engineering · introducing paper","url":"https://arxiv.org/abs/2604.12290"},{"kind":"repo","title":"Frontier-Engineering · official repository","url":"https://github.com/Einsia/Frontier-Engineering"},{"kind":"website","title":"Frontier-Engineering · author leaderboard","url":"https://lab.einsia.ai/frontier-eng/leaderboard/"}],"record_keys":["model-reports:frontier_engineering"],"reviewed_at":"2026-10-27","scope":{"en":"The 47-task v1 suite and fixed 10-task v1-lite subset. September 2026 corrected Medal Scores are separate from the original paper’s average ranks.","zh":"覆盖 v1 的 47 道题及固定的 10 题 v1-lite 子集。2026 年 9 月修正后的 Medal Score 与原论文平均名次分开阅读。"}},"publisher":null,"released":"2026-04-14","released_reference":{"basis":"benchmark_release","source_key":"model-reports:frontier_engineering","source_url":"https://lab.einsia.ai/frontier-eng/"},"repo_kind":null,"repo_resolution_status":null,"score_count":16,"score_direction":"higher_is_better","score_summary":{"display_max":0.533,"display_multiplier":1,"model_count":8,"model_count_basis":"source_model_id","numeric_count":16,"raw_max":0.533,"raw_min":0.066,"source_reference":{"obs_id":"curated\u0000frontier_engineering\u0000einsia_frontier_engineering_leaderboard_2026_09_15\u0000frontier_engineering_v1_2026_09_15\u0000OpenEvolve; 47 tasks; 2026-09-15 corrected evaluators and medal thresholds; released submissions rescored plus replacement candidates from shorter reruns, not a uniform 100-iteration rerun; mean gold/silver/bronze credit 1.00/0.67/0.33; invalid or unavailable-tier credit 0\u0000Claude Opus 4.6","observation_id":"curated\u0000frontier_engineering\u0000einsia_frontier_engineering_leaderboard_2026_09_15\u0000frontier_engineering_v1_2026_09_15\u0000OpenEvolve; 47 tasks; 2026-09-15 corrected evaluators and medal thresholds; released submissions rescored plus replacement candidates from shorter reruns, not a uniform 100-iteration rerun; mean gold/silver/bronze credit 1.00/0.67/0.33; invalid or unavailable-tier credit 0\u0000Claude Opus 4.6","reported_at":"2026-09-15","reported_date":"2026-09-15","source_id":"einsia_frontier_engineering_leaderboard_2026_09_15","source_url":"https://lab.einsia.ai/frontier-eng/leaderboard/"},"unit":"ratio"},"scored_models":[{"date":"2026-09-15","date_precision":"document_publication","name":"Claude Opus 4.6"},{"date":"2026-09-15","date_precision":"document_publication","name":"DeepSeek V3.2"},{"date":"2026-09-15","date_precision":"document_publication","name":"Gemini 3.1 Pro Preview"},{"date":"2026-09-15","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-09-15","date_precision":"document_publication","name":"GPT-5.4"},{"date":"2026-09-15","date_precision":"document_publication","name":"Grok 4.20"},{"date":"2026-09-15","date_precision":"document_publication","name":"Qwen3 Coder Next"},{"date":"2026-09-15","date_precision":"document_publication","name":"Seed 2.0 Pro"}],"slug":"frontier_engineering","source":"model_reports","source_url":"https://lab.einsia.ai/frontier-eng/","topics":["coding","agents","science"],"unit":"ratio"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5240,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":152,"join":"claire_record_id","repository":null,"source_record":"bm_frontierchallenge_5fbc977d"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates scientific agents on 97 released end-to-end workflows across six domains, using pass rate and average score to measure full delivery of required scientific deliverables.","descriptions":{"en":"Evaluates scientific agents on 97 released end-to-end workflows across six domains, using pass rate and average score to measure full delivery of required scientific deliverables."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24979","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_frontierchallenge_5fbc977d"},"modality":null,"name":"FrontierChallenge","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24979","source_url":"https://arxiv.org/abs/2608.24979"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24979","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24979","topics":["science"],"unit":null},{"aliases":["FrontierChallenge","Frontier Challenge"],"attention_signals":null,"categories":["scientific_agent"],"collected_at":null,"description":"Pass-rate results are model-scaffold measurements over the 97 released tasks with one trajectory per system-task pair, so a reported percentage is not a model-only number and does not estimate reliability beyond this release or under a different scaffold.","descriptions":{"en":"Pass-rate results are model-scaffold measurements over the 97 released tasks with one trajectory per system-task pair, so a reported percentage is not a model-only number and does not estimate reliability beyond this release or under a different scaffold."},"evidence_summary":{"document_count":1,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000frontier_challenge\u0000frontier_challenge_leaderboard_2026_08_25\u0000frontier_challenge_97_tasks_2026_08_25\u000097 released tasks; one trajectory per system-task pair; pass = native Score >= 99.9; agent scaffold: Claude Code\u0000Apodex 1.1","reported_at":"2026-08-25","source_url":"https://apodexai.github.io/FrontierAgent/benchmarks/FrontierChallenge/"},"first_score_reported_at":"2026-08-25","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000frontier_challenge\u0000frontier_challenge_leaderboard_2026_08_25\u0000frontier_challenge_97_tasks_2026_08_25\u000097 released tasks; one trajectory per system-task pair; pass = native Score >= 99.9; agent scaffold: Claude Code\u0000Apodex 1.1","reported_at":"2026-08-25","source_url":"https://apodexai.github.io/FrontierAgent/benchmarks/FrontierChallenge/"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:frontier_challenge","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["ai-rnd"],"join":"review:14H034160212:2026-10-15","primary":"ai-rnd","source_record":null},"modality":null,"name":"FrontierChallenge","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"model-reports:frontier_challenge","source_url":"https://apodexai.github.io/FrontierAgent/benchmarks/FrontierChallenge/"},"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":20.6,"display_multiplier":1,"model_count":12,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":20.6,"raw_min":3.1,"source_reference":{"obs_id":"curated\u0000frontier_challenge\u0000frontier_challenge_leaderboard_2026_08_25\u0000frontier_challenge_97_tasks_2026_08_25\u000097 released tasks; one trajectory per system-task pair; pass = native Score >= 99.9; agent scaffold: Claude Code\u0000Grok 4.6","observation_id":"curated\u0000frontier_challenge\u0000frontier_challenge_leaderboard_2026_08_25\u0000frontier_challenge_97_tasks_2026_08_25\u000097 released tasks; one trajectory per system-task pair; pass = native Score >= 99.9; agent scaffold: Claude Code\u0000Grok 4.6","reported_at":"2026-08-25","reported_date":"2026-08-25","source_id":"frontier_challenge_leaderboard_2026_08_25","source_url":"https://apodexai.github.io/FrontierAgent/benchmarks/FrontierChallenge/"},"unit":"percent"},"scored_models":[{"date":"2026-08-25","date_precision":"document_publication","name":"Apodex 1.1"},{"date":"2026-08-25","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2026-08-25","date_precision":"document_publication","name":"DeepSeek V4 Flash-0731"},{"date":"2026-08-25","date_precision":"document_publication","name":"DeepSeek V4 Pro-0813"},{"date":"2026-08-25","date_precision":"document_publication","name":"Gemini 3.7 Flash"},{"date":"2026-08-25","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-08-25","date_precision":"document_publication","name":"GPT-5.6 Sol"},{"date":"2026-08-25","date_precision":"document_publication","name":"GPT-5.6 Terra (max)"},{"date":"2026-08-25","date_precision":"document_publication","name":"Grok 4.6"},{"date":"2026-08-25","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-08-25","date_precision":"document_publication","name":"Qwen 3.8 Max"},{"date":"2026-08-25","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"frontier_challenge","source":"model_reports","source_url":"https://apodexai.github.io/FrontierAgent/benchmarks/FrontierChallenge/","topics":["agents","science"],"unit":"percent"},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FrontierCode is Cognition's coding evaluation that tests whether models can pass difficult coding tasks while meeting the standards of high-quality production codebases. The Diamond subset contains the hardest problems.","descriptions":{"en":"FrontierCode is Cognition's coding evaluation that tests whether models can pass difficult coding tasks while meeting the standards of high-quality production codebases. The Diamond subset contains the hardest problems."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontiercode:claude-fable-5","reported_at":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontiercode","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_99879b83a6d7430a"},"modality":"text","name":"FrontierCode","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":61.3,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.613,"raw_min":0.388,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontiercode:grok-4.6","reported_date":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"}],"slug":"llm-stats-frontiercode","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":["FrontierCode","FrontierCode (Diamond)"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Cognition's production-standard coding evaluation, asking whether models write *good* code rather than merely correct code. Unrelated to Terminal-Bench's Frontier-Bench despite the shared prefix. Reported per reasoning-effort setting (the Diamond subset at xhigh), so the effort level has to travel with the number. First-party to a competing coding agent's vendor, and revised as 1.1 on 2026-07-07, so the methodology version has to travel with the number too.","descriptions":{"en":"Cognition's production-standard coding evaluation, asking whether models write *good* code rather than merely correct code. Unrelated to Terminal-Bench's Frontier-Bench despite the shared prefix. Reported per reasoning-effort setting (the Diamond subset at xhigh), so the effort level has to travel with the number. First-party to a competing coding agent's vendor, and revised as 1.1 on 2026-07-07, so the methodology version has to travel with the number too."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:frontiercode","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"FrontierCode","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"model-reports:frontiercode","source_url":"https://cognition.com/blog/frontier-code"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"frontiercode","source":"model_reports","source_url":"https://cognition.com/blog/frontier-code","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FrontierCode 1.1 evaluates whether coding-agent changes are mergeable, using unit tests, maintainer-defined rubrics, and verifiers. Runs flagged for unfair internet use receive a zero score.","descriptions":{"en":"FrontierCode 1.1 evaluates whether coding-agent changes are mergeable, using unit tests, maintainer-defined rubrics, and verifiers. Runs flagged for unfair internet use receive a zero score."},"evidence_summary":{"document_count":1,"model_count":16,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontiercode-1.1:claude-opus-4-7","reported_at":"2026-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode-1.1?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontiercode-1.1","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_dd02ef5d25135618"},"modality":"text","name":"FrontierCode 1.1","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":16,"score_direction":"higher_is_better","score_summary":{"display_max":53.5,"display_multiplier":100,"model_count":16,"model_count_basis":"source_model_id","numeric_count":16,"raw_max":0.535,"raw_min":0.102,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontiercode-1.1:claude-fable-5","reported_date":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode-1.1?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-frontiercode-1-1","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode-1.1?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":326,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_dbd18e51dd070805"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FrontierCS is a benchmark of frontier computer-science problems requiring deep theoretical understanding and rigorous multi-step reasoning at the edge of the field.","descriptions":{"en":"FrontierCS is a benchmark of frontier computer-science problems requiring deep theoretical understanding and rigorous multi-step reasoning at the edge of the field."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontiercs:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercs?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontiercs","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_dbd18e51dd070805"},"modality":"text","name":"FrontierCS","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":50.8,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.508,"raw_min":0.463,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontiercs:seed-2.1-turbo","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercs?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-frontiercs","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercs?top_n=500","topics":["coding","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_frontierfinance_f2f44751"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Comprises 220 expert-crafted queries with 11,543 source-attributed rubrics across six finance use cases, evaluating agents on open-ended analyst-style answers via rubric-based grading.","descriptions":{"en":"Comprises 220 expert-crafted queries with 11,543 source-attributed rubrics across six finance use cases, evaluating agents on open-ended analyst-style answers via rubric-based grading."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.11683","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_frontierfinance_f2f44751"},"modality":null,"name":"FrontierFinance","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11683","source_url":"https://arxiv.org/abs/2608.11683"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11683","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11683","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":null,"source_record":"catalog_2f5bfab07dfbb484"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians, covering most major branches of modern mathematics from number theory and real analysis to algebraic geometry and category theory.","descriptions":{"en":"A benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians, covering most major branches of modern mathematics from number theory and real analysis to algebraic geometry and category theory."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontiermath:o1-2024-12-17","reported_at":"2024-12-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontiermath","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_2f5bfab07dfbb484"},"modality":"text","name":"FrontierMath","openness":"unknown","publication":{"evidence":[{"note":"Epoch AI identifies this as the original 2024-11-08 announcement of hundreds of expert-crafted mathematics problems across number theory and algebraic geometry, matching the source description. Its Read the paper link leads to arXiv:2411.04872. The source does not name a dataset revision or tier.","url":"https://epoch.ai/frontiermath/tiers-1-4/the-benchmark"}],"id":"frontiermath-publication","links":[{"kind":"website","title":"FrontierMath original announcement by Epoch AI","url":"https://epoch.ai/frontiermath/tiers-1-4/the-benchmark"},{"kind":"paper","title":"FrontierMath introducing paper","url":"https://arxiv.org/abs/2411.04872"}],"record_keys":["llm-stats:frontiermath"],"reviewed_at":"2026-10-26","scope":{"en":"The original FrontierMath expert-written mathematics benchmark. These links do not identify a particular score's dataset revision, tier or later Open Problems/Erdos extension.","zh":"原始 FrontierMath 专家编写数学评测的发布资料；这些链接不能确定某条成绩的数据修订版、难度层级，或将其归入后续 Open Problems、Erdős 扩展。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":89.0,"display_multiplier":100,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":0.89,"raw_min":0.055,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontiermath:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Instant"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Thinking"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 Pro"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"}],"slug":"llm-stats-frontiermath","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FrontierMath Tier 4 subset from the v2 evaluation release.","descriptions":{"en":"FrontierMath Tier 4 subset from the v2 evaluation release."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontiermath-tier-4-v2:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath-tier-4-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontiermath-tier-4-v2","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_9193b01245c728ca"},"modality":"text","name":"FrontierMath Tier 4 (v2)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":83.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.83,"raw_min":0.585,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontiermath-tier-4-v2:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath-tier-4-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-frontiermath-tier-4-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath-tier-4-v2?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":8467,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_845200828dcdd749"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","science"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FrontierScience Olympiad is a benchmark of olympiad-level scientific reasoning problems, testing expert understanding and multi-step reasoning across advanced natural-science domains.","descriptions":{"en":"FrontierScience Olympiad is a benchmark of olympiad-level scientific reasoning problems, testing expert understanding and multi-step reasoning across advanced natural-science domains."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontierscience-olympiad:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-olympiad?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontierscience-olympiad","languages":[],"library_direction":{"basis":"model-review","categories":["science","knowledge-expert-qa"],"join":"llm_stats_id","primary":"science","source_record":"catalog_845200828dcdd749"},"modality":"text","name":"FrontierScience Olympiad","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":76.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.76,"raw_min":0.748,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontierscience-olympiad:seed-2.1-turbo","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-olympiad?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-frontierscience-olympiad","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-olympiad?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":8146,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_24a4551d7f8a8078"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FrontierScience Research is a benchmark evaluating AI models on cutting-edge scientific research questions requiring deep domain expertise, multi-step reasoning, and synthesis of complex scientific concepts across disciplines.","descriptions":{"en":"FrontierScience Research is a benchmark evaluating AI models on cutting-edge scientific research questions requiring deep domain expertise, multi-step reasoning, and synthesis of complex scientific concepts across disciplines."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontierscience-research:muse-spark","reported_at":"2026-04-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-research?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontierscience-research","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_24a4551d7f8a8078"},"modality":"text","name":"FrontierScience Research","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":38.3,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.383,"raw_min":0.213,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontierscience-research:muse-spark","reported_date":"2026-04-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-research?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-frontierscience-research","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-research?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":232,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:proximal-labs/frontier-swe","source_record":"catalog_4d84bdec793147a5"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FrontierSWE measures whether an agent can complete open-ended technical projects at the scale of hours to tens of hours, spanning systems optimization, large-scale code construction, and applied ML research. Performance is reported as a dominance score, where higher is better.","descriptions":{"en":"FrontierSWE measures whether an agent can complete open-ended technical projects at the scale of hours to tens of hours, spanning systems optimization, large-scale code construction, and applied ML research. Performance is reported as a dominance score, where higher is better."},"evidence_summary":{"document_count":1,"model_count":16,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontierswe:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierswe?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontierswe","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","ai-rnd"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_4d84bdec793147a5"},"modality":"text","name":"FrontierSWE","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":16,"score_direction":"higher_is_better","score_summary":{"display_max":90.0,"display_multiplier":100,"model_count":16,"model_count_basis":"source_model_id","numeric_count":16,"raw_max":0.9,"raw_min":0.22,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontierswe:claude-fable-5","reported_date":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierswe?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-frontierswe","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierswe?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":232,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:proximal-labs/frontier-swe","source_record":"catalog_c74640a9725f90bd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"FrontierSWE (Impl.) evaluates software engineering implementation ability and reports model ranking on implementation tasks. Lower rank is better.","descriptions":{"en":"FrontierSWE (Impl.) evaluates software engineering implementation ability and reports model ranking on implementation tasks. Lower rank is better."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:frontier-swe-impl:mimo-v2.5-pro","reported_at":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-swe-impl?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:frontier-swe-impl","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_c74640a9725f90bd"},"modality":"text","name":"FrontierSWE (Impl.)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":3.4,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":3.4,"raw_min":3.4,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:frontier-swe-impl:mimo-v2.5-pro","reported_date":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-swe-impl?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"}],"slug":"llm-stats-frontier-swe-impl","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-swe-impl?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":122,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:bytedance/fullstackbench","source_record":"catalog_98f45d18494cedc9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"English subset of FullStackBench for evaluating end-to-end software engineering and full-stack development capability.","descriptions":{"en":"English subset of FullStackBench for evaluating end-to-end software engineering and full-stack development capability."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:fullstackbench-en:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-en?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:fullstackbench-en","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_98f45d18494cedc9"},"modality":"text","name":"FullStackBench en","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":62.6,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.626,"raw_min":0.581,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:fullstackbench-en:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-en?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-fullstackbench-en","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-en?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":122,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:bytedance/fullstackbench","source_record":"catalog_41a7e9d5dcfe0e8b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Chinese subset of FullStackBench for evaluating end-to-end software engineering and full-stack development capability.","descriptions":{"en":"Chinese subset of FullStackBench for evaluating end-to-end software engineering and full-stack development capability."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:fullstackbench-zh:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-zh?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:fullstackbench-zh","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_41a7e9d5dcfe0e8b"},"modality":"text","name":"FullStackBench zh","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":58.699999999999996,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.587,"raw_min":0.55,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:fullstackbench-zh:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-zh?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-fullstackbench-zh","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-zh?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":90,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:consequentai/fneval","source_record":"catalog_341e245f91f03a8b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A functional variant of the MATH benchmark that tests language models' ability to generalize reasoning patterns across different problem instances, revealing the reasoning gap between static and functional performance.","descriptions":{"en":"A functional variant of the MATH benchmark that tests language models' ability to generalize reasoning patterns across different problem instances, revealing the reasoning gap between static and functional performance."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:functionalmath:gemini-1.5-flash","reported_at":"2024-05-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/functionalmath?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:functionalmath","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_341e245f91f03a8b"},"modality":"text","name":"FunctionalMATH","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":64.60000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.646,"raw_min":0.536,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:functionalmath:gemini-1.5-pro","reported_date":"2024-05-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/functionalmath?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"}],"slug":"llm-stats-functionalmath","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/functionalmath?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:penway/funpiq","source_record":"bm_funpiq_ac17b6fd"},"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"FunPiQ is a benchmark for pixel-level fundus image quality assessment with pixel-level annotations for anatomical visibility, using a three-class scheme (good, usable, bad). It also introduces EFIQA-CP, an explainable method trained with pseudo-labels.","descriptions":{"en":"FunPiQ is a benchmark for pixel-level fundus image quality assessment with pixel-level annotations for anatomical visibility, using a three-class scheme (good, usable, bad). It also introduces EFIQA-CP, an explainable method trained with pseudo-labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.25915","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_funpiq_ac17b6fd"},"modality":null,"name":"FunPiQ","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25915","source_url":"https://arxiv.org/abs/2606.25915"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25915","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25915","topics":["medical"],"unit":null},{"aliases":["Future Rendering $\\neq$ Future Surface: A Benchmark and Dataset for Dynamic Surface Reconstruction Beyond the Observed Window"],"attention_signals":null,"categories":["Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A diagnostic benchmark for reconstructing dynamic surfaces beyond the observed time window. Methods train on 75% of a sequence and are scored by Chamfer distance on the held-out future frames.","descriptions":{"en":"A diagnostic benchmark for reconstructing dynamic surfaces beyond the observed time window. Methods train on 75% of a sequence and are scored by Chamfer distance on the held-out future frames."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.21471","languages":[],"library_direction":null,"modality":null,"name":"FutureSurf","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21471","source_url":"https://arxiv.org/abs/2607.21471"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21471","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21471","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":8,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fuzzingbrain/fuzzingbrain-bench","source_record":"bm_fuzzingbrain-bench_87c92c01"},"categories":["Cybersecurity","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on discovering distinct crashes in 77 open-source software challenges using sanitizer-instrumented Docker harnesses and deterministic scoring.","descriptions":{"en":"Evaluates LLMs on discovering distinct crashes in 77 open-source software challenges using sanitizer-instrumented Docker harnesses and deterministic scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.25158","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_fuzzingbrain-bench_87c92c01"},"modality":null,"name":"FuzzingBrain-Bench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25158","source_url":"https://arxiv.org/abs/2608.25158"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25158","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25158","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"G-IdiomAlign is a gloss-pivoted benchmark for cross-lingual idiom alignment, with protocols for multiple-choice equivalence and gloss-contrastive generation.","descriptions":{"en":"G-IdiomAlign is a gloss-pivoted benchmark for cross-lingual idiom alignment, with protocols for multiple-choice equivalence and gloss-contrastive generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18989","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_g-idiomalign_39650ae3"},"modality":null,"name":"G-IdiomAlign","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18989","source_url":"https://arxiv.org/abs/2606.18989"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18989","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18989","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"GABench evaluates LLM agents on graph analysis tasks across three graph types and four task categories: graph retrieval, graph theory, graph machine learning, and graph open-ended QA. It provides 84 executable tools and 10,400 tasks with verifiable ground truth.","descriptions":{"en":"GABench evaluates LLM agents on graph analysis tasks across three graph types and four task categories: graph retrieval, graph theory, graph machine learning, and graph open-ended QA. It provides 84 executable tools and 10,400 tasks with verifiable ground truth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01684","languages":[],"library_direction":{"basis":"model-review-2","categories":["tool-use","abstract-reasoning"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_gabench_d11355fd"},"modality":null,"name":"GABench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01684","source_url":"https://arxiv.org/abs/2608.01684"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01684","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01684","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":253,"join":"artifact:arxiv:2311.12983","repository":null,"source_record":"lib_gaia"},"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GAIA is a benchmark which aims at evaluating next-generation LLMs (LLMs with augmented capabilities due to added tooling, efficient prompting, access to search, etc), mading of more than 450 non-trivial question with an unambiguous answer, requiring different levels of tooling and autonomy to solve. GAIA 是一个旨在评估下一代LLMs（由于增加了工具、高效的提示、访问搜索等功能而具有增强能力的LLMs）的基准，由超过 450 个非平凡问题组成，这些问题有明确的答案，需要不同层次的工具和自主性来解决。","descriptions":{"en":"GAIA is a benchmark which aims at evaluating next-generation LLMs (LLMs with augmented capabilities due to added tooling, efficient prompting, access to search, etc), mading of more than 450 non-trivial question with an unambiguous answer, requiring different levels of tooling and autonomy to solve.","zh":"GAIA 是一个旨在评估下一代LLMs（由于增加了工具、高效的提示、访问搜索等功能而具有增强能力的LLMs）的基准，由超过 450 个非平凡问题组成，这些问题有明确的答案，需要不同层次的工具和自主性来解决。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1599","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","tool-use"],"join":"artifact:arxiv:2311.12983","primary":"deep-research","source_record":"lib_gaia"},"modality":null,"name":"GAIA","openness":"restricted","publisher":"Meta, Huggingface, AutoGPT","released":"2023-11-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1599-gaia","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GAIA","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"llm_stats_id","repository":null,"source_record":"catalog_0197a6c5271dada4"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GAIA2 evaluates general-purpose AI agents on real-world, multi-step questions that require reasoning, tool use, and information retrieval.","descriptions":{"en":"GAIA2 evaluates general-purpose AI agents on real-world, multi-step questions that require reasoning, tool use, and information retrieval."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gaia2:muse-glimmer-30b","reported_at":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gaia2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gaia2","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","deep-research"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_0197a6c5271dada4"},"modality":"multimodal","name":"GAIA2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":43.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.433,"raw_min":0.433,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gaia2:muse-glimmer-30b","reported_date":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gaia2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"}],"slug":"llm-stats-gaia2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gaia2?top_n=500","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":10,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:xufeiqiong/gama-bench","source_record":"bm_gama-bench_3cec2065"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"GAMA-Bench evaluates LLMs on gender-asymmetric moral framing across 1,298 paired conflict scenarios, measuring response differences in punitive, therapeutic, and blame dimensions.","descriptions":{"en":"GAMA-Bench evaluates LLMs on gender-asymmetric moral framing across 1,298 paired conflict scenarios, measuring response differences in punitive, therapeutic, and blame dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.14068","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_gama-bench_3cec2065"},"modality":null,"name":"GAMA-Bench","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14068","source_url":"https://arxiv.org/abs/2606.14068"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14068","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14068","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":10,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:areal-project/gameasg-bench","source_record":"bm_gameasg-bench_01bef6f8"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on generating complete browser games from natural-language requirements with static and behavioral tests.","descriptions":{"en":"Evaluates coding agents on generating complete browser games from natural-language requirements with static and behavioral tests."},"editorial_guide":{"id":"gameasg-bench","record_keys":["claire-radar:2609.21293"],"reviewed_at":"2026-10-26","scope":{"en":"Original GameASG-Bench paper v1 and the pinned public implementation. All tables retain the authors’ results and experimental settings; AI Benchmark Radar has not rerun them.","zh":"范围为 GameASG-Bench 原论文 v1 及固定版本的公开实现。各表保留作者结果和实验设置，AI Benchmark Radar 未重新运行这些实验。"},"sections":[{"id":"task","source_ids":["readme"],"text":{"en":"Each of the 47 tasks asks for one executable index.html, covering 12 genres with 32 2D and 15 3D games. Agents receive target.md, game-spec.md and tdd.md. First-party code must fit inside the HTML; only task-permitted CDN libraries may remain external. The evaluator serves no companion assets.","zh":"47 道任务要求各交付一个可运行的 index.html，覆盖 12 类游戏，其中 32 个为 2D、15 个为 3D。Agent 获得 target.md、game-spec.md 和 tdd.md。自有代码须包含在 HTML 中，仅可按任务要求引用 CDN 库；评估时不会提供本地配套素材。"},"title":{"en":"The deliverable: a complete browser game","zh":"交付物：完整的浏览器游戏"}},{"id":"interface","source_ids":["contract"],"text":{"en":"The window.__gameTest contract exposes reset, loadScenario, input and getSnapshot. A prepared scenario establishes a legal starting point; a player action must still cause the outcome. The interface must control the visible game itself, rather than a separate state that merely satisfies tests.","zh":"window.__gameTest 约定提供 reset、loadScenario、input 和 getSnapshot。场景加载只建立合法起点，待测结果仍须由玩家操作触发；接口必须控制画面中的真实游戏，不能另建一套只为通过检查而更新的状态。"},"title":{"en":"A test interface tied to real gameplay","zh":"测试接口必须连接真实玩法"}},{"id":"checks","source_ids":["readme","browser-runner"],"text":{"en":"L1 inspects the delivered source. L2 executes the game in Chromium, preparing a scenario, acting and observing the outcome on a fresh page for each check. P0 covers launch and interface prerequisites; P1 covers required gameplay; P2 covers extensions. Layer and priority answer different questions.","zh":"L1 检查交付源码；L2 在 Chromium 中运行游戏，通过准备场景、执行操作和观察结果进行验证，每项检查使用新页面。P0 对应启动与接口前提，P1 对应必需玩法，P2 对应扩展能力。检查层说明证据来自哪里，优先级说明需求的重要程度。"},"title":{"en":"Evidence layers and requirement priorities","zh":"检查层与需求优先级"}},{"id":"scoring","source_ids":["reporter","aggregation"],"text":{"en":"Strict success requires verified delivery, completed evaluation, every L1 check, and every applicable L2 P0/P1 check; P2 is excluded. Its denominator includes all 47 planned tasks, including failures and missing results. L1 and L2 Mean average task-level rates; priority columns pool applicable checks. NOT_APPLICABLE checks are excluded, including P1 when marked that way.","zh":"严格成功要求交付经确认、评估完成、全部 L1 与全部适用的 L2 P0/P1 检查通过，不要求 P2 全过。分母固定为全部 47 道计划任务，包含失败与缺失结果。L1 和 L2 Mean 对各任务通过率取平均；优先级列合并适用检查后计算。NOT_APPLICABLE 不计入适用检查，P1 被标为不适用时也会被排除。"},"title":{"en":"Read task success and check rates separately","zh":"任务成功率与检查通过率分别看"}},{"id":"main-results","source_ids":["paper","reporter"],"table":{"caption":{"en":"Paper v1, Table 3 · Task completion and check rates","zh":"论文 v1 表 3 · 任务完成与检查通过率"},"columns":[{"en":"Model","zh":"模型"},{"en":"Harness","zh":"运行框架"},{"en":"Strict success: count/47 (%)","zh":"严格成功：成功数/47（%）"},{"en":"L1 %","zh":"L1 通过率 %"},{"en":"L2 Mean %","zh":"L2 平均通过率 %"},{"en":"L2 P0 %","zh":"L2 P0 通过率 %"},{"en":"L2 P1 %","zh":"L2 P1 通过率 %"},{"en":"L2 P2 %","zh":"L2 P2 通过率 %"}],"note":{"en":"Parentheses contain percentages. A high mean check rate does not imply that a task passed every required check. Original model names, values and precision are retained; these are not reruns by this site.","zh":"括号内为百分比。平均检查通过率较高，并不等于整道任务的必需检查全部通过。模型名、数值与精度保留原表口径；本站未重新运行实验。"},"rows":[["GPT-6-Astra","Codex CLI","26/47 (55.3)","98.0","93.2","99.0","92.7","92.4"],["Claude-Opus-5","Claude Code","24/47 (51.1)","99.6","90.4","91.2","89.9","91.1"],["GPT-5.6-Sol","Codex CLI","21/47 (44.7)","98.9","91.3","100.0","89.7","91.9"],["DeepSeek-V4-Flash","Claude Code","18/47 (38.3)","99.4","89.3","98.0","86.7","90.2"],["DeepSeek-V4-Pro","Claude Code","15/47 (31.9)","99.6","85.2","85.3","83.9","88.9"],["Kimi-K3","Claude Code","15/47 (31.9)","97.7","88.5","98.0","86.3","89.7"],["GLM-5.2","Claude Code","11/47 (23.4)","98.8","84.6","91.2","83.3","83.0"],["Hunyuan-3","Claude Code","10/47 (21.3)","98.3","81.7","96.1","78.2","82.3"],["MiniMax-M3","Claude Code","7/47 (14.9)","98.3","70.6","91.2","64.5","75.4"]]},"text":{"en":"All nine configurations complete evaluation on every task at maximum reasoning effort. Each task is attempted once. These are the authors’ observations, with no repeated-run uncertainty estimate.","zh":"九组配置均在最高推理设置下完成全部任务的评估，每道任务尝试一次。以下为作者报告的观测结果，未提供重复运行的不确定性估计。"},"title":{"en":"Nine model–harness configurations","zh":"九组模型与运行框架的总体结果"}},{"id":"resource-results","source_ids":["paper","reporter"],"table":{"caption":{"en":"Paper v1, Table 4 · Mean artifact size and reported resources","zh":"论文 v1 表 4 · 平均文件大小与报告资源用量"},"columns":[{"en":"Model","zh":"模型"},{"en":"Harness","zh":"运行框架"},{"en":"Artifact KB","zh":"文件大小 KB"},{"en":"Input tokens k","zh":"输入 token（千）"},{"en":"Output tokens k","zh":"输出 token（千）"},{"en":"Estimated cost USD","zh":"估算费用 USD"}],"note":{"en":"Artifact sizes cover all 47 tasks per configuration. k = 1,000 tokens; KB = 1,024 bytes. These historical cost estimates are not current provider prices, and different accounting conventions limit efficiency comparisons.","zh":"每组配置的文件大小覆盖全部 47 道任务。k 为 1,000 个 token，KB 为 1,024 字节。金额是该实验的历史估算，不代表服务商现价；不同统计方式会影响效率比较。"},"rows":[["GPT-6-Astra","Codex CLI","78.82","896.1","55.5","4.4445"],["Claude-Opus-5","Claude Code","68.52","12,632.6","142.4","11.8500"],["GPT-5.6-Sol","Codex CLI","64.67","1,382.5","45.7","1.7867"],["DeepSeek-V4-Flash","Claude Code","85.21","13,038.4","203.6","0.2461"],["DeepSeek-V4-Pro","Claude Code","90.34","12,944.3","202.9","0.7431"],["Kimi-K3","Claude Code","57.51","1,263.9","72.3","2.5518"],["GLM-5.2","Claude Code","60.33","1,067.1","47.0","0.6184"],["Hunyuan-3","Claude Code","52.88","2,474.5","88.9","0.1779"],["MiniMax-M3","Claude Code","81.76","4,918.9","71.1","0.4069"]]},"text":{"en":"Resource means use available measurements; missing entries are excluded rather than treated as zero. Input tokens combine input, cache reads and cache writes. Monetary values are estimates from the report script’s fixed rates, excluding request-specific surcharges and tool fees.","zh":"资源均值按可用测量值计算，缺失项不按零处理。输入 token 合并输入、缓存读取和缓存写入。金额由报告脚本的固定费率估算，不含请求特定附加费与工具费用。"},"title":{"en":"Artifact size and resource use","zh":"交付文件大小与资源用量"}},{"id":"tool-results","source_ids":["paper"],"table":{"caption":{"en":"Paper v1, Table 5 · DeepSeek-V4-Flash tool configurations","zh":"论文 v1 表 5 · DeepSeek-V4-Flash 工具配置对照"},"columns":[{"en":"Tool setting","zh":"工具设置"},{"en":"Strict success: count/47 (%)","zh":"严格成功：成功数/47（%）"},{"en":"L1 %","zh":"L1 通过率 %"},{"en":"L2 P0 %","zh":"L2 P0 通过率 %"},{"en":"L2 P1 %","zh":"L2 P1 通过率 %"},{"en":"L2 P2 %","zh":"L2 P2 通过率 %"}],"note":{"en":"Every condition completes evaluation on 47 tasks. No tools = serialized API; file-only = read/write; syntax = files plus syntax checking; full = full tool access. The no-tool comparison also changes input delivery, so it does not isolate one tool alone.","zh":"各条件均完成 47 道任务评估。No tools 为序列化 API；File read/write only 为仅读写；Files + syntax checking 增加语法检查；Full tool access 为完整工具。无工具条件同时改变了输入流程，不能将差异只归因于某一种工具。"},"rows":[["No tools (serialized API)","7/47 (14.9)","98.5","83.3","65.3","80.1"],["File read/write only","6/47 (12.8)","98.8","87.3","66.2","73.4"],["Files + syntax checking","9/47 (19.1)","98.2","86.3","63.4","69.3"],["Full tool access","18/47 (38.3)","99.4","98.0","86.7","90.2"]]},"text":{"en":"DeepSeek-V4-Flash is tested at maximum reasoning effort. The full-tool baseline uses Claude Code and a nominal 120-turn budget; the no-tool condition instead sends serialized API input. Syntax-only access permits node --check, without a browser.","zh":"本组使用 DeepSeek-V4-Flash 的最高推理设置。完整工具基线使用 Claude Code、名义预算为 120 轮；无工具条件则改为序列化 API 输入。仅语法检查条件允许 node --check，但不允许浏览器。"},"title":{"en":"Tool access changes the workflow","zh":"工具访问会改变执行流程"}},{"id":"budget-results","source_ids":["paper","reporter"],"table":{"caption":{"en":"Paper v1, Table 6 · Nominal turn budgets","zh":"论文 v1 表 6 · 名义轮数预算"},"columns":[{"en":"Turns","zh":"轮数"},{"en":"Evaluated / planned","zh":"已评估/计划任务"},{"en":"Success / all planned (%)","zh":"成功/全部计划任务（%）"},{"en":"Success / evaluated (%)","zh":"成功/已评估任务（%）"},{"en":"L1 %","zh":"L1 通过率 %"},{"en":"L2 P0 %","zh":"L2 P0 通过率 %"},{"en":"L2 P1 %","zh":"L2 P1 通过率 %"},{"en":"L2 P2 %","zh":"L2 P2 通过率 %"}],"note":{"en":"The 30-turn result of 70.0% is 7 successes among 10 evaluated tasks; the primary result is 7/47 (14.9%). Check rates also describe the evaluated subset. Do not compare conditional rates as if all runs had completed.","zh":"30 轮的 70.0% 仅指 10 道已评估任务中成功 7 道；主指标为 7/47（14.9%）。检查通过率也只覆盖已评估子集，不能把条件成功率当作全部任务均完成后的结果比较。"},"rows":[["30","10/47","7/47 (14.9)","7/10 (70.0)","100.0","100.0","94.0","97.6"],["60","32/47","13/47 (27.7)","13/32 (40.6)","99.7","88.6","81.6","88.5"],["120","47/47","18/47 (38.3)","18/47 (38.3)","99.4","98.0","86.7","90.2"]]},"text":{"en":"This comparison fixes DeepSeek-V4-Flash, Claude Code, full tools and maximum reasoning effort. The conditional success column describes only tasks that reach evaluation; the all-planned column measures delivery across the full benchmark.","zh":"本组固定 DeepSeek-V4-Flash、Claude Code、完整工具和最高推理设置。条件成功率只描述进入评估的任务；全部计划任务成功率描述整个评测集上的交付情况。"},"title":{"en":"The denominator matters when runs stop early","zh":"提前停止时尤其要看分母"}},{"id":"effort-results","source_ids":["paper"],"table":{"caption":{"en":"Paper v1, Table 7 · Reasoning-effort comparison","zh":"论文 v1 表 7 · 推理强度对照"},"columns":[{"en":"Effort","zh":"推理强度"},{"en":"Strict success: count/47 (%)","zh":"严格成功：成功数/47（%）"},{"en":"L1 %","zh":"L1 通过率 %"},{"en":"L2 P0 %","zh":"L2 P0 通过率 %"},{"en":"L2 P1 %","zh":"L2 P1 通过率 %"},{"en":"L2 P2 %","zh":"L2 P2 通过率 %"},{"en":"Mean reasoning tokens","zh":"平均推理 token 数"}],"note":{"en":"Low, High and Max are the reported settings. Reasoning tokens are per-task means. The observed difference between High and Max comes from single runs; it is not evidence of a stable superiority across repetitions.","zh":"Low、High、Max 分别为原文的低、高、最高设置。推理 token 为各任务均值。High 与 Max 的观测差异来自单次运行，不能据此认定重复实验中仍会稳定领先。"},"rows":[["Low","7/47 (14.9)","98.3","93.1","71.7","80.2","37,699"],["High","19/47 (40.4)","99.4","97.1","82.7","88.9","138,232"],["Max","18/47 (38.3)","99.4","98.0","86.7","90.2","189,210"]]},"text":{"en":"The authors vary reasoning effort while keeping DeepSeek-V4-Flash, Claude Code, full tools and the nominal 120-turn setting. Read the task-success and token columns together.","zh":"作者在 DeepSeek-V4-Flash、Claude Code、完整工具与名义 120 轮设置下改变推理强度。阅读时应同时比较任务成功数和 token 用量。"},"title":{"en":"Reasoning settings and token use","zh":"推理设置与 token 用量"}},{"id":"harness-results","source_ids":["paper"],"table":{"caption":{"en":"Paper v1, Table 8 · Harness comparison","zh":"论文 v1 表 8 · 运行框架对照"},"columns":[{"en":"Harness","zh":"运行框架"},{"en":"Strict success: count/47 (%)","zh":"严格成功：成功数/47（%）"},{"en":"L1 %","zh":"L1 通过率 %"},{"en":"L2 P0 %","zh":"L2 P0 通过率 %"},{"en":"L2 P1 %","zh":"L2 P1 通过率 %"},{"en":"L2 P2 %","zh":"L2 P2 通过率 %"}],"note":{"en":"This compares complete harness configurations on the same 47 tasks. Equal strict-success totals do not imply identical artifacts or failure patterns.","zh":"此处比较同一 47 道任务上的完整框架配置。严格成功数相同，不代表交付文件或失败模式相同。"},"rows":[["Claude Code","18/47 (38.3)","99.4","98.0","86.7","90.2"],["Codex CLI","18/47 (38.3)","99.2","96.1","87.1","92.2"]]},"text":{"en":"With DeepSeek-V4-Flash, maximum effort and full tools, the two harnesses produce the same success count. Only 10 tasks succeed in both: each harness uniquely succeeds on 8, and neither succeeds on 21.","zh":"使用 DeepSeek-V4-Flash、最高推理强度和完整工具时，两种框架的成功总数相同。其中只有 10 道任务共同成功；各有 8 道仅由其中一种框架成功完成，另有 21 道均未成功。"},"title":{"en":"Equal totals can hide different failures","zh":"相同总成功数可能对应不同失败"}},{"id":"diagnostics","source_ids":["paper"],"text":{"en":"Table 9 describes two interaction defects. GPT-6-Astra/Codex CLI’s Diner Dasher passes L1 and P0/P2 but real mouse/touch drags fail to advance service. DeepSeek-V4-Flash/Claude Code’s Turbo Smash Beast stops natural-time simulation after reset or scenario loading, breaking acceleration and coasting checks.","zh":"表 9 给出两类交互缺陷：GPT-6-Astra/Codex CLI 的 Diner Dasher 虽通过 L1 与 P0/P2，真实鼠标和触摸拖动却无法推进服务流程；DeepSeek-V4-Flash/Claude Code 的 Turbo Smash Beast 在重置或加载场景后停止自然时间模拟，导致加速、滑行等检查失败。"},"title":{"en":"What the failure traces reveal","zh":"失败记录能揭示什么"}},{"id":"limits","source_ids":["readme","aggregation","contract"],"text":{"en":"Passing supports compliance with the published requirements under these checks. It does not certify enjoyment, visual quality or production readiness. The test code is public; standard generation containers omit it. NOT_APPLICABLE exclusions and model–harness differences should remain visible when interpreting a comparison.","zh":"通过评测说明在这些检查下满足了公开要求，不等于游戏有趣、视觉品质优秀或已经可上线。测试代码公开，但标准生成容器不挂载它。解释比较结果时，还应保留 NOT_APPLICABLE 排除情况及模型、框架差异。"},"title":{"en":"What passing can establish","zh":"通过评测能说明什么"}},{"id":"reproduce","source_ids":["runtime","launcher","readme"],"text":{"en":"Pin the repository revision, model, harness, reasoning setting, tool permissions and budget. The runtime pins Claude Code 2.1.206 and Codex CLI 0.153.4. Defaults allow one generation attempt, 3,600 seconds and 120 Claude Code turns. Use a fresh experiment ID; an existing HTML can be re-evaluated. README examples use different defaults from some paper rows and are not exact reproductions.","zh":"应固定仓库版本、模型、运行框架、推理设置、工具权限与预算。运行环境固定 Claude Code 2.1.206、Codex CLI 0.153.4；默认允许一次生成尝试、3,600 秒及 120 个 Claude Code 轮次。独立实验应使用新实验 ID，已有 HTML 可能被直接重新评估。README 示例的默认配置与部分论文行不同，不能直接视为精确复现。"},"title":{"en":"Record the full experiment setup","zh":"复现时记录完整实验配置"}}],"sources":[{"id":"paper","title":"GameASG-Bench paper v1 · Experimental settings and Tables 3–9","url":"https://arxiv.org/html/2609.21293v1"},{"id":"readme","title":"GameASG-Bench · Task format and execution guide (pinned release)","url":"https://github.com/areal-project/GameASG-Bench/blob/6427cb6517ce11ff350ec45fcae17b67c5aec245/README.md"},{"id":"contract","title":"GameASG-Bench · Armor Alley public gameplay contract (pinned release)","url":"https://github.com/areal-project/GameASG-Bench/blob/6427cb6517ce11ff350ec45fcae17b67c5aec245/task/armor-alley/tdd.md"},{"id":"reporter","title":"GameASG-Bench · Strict success, aggregation and cost estimation code","url":"https://github.com/areal-project/GameASG-Bench/blob/6427cb6517ce11ff350ec45fcae17b67c5aec245/scripts/gen_report.py"},{"id":"aggregation","title":"GameASG-Bench · Applicable-check and priority aggregation code","url":"https://github.com/areal-project/GameASG-Bench/blob/6427cb6517ce11ff350ec45fcae17b67c5aec245/scripts/l2/result_summary.js"},{"id":"browser-runner","title":"GameASG-Bench · Browser check isolation and execution code","url":"https://github.com/areal-project/GameASG-Bench/blob/6427cb6517ce11ff350ec45fcae17b67c5aec245/scripts/l2/run.js"},{"id":"runtime","title":"GameASG-Bench · Pinned runtime image","url":"https://github.com/areal-project/GameASG-Bench/blob/6427cb6517ce11ff350ec45fcae17b67c5aec245/Dockerfile"},{"id":"launcher","title":"GameASG-Bench · Generation budgets and container setup","url":"https://github.com/areal-project/GameASG-Bench/blob/6427cb6517ce11ff350ec45fcae17b67c5aec245/docker_run.sh"}],"summary":{"en":"A browser-game generation benchmark with 47 tasks. It checks whether a coding agent delivers the requested gameplay, not just valid source code; strict task success and average check rates describe different levels of completion.","zh":"一项包含 47 道任务的浏览器游戏生成评测，检查编码 Agent 是否交付了要求的玩法，而不仅是合法源码。严格任务成功率与平均检查通过率分别描述不同层次的完成情况。"},"title":"GameASG-Bench"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.21293","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_gameasg-bench_01bef6f8"},"modality":null,"name":"GameASG-Bench","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21293","source_url":"https://arxiv.org/abs/2609.21293"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21293","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21293","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Game commentary is an open-ended generation task requiring multimodal perception, strategic reasoning, and contextual knowledge.","descriptions":{"en":"Game commentary is an open-ended generation task requiring multimodal perception, strategic reasoning, and contextual knowledge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11129","languages":[],"library_direction":{"basis":"model-review-daily","categories":["chat-writing","video-understanding"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_gamecommbench_6b65b520"},"modality":null,"name":"GameCommBench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11129","source_url":"https://arxiv.org/abs/2610.11129"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11129","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11129","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark of 140 Godot engine tasks across 15 game families. Coding agents must build playable games end-to-end, and the games are scored through replayed demonstrations and rubric-guided multimodal judging.","descriptions":{"en":"A benchmark of 140 Godot engine tasks across 15 game families. Coding agents must build playable games end-to-end, and the games are scored through replayed demonstrations and rubric-guided multimodal judging."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17861","languages":[],"library_direction":null,"modality":null,"name":"GameCraft-Bench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17861","source_url":"https://arxiv.org/abs/2606.17861"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17861","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17861","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_gameenginebench_f2b34812"},"categories":["Code & Software","Manufacturing & Process Control","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"GameEngineBench evaluates coding agents on scoped C++ implementation tasks within Unreal Engine 5 projects, built from nine real-world game repositories. The 110 tasks span gameplay, multiplayer, AI, animation, UI, and other areas, requiring native C++ changes that compile and pass behavioral tests.","descriptions":{"en":"GameEngineBench evaluates coding agents on scoped C++ implementation tasks within Unreal Engine 5 projects, built from nine real-world game repositories. The 110 tasks span gameplay, multiplayer, AI, animation, UI, and other areas, requiring native C++ changes that compile and pass behavioral tests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.03525","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_gameenginebench_f2b34812"},"modality":null,"name":"GameEngineBench","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03525","source_url":"https://arxiv.org/abs/2607.03525"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03525","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03525","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":569,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tencentarc/gamehorizon","source_record":"bm_gamehorizon-bench_1643c19d"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A planned evaluation suite for gameplay across offline multiple-choice questions and online task environments.","descriptions":{"en":"A planned evaluation suite for gameplay across offline multiple-choice questions and online task environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.25001","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","multi-agent-games"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_gamehorizon-bench_1643c19d"},"modality":null,"name":"GameHorizon-Bench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25001","source_url":"https://arxiv.org/abs/2609.25001"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25001","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25001","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:nju-link/gamelogicbench","source_record":"bm_gamelogicbench_ad3aeeec"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on 72 gameplay-logic tasks in Godot projects using black-box behavioral assertions under deterministic simulation.","descriptions":{"en":"Evaluates coding agents on 72 gameplay-logic tasks in Godot projects using black-box behavioral assertions under deterministic simulation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.21562","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_gamelogicbench_ad3aeeec"},"modality":null,"name":"GameLogicBench","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21562","source_url":"https://arxiv.org/abs/2609.21562"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21562","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21562","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":224,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":62,"join":"llm_stats_id","repository":null,"source_record":"catalog_185897bc155c560e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GameWorld evaluates agents on interactive game environments, testing perception, planning, and sequential decision-making to accomplish in-game objectives.","descriptions":{"en":"GameWorld evaluates agents on interactive game environments, testing perception, planning, and sequential decision-making to accomplish in-game objectives."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gameworld:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gameworld?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gameworld","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"llm_stats_id","primary":"multi-agent-games","source_record":"catalog_185897bc155c560e"},"modality":"multimodal","name":"GameWorld","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":31.2,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.312,"raw_min":0.259,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gameworld:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gameworld?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-gameworld","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gameworld?top_n=500","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":null,"source_record":"bm_gamexpert-bench_691da8d8"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents across three game development lifecycle tracks: generation, bug diagnosis and repair, and multi-turn optimization, using live interaction, behavioral tests, and product criteria.","descriptions":{"en":"Evaluates coding agents across three game development lifecycle tracks: generation, bug diagnosis and repair, and multi-turn optimization, using live interaction, behavioral tests, and product criteria."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.21833","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_gamexpert-bench_691da8d8"},"modality":null,"name":"GameXpert-Bench","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.21833","source_url":"http://arxiv.org/abs/2608.21833v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-21833","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21833v1","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","大语言模型","LLM","知识储备","Knowledge","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GAOKAO-bench is an evaluation framework that utilizes Chinese high school entrance examination (GAOKAO) questions as a dataset to evaluate the language understanding and logical reasoning abilities of large language models. GAOKAO-bench是一个以中国高考题目为数据集，旨在提供和人类对齐的，直观，高效地测评大模型语言理解能力、逻辑推理能力的测评框架","descriptions":{"en":"GAOKAO-bench is an evaluation framework that utilizes Chinese high school entrance examination (GAOKAO) questions as a dataset to evaluate the language understanding and logical reasoning abilities of large language models.","zh":"GAOKAO-bench是一个以中国高考题目为数据集，旨在提供和人类对齐的，直观，高效地测评大模型语言理解能力、逻辑推理能力的测评框架"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:500","languages":["Chinese"],"library_direction":null,"modality":null,"name":"GAOKAO-Bench","openness":"unknown","publisher":null,"released":"2023-05-21","released_reference":{"basis":"paper_first_version","note":"First version of the GAOKAO benchmark paper, not a later paper revision.","source_key":"opencompass:500","source_url":"https://arxiv.org/abs/2305.12474"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-500-gaokao-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GAOKAO-Bench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","ACL 2024","多模态模型","VLM","知识储备","Knowledge","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GAOKAO-MM is a multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO), comprising of 8 subjects and 12 types of images, such as diagrams, function graphs, maps and photos. GAOKAO-MM 是一个基于中国高考的多模态基准，包含 8 个科目和 12 种图像类型，例如图表、函数图、地图和照片。GAOKAO-MM 源自本土中文语境，并对模型的能力设置了人类水平的要求，包括感知、理解、知识和推理。","descriptions":{"en":"GAOKAO-MM is a multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO), comprising of 8 subjects and 12 types of images, such as diagrams, function graphs, maps and photos.","zh":"GAOKAO-MM 是一个基于中国高考的多模态基准，包含 8 个科目和 12 种图像类型，例如图表、函数图、地图和照片。GAOKAO-MM 源自本土中文语境，并对模型的能力设置了人类水平的要求，包括感知、理解、知识和推理。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1083","languages":["Chinese"],"library_direction":null,"modality":null,"name":"GAOKAO-MM","openness":"unknown","publisher":"OpenMOSS","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1083-gaokao-mm","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GAOKAO-MM","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["gas-sensor-drift-ml-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:pesquisadoug/gas-sensor-drift-ml-benchmark","source_record":"bm_gas-sensor-drift-ml-benchmark_0e87fb0a"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A multi-task benchmark on the UCI Gas Sensor Array Drift dataset covering gas classification, concentration regression, clustering, and later-batch temporal evaluation. It fixes random hold-out and batches 1-5 to 6-10 split protocols, compares dummy, tree, SVM, and MLP baselines, and reports classification, regression, and clustering metrics.","descriptions":{"en":"A multi-task benchmark on the UCI Gas Sensor Array Drift dataset covering gas classification, concentration regression, clustering, and later-batch temporal evaluation. It fixes random hold-out and batches 1-5 to 6-10 split protocols, compares dummy, tree, SVM, and MLP baselines, and reports classification, regression, and clustering metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:pesquisadoug/gas-sensor-drift-ml-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Gas Sensor Drift ML Benchmark","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:pesquisadoug/gas-sensor-drift-ml-benchmark","source_url":"https://github.com/PesquisaDoug/gas-sensor-drift-ml-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-pesquisadoug-gas-sensor-drift-ml-benchmark","source":"claire_radar","source_url":"https://github.com/PesquisaDoug/gas-sensor-drift-ml-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":138,"github_stars_30d":-2,"hf_dataset_downloads":113,"hf_paper_upvotes":19,"join":"claire_record_id","repository":"gh:rzhub/gatemem","source_record":"bm_gatemem_7d143ff2"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"GateMem evaluates memory governance in multi-principal shared-memory agents across medical, office, education, and household domains. It measures utility, access control, and active forgetting via checkpoints and a composite score.","descriptions":{"en":"GateMem evaluates memory governance in multi-principal shared-memory agents across medical, office, education, and household domains. It measures utility, access control, and active forgetting via checkpoints and a composite score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.18829","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","safety-alignment"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_gatemem_7d143ff2"},"modality":null,"name":"GateMem","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18829","source_url":"https://arxiv.org/abs/2606.18829"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18829","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18829","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Finance","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"GAUGE evaluates agent-built financial valuation models against observed analyst practice using 56 facets, eight validity gates, and a failure-aware score over 196 tasks.","descriptions":{"en":"GAUGE evaluates agent-built financial valuation models against observed analyst practice using 56 facets, eight validity gates, and a failure-aware score over 196 tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24889","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_gauge_c5d48106"},"modality":null,"name":"GAUGE","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24889","source_url":"https://arxiv.org/abs/2607.24889"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24889","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24889","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"GAUGE evaluates physical fidelity of simulation engines and video world models using 22 task families covering rigid bodies, cables, textiles, and deformable objects, with real-world trajectories and calibrated metadata.","descriptions":{"en":"GAUGE evaluates physical fidelity of simulation engines and video world models using 22 task families covering rigid bodies, cables, textiles, and deformable objects, with real-world trajectories and calibrated metadata."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05948","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_gauge_6ce93fe5"},"modality":null,"name":"GAUGE","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05948","source_url":"https://arxiv.org/abs/2608.05948"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05948","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05948","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","物理保真"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GAUGE: A Physical‑Realism Evaluation Benchmark Leveraging real‑world replicated experimental data as ground‑truth references, it delivers systematic evaluation for world models and physics engines across rigid bodies, ropes, fabrics, and 3D soft bodies. 物理真实度评测基准GAUGE，以真实重复实验数据为对照标准，为世界模型与物理引擎提供跨刚体、绳索、织物和三维软体的系统化评测。","descriptions":{"en":"GAUGE: A Physical‑Realism Evaluation Benchmark Leveraging real‑world replicated experimental data as ground‑truth references, it delivers systematic evaluation for world models and physics engines across rigid bodies, ropes, fabrics, and 3D soft bodies.","zh":"物理真实度评测基准GAUGE，以真实重复实验数据为对照标准，为世界模型与物理引擎提供跨刚体、绳索、织物和三维软体的系统化评测。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:2574","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"artifact:arxiv:2608.05948","primary":"world-models","source_record":"bm_gauge_6ce93fe5"},"modality":null,"name":"GAUGE","openness":"unknown","publisher":"上海人工智能实验室","released":"2026-08-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2574-gauge","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GAUGE","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":20,"join":"claire_record_id","repository":"gh:gauntlet-benchmark/evaluation-harness","source_record":"bm_gauntletbench_be5725f6"},"categories":["Agents","General AI","Geometric reasoning","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"GauntletBench evaluates agent generalisation across five professional web applications with 100 vision-intensive tasks, probing temporal perception, graphical understanding, and 3D reasoning via automated objective scoring.","descriptions":{"en":"GauntletBench evaluates agent generalisation across five professional web applications with 100 vision-intensive tasks, probing temporal perception, graphical understanding, and 3D reasoning via automated objective scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.14397","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_gauntletbench_be5725f6"},"modality":null,"name":"GauntletBench","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14397","source_url":"https://arxiv.org/abs/2606.14397"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14397","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14397","topics":["computer-use","agents","reasoning-math"],"unit":null},{"aliases":["Gaze Target Estimation Anywhere with Concepts"],"attention_signals":{"as_of":"2026-10-11","github_stars":23,"github_stars_30d":0,"hf_dataset_downloads":102,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:irohxu/gazeanywhere","source_record":"bm_gaze-target-estimation-anywhere-with-conce_dea4f989"},"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates promptable gaze target estimation: given an image and a text or visual prompt identifying a subject, models output the subject's location, in/out-of-frame status, and a gaze target heatmap.","descriptions":{"en":"Evaluates promptable gaze target estimation: given an image and a text or visual prompt identifying a subject, models output the subject's location, in/out-of-frame status, and a gaze target heatmap."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11367","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_gaze-target-estimation-anywhere-with-conce_dea4f989"},"modality":null,"name":"Gaze-Co","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11367","source_url":"https://arxiv.org/abs/2608.11367"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11367","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11367","topics":[],"unit":null},{"aliases":["Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM capability for rule-intensive review of Chinese GB/T national standard documents, including error location, dimension, and type identification across 25 diagnostic error types.","descriptions":{"en":"Evaluates LLM capability for rule-intensive review of Chinese GB/T national standard documents, including error location, dimension, and type identification across 25 diagnostic error types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06312","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","document-ocr"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_benchmarking-and-enhancing-llms-for-rule-i_d857dd31"},"modality":null,"name":"GB/T-Bench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06312","source_url":"https://arxiv.org/abs/2608.06312"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06312","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06312","topics":[],"unit":null},{"aliases":["Urban Boundaries, Social Barriers: A Benchmark and Vision-Centric Framework for Mapping Gated Communities and Equity Implications"],"attention_signals":{"as_of":"2026-10-13","github_stars":7,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:minweizhao/gba-gcs","source_record":"bm_urban-boundaries-social-barriers-a-benchma_4d199b68"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal recognition of gated versus open residential communities in China's Greater Bay Area using satellite imagery, text metadata, and structured attributes.","descriptions":{"en":"Evaluates multimodal recognition of gated versus open residential communities in China's Greater Bay Area using satellite imagery, text metadata, and structured attributes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.03804","languages":[],"library_direction":null,"modality":null,"name":"GBA-GCs","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03804","source_url":"https://arxiv.org/abs/2609.03804"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03804","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03804","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"GBU-Palm is a large-scale multimodal video dataset for palm presentation attack detection, containing 21,326 videos from 105 subjects across six acquisition environments, including bona fide, Print, and Replay attacks, with 6,310 synchronized RGB-NIR samples. It evaluates video architectures under environment-matched and held-out-environment protocols, using metrics such as true accept, true reject, false accept, and false reject rates.","descriptions":{"en":"GBU-Palm is a large-scale multimodal video dataset for palm presentation attack detection, containing 21,326 videos from 105 subjects across six acquisition environments, including bona fide, Print, and Replay attacks, with 6,310 synchronized RGB-NIR samples. It evaluates video architectures under environment-matched and held-out-environment protocols, using metrics such as true accept, true reject, false accept, and false reject rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14389","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_gbu-palm_d39e8a5c"},"modality":null,"name":"GBU-Palm","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14389","source_url":"https://arxiv.org/abs/2608.14389"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14389","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14389","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Reasoning","Robotics & Embodied AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"GCA-Bench evaluates robotic grasping in scenarios requiring scene-level reasoning and semantic constraints, comparing large foundation models on complex action tasks.","descriptions":{"en":"GCA-Bench evaluates robotic grasping in scenarios requiring scene-level reasoning and semantic constraints, comparing large foundation models on complex action tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14341","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_gca-bench_71ae6cdc"},"modality":null,"name":"GCA-Bench","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14341","source_url":"https://arxiv.org/abs/2607.14341"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14341","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14341","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":17,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:surge-ai/gdp-pdf","source_record":"catalog_a2ed7249d0f2c4d1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GDP.pdf is a knowledge-work vision benchmark that evaluates models on economically valuable professional tasks presented as visual documents (PDFs), testing document-based reasoning, chart and table interpretation, and problem solving without tools.","descriptions":{"en":"GDP.pdf is a knowledge-work vision benchmark that evaluates models on economically valuable professional tasks presented as visual documents (PDFs), testing document-based reasoning, chart and table interpretation, and problem solving without tools."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gdp-pdf:claude-fable-5","reported_at":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdp-pdf?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gdp-pdf","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","professional-work"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_a2ed7249d0f2c4d1"},"modality":"multimodal","name":"GDP.pdf","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":81.6,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.816,"raw_min":0.227,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gdp-pdf:claude-sonnet-5","reported_date":"2026-06-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdp-pdf?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-gdp-pdf","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdp-pdf?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["GDP.pdf"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Vision-based knowledge work over 100 PDFs across 10 domains, reported no-tools. Published by Surge AI with a public dataset (huggingface.co/datasets/surgeai/GDP.pdf), harness (github.com/surge-ai/gdp-pdf) and paper (arXiv:2607.11192). The name resembles GDPval but the two are separate instruments and must not be merged.","descriptions":{"en":"Vision-based knowledge work over 100 PDFs across 10 domains, reported no-tools. Published by Surge AI with a public dataset (huggingface.co/datasets/surgeai/GDP.pdf), harness (github.com/surge-ai/gdp-pdf) and paper (arXiv:2607.11192). The name resembles GDPval but the two are separate instruments and must not be merged."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:gdp_pdf","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["document-ocr"],"join":"review:14H034160212:2026-10-15","primary":"document-ocr","source_record":null},"modality":null,"name":"GDP.pdf","openness":"unknown","publisher":null,"released":"2026-04-14","released_reference":{"basis":"benchmark_release","source_key":"model-reports:gdp_pdf","source_url":"https://surgehq.ai/blog/gdp-pdf-can-100b-ai-models-master-the-documents-that-run-the-world"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"gdp_pdf","source":"model_reports","source_url":"https://surgehq.ai/blog/gdp-pdf-can-100b-ai-models-master-the-documents-that-run-the-world","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":67,"github_stars_30d":4,"hf_dataset_downloads":null,"hf_paper_upvotes":26,"join":"claire_record_id","repository":"gh:prism-shadow/gdpevo","source_record":"bm_gdpevo_5e650b40"},"categories":["Agents","General AI","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"GDPevo evaluates agent self-evolution on real business tasks across 24 groups (240 tasks) in domains like CRM, ERP, finance, and healthcare. It uses rule hybridization to attribute test-time gains to training experience, with held-out test tasks.","descriptions":{"en":"GDPevo evaluates agent self-evolution on real business tasks across 24 groups (240 tasks) in domains like CRM, ERP, finance, and healthcare. It uses rule hybridization to attribute test-time gains to training experience, with held-out test tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.03764","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","professional-work"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_gdpevo_5e650b40"},"modality":null,"name":"GDPevo","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03764","source_url":"https://arxiv.org/abs/2608.03764"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03764","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03764","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":258803,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_8aacade59f3852c9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning","finance","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GDPval is an OpenAI benchmark evaluating AI models on economically valuable, real-world knowledge-work tasks spanning many professional occupations and industries.","descriptions":{"en":"GDPval is an OpenAI benchmark evaluating AI models on economically valuable, real-world knowledge-work tasks spanning many professional occupations and industries."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gdpval:nemotron-3-ultra-550b-a55b","reported_at":"2026-06-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gdpval","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_8aacade59f3852c9"},"modality":"text","name":"GDPval","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":87.9,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.879,"raw_min":0.467,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gdpval:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-gdpval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval?top_n=500","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":["GDPval","GDPval-AA","GDPval-AA v2"],"attention_signals":null,"categories":["professional"],"collected_at":null,"description":"Graded by expert human comparison against real deliverables, and the \"-AA\" variants are run by Artificial Analysis rather than the vendor, so an Elo here is not comparable to a vendor-run pass rate.","descriptions":{"en":"Graded by expert human comparison against real deliverables, and the \"-AA\" variants are run by Artificial Analysis rather than the vendor, so an Elo here is not comparable to a vendor-run pass rate."},"evidence_summary":{"document_count":7,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000gdpval\u0000deepseek_v4_model_card\u0000gdpval_aa\u0000think max, Elo\u0000DeepSeek-V4-Pro","reported_at":"2026-04-22","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro"},"first_score_reported_at":"2026-04-22","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000gdpval\u0000deepseek_v4_model_card\u0000gdpval_aa\u0000think max, Elo\u0000DeepSeek-V4-Pro","reported_at":"2026-04-22","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:gdpval","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"GDPval","openness":"unknown","publisher":null,"released":"2025-09-25","released_reference":{"basis":"benchmark_release","source_key":"model-reports:gdpval","source_url":"https://openai.com/index/gdpval/"},"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":1861.0,"display_multiplier":1,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":1861.0,"raw_min":1395.0,"source_reference":{"obs_id":"curated\u0000gdpval\u0000anthropic_claude_opus_5_system_card\u0000gdpval_aa_v2\u0000ELO rating, max effort, agentic loop with shell access and web browsing\u0000Claude Opus 5","observation_id":"curated\u0000gdpval\u0000anthropic_claude_opus_5_system_card\u0000gdpval_aa_v2\u0000ELO rating, max effort, agentic loop with shell access and web browsing\u0000Claude Opus 5","reported_at":"2026-07-24","reported_date":"2026-07-24","source_id":"anthropic_claude_opus_5_system_card","source_url":"https://www.anthropic.com/news/claude-opus-5"},"unit":"elo"},"scored_models":[{"date":"2026-07-24","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"}],"slug":"gdpval","source":"model_reports","source_url":"https://openai.com/index/gdpval/","topics":[],"unit":"elo"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":657,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:artificialanalysis/stirrup","source_record":"catalog_22b7ac366fb64f0a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning","finance","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GDPval-AA evaluates AI agents on economically valuable professional knowledge-work tasks and reports performance as an Elo score.","descriptions":{"en":"GDPval-AA evaluates AI agents on economically valuable professional knowledge-work tasks and reports performance as an Elo score."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gdpval-aa:inkling-small","reported_at":"2026-07-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-aa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gdpval-aa","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_22b7ac366fb64f0a"},"modality":"text","name":"GDPval-AA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":1769.0,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":1769.0,"raw_min":1269.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gdpval-aa:glm-5.3","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-aa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"}],"slug":"llm-stats-gdpval-aa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-aa?top_n=500","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","agentic"],"collected_at":"2026-08-25T10:41:06Z","description":"Agentic real-world work tasks","descriptions":{"en":"Agentic real-world work tasks"},"evidence_summary":{"document_count":1,"model_count":213,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:gdpval-aa-v2:raw_elo:6a7c0e25-1dcb-4b15-8495-a8536a9da051","reported_at":"2023-03-14","source_url":"https://artificialanalysis.ai/evaluations/gdpval-aa"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:gdpval-aa-v2:raw_elo","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"GDPval-AA v2 (Elo)","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"release_announcement","note":"The publisher launches GDPval-AA v2 with human-anchored Elo, new judges and longer agent turns. The original GDPval dataset and GDPval-AA v1 have different release dates.","source_key":"artificial-analysis:gdpval-aa-v2:raw_elo","source_url":"https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-1"},"repo_kind":null,"repo_resolution_status":null,"score_count":213,"score_direction":"higher_is_better","score_summary":{"display_max":1834.67,"display_multiplier":1,"model_count":213,"model_count_basis":"source_model_id","numeric_count":213,"raw_max":1834.67,"raw_min":-122.53,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:gdpval-aa-v2:raw_elo:b8fc61f7-5e9a-49e6-8547-6ac56db24627","reported_date":"2026-07-24","source_url":"https://artificialanalysis.ai/evaluations/gdpval-aa"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2023-03-14","date_precision":"model_announcement","name":"GPT-4"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-gdpval-aa-v2-raw-elo","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/gdpval-aa","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","agentic"],"collected_at":"2026-08-25T10:41:06Z","description":"Agentic real-world work tasks","descriptions":{"en":"Agentic real-world work tasks"},"evidence_summary":{"document_count":1,"model_count":213,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:gdpval-aa-v2:normalized_score:6a7c0e25-1dcb-4b15-8495-a8536a9da051","reported_at":"2023-03-14","source_url":"https://artificialanalysis.ai/evaluations/gdpval-aa"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:gdpval-aa-v2:normalized_score","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"GDPval-AA v2 (normalized score)","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"release_announcement","note":"The publisher launches GDPval-AA v2; this record is its normalized display score, kept separate from its raw-Elo source record.","source_key":"artificial-analysis:gdpval-aa-v2:normalized_score","source_url":"https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-1"},"repo_kind":null,"repo_resolution_status":null,"score_count":213,"score_direction":"higher_is_better","score_summary":{"display_max":66.7335,"display_multiplier":100,"model_count":213,"model_count_basis":"source_model_id","numeric_count":213,"raw_max":0.667335,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:gdpval-aa-v2:normalized_score:b8fc61f7-5e9a-49e6-8547-6ac56db24627","reported_date":"2026-07-24","source_url":"https://artificialanalysis.ai/evaluations/gdpval-aa"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2023-03-14","date_precision":"model_announcement","name":"GPT-4"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-gdpval-aa-v2-normalized-score","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/gdpval-aa","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":272285,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_44e326eb9a9e1060"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","finance","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GDPval-MM is the multimodal variant of the GDPval benchmark, evaluating AI model performance on real-world economically valuable tasks that require processing and generating multimodal content including documents, slides, diagrams, spreadsheets, images, and other professional deliverables across diverse industries.","descriptions":{"en":"GDPval-MM is the multimodal variant of the GDPval benchmark, evaluating AI model performance on real-world economically valuable tasks that require processing and generating multimodal content including documents, slides, diagrams, spreadsheets, images, and other professional deliverables across diverse industries."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gdpval-mm:minimax-m2.5","reported_at":"2026-02-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-mm?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gdpval-mm","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_44e326eb9a9e1060"},"modality":"multimodal","name":"GDPval-MM","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":84.89999999999999,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.849,"raw_min":0.59,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gdpval-mm:gpt-5.5","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-mm?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 Pro"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax M2.5"}],"slug":"llm-stats-gdpval-mm","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-mm?top_n=500","topics":["multimodal","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":272285,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_3a70d85f10b95f69"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning","finance","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GDPval-Rubrics evaluates AI model performance on economically valuable knowledge work tasks drawn from the public GDPval dataset. It uses pointwise scoring based on public rubrics, with the environment aligned to the GDPval-AA scaffolding.","descriptions":{"en":"GDPval-Rubrics evaluates AI model performance on economically valuable knowledge work tasks drawn from the public GDPval dataset. It uses pointwise scoring based on public rubrics, with the environment aligned to the GDPval-AA scaffolding."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gdpval-rubrics:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-rubrics?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gdpval-rubrics","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_3a70d85f10b95f69"},"modality":"text","name":"GDPval-Rubrics","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":74.78,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.7478,"raw_min":0.7478,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gdpval-rubrics:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-rubrics?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-gdpval-rubrics","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-rubrics?top_n=500","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Mathematics & Formal Science","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"GEB-Bench evaluates models on abstract structural motifs (e.g., self-reference, strange loops) presented in multiple modalities (natural scenes, folk stories, math theorems, programmatic skeletons) with tasks probing cross-modal mapping.","descriptions":{"en":"GEB-Bench evaluates models on abstract structural motifs (e.g., self-reference, strange loops) presented in multiple modalities (natural scenes, folk stories, math theorems, programmatic skeletons) with tasks probing cross-modal mapping."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04111","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","visual-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_geb-bench_e0eb6ac0"},"modality":null,"name":"GEB-Bench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04111","source_url":"https://arxiv.org/abs/2608.04111"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04111","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04111","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_gedit3d-bench_10c06cb7"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Although recent 3D generative models produce increasingly realistic assets, controllable 3D asset editing remains challenging.","descriptions":{"en":"Although recent 3D generative models produce increasingly realistic assets, controllable 3D asset editing remains challenging."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34271","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","spatial-3d"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_gedit3d-bench_10c06cb7"},"modality":null,"name":"GEdit3D-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34271","source_url":"https://arxiv.org/abs/2609.34271"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34271","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34271","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","其他","Other","多模态模型","VLM","视觉生成","Visual Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GenAI-Bench is a benchmark designed to benchmark MLLMs’s ability in judging the quality of AI generative contents, containing over 40,000 human ratings to evaluate the performance of MLLMs on aligning with human preferences. GenAI-Bench用于衡量MLLM判断AI生成内容质量的能力，包含超过40000个人工评分用以评估大模型与人类偏好的一致性。","descriptions":{"en":"GenAI-Bench is a benchmark designed to benchmark MLLMs’s ability in judging the quality of AI generative contents, containing over 40,000 human ratings to evaluate the performance of MLLMs on aligning with human preferences.","zh":"GenAI-Bench用于衡量MLLM判断AI生成内容质量的能力，包含超过40000个人工评分用以评估大模型与人类偏好的一致性。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1376","languages":[],"library_direction":null,"modality":"multimodal","name":"GenAI-Bench","openness":"open","publisher":"Carnegie Mellon University","released":"2024-06-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1376-genai-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GenAI-Bench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":46,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":50,"join":"claire_record_id","repository":"gh:darlednik/geneb","source_record":"bm_geneb_31b4a4f4"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"GENEB evaluates frozen representations from 40 genomic foundation models across 100 DNA classification tasks in 13 functional categories, using a unified linear probing protocol with full-data, 10-shot, and 1-shot regimes. Primary metric is Matthews correlation coefficient, with rankings at overall, category, and task levels.","descriptions":{"en":"GENEB evaluates frozen representations from 40 genomic foundation models across 100 DNA classification tasks in 13 functional categories, using a unified linear probing protocol with full-data, 10-shot, and 1-shot regimes. Primary metric is Matthews correlation coefficient, with rankings at overall, category, and task levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.04525","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_geneb_31b4a4f4"},"modality":null,"name":"GENEB","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04525","source_url":"https://arxiv.org/abs/2606.04525"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04525","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04525","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GeneBench is an evaluation focused on multi-stage scientific data analysis in genetics and quantitative biology. Tasks require reasoning about ambiguous or noisy data with minimal supervisory guidance, addressing realistic obstacles such as hidden confounders or QC failures, and correctly implementing and interpreting modern statistical methods.","descriptions":{"en":"GeneBench is an evaluation focused on multi-stage scientific data analysis in genetics and quantitative biology. Tasks require reasoning about ambiguous or noisy data with minimal supervisory guidance, addressing realistic obstacles such as hidden confounders or QC failures, and correctly implementing and interpreting modern statistical methods."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:genebench:gpt-5.5","reported_at":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:genebench","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_4180a59b1ba29e4a"},"modality":"text","name":"GeneBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":33.2,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.332,"raw_min":0.25,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:genebench:gpt-5.5-pro","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 Pro"}],"slug":"llm-stats-genebench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench?top_n=500","topics":["agents","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GeneBench-Pro is a research-level benchmark of 129 multi-stage computational-biology problems spanning genomics, quantitative biology, and translational biomedicine. Each problem gives the agent a messy dataset, brief context, and a target estimand, and requires navigating dependent inferential decision points to reach a verifiable answer.","descriptions":{"en":"GeneBench-Pro is a research-level benchmark of 129 multi-stage computational-biology problems spanning genomics, quantitative biology, and translational biomedicine. Each problem gives the agent a messy dataset, brief context, and a target estimand, and requires navigating dependent inferential decision points to reach a verifiable answer."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:genebench-pro:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench-pro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:genebench-pro","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_0a83fae6a9b340ef"},"modality":"text","name":"GeneBench-Pro","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":28.7,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.287,"raw_min":0.108,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:genebench-pro:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench-pro?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-genebench-pro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench-pro?top_n=500","topics":["agents","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"General Bench is a set of universal evaluation benchmarks for multimodal large models, covering language, image, video, audio, and 3D five modalities, with a total of 145 skills, over 700 tasks, and 325800 samples. General-Bench 是一套面向多模态大模型的通用评测基准，涵盖语言、图像、视频、音频和 3D 五大模态，共计 145 项技能、700 余个任务，包含 325 800 条样本。","descriptions":{"en":"General Bench is a set of universal evaluation benchmarks for multimodal large models, covering language, image, video, audio, and 3D five modalities, with a total of 145 skills, over 700 tasks, and 325800 samples.","zh":"General-Bench 是一套面向多模态大模型的通用评测基准，涵盖语言、图像、视频、音频和 3D 五大模态，共计 145 项技能、700 余个任务，包含 325 800 条样本。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2081","languages":["English","Chinese"],"library_direction":null,"modality":"multimodal","name":"General-Bench","openness":"unknown","publisher":"NUS , NTU , ZJU , KAUST , PKU , HFUT , UR , WHU , NJU , SJTU , Skywork AI","released":"2025-05-07","released_reference":{"basis":"paper_first_version","note":"First version introducing General-Level and General-Bench.","source_key":"opencompass:2081","source_url":"https://arxiv.org/abs/2505.04620"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2081-general-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/General-Bench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_generative-embedding-benchmark_a087748d"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"GEB evaluates embeddings by measuring answer-relevant content recoverable by a decoder, using a visual question-answering dataset with development and test splits, and scoring answer quality.","descriptions":{"en":"GEB evaluates embeddings by measuring answer-relevant content recoverable by a decoder, using a visual question-answering dataset with development and test splits, and scoring answer quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06972","languages":[],"library_direction":null,"modality":null,"name":"Generative Embedding Benchmark","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06972","source_url":"https://arxiv.org/abs/2608.06972"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06972","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06972","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":2,"hf_dataset_downloads":796,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:animeshshaw/geniac-secbench","source_record":"bm_geniac-secbench_393dfc82"},"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM-generated Infrastructure-as-Code security using 100 scenarios across Terraform, CloudFormation, and Kubernetes, scanned by Checkov, Trivy, and KICS, with metrics including vulnerability density, schema validity, and LLM-judge review.","descriptions":{"en":"Evaluates LLM-generated Infrastructure-as-Code security using 100 scenarios across Terraform, CloudFormation, and Kubernetes, scanned by Checkov, Trivy, and KICS, with metrics including vulnerability density, schema validity, and LLM-judge review."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.28021","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","code-generation"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_geniac-secbench_393dfc82"},"modality":null,"name":"GenIaC-SecBench","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28021","source_url":"http://arxiv.org/abs/2608.28021v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28021","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28021v1","topics":["coding","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates relative object scale in image generation and editing tasks using a benchmark and evaluation protocol.","descriptions":{"en":"Evaluates relative object scale in image generation and editing tasks using a benchmark and evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.00525","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","image-video-generation"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_genscale_fa84e6dd"},"modality":null,"name":"GenScale","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00525","source_url":"https://arxiv.org/abs/2609.00525"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00525","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00525","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"GEO-Bench evaluates ranking manipulation attacks in generative engine optimization under a unified protocol, measuring effectiveness and stealth across five datasets against a fixed ranker, with open-source attack implementations.","descriptions":{"en":"GEO-Bench evaluates ranking manipulation attacks in generative engine optimization under a unified protocol, measuring effectiveness and stealth across five datasets against a fixed ranker, with open-source attack implementations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29107","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_geo-bench_c7e53a2d"},"modality":null,"name":"GEO-Bench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29107","source_url":"https://arxiv.org/abs/2605.29107"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29107","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29107","topics":["agents","safety"],"unit":null},{"aliases":["GeoAgent: Evaluating VLM Geolocalization Through Embodied Navigation"],"attention_signals":null,"categories":["Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"GeoAgent evaluates VLM geolocalization performance through embodied navigation in Street View environments, comparing agentic exploration to static image baselines.","descriptions":{"en":"GeoAgent evaluates VLM geolocalization performance through embodied navigation in Street View environments, comparing agentic exploration to static image baselines."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29483","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","embodied-robotics"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_geoagent-evaluating-vlm-geolocalization-th_2dd205da"},"modality":null,"name":"GeoAgent","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29483","source_url":"https://arxiv.org/abs/2608.29483"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29483","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29483","topics":["reasoning-math"],"unit":null},{"aliases":["GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:rfr2003/geobenchllm","source_record":"bm_geobenchllm-a-comprehensive-benchmark-for-_dfb41079"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on 17 geo-related subdatasets across 8 tasks covering knowledge, reasoning, and application, with inference and evaluation scripts.","descriptions":{"en":"Evaluates LLMs on 17 geo-related subdatasets across 8 tasks covering knowledge, reasoning, and application, with inference and evaluation scripts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.07411","languages":[],"library_direction":{"basis":"model-review","categories":["science","knowledge-expert-qa"],"join":"claire_record_id","primary":"science","source_record":"bm_geobenchllm-a-comprehensive-benchmark-for-_dfb41079"},"modality":null,"name":"GeoBenchLLM","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07411","source_url":"https://arxiv.org/abs/2608.07411"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07411","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07411","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"GeoDisaster is an operational geospatial disaster reasoning benchmark with 2,921 instances across 43 question types and five task families, integrating EO/GIS evidence and grounding answers in executable geospatial workflows.","descriptions":{"en":"GeoDisaster is an operational geospatial disaster reasoning benchmark with 2,921 instances across 43 question types and five task families, integrating EO/GIS evidence and grounding answers in executable geospatial workflows."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17246","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_geodisaster_9f255b61"},"modality":null,"name":"GeoDisaster","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17246","source_url":"https://arxiv.org/abs/2606.17246"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17246","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17246","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"GeoFidelity-Bench evaluates segment-level geographic fidelity in text-to-image street-view generation using 7,117 Mapillary images across 109 road segments in 25 cities.","descriptions":{"en":"GeoFidelity-Bench evaluates segment-level geographic fidelity in text-to-image street-view generation using 7,117 Mapillary images across 109 road segments in 25 cities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.23669","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_geofidelity-bench_953b4352"},"modality":null,"name":"GeoFidelity-Bench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23669","source_url":"https://arxiv.org/abs/2606.23669"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23669","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23669","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":20,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:links-ads/geoid-flood","source_record":"bm_geoid-flood_b555af53"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"GEOID-Flood is a large-scale multi-modal benchmark for flood segmentation with over 14,000 tiles, co-registered SAR and optical data, and validated labels.","descriptions":{"en":"GEOID-Flood is a large-scale multi-modal benchmark for flood segmentation with over 14,000 tiles, co-registered SAR and optical data, and validated labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.02315","languages":[],"library_direction":null,"modality":null,"name":"GEOID-Flood","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02315","source_url":"https://arxiv.org/abs/2608.02315"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02315","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02315","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"GeoNatureAgent Benchmark evaluates LLM agents on environmental geospatial analysis through structured tool calls to a self-hostable API. It includes 93 tasks across 18 categories, with metrics for accuracy and cost.","descriptions":{"en":"GeoNatureAgent Benchmark evaluates LLM agents on environmental geospatial analysis through structured tool calls to a self-hostable API. It includes 93 tasks across 18 categories, with metrics for accuracy and cost."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12821","languages":[],"library_direction":{"basis":"model-review","categories":["science","tool-use"],"join":"claire_record_id","primary":"science","source_record":"bm_geonatureagent-benchmark_854d9ae1"},"modality":null,"name":"GeoNatureAgent Benchmark","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12821","source_url":"https://arxiv.org/abs/2606.12821"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12821","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12821","topics":["agents","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ucf-sage/geooutagebench","source_record":"bm_geooutagebench_f3aa487d"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"We introduce GeoOutageBench, a benchmark for assessing LLM-based geospatiotemporal KGQA for multimodal outage and resilience analysis.","descriptions":{"en":"We introduce GeoOutageBench, a benchmark for assessing LLM-based geospatiotemporal KGQA for multimodal outage and resilience analysis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.36082","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_geooutagebench_f3aa487d"},"modality":null,"name":"GeoOutageBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36082","source_url":"https://arxiv.org/abs/2609.36082"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36082","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36082","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models.","descriptions":{"en":"Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.12881","languages":[],"library_direction":{"basis":"model-review-daily","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_geospatialbench_47ec0f77"},"modality":null,"name":"GeoSpatialBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12881","source_url":"https://arxiv.org/abs/2610.12881"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12881","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12881","topics":["reasoning-math"],"unit":null},{"aliases":["GeoStore: Finding Small Storefronts in Large Scenes -- A Fine-Grained POI Localization Benchmark with Global-to-Local Asymmetric Matching"],"attention_signals":null,"categories":["Factuality","General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Focuses on finding small storefronts in large scenes through a fine-grained POI localization benchmark with global-to-local asymmetric matching.","descriptions":{"en":"Focuses on finding small storefronts in large scenes through a fine-grained POI localization benchmark with global-to-local asymmetric matching."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02012","languages":[],"library_direction":null,"modality":null,"name":"GeoStore","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02012","source_url":"https://arxiv.org/abs/2609.02012"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02012","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02012","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"GeoT2V-Bench evaluates 3D consistency in camera-prompted text-to-video models via 3D reconstruction, using metrics like static rendering error and flow agreement.","descriptions":{"en":"GeoT2V-Bench evaluates 3D consistency in camera-prompted text-to-video models via 3D reconstruction, using metrics like static rendering error and flow agreement."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.24829","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","spatial-3d"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_geot2v-bench_948c1065"},"modality":null,"name":"GeoT2V-Bench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24829","source_url":"https://arxiv.org/abs/2606.24829"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24829","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24829","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Geometric reasoning","Language & Knowledge","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"GeoVAD-Bench diagnoses visual chain-of-thought for geometry problem solving along five dimensions: perception, auxiliary quality, utilization, deductive reasoning, and final correctness, with controlled No-Aux, Auto-Aux, and GT-Aux intervention settings.","descriptions":{"en":"GeoVAD-Bench diagnoses visual chain-of-thought for geometry problem solving along five dimensions: perception, auxiliary quality, utilization, deductive reasoning, and final correctness, with controlled No-Aux, Auto-Aux, and GT-Aux intervention settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.12606","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning","health-medicine"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_geovad-bench_dadd2d8d"},"modality":null,"name":"GeoVAD-Bench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12606","source_url":"https://arxiv.org/abs/2609.12606"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12606","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12606","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":37,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:giantsteps/giantsteps-tempo-dataset","source_record":"catalog_38e7b95132e4409c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A dataset for tempo estimation in electronic dance music containing 664 2-minute audio previews from Beatport, annotated from user corrections for evaluating automatic tempo estimation algorithms.","descriptions":{"en":"A dataset for tempo estimation in electronic dance music containing 664 2-minute audio previews from Beatport, annotated from user corrections for evaluating automatic tempo estimation algorithms."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:giantsteps-tempo:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/giantsteps-tempo?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:giantsteps-tempo","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_38e7b95132e4409c"},"modality":"audio","name":"GiantSteps Tempo","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":88.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.88,"raw_min":0.88,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:giantsteps-tempo:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/giantsteps-tempo?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-giantsteps-tempo","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/giantsteps-tempo?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Reliable evaluation of image forgery localization (IFL) requires assessing models under diverse distribution changes, yet existing benchmarks often cover limited manipulation conditions or entangle multiple factors in cross-dataset evaluat…","descriptions":{"en":"Reliable evaluation of image forgery localization (IFL) requires assessing models under diverse distribution changes, yet existing benchmarks often cover limited manipulation conditions or entangle multiple factors in cross-dataset evaluat…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.01778","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_giftbench_30e212ba"},"modality":null,"name":"GIFTBench","openness":"unknown","publisher":null,"released":"2026-10-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.01778","source_url":"https://arxiv.org/abs/2610.01778"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-01778","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01778","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"GISAgentBench evaluates LLM agents on multi-step GIS tasks from practitioner sources, with 349 tasks and executable reference trajectories.","descriptions":{"en":"GISAgentBench evaluates LLM agents on multi-step GIS tasks from practitioner sources, with 349 tasks and executable reference trajectories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01645","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_gisagentbench_880052a5"},"modality":null,"name":"GISAgentBench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01645","source_url":"https://arxiv.org/abs/2608.01645"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01645","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01645","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"GIScholarBench evaluates LLM overconfidence in GIS research across three tasks: metadata retrieval, literature linking, and research direction generation, using 10,865 papers from 25 GIScience journals.","descriptions":{"en":"GIScholarBench evaluates LLM overconfidence in GIS research across three tasks: metadata retrieval, literature linking, and research direction generation, using 10,865 papers from 25 GIScience journals."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.08036","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","science"],"join":"claire_record_id","primary":"factuality","source_record":"bm_gischolarbench_a6158b85"},"modality":null,"name":"GIScholarBench","openness":"unknown","publisher":null,"released":"2026-06-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08036","source_url":"https://arxiv.org/abs/2606.08036"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08036","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08036","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","智能体","Agent","任务执行","Task Execution","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GitGoodBench Lite is a subset of 900 samples for evaluating the performance of AI agents in resolving git tasks (see Supported Scenarios). GitGoodBench Lite是一个包含900个样本的子集，用于评估AI智能体在解决git任务方面的性能（参见支持的场景）。数据集中的样本在编程语言Python、Java和Kotlin以及样本类型合并冲突解决和文件提交语法之间均匀分布。因此，该数据集包含每种样本类型和编程语言各150个样本。","descriptions":{"en":"GitGoodBench Lite is a subset of 900 samples for evaluating the performance of AI agents in resolving git tasks (see Supported Scenarios).","zh":"GitGoodBench Lite是一个包含900个样本的子集，用于评估AI智能体在解决git任务方面的性能（参见支持的场景）。数据集中的样本在编程语言Python、Java和Kotlin以及样本类型合并冲突解决和文件提交语法之间均匀分布。因此，该数据集包含每种样本类型和编程语言各150个样本。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1865","languages":["English"],"library_direction":null,"modality":null,"name":"GitGoodBench","openness":"open","publisher":"JetBrains","released":"2025-05-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1865-gitgoodbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GitGoodBench","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_1532170c64d8f2b9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["physics","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Global PIQA is a multilingual commonsense reasoning benchmark that evaluates physical interaction knowledge across 100 languages and cultures. It tests AI systems' understanding of physical world knowledge in diverse cultural contexts through multiple choice questions about everyday situations requiring physical commonsense.","descriptions":{"en":"Global PIQA is a multilingual commonsense reasoning benchmark that evaluates physical interaction knowledge across 100 languages and cultures. It tests AI systems' understanding of physical world knowledge in diverse cultural contexts through multiple choice questions about everyday situations requiring physical commonsense."},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:global-piqa:gemini-3-pro-preview","reported_at":"2025-11-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-piqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:global-piqa","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_1532170c64d8f2b9"},"modality":"text","name":"Global PIQA","openness":"unknown","publication":{"evidence":[{"note":"The project repository links the Global PIQA paper and distinguishes parallel and non-parallel culturally specific data. Its commonsense reasoning across more than 100 languages matches this record. The current repository documents an expanded release; that release is not assigned to historical scores.","url":"https://github.com/mrlbenchmarks/global-piqa/blob/main/README.md"},{"note":"The accessible introducing paper is Global PIQA, first submitted in October 2025 and revised in May 2026. Its task and participatory multilingual construction distinguish it from the earlier English PIQA paper.","url":"https://arxiv.org/abs/2510.24081"}],"id":"global-piqa-original","links":[{"kind":"paper","title":"Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures","url":"https://arxiv.org/abs/2510.24081"},{"kind":"repo","title":"Global PIQA authors' repository","url":"https://github.com/mrlbenchmarks/global-piqa"}],"record_keys":["llm-stats:global-piqa"],"reviewed_at":"2026-10-26","scope":{"en":"Global PIQA is the multilingual and multicultural benchmark introduced in 2025, separate from the original English PIQA. The source record does not identify a release, language or parallel/non-parallel split.","zh":"Global PIQA 是 2025 年发布的多语言、多文化基准，与原始英文 PIQA 区分。当前来源记录未明确数据版本、语言或平行／非平行划分。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":93.4,"display_multiplier":100,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":0.934,"raw_min":0.594,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:global-piqa:gemini-3-pro-preview","reported_date":"2025-11-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-piqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-global-piqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-piqa?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"llm_stats_id","repository":null,"source_record":"catalog_69b9582b979cfa0a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive multilingual benchmark covering 42 languages that addresses cultural and linguistic biases in evaluation, with improved translation quality and culturally sensitive question subsets.","descriptions":{"en":"A comprehensive multilingual benchmark covering 42 languages that addresses cultural and linguistic biases in evaluation, with improved translation quality and culturally sensitive question subsets."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:global-mmlu:gemma-3n-e2b-it-litert-preview","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:global-mmlu","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_69b9582b979cfa0a"},"modality":"text","name":"Global-MMLU","openness":"unknown","publisher":null,"released":"2024-12-04","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:global-mmlu","source_url":"https://arxiv.org/abs/2412.03304"},"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":83.6,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.836,"raw_min":0.551,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:global-mmlu:mimo-v2.5-pro","reported_date":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"}],"slug":"llm-stats-global-mmlu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":14525,"hf_paper_upvotes":21,"join":"llm_stats_id","repository":null,"source_record":"catalog_885d8f9fd59e8815"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A lightweight version of Global MMLU benchmark that evaluates language models across multiple languages while addressing cultural and linguistic biases in multilingual evaluation.","descriptions":{"en":"A lightweight version of Global MMLU benchmark that evaluates language models across multiple languages while addressing cultural and linguistic biases in multilingual evaluation."},"evidence_summary":{"document_count":1,"model_count":15,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:global-mmlu-lite:gemini-2.0-flash-lite","reported_at":"2025-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu-lite?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:global-mmlu-lite","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_885d8f9fd59e8815"},"modality":"text","name":"Global-MMLU-Lite","openness":"unknown","publication":{"evidence":[{"note":"Cohere Labs' official card identifies Lite separately from full Global MMLU and links the paper. It records the original 15-language release and later v2/v3 expansions, and explicitly says paper results use the original 15 languages. Its human-translated culturally sensitive/agnostic subset matches the source record; no revision is inferred.","url":"https://huggingface.co/datasets/CohereLabs/Global-MMLU-Lite"},{"note":"The accessible authors' paper introduces Global MMLU; the official Lite card connects the Lite evaluation to this paper rather than equating Lite with the full dataset.","url":"https://arxiv.org/abs/2412.03304"}],"id":"global-mmlu-lite-original","links":[{"kind":"paper","title":"Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation","url":"https://arxiv.org/abs/2412.03304"},{"kind":"dataset","title":"Cohere Labs Global-MMLU-Lite dataset","url":"https://huggingface.co/datasets/CohereLabs/Global-MMLU-Lite"}],"record_keys":["llm-stats:global-mmlu-lite"],"reviewed_at":"2026-10-26","scope":{"en":"Global-MMLU-Lite has its own human-translated dataset release, distinct from full Global MMLU. The source record does not identify a Lite revision, language set or cultural-sensitivity subset.","zh":"Global-MMLU-Lite 是独立发布的人工翻译精简数据集，与完整版 Global MMLU 区分。当前来源记录未注明 Lite 版本、语言范围或文化敏感性子集。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":15,"score_direction":"higher_is_better","score_summary":{"display_max":89.2,"display_multiplier":100,"model_count":15,"model_count_basis":"source_model_id","numeric_count":15,"raw_max":0.892,"raw_min":0.342,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:global-mmlu-lite:gemini-2.5-pro-preview-06-05","reported_date":"2025-06-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu-lite?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"}],"slug":"llm-stats-global-mmlu-lite","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu-lite?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"GlobalDentBench evaluates LLM clinical reasoning in dentistry with 8,978 expert-validated questions across 14 specialties and 88 countries, covering multiple-choice, short-answer, and case-based formats at three reasoning levels.","descriptions":{"en":"GlobalDentBench evaluates LLM clinical reasoning in dentistry with 8,978 expert-validated questions across 14 specialties and 88 countries, covering multiple-choice, short-answer, and case-based formats at three reasoning levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24636","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_globaldentbench_f42d9d35"},"modality":null,"name":"GlobalDentBench","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24636","source_url":"https://arxiv.org/abs/2605.24636"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24636","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24636","topics":["reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":238,"hf_paper_upvotes":4,"join":"claire_record_id","repository":null,"source_record":"bm_globeaudio_abb3482b"},"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"GlobeAudio evaluates audio-language models on naturalistic audio understanding across six languages. It includes 5,637 multiple-choice questions with naturally occurring audio, testing auditory reasoning and cultural interpretation.","descriptions":{"en":"GlobeAudio evaluates audio-language models on naturalistic audio understanding across six languages. It includes 5,637 multiple-choice questions with naturally occurring audio, testing auditory reasoning and cultural interpretation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.08194","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_globeaudio_abb3482b"},"modality":null,"name":"GlobeAudio","openness":"unknown","publisher":null,"released":"2026-06-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08194","source_url":"https://arxiv.org/abs/2606.08194"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08194","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08194","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"GlucoFM-Bench evaluates time-series foundation models for blood glucose forecasting across 15 datasets, protocols including zero-shot, few-shot, and full-shot, with metrics like RMSE.","descriptions":{"en":"GlucoFM-Bench evaluates time-series foundation models for blood glucose forecasting across 15 datasets, protocols including zero-shot, few-shot, and full-shot, with metrics like RMSE."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06881","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_glucofm-bench_d664b863"},"modality":null,"name":"GlucoFM-Bench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06881","source_url":"https://arxiv.org/abs/2606.06881"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06881","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06881","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","其他","Other","医疗","科学智能","AI for Science","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GMAI-MMBench is the most comprehensive and structured benchmark developed to evaluate Large Vision-Language Models (LVLMs) in general medical artificial intelligence (GMAI) applications. It addresses the limitations of existing benchmarks that typically focus on narrow domains and lack perceptual di GMAI-MMBench 是目前最全面、结构化最完善的通用医学人工智能（GMAI）基准数据集，专为评估大规模视觉语言模型（LVLMs）在医疗领域中的表现而设计。针对现有医学基准通常局限于特定学科、感知粒度单一的问题，GMAI-MMBench 从285个真实医学数据集构建而成，涵盖39种医学影像模态、18个临床任务、18个医学科室，以及4种不同的感知粒度。所有任务采用视觉问答（VQA）形式组织，具备良好的交互性和通用性。其独特的词汇树结构允许用户针对具体研究需求灵活定制评估路径，支持多样化的评测场景。我们对50个主流LVLMs进行了系统测试，发现即使是目前最先进的 GPT-4o，其准确率也仅为5","descriptions":{"en":"GMAI-MMBench is the most comprehensive and structured benchmark developed to evaluate Large Vision-Language Models (LVLMs) in general medical artificial intelligence (GMAI) applications. It addresses the limitations of existing benchmarks that typically focus on narrow domains and lack perceptual di","zh":"GMAI-MMBench 是目前最全面、结构化最完善的通用医学人工智能（GMAI）基准数据集，专为评估大规模视觉语言模型（LVLMs）在医疗领域中的表现而设计。针对现有医学基准通常局限于特定学科、感知粒度单一的问题，GMAI-MMBench 从285个真实医学数据集构建而成，涵盖39种医学影像模态、18个临床任务、18个医学科室，以及4种不同的感知粒度。所有任务采用视觉问答（VQA）形式组织，具备良好的交互性和通用性。其独特的词汇树结构允许用户针对具体研究需求灵活定制评估路径，支持多样化的评测场景。我们对50个主流LVLMs进行了系统测试，发现即使是目前最先进的 GPT-4o，其准确率也仅为5"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1927","languages":[],"library_direction":null,"modality":"multimodal","name":"GMAIMMBench","openness":"unknown","publisher":"上海人工智能实验室","released":"2024-08-31","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1927-gmaimmbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GMAIMMBench","topics":["multimodal","reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored.","descriptions":{"en":"Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.05387","languages":[],"library_direction":{"basis":"model-review-daily","categories":["code-generation","ai-rnd"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_gnn-cb_6e6710a9"},"modality":null,"name":"GNN-CB","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05387","source_url":"https://arxiv.org/abs/2610.05387"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05387","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05387","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Gorilla enables LLMs to use tools by invoking APIs. Given a natural language query, Gorilla comes up with the semantically- and syntactically- correct API to invoke. Gorilla 使大语言模型能够通过调用 API 使用工具。针对自然语言查询，Gorilla 能够生成语义和语法上正确的 API 调用。","descriptions":{"en":"Gorilla enables LLMs to use tools by invoking APIs. Given a natural language query, Gorilla comes up with the semantically- and syntactically- correct API to invoke.","zh":"Gorilla 使大语言模型能够通过调用 API 使用工具。针对自然语言查询，Gorilla 能够生成语义和语法上正确的 API 调用。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1128","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"artifact:arxiv:2305.15334","primary":"tool-use","source_record":"catalog_072cecd64f67d822"},"modality":null,"name":"Gorilla","openness":"unknown","publisher":"Microsoft Research","released":"2023-05-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1128-gorilla","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Gorilla","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"llm_stats_id","repository":null,"source_record":"catalog_66d5dc0b03474e67"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"APIBench, a comprehensive dataset of over 11,000 instruction-API pairs from HuggingFace, TorchHub, and TensorHub APIs for evaluating language models' ability to generate accurate API calls.","descriptions":{"en":"APIBench, a comprehensive dataset of over 11,000 instruction-API pairs from HuggingFace, TorchHub, and TensorHub APIs for evaluating language models' ability to generate accurate API calls."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gorilla-benchmark-api-bench:llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gorilla-benchmark-api-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gorilla-benchmark-api-bench","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_66d5dc0b03474e67"},"modality":"text","name":"Gorilla Benchmark API Bench","openness":"unknown","publisher":null,"released":"2023-05-24","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Gorilla: Large Language Model Connected with Massive APIs. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:gorilla-benchmark-api-bench","source_url":"https://arxiv.org/abs/2305.15334"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":35.3,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.353,"raw_min":0.082,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gorilla-benchmark-api-bench:llama-3.1-405b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gorilla-benchmark-api-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"}],"slug":"llm-stats-gorilla-benchmark-api-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gorilla-benchmark-api-bench?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":68,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_6513c0b4e0e4c6ed"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","summarization"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A long document summarization dataset consisting of reports from government research agencies including Congressional Research Service and U.S. Government Accountability Office, with significantly longer documents and summaries than other datasets.","descriptions":{"en":"A long document summarization dataset consisting of reports from government research agencies including Congressional Research Service and U.S. Government Accountability Office, with significantly longer documents and summaries than other datasets."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:govreport:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/govreport?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:govreport","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_6513c0b4e0e4c6ed"},"modality":"text","name":"GovReport","openness":"unknown","publisher":null,"released":"2021-04-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Efficient Attentions for Long Document Summarization. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:govreport","source_url":"https://arxiv.org/abs/2104.02112"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":26.400000000000002,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.264,"raw_min":0.259,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:govreport:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/govreport?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-govreport","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/govreport?top_n=500","topics":["long-context","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":540,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":"gh:idavidrein/gpqa","source_record":"lib_gpqa"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["physics","reasoning","general","biology","chemistry"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.","descriptions":{"en":"A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy."},"editorial_guide":{"id":"gpqa-splits","record_keys":["model-reports:gpqa","model-reports:gpqa_diamond","llm-stats:gpqa","opencompass:1135","artificial-analysis:gpqa-diamond"],"reviewed_at":"2026-10-25","scope":{"en":"The original GPQA family, including the main, Extended and Diamond sets. A shared paper does not establish that two reported scores use the same split.","zh":"覆盖原版 GPQA 的 main、Extended 和 Diamond 题集。引用同一篇论文，并不代表两条成绩使用同一划分。"},"sections":[{"id":"task","source_ids":["paper","dataset"],"text":{"en":"Experts wrote and validated questions in biology, chemistry and physics. Each question has a correct option and distractors. The task tests choosing the answer to a specialized problem, rather than performing a laboratory experiment or producing a research contribution.","zh":"题目由生物、化学和物理领域专家编写并验证，包含正确选项和干扰项。任务考查对专业问题的作答，不能直接等同于完成实验或产出研究成果。"},"title":{"en":"Specialist science questions","zh":"专业科学选择题"}},{"id":"splits","source_ids":["paper"],"text":{"en":"The paper distinguishes 546 Extended questions, 448 main questions and 198 Diamond questions. Diamond applies stricter expert-agreement and non-expert-difficulty criteria. A label such as full GPQA is insufficient: check the dataset configuration or reported question count.","zh":"论文区分 Extended 的 546 题、main 的 448 题和 Diamond 的 198 题。Diamond 对专家一致性和非专家难度采用更严格条件。仅标为完整 GPQA 仍不够，须核对配置或题数。"},"title":{"en":"Main, Extended and Diamond","zh":"三个题集不能混用"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"Accuracy measures the fraction of selected answers that are correct. The released baselines include different prompting and open-book configurations. Keep answer extraction, examples, search access and repeat policy visible when comparing scores from separate reports.","zh":"正确率表示所选答案中答对的比例。官方基线包含不同提示方式和允许检索的配置。比较不同报告时，应保留答案提取、示例、搜索权限和重复运行规则。"},"title":{"en":"Accuracy needs an evaluation setup","zh":"正确率需要对应评测条件"}},{"id":"google-proof","source_ids":["dataset"],"text":{"en":"The name reflects difficulty for skilled non-experts even when they could search the web during construction. It is not a guarantee against training contamination, nor evidence that retrieval never helps a later system.","zh":"这一称呼来自建集时的实验：熟练的非领域专家即使可以搜索网页，仍觉得题目困难。它不保证没有训练污染，也不表示检索对后来的系统永远无效。"},"title":{"en":"What Google-proof means here","zh":"如何理解 Google-proof"}},{"id":"limits","source_ids":["paper","dataset"],"text":{"en":"The authors flag limited statistical power for small accuracy differences. The dataset page also asks users not to republish questions to reduce training leakage. Compare uncertainty and failure cases, and avoid interpreting a small lead as general scientific competence.","zh":"作者指出，小幅正确率差异受限于题量，统计辨别力有限。数据页还要求不要重新公开题目，以减少训练泄漏。应结合不确定性和失败样例理解成绩，避免将微小领先解释为全面科学能力。"},"title":{"en":"Small datasets and exposure","zh":"小题集与公开暴露"}}],"sources":[{"id":"paper","title":"GPQA paper · Dataset splits and limitations","url":"https://arxiv.org/html/2311.12022v1"},{"id":"repository","title":"GPQA · Official baselines and prompts","url":"https://github.com/idavidrein/gpqa"},{"id":"dataset","title":"GPQA · Author dataset card and exposure notice","url":"https://huggingface.co/datasets/Idavidrein/gpqa"}],"summary":{"en":"GPQA tests difficult specialist science questions with multiple-choice answers. Diamond is a selected subset, so identifying the exact split is more useful than comparing the benchmark name alone.","zh":"GPQA 用选择题考查较难的专业科学问题。Diamond 是经过筛选的子集，先确定具体划分，比仅按评测名称比较更重要。"},"title":"GPQA and GPQA Diamond"},"evidence_summary":{"document_count":1,"model_count":239,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gpqa:gpt-3.5-turbo-0125","reported_at":"2023-03-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:gpqa","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"lib_gpqa"},"modality":"text","name":"GPQA","openness":"restricted","publisher":"Anthropic","released":"2023-11-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":239,"score_direction":"higher_is_better","score_summary":{"display_max":94.6,"display_multiplier":100,"model_count":239,"model_count_basis":"source_model_id","numeric_count":239,"raw_max":0.946,"raw_min":0.119,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gpqa:claude-mythos-preview","reported_date":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-13","date_precision":"model_announcement","name":"ChatGPT-4o Latest"},{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Opus 4"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude Opus 4.1"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Sonnet 4"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude Sonnet 4.5"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 7B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Zero"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek-V3 0324"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2026-01-22","date_precision":"model_announcement","name":"ERNIE 5.0"},{"date":"2024-02-15","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Gemini 2.0 Flash Thinking"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B High"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 20B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 20B High"},{"date":"2023-03-21","date_precision":"model_announcement","name":"GPT-3.5 Turbo"},{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"},{"date":"2024-04-09","date_precision":"model_announcement","name":"GPT-4 Turbo"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 High"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 Medium"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 High"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Instant"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Thinking"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Pro"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok 4 Fast"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2024-03-28","date_precision":"model_announcement","name":"Grok-1.5"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2 mini"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Grok-3"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Grok-3 Mini"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4 Heavy"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 8B V1"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 70B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama-3.3 Nemotron Super 49B v1"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 2506"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Min  istral 3 (3B Reasoning 2512)"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Reasoning 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Reasoning 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Base)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 Eagle)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 NVFP4)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Base"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Instruct"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Base"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Instruct"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"Nemotron Nano 9B v2"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1-pro"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Mini Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 30B A3B"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ-32B"},{"date":"2024-11-28","date_precision":"model_announcement","name":"QwQ-32B-Preview"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Lite"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Pro"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"}],"slug":"llm-stats-gpqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":540,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"artifact:arxiv:2311.12022","repository":null,"source_record":"catalog_8fcb539c7245dd97"},"categories":["知识","Knowledge","大语言模型","LLM","知识储备","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. GPQA 包含 448 道由生物学、物理学和化学领域专家撰写的多项选择题。","descriptions":{"en":"GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.","zh":"GPQA 包含 448 道由生物学、物理学和化学领域专家撰写的多项选择题。"},"editorial_guide":{"id":"gpqa-splits","record_keys":["model-reports:gpqa","model-reports:gpqa_diamond","llm-stats:gpqa","opencompass:1135","artificial-analysis:gpqa-diamond"],"reviewed_at":"2026-10-25","scope":{"en":"The original GPQA family, including the main, Extended and Diamond sets. A shared paper does not establish that two reported scores use the same split.","zh":"覆盖原版 GPQA 的 main、Extended 和 Diamond 题集。引用同一篇论文，并不代表两条成绩使用同一划分。"},"sections":[{"id":"task","source_ids":["paper","dataset"],"text":{"en":"Experts wrote and validated questions in biology, chemistry and physics. Each question has a correct option and distractors. The task tests choosing the answer to a specialized problem, rather than performing a laboratory experiment or producing a research contribution.","zh":"题目由生物、化学和物理领域专家编写并验证，包含正确选项和干扰项。任务考查对专业问题的作答，不能直接等同于完成实验或产出研究成果。"},"title":{"en":"Specialist science questions","zh":"专业科学选择题"}},{"id":"splits","source_ids":["paper"],"text":{"en":"The paper distinguishes 546 Extended questions, 448 main questions and 198 Diamond questions. Diamond applies stricter expert-agreement and non-expert-difficulty criteria. A label such as full GPQA is insufficient: check the dataset configuration or reported question count.","zh":"论文区分 Extended 的 546 题、main 的 448 题和 Diamond 的 198 题。Diamond 对专家一致性和非专家难度采用更严格条件。仅标为完整 GPQA 仍不够，须核对配置或题数。"},"title":{"en":"Main, Extended and Diamond","zh":"三个题集不能混用"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"Accuracy measures the fraction of selected answers that are correct. The released baselines include different prompting and open-book configurations. Keep answer extraction, examples, search access and repeat policy visible when comparing scores from separate reports.","zh":"正确率表示所选答案中答对的比例。官方基线包含不同提示方式和允许检索的配置。比较不同报告时，应保留答案提取、示例、搜索权限和重复运行规则。"},"title":{"en":"Accuracy needs an evaluation setup","zh":"正确率需要对应评测条件"}},{"id":"google-proof","source_ids":["dataset"],"text":{"en":"The name reflects difficulty for skilled non-experts even when they could search the web during construction. It is not a guarantee against training contamination, nor evidence that retrieval never helps a later system.","zh":"这一称呼来自建集时的实验：熟练的非领域专家即使可以搜索网页，仍觉得题目困难。它不保证没有训练污染，也不表示检索对后来的系统永远无效。"},"title":{"en":"What Google-proof means here","zh":"如何理解 Google-proof"}},{"id":"limits","source_ids":["paper","dataset"],"text":{"en":"The authors flag limited statistical power for small accuracy differences. The dataset page also asks users not to republish questions to reduce training leakage. Compare uncertainty and failure cases, and avoid interpreting a small lead as general scientific competence.","zh":"作者指出，小幅正确率差异受限于题量，统计辨别力有限。数据页还要求不要重新公开题目，以减少训练泄漏。应结合不确定性和失败样例理解成绩，避免将微小领先解释为全面科学能力。"},"title":{"en":"Small datasets and exposure","zh":"小题集与公开暴露"}}],"sources":[{"id":"paper","title":"GPQA paper · Dataset splits and limitations","url":"https://arxiv.org/html/2311.12022v1"},{"id":"repository","title":"GPQA · Official baselines and prompts","url":"https://github.com/idavidrein/gpqa"},{"id":"dataset","title":"GPQA · Author dataset card and exposure notice","url":"https://huggingface.co/datasets/Idavidrein/gpqa"}],"summary":{"en":"GPQA tests difficult specialist science questions with multiple-choice answers. Diamond is a selected subset, so identifying the exact split is more useful than comparing the benchmark name alone.","zh":"GPQA 用选择题考查较难的专业科学问题。Diamond 是经过筛选的子集，先确定具体划分，比仅按评测名称比较更重要。"},"title":"GPQA and GPQA Diamond"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1135","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"artifact:arxiv:2311.12022","primary":"knowledge-expert-qa","source_record":"catalog_8fcb539c7245dd97"},"modality":null,"name":"GPQA","openness":"restricted","publisher":"Anthropic","released":"2023-11-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1135-gpqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GPQA","topics":[],"unit":null},{"aliases":["GPQA main","GPQA full"],"attention_signals":{"as_of":"2026-10-10","github_stars":540,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"artifact:arxiv:2311.12022","repository":null,"source_record":"catalog_8fcb539c7245dd97"},"categories":["science"],"collected_at":null,"description":"The full split, distinct from the Diamond subset most frontier cards report.","descriptions":{"en":"The full split, distinct from the Diamond subset most frontier cards report."},"editorial_guide":{"id":"gpqa-splits","record_keys":["model-reports:gpqa","model-reports:gpqa_diamond","llm-stats:gpqa","opencompass:1135","artificial-analysis:gpqa-diamond"],"reviewed_at":"2026-10-25","scope":{"en":"The original GPQA family, including the main, Extended and Diamond sets. A shared paper does not establish that two reported scores use the same split.","zh":"覆盖原版 GPQA 的 main、Extended 和 Diamond 题集。引用同一篇论文，并不代表两条成绩使用同一划分。"},"sections":[{"id":"task","source_ids":["paper","dataset"],"text":{"en":"Experts wrote and validated questions in biology, chemistry and physics. Each question has a correct option and distractors. The task tests choosing the answer to a specialized problem, rather than performing a laboratory experiment or producing a research contribution.","zh":"题目由生物、化学和物理领域专家编写并验证，包含正确选项和干扰项。任务考查对专业问题的作答，不能直接等同于完成实验或产出研究成果。"},"title":{"en":"Specialist science questions","zh":"专业科学选择题"}},{"id":"splits","source_ids":["paper"],"text":{"en":"The paper distinguishes 546 Extended questions, 448 main questions and 198 Diamond questions. Diamond applies stricter expert-agreement and non-expert-difficulty criteria. A label such as full GPQA is insufficient: check the dataset configuration or reported question count.","zh":"论文区分 Extended 的 546 题、main 的 448 题和 Diamond 的 198 题。Diamond 对专家一致性和非专家难度采用更严格条件。仅标为完整 GPQA 仍不够，须核对配置或题数。"},"title":{"en":"Main, Extended and Diamond","zh":"三个题集不能混用"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"Accuracy measures the fraction of selected answers that are correct. The released baselines include different prompting and open-book configurations. Keep answer extraction, examples, search access and repeat policy visible when comparing scores from separate reports.","zh":"正确率表示所选答案中答对的比例。官方基线包含不同提示方式和允许检索的配置。比较不同报告时，应保留答案提取、示例、搜索权限和重复运行规则。"},"title":{"en":"Accuracy needs an evaluation setup","zh":"正确率需要对应评测条件"}},{"id":"google-proof","source_ids":["dataset"],"text":{"en":"The name reflects difficulty for skilled non-experts even when they could search the web during construction. It is not a guarantee against training contamination, nor evidence that retrieval never helps a later system.","zh":"这一称呼来自建集时的实验：熟练的非领域专家即使可以搜索网页，仍觉得题目困难。它不保证没有训练污染，也不表示检索对后来的系统永远无效。"},"title":{"en":"What Google-proof means here","zh":"如何理解 Google-proof"}},{"id":"limits","source_ids":["paper","dataset"],"text":{"en":"The authors flag limited statistical power for small accuracy differences. The dataset page also asks users not to republish questions to reduce training leakage. Compare uncertainty and failure cases, and avoid interpreting a small lead as general scientific competence.","zh":"作者指出，小幅正确率差异受限于题量，统计辨别力有限。数据页还要求不要重新公开题目，以减少训练泄漏。应结合不确定性和失败样例理解成绩，避免将微小领先解释为全面科学能力。"},"title":{"en":"Small datasets and exposure","zh":"小题集与公开暴露"}}],"sources":[{"id":"paper","title":"GPQA paper · Dataset splits and limitations","url":"https://arxiv.org/html/2311.12022v1"},{"id":"repository","title":"GPQA · Official baselines and prompts","url":"https://github.com/idavidrein/gpqa"},{"id":"dataset","title":"GPQA · Author dataset card and exposure notice","url":"https://huggingface.co/datasets/Idavidrein/gpqa"}],"summary":{"en":"GPQA tests difficult specialist science questions with multiple-choice answers. Diamond is a selected subset, so identifying the exact split is more useful than comparing the benchmark name alone.","zh":"GPQA 用选择题考查较难的专业科学问题。Diamond 是经过筛选的子集，先确定具体划分，比仅按评测名称比较更重要。"},"title":"GPQA and GPQA Diamond"},"evidence_summary":{"document_count":3,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000gpqa\u0000qwen3_5_model_card\u0000gpqa\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000gpqa\u0000qwen3_5_model_card\u0000gpqa\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:gpqa","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"artifact:arxiv:2311.12022","primary":"knowledge-expert-qa","source_record":"catalog_8fcb539c7245dd97"},"modality":null,"name":"GPQA (full)","openness":"unknown","publisher":null,"released":"2023-11-20","released_reference":{"basis":"benchmark_release","source_key":"model-reports:gpqa","source_url":"https://arxiv.org/abs/2311.12022"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":88.4,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":88.4,"raw_min":88.4,"source_reference":{"obs_id":"curated\u0000gpqa\u0000qwen3_5_model_card\u0000gpqa\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000gpqa\u0000qwen3_5_model_card\u0000gpqa\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"gpqa","source":"model_reports","source_url":"https://arxiv.org/abs/2311.12022","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":540,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":null,"source_record":"catalog_9b2670faedafe987"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","healthcare","biology"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Biology subset of GPQA, containing challenging multiple-choice questions written by domain experts in biology. These Google-proof questions require graduate-level knowledge and reasoning.","descriptions":{"en":"Biology subset of GPQA, containing challenging multiple-choice questions written by domain experts in biology. These Google-proof questions require graduate-level knowledge and reasoning."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gpqa-biology:o1-2024-12-17","reported_at":"2024-12-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-biology?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gpqa-biology","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_9b2670faedafe987"},"modality":"text","name":"GPQA Biology","openness":"unknown","publisher":null,"released":"2023-11-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: GPQA: A Graduate-Level Google-Proof Q&A Benchmark. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:gpqa-biology","source_url":"https://arxiv.org/abs/2311.12022"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":69.19999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.692,"raw_min":0.692,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gpqa-biology:o1-2024-12-17","reported_date":"2024-12-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-biology?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"}],"slug":"llm-stats-gpqa-biology","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-biology?top_n=500","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":540,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":null,"source_record":"catalog_de8024fbe9575be8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","chemistry"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Chemistry subset of GPQA, containing challenging multiple-choice questions written by domain experts in chemistry. These Google-proof questions require graduate-level knowledge and reasoning.","descriptions":{"en":"Chemistry subset of GPQA, containing challenging multiple-choice questions written by domain experts in chemistry. These Google-proof questions require graduate-level knowledge and reasoning."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gpqa-chemistry:o1-2024-12-17","reported_at":"2024-12-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-chemistry?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gpqa-chemistry","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_de8024fbe9575be8"},"modality":"text","name":"GPQA Chemistry","openness":"unknown","publisher":null,"released":"2023-11-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: GPQA: A Graduate-Level Google-Proof Q&A Benchmark. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:gpqa-chemistry","source_url":"https://arxiv.org/abs/2311.12022"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":64.7,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.647,"raw_min":0.647,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gpqa-chemistry:o1-2024-12-17","reported_date":"2024-12-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-chemistry?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"}],"slug":"llm-stats-gpqa-chemistry","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-chemistry?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","reasoning","science"],"collected_at":"2026-08-25T10:41:06Z","description":"Scientific reasoning","descriptions":{"en":"Scientific reasoning"},"editorial_guide":{"id":"gpqa-splits","record_keys":["model-reports:gpqa","model-reports:gpqa_diamond","llm-stats:gpqa","opencompass:1135","artificial-analysis:gpqa-diamond"],"reviewed_at":"2026-10-25","scope":{"en":"The original GPQA family, including the main, Extended and Diamond sets. A shared paper does not establish that two reported scores use the same split.","zh":"覆盖原版 GPQA 的 main、Extended 和 Diamond 题集。引用同一篇论文，并不代表两条成绩使用同一划分。"},"sections":[{"id":"task","source_ids":["paper","dataset"],"text":{"en":"Experts wrote and validated questions in biology, chemistry and physics. Each question has a correct option and distractors. The task tests choosing the answer to a specialized problem, rather than performing a laboratory experiment or producing a research contribution.","zh":"题目由生物、化学和物理领域专家编写并验证，包含正确选项和干扰项。任务考查对专业问题的作答，不能直接等同于完成实验或产出研究成果。"},"title":{"en":"Specialist science questions","zh":"专业科学选择题"}},{"id":"splits","source_ids":["paper"],"text":{"en":"The paper distinguishes 546 Extended questions, 448 main questions and 198 Diamond questions. Diamond applies stricter expert-agreement and non-expert-difficulty criteria. A label such as full GPQA is insufficient: check the dataset configuration or reported question count.","zh":"论文区分 Extended 的 546 题、main 的 448 题和 Diamond 的 198 题。Diamond 对专家一致性和非专家难度采用更严格条件。仅标为完整 GPQA 仍不够，须核对配置或题数。"},"title":{"en":"Main, Extended and Diamond","zh":"三个题集不能混用"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"Accuracy measures the fraction of selected answers that are correct. The released baselines include different prompting and open-book configurations. Keep answer extraction, examples, search access and repeat policy visible when comparing scores from separate reports.","zh":"正确率表示所选答案中答对的比例。官方基线包含不同提示方式和允许检索的配置。比较不同报告时，应保留答案提取、示例、搜索权限和重复运行规则。"},"title":{"en":"Accuracy needs an evaluation setup","zh":"正确率需要对应评测条件"}},{"id":"google-proof","source_ids":["dataset"],"text":{"en":"The name reflects difficulty for skilled non-experts even when they could search the web during construction. It is not a guarantee against training contamination, nor evidence that retrieval never helps a later system.","zh":"这一称呼来自建集时的实验：熟练的非领域专家即使可以搜索网页，仍觉得题目困难。它不保证没有训练污染，也不表示检索对后来的系统永远无效。"},"title":{"en":"What Google-proof means here","zh":"如何理解 Google-proof"}},{"id":"limits","source_ids":["paper","dataset"],"text":{"en":"The authors flag limited statistical power for small accuracy differences. The dataset page also asks users not to republish questions to reduce training leakage. Compare uncertainty and failure cases, and avoid interpreting a small lead as general scientific competence.","zh":"作者指出，小幅正确率差异受限于题量，统计辨别力有限。数据页还要求不要重新公开题目，以减少训练泄漏。应结合不确定性和失败样例理解成绩，避免将微小领先解释为全面科学能力。"},"title":{"en":"Small datasets and exposure","zh":"小题集与公开暴露"}}],"sources":[{"id":"paper","title":"GPQA paper · Dataset splits and limitations","url":"https://arxiv.org/html/2311.12022v1"},{"id":"repository","title":"GPQA · Official baselines and prompts","url":"https://github.com/idavidrein/gpqa"},{"id":"dataset","title":"GPQA · Author dataset card and exposure notice","url":"https://huggingface.co/datasets/Idavidrein/gpqa"}],"summary":{"en":"GPQA tests difficult specialist science questions with multiple-choice answers. Diamond is a selected subset, so identifying the exact split is more useful than comparing the benchmark name alone.","zh":"GPQA 用选择题考查较难的专业科学问题。Diamond 是经过筛选的子集，先确定具体划分，比仅按评测名称比较更重要。"},"title":"GPQA and GPQA Diamond"},"evidence_summary":{"document_count":1,"model_count":586,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:gpqa-diamond:037dec2f-51e8-4127-a1f1-85155dae7a1d","reported_at":"2022-11-30","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:gpqa-diamond","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["knowledge-expert-qa"],"join":"review:14H034160212:2026-10-15","primary":"knowledge-expert-qa","source_record":null},"modality":null,"name":"GPQA Diamond","openness":"unknown","publisher":null,"released":"2023-11-20","released_reference":{"basis":"paper_first_version","note":"First version of the GPQA paper defining its Diamond partition. This remains before the requested 2024 cohort.","source_key":"artificial-analysis:gpqa-diamond","source_url":"https://arxiv.org/abs/2311.12022"},"repo_kind":null,"repo_resolution_status":null,"score_count":586,"score_direction":"higher_is_better","score_summary":{"display_max":94.94949494949499,"display_multiplier":100,"model_count":586,"model_count_basis":"source_model_id","numeric_count":586,"raw_max":0.94949494949495,"raw_min":0.097979797979798,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:gpqa-diamond:c8adc5cf-fd5a-407b-af51-dc3bede3e49c","reported_date":"2026-08-12","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2023-07-11","date_precision":"model_announcement","name":"Claude 2.0"},{"date":"2023-11-21","date_precision":"model_announcement","name":"Claude 2.1"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-06-21","date_precision":"model_announcement","name":"Claude 3.5 Sonnet (June '24)"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet (Oct '24)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2023-03-14","date_precision":"model_announcement","name":"Claude Instant"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2024-03-12","date_precision":"model_announcement","name":"Command-R (Mar '24)"},{"date":"2024-04-04","date_precision":"model_announcement","name":"Command-R+ (Apr '24)"},{"date":"2024-03-27","date_precision":"model_announcement","name":"DBRX Instruct"},{"date":"2025-02-13","date_precision":"model_announcement","name":"DeepHermes 3 - Llama-3.1 8B Preview (Non-reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"DeepHermes 3 - Mistral 24B Preview (Non-reasoning)"},{"date":"2024-06-17","date_precision":"model_announcement","name":"DeepSeek Coder V2 Lite Instruct"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2024-12-10","date_precision":"model_announcement","name":"DeepSeek-V2.5 (Dec '24)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2023-12-06","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-14","date_precision":"model_announcement","name":"Gemini 1.5 Flash (May '24)"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Flash (Sep '24)"},{"date":"2024-10-03","date_precision":"model_announcement","name":"Gemini 1.5 Flash-8B"},{"date":"2024-05-15","date_precision":"model_announcement","name":"Gemini 1.5 Pro (May '24)"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Pro (Sep '24)"},{"date":"2024-12-11","date_precision":"model_announcement","name":"Gemini 2.0 Flash (experimental)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Gemini 2.0 Flash Thinking Experimental (Jan '25)"},{"date":"2025-02-25","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite (Feb '25)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite (Preview)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Pro Experimental (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-04-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Non-reasoning)"},{"date":"2025-04-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-03-25","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview (Mar' 25)"},{"date":"2025-05-06","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview (May' 25)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct Preview (May '25)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2022-11-30","date_precision":"model_announcement","name":"GPT-3.5 Turbo"},{"date":"2023-03-14","date_precision":"model_announcement","name":"GPT-4"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2025-02-15","date_precision":"model_announcement","name":"GPT-4o (ChatGPT)"},{"date":"2025-03-27","date_precision":"model_announcement","name":"GPT-4o (March 2025, chatgpt-4o-latest)"},{"date":"2024-05-13","date_precision":"model_announcement","name":"GPT-4o (May '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (ChatGPT)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Grok 2 (Dec '24)"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok Beta"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 - Llama-3.1 70B"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-03-06","date_precision":"model_announcement","name":"Jamba 1.6 Large"},{"date":"2025-03-06","date_precision":"model_announcement","name":"Jamba 1.6 Mini"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2025-10-30","date_precision":"model_announcement","name":"Kimi Linear 48B A3B Instruct"},{"date":"2024-09-30","date_precision":"model_announcement","name":"LFM 40B"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 13B"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 70B"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 7B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 4B v1.1 (Reasoning)"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Llama 3.1 Tulu3 405B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 3B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 90B (Vision)"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 40k"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Mistral Large (Feb '24)"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2 (Jul '24)"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2023-12-11","date_precision":"model_announcement","name":"Mistral Medium"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Mistral Saba"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Mistral Small (Feb '24)"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Mistral Small (Sep '24)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2024-04-17","date_precision":"model_announcement","name":"Mixtral 8x22B Instruct"},{"date":"2023-12-11","date_precision":"model_announcement","name":"Mixtral 8x7B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-06-10","date_precision":"model_announcement","name":"o3-pro"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"OLMo 2 32B"},{"date":"2024-11-26","date_precision":"model_announcement","name":"OLMo 2 7B"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2023-12-18","date_precision":"model_announcement","name":"OpenChat 3.5 (1210)"},{"date":"2024-04-23","date_precision":"model_announcement","name":"Phi-3 Mini Instruct 3.8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2025-02-26","date_precision":"model_announcement","name":"Phi-4 Multimodal Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-04-25","date_precision":"model_announcement","name":"Qwen1.5 Chat 110B"},{"date":"2024-06-07","date_precision":"model_announcement","name":"Qwen2 Instruct 72B"},{"date":"2024-11-11","date_precision":"model_announcement","name":"Qwen2.5 Coder Instruct 32B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Coder Instruct 7B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 32B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-01-28","date_precision":"model_announcement","name":"Qwen2.5 Max"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Qwen2.5 Turbo"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ 32B"},{"date":"2024-11-27","date_precision":"model_announcement","name":"QwQ 32B-Preview"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Solar Pro 2 (Preview) (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Solar Pro 2 (Preview) (Reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Sonar"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Sonar Pro"},{"date":"2025-01-28","date_precision":"model_announcement","name":"Sonar Reasoning"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-gpqa-diamond","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond","topics":["reasoning-math","science"],"unit":null},{"aliases":["GPQA","GPQA-Diamond","GPQA Diamond"],"attention_signals":{"as_of":"2026-10-10","github_stars":540,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"artifact:arxiv:2311.12022","repository":null,"source_record":"catalog_8fcb539c7245dd97"},"categories":["science"],"collected_at":null,"description":"198 questions in the Diamond split, so run-to-run variance is wide and a single reported number hides it. Approaching saturation at the 2026 frontier, where reported scores cluster in the high 80s and 90s.","descriptions":{"en":"198 questions in the Diamond split, so run-to-run variance is wide and a single reported number hides it. Approaching saturation at the 2026 frontier, where reported scores cluster in the high 80s and 90s."},"editorial_guide":{"id":"gpqa-splits","record_keys":["model-reports:gpqa","model-reports:gpqa_diamond","llm-stats:gpqa","opencompass:1135","artificial-analysis:gpqa-diamond"],"reviewed_at":"2026-10-25","scope":{"en":"The original GPQA family, including the main, Extended and Diamond sets. A shared paper does not establish that two reported scores use the same split.","zh":"覆盖原版 GPQA 的 main、Extended 和 Diamond 题集。引用同一篇论文，并不代表两条成绩使用同一划分。"},"sections":[{"id":"task","source_ids":["paper","dataset"],"text":{"en":"Experts wrote and validated questions in biology, chemistry and physics. Each question has a correct option and distractors. The task tests choosing the answer to a specialized problem, rather than performing a laboratory experiment or producing a research contribution.","zh":"题目由生物、化学和物理领域专家编写并验证，包含正确选项和干扰项。任务考查对专业问题的作答，不能直接等同于完成实验或产出研究成果。"},"title":{"en":"Specialist science questions","zh":"专业科学选择题"}},{"id":"splits","source_ids":["paper"],"text":{"en":"The paper distinguishes 546 Extended questions, 448 main questions and 198 Diamond questions. Diamond applies stricter expert-agreement and non-expert-difficulty criteria. A label such as full GPQA is insufficient: check the dataset configuration or reported question count.","zh":"论文区分 Extended 的 546 题、main 的 448 题和 Diamond 的 198 题。Diamond 对专家一致性和非专家难度采用更严格条件。仅标为完整 GPQA 仍不够，须核对配置或题数。"},"title":{"en":"Main, Extended and Diamond","zh":"三个题集不能混用"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"Accuracy measures the fraction of selected answers that are correct. The released baselines include different prompting and open-book configurations. Keep answer extraction, examples, search access and repeat policy visible when comparing scores from separate reports.","zh":"正确率表示所选答案中答对的比例。官方基线包含不同提示方式和允许检索的配置。比较不同报告时，应保留答案提取、示例、搜索权限和重复运行规则。"},"title":{"en":"Accuracy needs an evaluation setup","zh":"正确率需要对应评测条件"}},{"id":"google-proof","source_ids":["dataset"],"text":{"en":"The name reflects difficulty for skilled non-experts even when they could search the web during construction. It is not a guarantee against training contamination, nor evidence that retrieval never helps a later system.","zh":"这一称呼来自建集时的实验：熟练的非领域专家即使可以搜索网页，仍觉得题目困难。它不保证没有训练污染，也不表示检索对后来的系统永远无效。"},"title":{"en":"What Google-proof means here","zh":"如何理解 Google-proof"}},{"id":"limits","source_ids":["paper","dataset"],"text":{"en":"The authors flag limited statistical power for small accuracy differences. The dataset page also asks users not to republish questions to reduce training leakage. Compare uncertainty and failure cases, and avoid interpreting a small lead as general scientific competence.","zh":"作者指出，小幅正确率差异受限于题量，统计辨别力有限。数据页还要求不要重新公开题目，以减少训练泄漏。应结合不确定性和失败样例理解成绩，避免将微小领先解释为全面科学能力。"},"title":{"en":"Small datasets and exposure","zh":"小题集与公开暴露"}}],"sources":[{"id":"paper","title":"GPQA paper · Dataset splits and limitations","url":"https://arxiv.org/html/2311.12022v1"},{"id":"repository","title":"GPQA · Official baselines and prompts","url":"https://github.com/idavidrein/gpqa"},{"id":"dataset","title":"GPQA · Author dataset card and exposure notice","url":"https://huggingface.co/datasets/Idavidrein/gpqa"}],"summary":{"en":"GPQA tests difficult specialist science questions with multiple-choice answers. Diamond is a selected subset, so identifying the exact split is more useful than comparing the benchmark name alone.","zh":"GPQA 用选择题考查较难的专业科学问题。Diamond 是经过筛选的子集，先确定具体划分，比仅按评测名称比较更重要。"},"title":"GPQA and GPQA Diamond"},"evidence_summary":{"document_count":30,"model_count":31,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000gpqa_diamond\u0000deepseek_v3_report\u0000gpqa_diamond\u0000Pass@1, 8K output limit\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"first_score_reported_at":"2024-12-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000gpqa_diamond\u0000deepseek_v3_report\u0000gpqa_diamond\u0000Pass@1, 8K output limit\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:gpqa_diamond","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"artifact:arxiv:2311.12022","primary":"knowledge-expert-qa","source_record":"catalog_8fcb539c7245dd97"},"modality":null,"name":"GPQA Diamond","openness":"unknown","publisher":null,"released":"2023-11-20","released_reference":{"basis":"benchmark_release","source_key":"model-reports:gpqa_diamond","source_url":"https://arxiv.org/abs/2311.12022"},"repo_kind":null,"repo_resolution_status":null,"score_count":33,"score_direction":"higher_is_better","score_summary":{"display_max":94.3,"display_multiplier":1,"model_count":31,"model_count_basis":"source_model_id","numeric_count":33,"raw_max":94.3,"raw_min":25.3,"source_reference":{"obs_id":"curated\u0000gpqa_diamond\u0000google_gemini_3_1_pro_model_card\u0000gpqa_diamond\u0000Thinking (High), No tools\u0000Gemini 3.1 Pro","observation_id":"curated\u0000gpqa_diamond\u0000google_gemini_3_1_pro_model_card\u0000gpqa_diamond\u0000Thinking (High), No tools\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","reported_date":"2026-02-19","source_id":"google_gemini_3_1_pro_model_card","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"unit":"percent"},"scored_models":[{"date":"2025-04-30","date_precision":"document_publication","name":"Bespoke-Stratos-7B"},{"date":"2025-06-17","date_precision":"document_publication","name":"Claude 4 Opus"},{"date":"2025-06-17","date_precision":"document_publication","name":"Claude 4 Sonnet"},{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Opus 4"},{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Sonnet 4"},{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2025-04-30","date_precision":"document_publication","name":"DeepSeek-R1-Distill-Llama-8B"},{"date":"2025-04-30","date_precision":"document_publication","name":"DeepSeek-R1-Distill-Qwen-7B"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2025-06-17","date_precision":"document_publication","name":"Gemini 2.5 Pro"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2025-06-17","date_precision":"document_publication","name":"Grok 3 Beta Extended Thinking"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2025-04-30","date_precision":"document_publication","name":"Llama-3.2-3B-Instruct"},{"date":"2025-04-30","date_precision":"document_publication","name":"o1-mini"},{"date":"2025-06-17","date_precision":"document_publication","name":"o3 high"},{"date":"2025-04-30","date_precision":"document_publication","name":"OpenThinker-7B"},{"date":"2025-04-30","date_precision":"document_publication","name":"Phi-4-Mini (base model, 3.8B)"},{"date":"2025-04-30","date_precision":"document_publication","name":"Phi-4-mini-reasoning (3.8B)"},{"date":"2025-05-14","date_precision":"document_publication","name":"Qwen3-235B-A22B (Thinking)"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"gpqa_diamond","source":"model_reports","source_url":"https://arxiv.org/abs/2311.12022","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":540,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":null,"source_record":"catalog_8fcb539c7245dd97"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["physics","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Physics subset of GPQA, containing challenging multiple-choice questions written by domain experts in physics. These Google-proof questions require graduate-level knowledge and reasoning.","descriptions":{"en":"Physics subset of GPQA, containing challenging multiple-choice questions written by domain experts in physics. These Google-proof questions require graduate-level knowledge and reasoning."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gpqa-physics:o1-2024-12-17","reported_at":"2024-12-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-physics?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gpqa-physics","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_8fcb539c7245dd97"},"modality":"text","name":"GPQA Physics","openness":"unknown","publisher":null,"released":"2023-11-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: GPQA: A Graduate-Level Google-Proof Q&A Benchmark. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:gpqa-physics","source_url":"https://arxiv.org/abs/2311.12022"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":92.80000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.928,"raw_min":0.928,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gpqa-physics:o1-2024-12-17","reported_date":"2024-12-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-physics?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"}],"slug":"llm-stats-gpqa-physics","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-physics?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","视觉生成","Visual Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GPT-ImgEval, quantitatively and qualitatively diagnoses GPT-4o's performance across three critical dimensions: (1) generation quality, (2) editing proficiency, and (3) world knowledge-informed semantic synthesis. GPT-ImgEval，从三个关键维度对GPT-4o的性能进行定量和定性诊断：（1）生成质量，（2）编辑能力，以及（3）基于世界知识的语义合成能力。","descriptions":{"en":"GPT-ImgEval, quantitatively and qualitatively diagnoses GPT-4o's performance across three critical dimensions: (1) generation quality, (2) editing proficiency, and (3) world knowledge-informed semantic synthesis.","zh":"GPT-ImgEval，从三个关键维度对GPT-4o的性能进行定量和定性诊断：（1）生成质量，（2）编辑能力，以及（3）基于世界知识的语义合成能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1740","languages":[],"library_direction":null,"modality":"multimodal","name":"GPT-ImgEval","openness":"open","publisher":"Peking University, Sun Yat-sen University, etc.","released":"2025-04-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1740-gpt-imgeval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GPT-ImgEval","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_gptnt_3088efa1"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"GPTNT evaluates multimodal agents on real-time collaborative bomb defusal in the game Keep Talking and Nobody Explodes, requiring asynchronous communication under time pressure and information asymmetry. Success is measured by defusing procedurally generated bombs.","descriptions":{"en":"GPTNT evaluates multimodal agents on real-time collaborative bomb defusal in the game Keep Talking and Nobody Explodes, requiring asynchronous communication under time pressure and information asymmetry. Success is measured by defusing procedurally generated bombs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.28514","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","visual-reasoning"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_gptnt_3088efa1"},"modality":null,"name":"GPTNT","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.28514","source_url":"https://arxiv.org/abs/2606.28514"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-28514","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28514","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jyotydivya/gpu-accelerated-ml-inference-benchmark","source_record":"bm_gpu-accelerated-ml-inference-benchmark_af49dfaf"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Compares inference latency, throughput, and GPU utilization for PyTorch CPU, PyTorch CUDA, and TensorRT across batch sizes.","descriptions":{"en":"Compares inference latency, throughput, and GPU utilization for PyTorch CPU, PyTorch CUDA, and TensorRT across batch sizes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:jyotydivya/gpu-accelerated-ml-inference-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_gpu-accelerated-ml-inference-benchmark_af49dfaf"},"modality":null,"name":"GPU-Accelerated ML Inference Benchmark","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:jyotydivya/gpu-accelerated-ml-inference-benchmark","source_url":"https://github.com/jyotydivya/GPU-Accelerated-ML-Inference-Benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-jyotydivya-gpu-accelerated-ml-inference-benchmark","source":"claire_radar","source_url":"https://github.com/jyotydivya/GPU-Accelerated-ML-Inference-Benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DC"],"collected_at":"2026-10-14T09:44:51Z","description":"Writing fast GPU code for physical simulation is difficult: implementations must preserve numerical accuracy while handling irregular data access, synchronization, and iterative solvers.","descriptions":{"en":"Writing fast GPU code for physical simulation is difficult: implementations must preserve numerical accuracy while handling irregular data access, synchronization, and iterative solvers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35639","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","science"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_gpuphysbench_33c91da2"},"modality":null,"name":"GPUPhysBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35639","source_url":"https://arxiv.org/abs/2609.35639"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35639","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35639","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_grace_1b12b544"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"GRACE evaluates step-level faithfulness of chain-of-thought reasoning in context-grounded tasks, with human annotations and a taxonomy of error categories.","descriptions":{"en":"GRACE evaluates step-level faithfulness of chain-of-thought reasoning in context-grounded tasks, with human annotations and a taxonomy of error categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16151","languages":[],"library_direction":{"basis":"model-review-2","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_grace_1b12b544"},"modality":null,"name":"GRACE","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16151","source_url":"https://arxiv.org/abs/2606.16151"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16151","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16151","topics":["reasoning-math"],"unit":null},{"aliases":["baoyan-agent-benchmark"],"attention_signals":null,"categories":["Agents","Language & Knowledge","Planning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates Chinese-language agent products on graduate admissions advising, covering recommendation, planning, and two-stage information-update tasks with per-task rubrics.","descriptions":{"en":"Evaluates Chinese-language agent products on graduate admissions advising, covering recommendation, planning, and two-stage information-update tasks with per-task rubrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:yeyuxin777/baoyan-agent-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning","multilingual"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_baoyan-agent-benchmark_947b1972"},"modality":null,"name":"Graduate Admissions Agent Benchmark","openness":"unknown","publisher":null,"released":"2026-10-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:yeyuxin777/baoyan-agent-benchmark","source_url":"https://github.com/yeyuxin777/baoyan-agent-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-yeyuxin777-baoyan-agent-benchmark","source":"claire_radar","source_url":"https://github.com/yeyuxin777/baoyan-agent-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GrailQA is a new large-scale, high-quality dataset for question answering on knowledge bases (KBQA) on Freebase with 64,331 questions annotated with both answers and corresponding logical forms in different syntax (i.e., SPARQL, S-expression, etc.). GrailQA 是一个大规模高质量数据集，用于知识库问答，包含 64,331 个问题，并附有答案和不同语法的相应逻辑形式。","descriptions":{"en":"GrailQA is a new large-scale, high-quality dataset for question answering on knowledge bases (KBQA) on Freebase with 64,331 questions annotated with both answers and corresponding logical forms in different syntax (i.e., SPARQL, S-expression, etc.).","zh":"GrailQA 是一个大规模高质量数据集，用于知识库问答，包含 64,331 个问题，并附有答案和不同语法的相应逻辑形式。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1098","languages":[],"library_direction":null,"modality":null,"name":"GrailQA","openness":"unknown","publisher":"The Ohio State University","released":"2021-02-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1098-grailqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GrailQA","topics":[],"unit":null},{"aliases":["Reliability-Prioritized Fine-Grained Generation in Multimodal Large"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"A granularity-aware evaluation of whether multimodal models produce visually correct descriptions at reliably specific category levels.","descriptions":{"en":"A granularity-aware evaluation of whether multimodal models produce visually correct descriptions at reliably specific category levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.29573","languages":[],"library_direction":null,"modality":null,"name":"GranFact","openness":"unknown","publisher":null,"released":"2026-06-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29573","source_url":"https://arxiv.org/abs/2606.29573"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29573","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29573","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"GraphARC is a benchmark for abstract reasoning on graphs, generalizing ARC few-shot learning. No artifacts are provided in this article.","descriptions":{"en":"GraphARC is a benchmark for abstract reasoning on graphs, generalizing ARC few-shot learning. No artifacts are provided in this article."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.31031","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_grapharc_4326342c"},"modality":null,"name":"GraphARC","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31031","source_url":"https://arxiv.org/abs/2605.31031"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31031","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31031","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":54,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:graphinfer/graphinfer-bench","source_record":"bm_graphinfer-bench_ae7fff34"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"GraphInfer-Bench evaluates LLMs on graph inference tasks where answers reside in no single node or path, covering five task types over six real-world graphs with 42,000 samples. Tasks include masked-node prediction, edge inference, theme description, outlier detection, and community partition.","descriptions":{"en":"GraphInfer-Bench evaluates LLMs on graph inference tasks where answers reside in no single node or path, covering five task types over six real-world graphs with 42,000 samples. Tasks include masked-node prediction, edge inference, theme description, outlier detection, and community partition."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.11562","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_graphinfer-bench_ae7fff34"},"modality":null,"name":"GraphInfer-Bench","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11562","source_url":"https://arxiv.org/abs/2606.11562"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11562","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11562","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"LLM-based multi-agent systems coordinate specialized reasoning through aggregation, interaction, and adaptive control, yet their potential for graph learning remains unexplored.","descriptions":{"en":"LLM-based multi-agent systems coordinate specialized reasoning through aggregation, interaction, and adaptive control, yet their potential for graph learning remains unexplored."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.39777","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","abstract-reasoning"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_graphmas_7e744962"},"modality":null,"name":"GraphMAS","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39777","source_url":"https://arxiv.org/abs/2609.39777"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39777","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39777","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:gui0609/graphrarebench","source_record":"bm_graphrarebench_9ff01b1d"},"categories":["General AI","Language & Knowledge","q-bio.QM"],"collected_at":"2026-10-14T09:44:51Z","description":"GraphRareBench evaluates phenotype-driven rare-disease ranking with 2,365 cases, 18,093 target-confounder pairs, and auditable evidence records.","descriptions":{"en":"GraphRareBench evaluates phenotype-driven rare-disease ranking with 2,365 cases, 18,093 target-confounder pairs, and auditable evidence records."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.24878","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_graphrarebench_9ff01b1d"},"modality":null,"name":"GraphRareBench","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24878","source_url":"https://arxiv.org/abs/2607.24878"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24878","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24878","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sunyuanfu/graphverse","source_record":"bm_graphverse_5c59831b"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"GraphVerse evaluates multimodal large language models on visual graph reasoning, covering perception, reasoning, and text-based graph reasoning in single and paired image settings, with process-sensitive scoring.","descriptions":{"en":"GraphVerse evaluates multimodal large language models on visual graph reasoning, covering perception, reasoning, and text-based graph reasoning in single and paired image settings, with process-sensitive scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.06769","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_graphverse_5c59831b"},"modality":null,"name":"GraphVerse","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06769","source_url":"https://arxiv.org/abs/2608.06769"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06769","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06769","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":4336,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_9b96b90782b4c763"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GraphWalks is a synthetic multi-hop long-context reasoning benchmark in which a model is given an edge-list representation of a graph and must traverse it to find neighboring nodes (via breadth-first search) or parent nodes for a given start node. Performance is reported as F1 of the model-predicted answer set versus the ground truth.","descriptions":{"en":"GraphWalks is a synthetic multi-hop long-context reasoning benchmark in which a model is given an edge-list representation of a graph and must traverse it to find neighboring nodes (via breadth-first search) or parent nodes for a given start node. Performance is reported as F1 of the model-predicted answer set versus the ground truth."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:graphwalks:mimo-v2.5","reported_at":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:graphwalks","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","abstract-reasoning"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_9b96b90782b4c763"},"modality":"text","name":"GraphWalks","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":90.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.9,"raw_min":0.62,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:graphwalks:mai-thinking-1","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"}],"slug":"llm-stats-graphwalks","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":4792,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_53d58e9c28a37225"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","spatial_reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"GraphWalks BFS variant evaluated on 1M-token contexts.","descriptions":{"en":"GraphWalks BFS variant evaluated on 1M-token contexts."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:graphwalks-bfs-1m:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-1m?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:graphwalks-bfs-1m","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","abstract-reasoning"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_53d58e9c28a37225"},"modality":"text","name":"Graphwalks BFS 1M","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":77.10000000000001,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.771,"raw_min":0.512,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:graphwalks-bfs-1m:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-1m?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-graphwalks-bfs-1m","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-1m?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":4792,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_6d2eb34ec8f6b584"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","spatial_reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length under 128k tokens, returning nodes reachable at specified depths.","descriptions":{"en":"A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length under 128k tokens, returning nodes reachable at specified depths."},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:graphwalks-bfs-<128k:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3C128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:graphwalks-bfs-<128k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","abstract-reasoning"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_6d2eb34ec8f6b584"},"modality":"text","name":"Graphwalks BFS <128k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":94.0,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.94,"raw_min":0.25,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:graphwalks-bfs-<128k:gpt-5.2-2025-12-11","reported_date":"2025-12-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3C128k?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"}],"slug":"llm-stats-graphwalks-bfs-128k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3C128k?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":4792,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_6d2eb34ec8f6b584"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","spatial_reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length over 128k tokens, testing long-context reasoning capabilities.","descriptions":{"en":"A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length over 128k tokens, testing long-context reasoning capabilities."},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:graphwalks-bfs->128k:gpt-4.1-2025-04-14","reported_at":"2025-04-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3E128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:graphwalks-bfs->128k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","abstract-reasoning"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_6d2eb34ec8f6b584"},"modality":"text","name":"Graphwalks BFS >128k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":90.7,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.907,"raw_min":0.029,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:graphwalks-bfs->128k:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3E128k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-graphwalks-bfs-128k-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3E128k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":4792,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_2afafaa160f04bd4"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","spatial_reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length under 128k tokens, requiring understanding of graph structure and edge relationships.","descriptions":{"en":"A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length under 128k tokens, requiring understanding of graph structure and edge relationships."},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:graphwalks-parents-<128k:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3C128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:graphwalks-parents-<128k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","abstract-reasoning"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_2afafaa160f04bd4"},"modality":"text","name":"Graphwalks parents <128k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":89.8,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.898,"raw_min":0.094,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:graphwalks-parents-<128k:gpt-5.4","reported_date":"2026-03-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3C128k?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"}],"slug":"llm-stats-graphwalks-parents-128k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3C128k?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":4792,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_2afafaa160f04bd4"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","spatial_reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length over 128k tokens, testing long-context reasoning and graph structure understanding.","descriptions":{"en":"A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length over 128k tokens, testing long-context reasoning and graph structure understanding."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:graphwalks-parents->128k:gpt-4.1-2025-04-14","reported_at":"2025-04-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3E128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:graphwalks-parents->128k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","abstract-reasoning"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_2afafaa160f04bd4"},"modality":"text","name":"Graphwalks parents >128k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":95.39999999999999,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.954,"raw_min":0.056,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:graphwalks-parents->128k:claude-opus-4-6","reported_date":"2026-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3E128k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"}],"slug":"llm-stats-graphwalks-parents-128k-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3E128k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_greekbarretrieval_75c379ac"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"GreekBarRetrieval is a retrieval benchmark for Greek statutory articles, comprising bar-exam questions with case facts and candidate articles.","descriptions":{"en":"GreekBarRetrieval is a retrieval benchmark for Greek statutory articles, comprising bar-exam questions with case facts and candidate articles."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.18752","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","deep-research"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_greekbarretrieval_75c379ac"},"modality":null,"name":"GreekBarRetrieval","openness":"unknown","publisher":null,"released":"2026-08-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.18752","source_url":"https://arxiv.org/abs/2608.18752"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-18752","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18752","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:guptabhishekumar/grit-benchmark","source_record":"bm_grit-benchmark_0cf59789"},"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM planners on robot skill execution under turbulence, scoring recovery from disruptions across tasks, tiers, and seeds.","descriptions":{"en":"Evaluates LLM planners on robot skill execution under turbulence, scoring recovery from disruptions across tasks, tiers, and seeds."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:guptabhishekumar/grit-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-skills-evolution","embodied-robotics"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_grit-benchmark_0cf59789"},"modality":null,"name":"GRIT","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:guptabhishekumar/grit-benchmark","source_url":"https://github.com/guptabhishekumar/grit-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-guptabhishekumar-grit-benchmark","source":"claire_radar","source_url":"https://github.com/guptabhishekumar/grit-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GroundingSuite is designed to test the localization capabilities of multimodal models. It created 3,720 pixel-level data entries based on COCO Unlabeled images. This dataset covers four dimensions: Stuff Class Object, Multi-Object, Part-Level Object, and Single Object. GroundingSuite 用来测试多模态模型的定位能力。它通过半自动标注和人工筛选在COCO Unlabel的图片基础上创建了3720条pixel-level的数据，覆盖Stuff Class Object, Multi Object, Part Level Object, Single Object四个维度，是一个全面评测多模态模型定位能力的测试集。","descriptions":{"en":"GroundingSuite is designed to test the localization capabilities of multimodal models. It created 3,720 pixel-level data entries based on COCO Unlabeled images. This dataset covers four dimensions: Stuff Class Object, Multi-Object, Part-Level Object, and Single Object.","zh":"GroundingSuite 用来测试多模态模型的定位能力。它通过半自动标注和人工筛选在COCO Unlabel的图片基础上创建了3720条pixel-level的数据，覆盖Stuff Class Object, Multi Object, Part Level Object, Single Object四个维度，是一个全面评测多模态模型定位能力的测试集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2027","languages":[],"library_direction":null,"modality":"multimodal","name":"GroundingSuite","openness":"open","publisher":"Huazhong University of Science and Technology","released":"2025-07-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2027-groundingsuite","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GroundingSuite","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Geology question sets and grading rubrics for evaluating LLMs on real-world geological reasoning, with 153 questions across four configs, each grounded in source corpora with evidence locators and a harness for generation and scoring.","descriptions":{"en":"Geology question sets and grading rubrics for evaluating LLMs on real-world geological reasoning, with 153 questions across four configs, each grounded in source corpora with evidence locators and a harness for generation and scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:eigenformai/groundtruth-dynamic-benchmarking","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning","agent-skills-evolution"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_groundtruth-dynamic-benchmarking_4f938e9e"},"modality":null,"name":"Groundtruth Dynamic Benchmarking","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:eigenformai/groundtruth-dynamic-benchmarking","source_url":"https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-eigenformai-groundtruth-dynamic-benchmarking","source":"claire_radar","source_url":"https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":233,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:ltzheng/agent-studio","source_record":"catalog_bc257a1b3101252a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","grounding","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A subset of GroundUI-18K for UI grounding evaluation, where models must predict action coordinates on screenshots based on single-step instructions across web, desktop, and mobile platforms.","descriptions":{"en":"A subset of GroundUI-18K for UI grounding evaluation, where models must predict action coordinates on screenshots based on single-step instructions across web, desktop, and mobile platforms."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:groundui-1k:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/groundui-1k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:groundui-1k","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_bc257a1b3101252a"},"modality":"multimodal","name":"GroundUI-1K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":81.39999999999999,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.814,"raw_min":0.802,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:groundui-1k:nova-pro","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/groundui-1k?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"}],"slug":"llm-stats-groundui-1k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/groundui-1k?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"GroupToM-Bench is a multimodal benchmark for evaluating group-level theory of mind through a seven-level cognitive audit framework.","descriptions":{"en":"GroupToM-Bench is a multimodal benchmark for evaluating group-level theory of mind through a seven-level cognitive audit framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04184","languages":[],"library_direction":null,"modality":null,"name":"GroupToM-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04184","source_url":"https://arxiv.org/abs/2606.04184"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04184","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04184","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_grouptravelbench_058d38b0"},"categories":["General AI","Language & Knowledge","Planning","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for multi-user, multi-turn travel planning with 650 tasks and a synchronous group-chat sandbox, evaluating elicitation, coordination, and fairness-aware planning.","descriptions":{"en":"A benchmark for multi-user, multi-turn travel planning with 650 tasks and a synchronous group-chat sandbox, evaluating elicitation, coordination, and fairness-aware planning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.25200","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","multi-agent-games"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_grouptravelbench_058d38b0"},"modality":null,"name":"GroupTravelBench","openness":"unknown","publisher":null,"released":"2026-05-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25200","source_url":"https://arxiv.org/abs/2605.25200"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25200","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25200","topics":["agents"],"unit":null},{"aliases":["Gsci-Fact","GsciFact","Gscifact"],"attention_signals":null,"categories":["science"],"collected_at":null,"description":"About 32.2K general-science claims, balanced across Supported, Refuted and Not-Enough-Info, generated automatically from SciQ crowd-sourced exam questions by the Multi2Claim pipeline (EACL 2023). Annotators judged a quarter of claims answerable without the evidence, so part of a score can come from world knowledge rather than verification. Only 50 examples are public; the full set is on request.","descriptions":{"en":"About 32.2K general-science claims, balanced across Supported, Refuted and Not-Enough-Info, generated automatically from SciQ crowd-sourced exam questions by the Multi2Claim pipeline (EACL 2023). Annotators judged a quarter of claims answerable without the evidence, so part of a score can come from world knowledge rather than verification. Only 50 examples are public; the full set is on request."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:gsci_fact","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["factuality"],"join":"review:14H034160212:2026-10-15","primary":"factuality","source_record":null},"modality":null,"name":"Gsci-Fact","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"gsci_fact","source":"model_reports","source_url":"https://aclanthology.org/2023.eacl-main.194/","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1459,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"llm_stats_id","repository":null,"source_record":"catalog_e0d03a02444be465"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Grade School Math 8K with Chain-of-Thought prompting, featuring 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations.","descriptions":{"en":"Grade School Math 8K with Chain-of-Thought prompting, featuring 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gsm-8k-(cot):llama-3.1-70b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm-8k-%28cot%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gsm-8k-(cot)","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_e0d03a02444be465"},"modality":"text","name":"GSM-8K (CoT)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":95.1,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.951,"raw_min":0.845,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gsm-8k-(cot):llama-3.1-70b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm-8k-%28cot%29?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"}],"slug":"llm-stats-gsm-8k-cot","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm-8k-%28cot%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["数学","Math","NeurIPS 2024","大语言模型","LLM","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GSM1k is meant for evaluating LLM's math reasoning ability. It mirrors the style and complexity of the established GSM8k benchmark while consider the problem of data-leaking and overfitting GSM1k可用于评估LLM的数学推理能力；它与GSM8k保持了风格和复杂性的一致，同时考虑了数据泄露和过拟合的问题。","descriptions":{"en":"GSM1k is meant for evaluating LLM's math reasoning ability. It mirrors the style and complexity of the established GSM8k benchmark while consider the problem of data-leaking and overfitting","zh":"GSM1k可用于评估LLM的数学推理能力；它与GSM8k保持了风格和复杂性的一致，同时考虑了数据泄露和过拟合的问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1272","languages":[],"library_direction":null,"modality":null,"name":"GSM1k","openness":"unknown","publisher":null,"released":"2024-05-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1272-gsm1k","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GSM1k","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1459,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"llm_stats_id","repository":null,"source_record":"catalog_0eb1f54b7f307458"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Grade School Math 8K, a dataset of 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations.","descriptions":{"en":"Grade School Math 8K, a dataset of 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations."},"evidence_summary":{"document_count":1,"model_count":48,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gsm8k:claude-3-opus-20240229","reported_at":"2024-02-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:gsm8k","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_0eb1f54b7f307458"},"modality":"text","name":"GSM8k","openness":"unknown","publisher":null,"released":"2021-04-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":48,"score_direction":"higher_is_better","score_summary":{"display_max":99.6,"display_multiplier":100,"model_count":48,"model_count_basis":"source_model_id","numeric_count":48,"raw_max":0.996,"raw_min":0.252,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gsm8k:mimo-v2.5-pro","reported_date":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k?top_n=500"},"unit":null},"scored_models":[{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-08-30","date_precision":"model_announcement","name":"Command R+"},{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2024-03-28","date_precision":"model_announcement","name":"Grok-1.5"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Base"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"}],"slug":"llm-stats-gsm8k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["GSM8K"],"attention_signals":null,"categories":["math"],"collected_at":null,"description":"Saturated. A legacy baseline for small open models.","descriptions":{"en":"Saturated. A legacy baseline for small open models."},"evidence_summary":{"document_count":7,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000gsm8k\u0000google_gemini_1_5_report\u0000gsm8k\u000011-shot\u0000Gemini 1.5 Pro","reported_at":"2024-03-08","source_url":"https://arxiv.org/abs/2403.05530"},"first_score_reported_at":"2024-03-08","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000gsm8k\u0000google_gemini_1_5_report\u0000gsm8k\u000011-shot\u0000Gemini 1.5 Pro","reported_at":"2024-03-08","source_url":"https://arxiv.org/abs/2403.05530"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:gsm8k","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["math"],"join":"review:14H034160212:2026-10-15","primary":"math","source_record":null},"modality":null,"name":"GSM8K","openness":"unknown","publisher":null,"released":"2021-10-27","released_reference":{"basis":"benchmark_release","source_key":"model-reports:gsm8k","source_url":"https://github.com/openai/grade-school-math"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":90.8,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":90.8,"raw_min":90.8,"source_reference":{"obs_id":"curated\u0000gsm8k\u0000google_gemini_1_5_report\u0000gsm8k\u000011-shot\u0000Gemini 1.5 Pro","observation_id":"curated\u0000gsm8k\u0000google_gemini_1_5_report\u0000gsm8k\u000011-shot\u0000Gemini 1.5 Pro","reported_at":"2024-03-08","reported_date":"2024-03-08","source_id":"google_gemini_1_5_report","source_url":"https://arxiv.org/abs/2403.05530"},"unit":"percent"},"scored_models":[{"date":"2024-03-08","date_precision":"document_publication","name":"Gemini 1.5 Pro"}],"slug":"gsm8k","source":"model_reports","source_url":"https://github.com/openai/grade-school-math","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1459,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"artifact:arxiv:2110.14168","repository":null,"source_record":"catalog_0eb1f54b7f307458"},"categories":["数学","Math","大语言模型","LLM","数理能力","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GSM8K is a dataset of 8,500 high quality linguistically diverse grade school math word problems created by human problem writers. The dataset is segmented into 7,500 training problems and 1,000 test problems. These problems take between 2 and 8 steps to solve, and solutions primarily involve performing a sequence of elementary calculations using basic arithmetic operations (+ − × ÷) to reach the final answer. GSM8K 是一个包含 8,500 个高质量、语言多样化的小学数学单词问题的数据集，由人类问题编写者创建。该数据集分为 7,500 个训练问题和 1,000 个测试问题。这些问题的解题步骤在 2 到 8 步之间，解题过程主要涉及使用基本算术运算（+ - × ÷）进行一连串的基本计算，从而得出最终答案。一个聪明的初中生应该能够解决每一个问题。它可用于多步数学推理。","descriptions":{"en":"GSM8K is a dataset of 8,500 high quality linguistically diverse grade school math word problems created by human problem writers. The dataset is segmented into 7,500 training problems and 1,000 test problems. These problems take between 2 and 8 steps to solve, and solutions primarily involve performing a sequence of elementary calculations using basic arithmetic operations (+ − × ÷) to reach the final answer.","zh":"GSM8K 是一个包含 8,500 个高质量、语言多样化的小学数学单词问题的数据集，由人类问题编写者创建。该数据集分为 7,500 个训练问题和 1,000 个测试问题。这些问题的解题步骤在 2 到 8 步之间，解题过程主要涉及使用基本算术运算（+ - × ÷）进行一连串的基本计算，从而得出最终答案。一个聪明的初中生应该能够解决每一个问题。它可用于多步数学推理。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:535","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"artifact:arxiv:2110.14168","primary":"math","source_record":"catalog_0eb1f54b7f307458"},"modality":null,"name":"GSM8K","openness":"unknown","publisher":null,"released":"2021-04-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-535-gsm8k","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GSM8K","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1459,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"llm_stats_id","repository":null,"source_record":"catalog_dce7019d8b7b470d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Grade School Math 8K adapted for chat format evaluation, featuring 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations.","descriptions":{"en":"Grade School Math 8K adapted for chat format evaluation, featuring 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:gsm8k-chat:llama-3.1-nemotron-70b-instruct","reported_at":"2024-10-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k-chat?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:gsm8k-chat","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_dce7019d8b7b470d"},"modality":"text","name":"GSM8K Chat","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":81.88,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.8188,"raw_min":0.8188,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:gsm8k-chat:llama-3.1-nemotron-70b-instruct","reported_date":"2024-10-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k-chat?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"}],"slug":"llm-stats-gsm8k-chat","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k-chat?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","数学","Math","visual mathematical reasoning","gsm8k","multi-image math benchmark","多模态模型","VLM","逻辑推理","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GSM8K-V is a multi-image, purely visual mathematical reasoning benchmark built by rendering GSM8K textual problems into visual scenes. It exposes substantial gaps in current vision-language models’ visual reasoning despite their near-saturated performance on text-based GSM8K. GSM8K-V 是一个多图视觉数学推理基准，通过将 GSM8K 的文本题系统性转换为多场景图像构建而成。它揭示了当前 VLMs 在视觉数学推理上的表现与其在文本数学推理上的表现之间存在显著差距。","descriptions":{"en":"GSM8K-V is a multi-image, purely visual mathematical reasoning benchmark built by rendering GSM8K textual problems into visual scenes. It exposes substantial gaps in current vision-language models’ visual reasoning despite their near-saturated performance on text-based GSM8K.","zh":"GSM8K-V 是一个多图视觉数学推理基准，通过将 GSM8K 的文本题系统性转换为多场景图像构建而成。它揭示了当前 VLMs 在视觉数学推理上的表现与其在文本数学推理上的表现之间存在显著差距。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2233","languages":[],"library_direction":null,"modality":"multimodal","name":"GSM8K-V","openness":"open","publisher":"浙江大学","released":"2025-09-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2233-gsm8k-v","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GSM8K-V","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yuxieharrison/gso-net","source_record":"bm_gso-net_911a783f"},"categories":["Logistics & Operations","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A large-scale industrial vision benchmark for standard operating procedure understanding in petrochemical unloading, built from 50,325 images across 64 stations with 321,432 bounding boxes. It links nine macroscopic procedural steps and 15 microscopic operational states across a core joint step-state detection task and a diagnostic frame-level step classification task, using a strict cross-site split.","descriptions":{"en":"A large-scale industrial vision benchmark for standard operating procedure understanding in petrochemical unloading, built from 50,325 images across 64 stations with 321,432 bounding boxes. It links nine macroscopic procedural steps and 15 microscopic operational states across a core joint step-state detection task and a diagnostic frame-level step classification task, using a strict cross-site split."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.12408","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","safety-alignment","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_gso-net_911a783f"},"modality":null,"name":"GSO-Net","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12408","source_url":"https://arxiv.org/abs/2609.12408"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12408","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12408","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":47,"join":"claire_record_id","repository":null,"source_record":"bm_gst-bench_e194aeb7"},"categories":["Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"GST-Bench is a VQA benchmark for global spatial intelligence in video understanding, covering synthetic videos and human-verified questions. It evaluates VLMs' ability to infer spatial relations from novel viewpoints and map egocentric observations to top-down views.","descriptions":{"en":"GST-Bench is a VQA benchmark for global spatial intelligence in video understanding, covering synthetic videos and human-verified questions. It evaluates VLMs' ability to infer spatial relations from novel viewpoints and map egocentric observations to top-down views."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05747","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","video-understanding"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_gst-bench_e194aeb7"},"modality":null,"name":"GST-Bench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05747","source_url":"https://arxiv.org/abs/2608.05747"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05747","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05747","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","NeurIPS 2024","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"GTA is meant for LLM's tool-use evaluations under real-world scenarios, including 229 real-world tasks and executable tool chains. GTA用于评估LLM调用工具解决实际问题的能力，由229个真实任务和可执行工具链组成。","descriptions":{"en":"GTA is meant for LLM's tool-use evaluations under real-world scenarios, including 229 real-world tasks and executable tool chains.","zh":"GTA用于评估LLM调用工具解决实际问题的能力，由229个真实任务和可执行工具链组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1328","languages":[],"library_direction":null,"modality":null,"name":"GTA","openness":"open","publisher":"Shanghai AI Laboratory","released":"2024-07-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1328-gta","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GTA","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"GTBench is a curriculum-grounded benchmark for evaluating LLMs as mathematical research assistants in graph theory, with problems across three difficulty groups.","descriptions":{"en":"GTBench is a curriculum-grounded benchmark for evaluating LLMs as mathematical research assistants in graph theory, with problems across three difficulty groups."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03144","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_gtbench_117f8151"},"modality":null,"name":"GTBench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03144","source_url":"https://arxiv.org/abs/2606.03144"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03144","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03144","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Information retrieval","Language & Knowledge","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"Guardian Crawler is a retrieval-first testbed for knowledge discovery over synthetic web-like corpora, combining BM25 retrieval with reranking and constrained generation.","descriptions":{"en":"Guardian Crawler is a retrieval-first testbed for knowledge discovery over synthetic web-like corpora, combining BM25 retrieval with reranking and constrained generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08994","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_guardian-crawler_0a6465c9"},"modality":null,"name":"Guardian Crawler","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08994","source_url":"https://arxiv.org/abs/2608.08994"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08994","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08994","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_guardianagentbench_89ff40f2"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"GuardianAgentBench (GABench) is a benchmark of 580 agent scenarios across six domains, evaluated on three production frameworks (LangChain, LlamaIndex, Vectara), with multi-stage validation and five adversarial attack modes to assess tool-use correctness and safety.","descriptions":{"en":"GuardianAgentBench (GABench) is a benchmark of 580 agent scenarios across six domains, evaluated on three production frameworks (LangChain, LlamaIndex, Vectara), with multi-stage validation and five adversarial attack modes to assess tool-use correctness and safety."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20982","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_guardianagentbench_89ff40f2"},"modality":null,"name":"GuardianAgentBench","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20982","source_url":"https://arxiv.org/abs/2607.20982"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20982","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20982","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"3,024 instruction-scene examples organized as same-scene Safe/Unsafe contrastive pairs across hazard categories, based on international safety standards. Evaluates VLM safety reasoning under latent contextual risk.","descriptions":{"en":"3,024 instruction-scene examples organized as same-scene Safe/Unsafe contrastive pairs across hazard categories, based on international safety standards. Evaluates VLM safety reasoning under latent contextual risk."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.21928","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_guardianbench_debc3a47"},"modality":null,"name":"GuardianBench","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.21928","source_url":"http://arxiv.org/abs/2608.21928v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-21928","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21928v1","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":null,"source_record":"bm_gui-cc_ad8b1116"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"GUI-CC evaluates whether GUI world models maintain contextual consistency in multi-step agent interactions using offline reference trajectories and online probing agents.","descriptions":{"en":"GUI-CC evaluates whether GUI world models maintain contextual consistency in multi-step agent interactions using offline reference trajectories and online probing agents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.00048","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","world-models"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_gui-cc_ad8b1116"},"modality":null,"name":"GUI-CC","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00048","source_url":"https://arxiv.org/abs/2609.00048"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00048","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00048","topics":["computer-use","agents"],"unit":null},{"aliases":["GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"claire_record_id","repository":null,"source_record":"bm_gui-primitives-diagnosing-spatial-reasonin_a22b1551"},"categories":["General AI","Multimodal","Reasoning","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"994-item benchmark of contrastive instruction pairs over seven spatial relations in GUIs, isolating whether vision-language models bind relational language to UI elements correctly.","descriptions":{"en":"994-item benchmark of contrastive instruction pairs over seven spatial relations in GUIs, isolating whether vision-language models bind relational language to UI elements correctly."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.21832","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","spatial-3d"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_gui-primitives-diagnosing-spatial-reasonin_a22b1551"},"modality":null,"name":"GUI-Primitives","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.21832","source_url":"http://arxiv.org/abs/2608.21832v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-21832","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21832v1","topics":["computer-use","agents","multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:fawnliu/guideme","source_record":"bm_guideme_feb34852"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"GuideMe is a benchmark for evaluating multimodal large language models on streaming video task guidance. It includes 2,458 videos (223.7 hours) with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance. Assessment uses temporal-semantic matching, behavioral classification, and LLM-as-a-Judge.","descriptions":{"en":"GuideMe is a benchmark for evaluating multimodal large language models on streaming video task guidance. It includes 2,458 videos (223.7 hours) with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance. Assessment uses temporal-semantic matching, behavioral classification, and LLM-as-a-Judge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.02991","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_guideme_feb34852"},"modality":null,"name":"GuideMe","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02991","source_url":"https://arxiv.org/abs/2607.02991"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02991","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02991","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","Mobile & Personal Computing"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates exploratory GUI testing agents on 61 Android apps with 508 preset defects, using an open-set evaluator that decomposes trajectories into independently diagnosable capabilities.","descriptions":{"en":"Evaluates exploratory GUI testing agents on 61 Android apps with 508 preset defects, using an open-set evaluator that decomposes trajectories into independently diagnosable capabilities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29532","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","agentic-coding"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_guitestscape_3504136d"},"modality":null,"name":"GUITestScape","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29532","source_url":"https://arxiv.org/abs/2605.29532"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29532","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29532","topics":["coding","computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","强化学习","智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Gym4ReaL is a benchmark suite designed to evaluate reinforcement learning algorithms in real-world scenarios, addressing challenges such as non-stationarity, partial observability, and large state-action spaces. Gym4ReaL 是一个用于评估强化学习算法在真实世界场景中表现的基准套件，涵盖非平稳性、部分可观测性和大状态-动作空间等挑战。","descriptions":{"en":"Gym4ReaL is a benchmark suite designed to evaluate reinforcement learning algorithms in real-world scenarios, addressing challenges such as non-stationarity, partial observability, and large state-action spaces.","zh":"Gym4ReaL 是一个用于评估强化学习算法在真实世界场景中表现的基准套件，涵盖非平稳性、部分可观测性和大状态-动作空间等挑战。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2035","languages":[],"library_direction":null,"modality":null,"name":"Gym4ReaL","openness":"unknown","publisher":"Politecnico di Milano","released":"2025-06-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2035-gym4real","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Gym4ReaL","topics":["agents"],"unit":null},{"aliases":["GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.HC"],"collected_at":"2026-10-14T09:44:51Z","description":"An evaluation of large language models' cultural value orientations across the two Inglehart-Welzel axes, covering several domains and roles, to quantify their alignment along traditional vs secular-rational and survival vs self-expression values.","descriptions":{"en":"An evaluation of large language models' cultural value orientations across the two Inglehart-Welzel axes, covering several domains and roles, to quantify their alignment along traditional vs secular-rational and survival vs self-expression values."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.18384","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_gyroval-a-robust-benchmark-for-cultural-va_e114f7a2"},"modality":null,"name":"GYROval","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18384","source_url":"https://arxiv.org/abs/2609.18384"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18384","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18384","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_h2hmem_6775c266"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"H2HMem is a benchmark for evaluating memory capabilities of agents in human-human multimodal interactions, covering dyadic and multi-party conversations with tasks in memory recall, reasoning, and application.","descriptions":{"en":"H2HMem is a benchmark for evaluating memory capabilities of agents in human-human multimodal interactions, covering dyadic and multi-party conversations with tasks in memory recall, reasoning, and application."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09461","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_h2hmem_6775c266"},"modality":null,"name":"H2HMem","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09461","source_url":"https://arxiv.org/abs/2606.09461"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09461","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09461","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"claire_record_id","repository":"gh:rongdingyi/h2r-bench","source_record":"bm_h2r-bench_da8120f6"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"H2R-Bench evaluates cross-embodiment human-to-robot manipulation video generation. Models convert egocentric human demonstrations into robot manipulation videos under specified target embodiments. Scoring covers five dimensions: goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and general video quality, aggregated into H2RCore.","descriptions":{"en":"H2R-Bench evaluates cross-embodiment human-to-robot manipulation video generation. Models convert egocentric human demonstrations into robot manipulation videos under specified target embodiments. Scoring covers five dimensions: goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and general video quality, aggregated into H2RCore."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.13049","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","embodied-robotics"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_h2r-bench_da8120f6"},"modality":null,"name":"H2R-Bench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13049","source_url":"https://arxiv.org/abs/2608.13049"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13049","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13049","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark proposed for evaluating human-to-robot transfer methods in a real-to-sim setting.","descriptions":{"en":"A benchmark proposed for evaluating human-to-robot transfer methods in a real-to-sim setting."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24778","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_h2rbench_f1bb8dfe"},"modality":null,"name":"H2RBench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24778","source_url":"https://arxiv.org/abs/2609.24778"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24778","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24778","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Adapts Terminal Bench with hack-verifiable environments to automatically detect reward hacking in agent trajectories.","descriptions":{"en":"Adapts Terminal Bench with hack-verifiable environments to automatically detect reward hacking in agent trajectories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22103","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","agent-security"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_hack-verifiable-terminal-bench_a02835b2"},"modality":null,"name":"Hack-Verifiable Terminal Bench","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22103","source_url":"http://arxiv.org/abs/2608.22103v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22103","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22103v1","topics":["coding","agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":32,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":"gh:hakari-bench/hakari-bench","source_record":"bm_hakari-bench_2bd08ca4"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates retrieval architectures and efficiency settings (dimensionality reduction, quantization, reranking) across 35 benchmarks and 551 tasks in 43 languages, with unified conditions and metrics for BM25, dense, sparse, late interaction, and reranker models.","descriptions":{"en":"Evaluates retrieval architectures and efficiency settings (dimensionality reduction, quantization, reranking) across 35 benchmarks and 551 tasks in 43 languages, with unified conditions and metrics for BM25, dense, sparse, late interaction, and reranker models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22778","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_hakari-bench_2bd08ca4"},"modality":null,"name":"HAKARI-Bench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22778","source_url":"https://arxiv.org/abs/2606.22778"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22778","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22778","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_hakushobench_550d3cd1"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"HakushoBench is a Japanese chart and table VQA benchmark built from 33 governmental white papers, containing 2,053 images across over 10 image types with manually annotated QA pairs. It evaluates vision-language models on deep holistic understanding of charts and tables.","descriptions":{"en":"HakushoBench is a Japanese chart and table VQA benchmark built from 33 governmental white papers, containing 2,053 images across over 10 image types with manually annotated QA pairs. It evaluates vision-language models on deep holistic understanding of charts and tables."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01132","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","multilingual"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_hakushobench_550d3cd1"},"modality":null,"name":"HakushoBench","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01132","source_url":"https://arxiv.org/abs/2606.01132"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01132","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01132","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":344,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":27,"join":"llm_stats_id","repository":"gh:tianyi-lab/hallusionbench","source_record":"catalog_8b46f9ccf66ae297"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive benchmark designed to evaluate image-context reasoning in large visual-language models (LVLMs) by challenging models with 346 images and 1,129 carefully crafted questions to assess language hallucination and visual illusion","descriptions":{"en":"A comprehensive benchmark designed to evaluate image-context reasoning in large visual-language models (LVLMs) by challenging models with 346 images and 1,129 carefully crafted questions to assess language hallucination and visual illusion"},"evidence_summary":{"document_count":1,"model_count":18,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hallusion-bench:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hallusion-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:hallusion-bench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_8b46f9ccf66ae297"},"modality":"multimodal","name":"Hallusion Bench","openness":"unknown","publication":{"evidence":[{"note":"The author repository explicitly introduces HallusionBench as an image-context reasoning benchmark for language hallucination and visual illusion, matching the source record's task description, and links arXiv:2310.14566 and its CVPR 2024 publication. This is HallusionBench, not the separately linked AutoHallusion benchmark.","url":"https://github.com/tianyi-lab/HallusionBench/blob/main/README.md"}],"id":"hallusionbench-publication","links":[{"kind":"paper","title":"HallusionBench introducing paper","url":"https://arxiv.org/abs/2310.14566"},{"kind":"repo","title":"HallusionBench official repository","url":"https://github.com/tianyi-lab/HallusionBench"}],"record_keys":["llm-stats:hallusion-bench"],"reviewed_at":"2026-10-26","scope":{"en":"HallusionBench's original image-context reasoning benchmark, paper and author-maintained code/data repository.","zh":"HallusionBench 图像情境推理评测的原论文及作者维护的代码、数据仓库。"}},"publisher":null,"released":"2023-10-23","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:hallusion-bench","source_url":"https://arxiv.org/abs/2310.14566"},"repo_kind":null,"repo_resolution_status":null,"score_count":18,"score_direction":"higher_is_better","score_summary":{"display_max":70.0,"display_multiplier":100,"model_count":18,"model_count_basis":"source_model_id","numeric_count":18,"raw_max":0.7,"raw_min":0.472,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hallusion-bench:qwen3.5-27b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hallusion-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-hallusion-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hallusion-bench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":344,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":27,"join":"artifact:arxiv:2310.14566","repository":"gh:tianyi-lab/hallusionbench","source_record":"catalog_8b46f9ccf66ae297"},"categories":["多模态","Multimodal","推理","Reasoning","多模态模型","VLM","图像理解","Image Understanding","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HallusionBench is a comprehensive benchmark designed for the evaluation of image-context reasoning, comprising 346 images paired with 1129 questions. HallusionBench是一个专为评估图像上下文推理而设计的综合基准测试，包括346张图像和1129个问题。","descriptions":{"en":"HallusionBench is a comprehensive benchmark designed for the evaluation of image-context reasoning, comprising 346 images paired with 1129 questions.","zh":"HallusionBench是一个专为评估图像上下文推理而设计的综合基准测试，包括346张图像和1129个问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1355","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"artifact:arxiv:2310.14566","primary":"visual-reasoning","source_record":"catalog_8b46f9ccf66ae297"},"modality":"multimodal","name":"HallusionBench","openness":"unknown","publisher":"University of Maryland","released":"2023-10-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1355-hallusionbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HallusionBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_hallutruthqa_4b25f1be"},"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"HalluTruthQA is a fine-grained benchmark for Arabic QA hallucination detection, localization, and explanation, containing 2,400 expert-curated examples across four knowledge domains with span-level and explanation annotations.","descriptions":{"en":"HalluTruthQA is a fine-grained benchmark for Arabic QA hallucination detection, localization, and explanation, containing 2,400 expert-curated examples across four knowledge domains with span-level and explanation annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20219","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","multilingual"],"join":"claire_record_id","primary":"factuality","source_record":"bm_hallutruthqa_4b25f1be"},"modality":null,"name":"HalluTruthQA","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20219","source_url":"https://arxiv.org/abs/2607.20219"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20219","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20219","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","大语言模型","LLM","安全对齐","Safety Alignment","事实可靠性","Factual Reliability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HaluEval evaluates the performance of LLMs in recognizing hallucination. It includes 5,000 general user queries with ChatGPT responses and 30,000 task-specific examples from three tasks, i.e., question answering, knowledge-grounded dialogue, and text summarization. HaluEval用于评估大语言模型识别幻觉的能力，包含 5,000 条普通用户查询及 ChatGPT 的回答，以及来自三个任务的 30,000 个特定任务示例，即问答、基于知识的对话和文本摘要。","descriptions":{"en":"HaluEval evaluates the performance of LLMs in recognizing hallucination. It includes 5,000 general user queries with ChatGPT responses and 30,000 task-specific examples from three tasks, i.e., question answering, knowledge-grounded dialogue, and text summarization.","zh":"HaluEval用于评估大语言模型识别幻觉的能力，包含 5,000 条普通用户查询及 ChatGPT 的回答，以及来自三个任务的 30,000 个特定任务示例，即问答、基于知识的对话和文本摘要。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1121","languages":[],"library_direction":null,"modality":null,"name":"HaluEval","openness":"unknown","publisher":"Beijing Key Laboratory of Big Data Management and Analysis Methods","released":"2023-10-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1121-halueval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HaluEval","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Quantum Computing & Control","quant-ph"],"collected_at":"2026-10-14T09:44:51Z","description":"HamQASBench evaluates Quantum Architecture Search methods across 11 molecules organized in five structural tiers, using energy accuracy, per-qubit entanglement, and pairwise state fidelity to diagnose structural failures.","descriptions":{"en":"HamQASBench evaluates Quantum Architecture Search methods across 11 molecules organized in five structural tiers, using energy accuracy, per-qubit entanglement, and pairwise state fidelity to diagnose structural failures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.04845","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_hamqasbench_9db953d2"},"modality":null,"name":"HamQASBench","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04845","source_url":"https://arxiv.org/abs/2607.04845"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04845","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04845","topics":["science"],"unit":null},{"aliases":["HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following"],"attention_signals":null,"categories":["Biology & Drug Discovery","Finance","Language & Knowledge","Logistics & Operations","Long Context"],"collected_at":"2026-09-25T04:40:22Z","description":"HANDBOOK.md evaluates whether tool-using agents consistently obey lengthy standing procedures while completing workplace tasks in self-contained company environments.","descriptions":{"en":"HANDBOOK.md evaluates whether tool-using agents consistently obey lengthy standing procedures while completing workplace tasks in self-contained company environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.25398","languages":[],"library_direction":null,"modality":null,"name":"HANDBOOK.md","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25398","source_url":"https://arxiv.org/abs/2607.25398"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25398","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25398","topics":["long-context","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark that evaluates whether generated worlds remain reliable as agents interact with them.","descriptions":{"en":"A benchmark that evaluates whether generated worlds remain reliable as agents interact with them."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24308","languages":[],"library_direction":null,"modality":null,"name":"HappyWorld-Bench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24308","source_url":"https://arxiv.org/abs/2609.24308"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24308","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24308","topics":[],"unit":null},{"aliases":["Harbor-Index","Harbor Index"],"attention_signals":{"as_of":"2026-10-13","github_stars":40,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"artifact:gh:harbor-framework/harbor-index","repository":"gh:harbor-framework/harbor-index","source_record":"catalog_4f8c197b84c2e7d5"},"categories":["coding_agent"],"collected_at":null,"description":"Compact high-signal frontier-agent evaluation; the environment and harness define the score.","descriptions":{"en":"Compact high-signal frontier-agent evaluation; the environment and harness define the score."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000harbor_index\u0000tencent_hy4_preview\u0000harbor_index\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000harbor_index\u0000tencent_hy4_preview\u0000harbor_index\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:harbor_index","languages":[],"library_direction":{"basis":"model-review-2","categories":["composite-index","agentic-coding"],"join":"artifact:gh:harbor-framework/harbor-index","primary":"composite-index","source_record":"catalog_4f8c197b84c2e7d5"},"modality":null,"name":"Harbor-Index","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":39.6,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":39.6,"raw_min":39.6,"source_reference":{"obs_id":"curated\u0000harbor_index\u0000tencent_hy4_preview\u0000harbor_index\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000harbor_index\u0000tencent_hy4_preview\u0000harbor_index\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"harbor_index","source":"model_reports","source_url":"https://github.com/harbor-framework/harbor-index","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","代码","Code","数学","Math","Strong Reasoning","大语言模型","LLM","逻辑推理","代码工程","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HARDMath2 is a benchmark for applied mathematics created by students in a graduate class at Harvard University, featuring 211 original problems covering core topics such as boundary-layer analysis, WKB methods, asymptotic solutions of nonlinear partial differential equations, and the asymptotics. HARDMath2是由哈佛大学研究生课程的学生创建的一项应用数学基准测试，包含211道原创问题，涵盖边界层分析、WKB方法、非线性偏微分方程的渐近解以及振荡积分的渐近性等核心主题。该基准通过一种创新的协作方式构建，学生不仅设计并改进符合课程大纲的高难度问题，还对解决方案进行同行验证，同时测试不同模型的表现。最终，LLM生成的解答会与学生的答案以及数值真值进行自动对比，以评估模型的准确性和能力。","descriptions":{"en":"HARDMath2 is a benchmark for applied mathematics created by students in a graduate class at Harvard University, featuring 211 original problems covering core topics such as boundary-layer analysis, WKB methods, asymptotic solutions of nonlinear partial differential equations, and the asymptotics.","zh":"HARDMath2是由哈佛大学研究生课程的学生创建的一项应用数学基准测试，包含211道原创问题，涵盖边界层分析、WKB方法、非线性偏微分方程的渐近解以及振荡积分的渐近性等核心主题。该基准通过一种创新的协作方式构建，学生不仅设计并改进符合课程大纲的高难度问题，还对解决方案进行同行验证，同时测试不同模型的表现。最终，LLM生成的解答会与学生的答案以及数值真值进行自动对比，以评估模型的准确性和能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1842","languages":[],"library_direction":null,"modality":null,"name":"HARDMath2","openness":"unknown","publisher":"Harvard University","released":"2025-05-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1842-hardmath2","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HARDMath2","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jasonnlp/hardmtbench","source_record":"bm_hardmtbench_3eaf8a99"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"HardMTBench is a difficulty-aware diagnostic benchmark for Chinese-English domain translation, covering 12 domains with 20,000 directional test items and annotated hardness scores based on domain knowledge, translation difficulty, and terminology load.","descriptions":{"en":"HardMTBench is a difficulty-aware diagnostic benchmark for Chinese-English domain translation, covering 12 domains with 20,000 directional test items and annotated hardness scores based on domain knowledge, translation difficulty, and terminology load."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.28315","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_hardmtbench_3eaf8a99"},"modality":null,"name":"HardMTBench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28315","source_url":"https://arxiv.org/abs/2605.28315"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28315","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28315","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Information retrieval","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"AgentREVEAL is a diagnostic framework that evaluates safety alignment degradation in LLM agents when web retrieval is integrated. It assesses the impact of retrieval integration and content properties on harmful compliance, using a set of harmful behaviors and retrieval sources.","descriptions":{"en":"AgentREVEAL is a diagnostic framework that evaluates safety alignment degradation in LLM agents when web retrieval is integrated. It assesses the impact of retrieval integration and content properties on harmful compliance, using a set of harmful behaviors and retrieval sources."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29224","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","safety-alignment"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_harmurlbench_cbd88280"},"modality":null,"name":"HarmURLBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29224","source_url":"https://arxiv.org/abs/2605.29224"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29224","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29224","topics":["agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates harmful video understanding in large multimodal models using 1,379 videos and 4,137 multiple-choice questions across three hierarchical dimensions: Observable Evidence, Clip-Internal Meaning, and Beyond-Clip Reasoning.","descriptions":{"en":"Evaluates harmful video understanding in large multimodal models using 1,379 videos and 4,137 multiple-choice questions across three hierarchical dimensions: Observable Evidence, Clip-Internal Meaning, and Beyond-Clip Reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.27187","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","safety-alignment"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_harmvideobench_cc5648e1"},"modality":null,"name":"HarmVideoBench","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27187","source_url":"https://arxiv.org/abs/2606.27187"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27187","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27187","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_harness-bench_b6a94bb9"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Harness-Bench evaluates configuration-level harness effects in agent workflows with 106 sandboxed tasks, measuring completion, process quality, efficiency, and failure behavior across model-harness pairings.","descriptions":{"en":"Harness-Bench evaluates configuration-level harness effects in agent workflows with 106 sandboxed tasks, measuring completion, process quality, efficiency, and failure behavior across model-harness pairings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27922","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_harness-bench_b6a94bb9"},"modality":null,"name":"Harness-Bench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27922","source_url":"https://arxiv.org/abs/2605.27922"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27922","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27922","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":568,"join":"claire_record_id","repository":null,"source_record":"bm_harnessdev_06493943"},"categories":["Agents","Code & Software","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"HarnessDev evaluates LLMs on creating and evolving runnable agent harnesses, shifting evaluation from task outputs to the infrastructure capabilities themselves.","descriptions":{"en":"HarnessDev evaluates LLMs on creating and evolving runnable agent harnesses, shifting evaluation from task outputs to the infrastructure capabilities themselves."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01437","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","agentic-coding"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_harnessdev_06493943"},"modality":null,"name":"HarnessDev","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01437","source_url":"https://arxiv.org/abs/2609.01437"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01437","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01437","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":35,"join":"claire_record_id","repository":null,"source_record":"bm_harnessopt-bench_24067463"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs optimizing a target agent's harness (prompts, tools, control flow) under budgeted, stochastic evaluation. Scoring is normalized gain over seed on a held-out test partition, with trusted execution environment enforcing evaluation boundary.","descriptions":{"en":"Evaluates LLMs optimizing a target agent's harness (prompts, tools, control flow) under budgeted, stochastic evaluation. Scoring is normalized gain over seed on a held-out test partition, with trusted execution environment enforcing evaluation boundary."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06301","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","ai-rnd"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_harnessopt-bench_24067463"},"modality":null,"name":"HarnessOpt-Bench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06301","source_url":"https://arxiv.org/abs/2608.06301"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06301","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06301","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":22,"github_stars_30d":7,"hf_dataset_downloads":96,"hf_paper_upvotes":10,"join":"claire_record_id","repository":"gh:baiyajing/harnessrisk","source_record":"bm_harnessrisk_e838b2b9"},"categories":["Agent safety","Agents","Cybersecurity","Persistent-state safety","Prompt-injection resistance","Safety","Safety & Trustworthiness","Tool-use safety"],"collected_at":"2026-10-14T09:44:51Z","description":"HarnessRisk evaluates agent harness safety across six lifecycle phases with 128 sandboxed cases. It measures Utility, Attack Success Rate, Persistence, and Detection for each trajectory.","descriptions":{"en":"HarnessRisk evaluates agent harness safety across six lifecycle phases with 128 sandboxed cases. It measures Utility, Attack Success Rate, Persistence, and Detection for each trajectory."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.17597","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_harnessrisk_e838b2b9"},"modality":null,"name":"HarnessRisk","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17597","source_url":"https://arxiv.org/abs/2608.17597"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17597","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17597","topics":["agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"HarnessSafe evaluates safety of agent harnesses through 328 executable cases across seven persistent-carrier families, using trace-based evaluation to track attack chains from entry to potential violation.","descriptions":{"en":"HarnessSafe evaluates safety of agent harnesses through 328 executable cases across seven persistent-carrier families, using trace-based evaluation to track attack chains from entry to potential violation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06984","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_harnesssafe_835e5e2c"},"modality":null,"name":"HarnessSafe","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06984","source_url":"https://arxiv.org/abs/2608.06984"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06984","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06984","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Coding agent harnesses mediate tool use and authorize actions, yet their security mechanisms and runtime effects remain incompletely characterized.","descriptions":{"en":"Coding agent harnesses mediate tool use and authorize actions, yet their security mechanisms and runtime effects remain incompletely characterized."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.07639","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-security","agentic-coding"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_harnesssecurity-bench_02d03451"},"modality":null,"name":"HarnessSecurity-Bench","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.07639","source_url":"https://arxiv.org/abs/2610.07639"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-07639","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07639","topics":["coding","agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1446,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:harveyai/harvey-labs","source_record":"catalog_45bbbc773f376558"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["knowledge","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Harvey LAB (Vals) is a professional legal-work evaluation of AI systems on complex law-firm style tasks, reported by Vals.","descriptions":{"en":"Harvey LAB (Vals) is a professional legal-work evaluation of AI systems on complex law-firm style tasks, reported by Vals."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:harvey-lab:grok-4.6","reported_at":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:harvey-lab","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_45bbbc773f376558"},"modality":"text","name":"Harvey LAB (Vals)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":15.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.158,"raw_min":0.158,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:harvey-lab:grok-4.6","reported_date":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"}],"slug":"llm-stats-harvey-lab","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["agentic","legal"],"collected_at":"2026-08-25T10:41:06Z","description":"Legal agentic work, criterion pass rate","descriptions":{"en":"Legal agentic work, criterion pass rate"},"evidence_summary":{"document_count":1,"model_count":36,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:harvey-lab-aa:f0083258-8646-45b8-8082-7aaf6c2ea82a","reported_at":"2025-08-05","source_url":"https://artificialanalysis.ai/evaluations/harvey-lab-aa"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:harvey-lab-aa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["finance-legal"],"join":"review:14H034160212:2026-10-15","primary":"finance-legal","source_record":null},"modality":null,"name":"Harvey LAB-AA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":36,"score_direction":"higher_is_better","score_summary":{"display_max":94.6446663771892,"display_multiplier":100,"model_count":36,"model_count_basis":"source_model_id","numeric_count":36,"raw_max":0.946446663771892,"raw_min":0.138949196699957,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:harvey-lab-aa:f7d2fc3e-1f7b-405f-818c-07952a4af78f","reported_date":"2026-07-16","source_url":"https://artificialanalysis.ai/evaluations/harvey-lab-aa"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"}],"slug":"artificial-analysis-harvey-lab-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/harvey-lab-aa","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1446,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:harveyai/harvey-labs","source_record":"catalog_09556350c5c8a0a0"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["knowledge","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Harvey LAB-AA evaluates model performance on complex legal workflows.","descriptions":{"en":"Harvey LAB-AA evaluates model performance on complex legal workflows."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:harvey-lab-aa:gemini-3.7-flash","reported_at":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab-aa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:harvey-lab-aa","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_09556350c5c8a0a0"},"modality":"text","name":"Harvey LAB-AA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":90.7,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.907,"raw_min":0.907,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:harvey-lab-aa:gemini-3.7-flash","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab-aa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"}],"slug":"llm-stats-harvey-lab-aa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab-aa?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yahyaaalaila/hawkesnest","source_record":"bm_hawkesnest_c85cfbd3"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"HawkesNest is a synthetic benchmark for spatiotemporal point process models, providing controlled generators and complexity ladders across four axes: space-time entanglement, background heterogeneity, cross-type interaction, and domain topology. It includes simulation, export, and visualization tools.","descriptions":{"en":"HawkesNest is a synthetic benchmark for spatiotemporal point process models, providing controlled generators and complexity ladders across four axes: space-time entanglement, background heterogeneity, cross-type interaction, and domain topology. It includes simulation, export, and visualization tools."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.16863","languages":[],"library_direction":{"basis":"model-review-2","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_hawkesnest_c85cfbd3"},"modality":null,"name":"HawkesNest","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16863","source_url":"https://arxiv.org/abs/2606.16863"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16863","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16863","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multilingual medical LLMs on 1,000 examples per language across nine languages, covering MCQA, NLI, and open-ended QA tasks.","descriptions":{"en":"Evaluates multilingual medical LLMs on 1,000 examples per language across nine languages, covering MCQA, NLI, and open-ended QA tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.19981","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","multilingual"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_healmed_cb6026bf"},"modality":null,"name":"HealMed","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.19981","source_url":"https://arxiv.org/abs/2608.19981"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-19981","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19981","topics":["language","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":58,"github_stars_30d":8,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:microsoft/healthagentbench","source_record":"bm_healthagentbench_400912dd"},"categories":["Agents & Tool Use","Biology & Drug Discovery","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents on 54 realistic healthcare tasks across 7 categories, including medical imaging, EHR analysis, and clinical trial matching. Agents operate in terminal environments with task-specific verifiers and a final task success rate.","descriptions":{"en":"Evaluates AI agents on 54 realistic healthcare tasks across 7 categories, including medical imaging, EHR analysis, and clinical trial matching. Agents operate in terminal environments with task-specific verifiers and a final task success rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.31179","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","tool-use"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_healthagentbench_400912dd"},"modality":null,"name":"HealthAgentBench","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31179","source_url":"https://arxiv.org/abs/2606.31179"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31179","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31179","topics":["agents","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:openai/healthbench","source_record":"lib_healthbench"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"An open-source benchmark for measuring performance and safety of large language models in healthcare, consisting of 5,000 multi-turn conversations evaluated by 262 physicians using 48,562 unique rubric criteria across health contexts and behavioral dimensions","descriptions":{"en":"An open-source benchmark for measuring performance and safety of large language models in healthcare, consisting of 5,000 multi-turn conversations evaluated by 262 physicians using 48,562 unique rubric criteria across health contexts and behavioral dimensions"},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:healthbench:gpt-oss-120b","reported_at":"2025-08-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:healthbench","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"llm_stats_id","primary":"health-medicine","source_record":"lib_healthbench"},"modality":"text","name":"HealthBench","openness":"unknown","publisher":null,"released":"2025-05-12","released_reference":{"basis":"release_announcement","note":"The dated May 12 announcement introduces HealthBench and its Hard and Consensus subsets; prefer this explicit release to the May 13 preprint.","source_key":"llm-stats:healthbench","source_url":"https://openai.com/index/healthbench/"},"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":60.199999999999996,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.602,"raw_min":0.425,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:healthbench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 20B"},{"date":"2026-03-04","date_precision":"model_announcement","name":"GPT-5.3 Chat"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-healthbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench?top_n=500","topics":["medical"],"unit":null},{"aliases":["HealthBench","HealthBench Hard","HealthBench Consensus"],"attention_signals":null,"categories":["health"],"collected_at":null,"description":"Physician-written rubrics scored by an LLM grader, and the length-adjusted variant is what recent cards report. Published by OpenAI.","descriptions":{"en":"Physician-written rubrics scored by an LLM grader, and the length-adjusted variant is what recent cards report. Published by OpenAI."},"evidence_summary":{"document_count":3,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000healthbench\u0000bytedance_seed2_0_model_card\u0000healthbench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000healthbench\u0000bytedance_seed2_0_model_card\u0000healthbench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:healthbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["health-medicine"],"join":"review:14H034160212:2026-10-15","primary":"health-medicine","source_record":null},"modality":null,"name":"HealthBench","openness":"unknown","publisher":null,"released":"2025-05-12","released_reference":{"basis":"benchmark_release","source_key":"model-reports:healthbench","source_url":"https://openai.com/index/healthbench/"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":57.7,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":57.7,"raw_min":46.2,"source_reference":{"obs_id":"curated\u0000healthbench\u0000bytedance_seed2_0_model_card\u0000healthbench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000healthbench\u0000bytedance_seed2_0_model_card\u0000healthbench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"healthbench","source":"model_reports","source_url":"https://openai.com/index/healthbench/","topics":["medical"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4665,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:openai/simple-evals","source_record":"catalog_bd8fc219b7dfc4b0"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"HealthBench Consensus is a HealthBench subset focused on questions where physician-created rubric criteria have especially high agreement, measuring healthcare performance and safety on consensus-evaluable conversations.","descriptions":{"en":"HealthBench Consensus is a HealthBench subset focused on questions where physician-created rubric criteria have especially high agreement, measuring healthcare performance and safety on consensus-evaluable conversations."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:healthbench-consensus:gpt-5.5-instant","reported_at":"2026-05-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-consensus?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:healthbench-consensus","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_bd8fc219b7dfc4b0"},"modality":"text","name":"HealthBench Consensus","openness":"unknown","publisher":null,"released":"2025-05-12","released_reference":{"basis":"release_announcement","note":"The dated May 12 announcement introduces HealthBench and its Hard and Consensus subsets; prefer this explicit release to the May 13 preprint.","source_key":"llm-stats:healthbench-consensus","source_url":"https://openai.com/index/healthbench/"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":95.5,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.955,"raw_min":0.947,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:healthbench-consensus:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-consensus?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-healthbench-consensus","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-consensus?top_n=500","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:openai/healthbench","source_record":"lib_healthbench_hard"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A challenging variation of HealthBench that evaluates large language models' performance and safety in healthcare through 5,000 multi-turn conversations with particularly rigorous evaluation criteria validated by 262 physicians from 60 countries","descriptions":{"en":"A challenging variation of HealthBench that evaluates large language models' performance and safety in healthcare through 5,000 multi-turn conversations with particularly rigorous evaluation criteria validated by 262 physicians from 60 countries"},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:healthbench-hard:gpt-oss-120b","reported_at":"2025-08-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-hard?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:healthbench-hard","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"llm_stats_id","primary":"health-medicine","source_record":"lib_healthbench_hard"},"modality":"text","name":"HealthBench Hard","openness":"unknown","publisher":null,"released":"2025-05-12","released_reference":{"basis":"release_announcement","note":"The dated May 12 announcement introduces HealthBench and its Hard and Consensus subsets; prefer this explicit release to the May 13 preprint.","source_key":"llm-stats:healthbench-hard","source_url":"https://openai.com/index/healthbench/"},"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":42.8,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.428,"raw_min":0.016,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:healthbench-hard:muse-spark","reported_date":"2026-04-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-hard?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 20B"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2026-03-04","date_precision":"model_announcement","name":"GPT-5.3 Chat"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"}],"slug":"llm-stats-healthbench-hard","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-hard?top_n=500","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":5736,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"lib_healthbench_professional"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"HealthBench Professional evaluates model capability and safety for clinician use cases using real clinician-style chats and physician-authored grading rubrics.","descriptions":{"en":"HealthBench Professional evaluates model capability and safety for clinician use cases using real clinician-style chats and physician-authored grading rubrics."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:healthbench-professional:gpt-5.5-instant","reported_at":"2026-05-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-professional?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:healthbench-professional","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"llm_stats_id","primary":"health-medicine","source_record":"lib_healthbench_professional"},"modality":"text","name":"HealthBench Professional","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":66.0,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.66,"raw_min":0.35,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:healthbench-professional:claude-fable-5","reported_date":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-professional?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"}],"slug":"llm-stats-healthbench-professional","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-professional?top_n=500","topics":["medical"],"unit":null},{"aliases":["HealthBench Professional"],"attention_signals":null,"categories":["health"],"collected_at":null,"description":"A distinct, harder split from the HealthBench variants recorded separately in this registry, so its scores are not comparable to them. Published as an OpenAI paper (525 examples, 3 clinician use cases) with a public dataset; released 2026-04-22 per PDF metadata. Graded by an LLM against physician-written rubrics.","descriptions":{"en":"A distinct, harder split from the HealthBench variants recorded separately in this registry, so its scores are not comparable to them. Published as an OpenAI paper (525 examples, 3 clinician use cases) with a public dataset; released 2026-04-22 per PDF metadata. Graded by an LLM against physician-written rubrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:healthbench_professional","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["health-medicine"],"join":"review:14H034160212:2026-10-15","primary":"health-medicine","source_record":null},"modality":null,"name":"HealthBench Professional","openness":"unknown","publisher":null,"released":"2026-04-22","released_reference":{"basis":"benchmark_release","source_key":"model-reports:healthbench_professional","source_url":"https://cdn.openai.com/dd128428-0184-4e25-b155-3a7686c7d744/HealthBench-Professional.pdf"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"healthbench_professional","source":"model_reports","source_url":"https://cdn.openai.com/dd128428-0184-4e25-b155-3a7686c7d744/HealthBench-Professional.pdf","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":266,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:mindbench-ai/healthbench-psych","source_record":"bm_healthbench-psych_83387771"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on 610 mental-health conversations from HealthBench using a three-judge panel and physician rubrics, with a harder 119-conversation subset.","descriptions":{"en":"Evaluates LLMs on 610 mental-health conversations from HealthBench using a three-judge panel and physician rubrics, with a harder 119-conversation subset."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.25071","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_healthbench-psych_83387771"},"modality":null,"name":"HealthBench-Psych","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25071","source_url":"https://arxiv.org/abs/2608.25071"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25071","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25071","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"HealthLoopQA evaluates LLM reasoning over continuous diabetes monitoring data through 127 tasks and 1,500 QA instances covering process mining, anomaly detection, and prediction over 30-day horizons, with a fault-injected simulation testbed for safety scenarios.","descriptions":{"en":"HealthLoopQA evaluates LLM reasoning over continuous diabetes monitoring data through 127 tasks and 1,500 QA instances covering process mining, anomaly detection, and prediction over 30-day horizons, with a fault-injected simulation testbed for safety scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.06976","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_healthloopqa_7ef7035f"},"modality":null,"name":"HealthLoopQA","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06976","source_url":"https://arxiv.org/abs/2609.06976"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06976","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06976","topics":["reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:oppo-mente-lab/hearincontext","source_record":"bm_hearincontext_cb37359f"},"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for implicit context in speech recognition, pairing Mandarin-English synthetic speech with assistant replies that support varying interpretations, to test models' handling of contextual ambiguity.","descriptions":{"en":"A benchmark for implicit context in speech recognition, pairing Mandarin-English synthetic speech with assistant replies that support varying interpretations, to test models' handling of contextual ambiguity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.18680","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_hearincontext_cb37359f"},"modality":null,"name":"HearInContext","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18680","source_url":"https://arxiv.org/abs/2609.18680"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18680","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18680","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:peng-weihan/heart-bench","source_record":"bm_heart-bench_230e27fe"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"HEART-Bench evaluates whether LLM agents can simulate coherent, human-like psychology. It provides 11 fictional characters with raw episodic memories, 64 decision-making scenarios based on the DIAMONDS taxonomy, and 673 multiple-choice questions with expert-annotated ground truth. Two evaluation tracks: MCQ and open-ended consciousness-narrative.","descriptions":{"en":"HEART-Bench evaluates whether LLM agents can simulate coherent, human-like psychology. It provides 11 fictional characters with raw episodic memories, 64 decision-making scenarios based on the DIAMONDS taxonomy, and 673 multiple-choice questions with expert-annotated ground truth. Two evaluation tracks: MCQ and open-ended consciousness-narrative."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.30058","languages":[],"library_direction":null,"modality":null,"name":"HEART-Bench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30058","source_url":"https://arxiv.org/abs/2605.30058"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30058","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30058","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"HeatCast is a benchmark for monthly Land Surface Temperature forecasting at 30m resolution across 124 U.S. cities. It provides Landsat-based tiles, fixed temporal split, LCZ-stratified metrics, and a reference evaluation harness.","descriptions":{"en":"HeatCast is a benchmark for monthly Land Surface Temperature forecasting at 30m resolution across 124 U.S. cities. It provides Landsat-based tiles, fixed temporal split, LCZ-stratified metrics, and a reference evaluation harness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07640","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_heatcast_ee7ee561"},"modality":null,"name":"HeatCast","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07640","source_url":"https://arxiv.org/abs/2608.07640"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07640","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07640","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":101,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":"gh:trata-inc/trata-hedge-bench","source_record":"bm_hedge-bench_8e798fcd"},"categories":["Finance","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Hedge-Bench 1.0 evaluates agents on 102 financial reasoning tasks with deterministic grading against expert reasoning traces.","descriptions":{"en":"Hedge-Bench 1.0 evaluates agents on 102 financial reasoning tasks with deterministic grading against expert reasoning traces."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.03918","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_hedge-bench_8e798fcd"},"modality":null,"name":"Hedge-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03918","source_url":"https://arxiv.org/abs/2606.03918"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03918","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03918","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Materials & Chemistry","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"HEDGEHOG is a six-stage filtration benchmark for evaluating molecular generators on medicinal plausibility, including physicochemical, structural, synthesis, docking, and 3D pose checks, applied to 23 generators and 230,000 molecules.","descriptions":{"en":"HEDGEHOG is a six-stage filtration benchmark for evaluating molecular generators on medicinal plausibility, including physicochemical, structural, synthesis, docking, and 3D pose checks, applied to 23 generators and 230,000 molecules."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.13155","languages":[],"library_direction":{"basis":"model-review","categories":["science","health-medicine"],"join":"claire_record_id","primary":"science","source_record":"bm_hedgehog_997bad40"},"modality":null,"name":"HEDGEHOG","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.13155","source_url":"https://arxiv.org/abs/2607.13155"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-13155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13155","topics":["medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":41,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"llm_stats_id","repository":null,"source_record":"catalog_99665bad41328c17"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A challenging commonsense natural language inference dataset that uses Adversarial Filtering to create questions trivial for humans (>95% accuracy) but difficult for state-of-the-art models, requiring completion of sentence endings based on physical situations and everyday activities","descriptions":{"en":"A challenging commonsense natural language inference dataset that uses Adversarial Filtering to create questions trivial for humans (>95% accuracy) but difficult for state-of-the-art models, requiring completion of sentence endings based on physical situations and everyday activities"},"evidence_summary":{"document_count":1,"model_count":27,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hellaswag:gpt-4-0613","reported_at":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hellaswag?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:hellaswag","languages":[],"library_direction":{"basis":"model-review-2","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_99665bad41328c17"},"modality":"text","name":"HellaSwag","openness":"restricted","publisher":null,"released":"2019-05-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":27,"score_direction":"higher_is_better","score_summary":{"display_max":95.39999999999999,"display_multiplier":100,"model_count":27,"model_count_basis":"source_model_id","numeric_count":27,"raw_max":0.954,"raw_min":0.33,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hellaswag:claude-3-opus-20240229","reported_date":"2024-02-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hellaswag?top_n=500"},"unit":null},"scored_models":[{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-08-30","date_precision":"model_announcement","name":"Command R+"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2024-07-18","date_precision":"model_announcement","name":"Mistral NeMo Instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-hellaswag","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hellaswag?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":41,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"artifact:arxiv:1905.07830","repository":null,"source_record":"catalog_99665bad41328c17"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HellaSwag is a challenge dataset for evaluating commonsense natural language inference, which is specially hard for state-of-the-art models, though its questions are trivial for humans (>95% accuracy). It consists of 70k multiple choice questions, each with a scenario and four possible endings, which requires to select the most reasonable ending. These questions come from two domains:activitynet and wikihow, involving video and text scenarios respectively. The correct answers of these questions are the real sentences for the next event, while the incorrect answers are adversarially generated and human verified, so as to fool machines but not humans. HellaSwag 是一个用于评估常识性自然语言推理的数据集，HellaSwag的问题对于最先进的模型来说是特别困难的，尽管它的问题对于人类来说非常轻松就能回答的（> 95% 的准确率）。它由7万多道多项选择题组成，每道题都有一个场景和四种可能的答案，需要选择最合理的答案。这些问题来自两个领域：activitynet和wikihow，分别涉及视频和文本场景。这些问题的正确答案是下一个事件的真实句子，而错误答案是通过对抗技术生成的并经过人类验证，这些答案可以欺骗机器但不能欺骗人类。","descriptions":{"en":"HellaSwag is a challenge dataset for evaluating commonsense natural language inference, which is specially hard for state-of-the-art models, though its questions are trivial for humans (>95% accuracy). It consists of 70k multiple choice questions, each with a scenario and four possible endings, which requires to select the most reasonable ending. These questions come from two domains:activitynet and wikihow, involving video and text scenarios respectively. The correct answers of these questions are the real sentences for the next event, while the incorrect answers are adversarially generated and human verified, so as to fool machines but not humans.","zh":"HellaSwag 是一个用于评估常识性自然语言推理的数据集，HellaSwag的问题对于最先进的模型来说是特别困难的，尽管它的问题对于人类来说非常轻松就能回答的（> 95% 的准确率）。它由7万多道多项选择题组成，每道题都有一个场景和四种可能的答案，需要选择最合理的答案。这些问题来自两个领域：activitynet和wikihow，分别涉及视频和文本场景。这些问题的正确答案是下一个事件的真实句子，而错误答案是通过对抗技术生成的并经过人类验证，这些答案可以欺骗机器但不能欺骗人类。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:531","languages":[],"library_direction":{"basis":"model-review-2","categories":["abstract-reasoning"],"join":"artifact:arxiv:1905.07830","primary":"abstract-reasoning","source_record":"catalog_99665bad41328c17"},"modality":null,"name":"HellaSwag","openness":"restricted","publisher":null,"released":"2019-05-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-531-hellaswag","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HellaSwag","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["长文本","Long-Context","大语言模型","LLM","语言生成","Generation","长上下文","Long Context","官方自建","Official"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HelloBench is a hierarchical long text generation benchmark to evaluate LLMs' performance in generating long text.  Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation. HelloBench为长文本生成基准，这是一个全面的、开放式的基准，用于评估LLM在生成长文本方面的性能。基于Bloom的分类法，HelloBench将长文本生成任务分为五个子任务：开放式QA、摘要、聊天、文本完成和启发式文本生成。","descriptions":{"en":"HelloBench is a hierarchical long text generation benchmark to evaluate LLMs' performance in generating long text.  Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation.","zh":"HelloBench为长文本生成基准，这是一个全面的、开放式的基准，用于评估LLM在生成长文本方面的性能。基于Bloom的分类法，HelloBench将长文本生成任务分为五个子任务：开放式QA、摘要、聊天、文本完成和启发式文本生成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1097","languages":[],"library_direction":null,"modality":null,"name":"HelloBench","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-09-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1097-hellobench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HelloBench","topics":["long-context","language"],"unit":null},{"aliases":["Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains"],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"claire_record_id","repository":null,"source_record":"bm_teaching-nemotron-greek-mining-a-corpus-ad_afa07bfd"},"categories":["General AI","Information retrieval","Language & Knowledge","eess.AS"],"collected_at":"2026-10-14T09:44:51Z","description":"HERA evaluates retrieval-augmented generation for Modern Greek across legal, energy, financial, and medical specialist domains using mined corpora and expert-judged answers.","descriptions":{"en":"HERA evaluates retrieval-augmented generation for Modern Greek across legal, energy, financial, and medical specialist domains using mined corpora and expert-judged answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05138","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","multilingual"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_teaching-nemotron-greek-mining-a-corpus-ad_afa07bfd"},"modality":null,"name":"HERA","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05138","source_url":"https://arxiv.org/abs/2608.05138"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05138","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05138","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_herabench_8a4ad4a5"},"categories":["Agents","Agents & Tool Use","Mobile & Personal Computing"],"collected_at":"2026-10-14T09:44:51Z","description":"HeraBench is a fault-injected benchmark for multi-device agent workflows on Linux and Android, evaluating hierarchical replanning under injected strategy- and device-level failures.","descriptions":{"en":"HeraBench is a fault-injected benchmark for multi-device agent workflows on Linux and Android, evaluating hierarchical replanning under injected strategy- and device-level failures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.20487","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","tool-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_herabench_8a4ad4a5"},"modality":null,"name":"HeraBench","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20487","source_url":"https://arxiv.org/abs/2606.20487"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20487","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20487","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","RAG","大语言模型","LLM","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HERB comprises 39,190 artifacts—including documents, meeting transcripts, Slack messages, GitHub content, and URLs—simulating business workflows across product planning, development, and support stages, with noisy, multi-hop QA tasks featuring both answerable and unanswerable queries. HERB基准包含39,190个企业文档、会议记录、Slack消息、GitHub内容和网页链接，模拟产品规划、开发与支持等业务流程，生成包含噪声的多跳问答任务，涵盖可回答与不可回答查询。","descriptions":{"en":"HERB comprises 39,190 artifacts—including documents, meeting transcripts, Slack messages, GitHub content, and URLs—simulating business workflows across product planning, development, and support stages, with noisy, multi-hop QA tasks featuring both answerable and unanswerable queries.","zh":"HERB基准包含39,190个企业文档、会议记录、Slack消息、GitHub内容和网页链接，模拟产品规划、开发与支持等业务流程，生成包含噪声的多跳问答任务，涵盖可回答与不可回答查询。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:2031","languages":[],"library_direction":null,"modality":null,"name":"HERB","openness":"unknown","publisher":"Salesforce AI Research","released":"2025-06-29","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2031-herb","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HERB","topics":[],"unit":null},{"aliases":["hermes-code-agent-benchmark"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:flappearlabs/hermes-code-agent-benchmark","source_record":"bm_hermes-code-agent-benchmark_52e255be"},"categories":["Agents","Code","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates software engineering task completion across 20 selected SWE-bench Verified, SWE-bench Pro V2, and Terminal-Bench instances using official verifiers.","descriptions":{"en":"Evaluates software engineering task completion across 20 selected SWE-bench Verified, SWE-bench Pro V2, and Terminal-Bench instances using official verifiers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:flappearlabs/hermes-code-agent-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_hermes-code-agent-benchmark_52e255be"},"modality":null,"name":"Hermes Code Agent Benchmark","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:flappearlabs/hermes-code-agent-benchmark","source_url":"https://github.com/FlapPearLabs/hermes-code-agent-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-flappearlabs-hermes-code-agent-benchmark","source":"claire_radar","source_url":"https://github.com/FlapPearLabs/hermes-code-agent-benchmark","topics":["coding","agents"],"unit":null},{"aliases":["HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"HERO is a heterogeneity-aware benchmark library for federated continual learning that separates task splits, client data splits, and client task sequences.","descriptions":{"en":"HERO is a heterogeneity-aware benchmark library for federated continual learning that separates task splits, client data splits, and client task sequences."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.08784","languages":[],"library_direction":null,"modality":null,"name":"HERO","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08784","source_url":"https://arxiv.org/abs/2607.08784"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08784","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08784","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Tests rule induction in goal-directed episodic tasks through text games, covering eight tasks across attribute and procedural induction families with controllable lexical grounding and identifiability conditions.","descriptions":{"en":"Tests rule induction in goal-directed episodic tasks through text games, covering eight tasks across attribute and procedural induction families with controllable lexical grounding and identifiability conditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02556","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","multi-agent-games"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_hero-s-journey_260791d3"},"modality":null,"name":"HERO'S JOURNEY","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02556","source_url":"https://arxiv.org/abs/2606.02556"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02556","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02556","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"HG-Bench evaluates page-aware, two-level answer-region grounding: given multi-page handwritten homework images, models must localize complete answer regions and ordered step-level subregions, with question- and step-level boxes under a hierarchical constraint.","descriptions":{"en":"HG-Bench evaluates page-aware, two-level answer-region grounding: given multi-page handwritten homework images, models must localize complete answer regions and ordered step-level subregions, with question- and step-level boxes under a hierarchical constraint."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.25491","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","visual-reasoning"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_hg-bench_1e72857e"},"modality":null,"name":"HG-Bench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25491","source_url":"https://arxiv.org/abs/2606.25491"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25491","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25491","topics":[],"unit":null},{"aliases":["Escaping Python Dependency Hell: A Hybrid Replay-and-Repair Pipeline for Python Dependency Resolution"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates a Python dependency resolution pipeline on dependency-failing snippets.","descriptions":{"en":"Evaluates a Python dependency resolution pipeline on dependency-failing snippets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.26952","languages":[],"library_direction":null,"modality":null,"name":"HG2.9K","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.26952","source_url":"https://arxiv.org/abs/2609.26952"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-26952","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26952","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Google DeepMind's internal mathematical reasoning benchmark that introduces novel problems not encountered during model training to evaluate true mathematical reasoning capabilities rather than memorization","descriptions":{"en":"Google DeepMind's internal mathematical reasoning benchmark that introduces novel problems not encountered during model training to evaluate true mathematical reasoning capabilities rather than memorization"},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hiddenmath:gemini-1.5-flash-8b","reported_at":"2024-03-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hiddenmath?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:hiddenmath","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_e065ced955caaca5"},"modality":"text","name":"HiddenMath","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":63.0,"display_multiplier":100,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":0.63,"raw_min":0.158,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hiddenmath:gemini-2.0-flash","reported_date":"2024-12-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hiddenmath?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"}],"slug":"llm-stats-hiddenmath","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hiddenmath?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification"],"attention_signals":null,"categories":["Chip Design & EDA","Language & Knowledge","cs.AR"],"collected_at":"2026-09-25T04:40:22Z","description":"HierSVA-B evaluates LLM-generated SystemVerilog assertions for hierarchical hardware formal verification across six quality axes.","descriptions":{"en":"HierSVA-B evaluates LLM-generated SystemVerilog assertions for hierarchical hardware formal verification across six quality axes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.13706","languages":[],"library_direction":null,"modality":null,"name":"HierSVA-B","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13706","source_url":"https://arxiv.org/abs/2606.13706"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13706","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13706","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Chip Design & EDA","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"HighTide is an AI-assisted VLSI benchmark suite with open-source designs, Bazel-based compilation, and agent skills for design curation.","descriptions":{"en":"HighTide is an AI-assisted VLSI benchmark suite with open-source designs, Bazel-based compilation, and agent skills for design curation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04126","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_hightide_5b4de3e6"},"modality":null,"name":"HighTide","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04126","source_url":"https://arxiv.org/abs/2606.04126"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04126","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04126","topics":["agents"],"unit":null},{"aliases":["Beam-Wise Statistical Background Subtraction for Static Roadside LiDAR: A Cross-Sensor Benchmark Study"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_beam-wise-statistical-background-subtracti_69023b31"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"HighwayScene is a multi-LiDAR dataset recorded in a static roadside setup for evaluating background subtraction, with public annotations and implementations released.","descriptions":{"en":"HighwayScene is a multi-LiDAR dataset recorded in a static roadside setup for evaluating background subtraction, with public annotations and implementations released."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14868","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_beam-wise-statistical-background-subtracti_69023b31"},"modality":null,"name":"HighwayScene","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14868","source_url":"https://arxiv.org/abs/2608.14868"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14868","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14868","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:freedomintelligence/himed","source_record":"bm_himed_f6226988"},"categories":["Biology & Drug Discovery","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"HiMed is a Hindi medical dataset and benchmark suite covering Western and Indian medicine. It evaluates medical reasoning in Hindi across multiple tasks, with data and evaluation scripts publicly released on GitHub.","descriptions":{"en":"HiMed is a Hindi medical dataset and benchmark suite covering Western and Indian medicine. It evaluates medical reasoning in Hindi across multiple tasks, with data and evaluation scripts publicly released on GitHub."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.24635","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","multilingual"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_himed_f6226988"},"modality":null,"name":"HiMed","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24635","source_url":"https://arxiv.org/abs/2605.24635"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24635","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24635","topics":["reasoning-math","language","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Robotics & Embodied AI","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"A black-box audit protocol that measures parametric hindsight leakage in time-indexed LLM decision tasks. It uses date manipulations, memory probes, and six defined metrics, and the paper applies it to 15 models.","descriptions":{"en":"A black-box audit protocol that measures parametric hindsight leakage in time-indexed LLM decision tasks. It uses date manipulations, memory probes, and six defined metrics, and the paper applies it to 15 models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.18867","languages":[],"library_direction":null,"modality":null,"name":"HindsightBench","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18867","source_url":"https://arxiv.org/abs/2607.18867"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18867","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18867","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":26,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":32,"join":"llm_stats_id","repository":null,"source_record":"catalog_c2c4f815df1512b3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","science","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"HiPhO is a high-school physics olympiad benchmark evaluating multimodal reasoning over physics problems that include diagrams and figures.","descriptions":{"en":"HiPhO is a high-school physics olympiad benchmark evaluating multimodal reasoning over physics problems that include diagrams and figures."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hipho:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hipho?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:hipho","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"llm_stats_id","primary":"science","source_record":"catalog_c2c4f815df1512b3"},"modality":"multimodal","name":"HiPhO","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":84.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.841,"raw_min":0.841,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hipho:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hipho?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-hipho","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hipho?top_n=500","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":16,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"llm_stats_id","repository":null,"source_record":"catalog_e90605fef1281a39"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","knowledge"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"HLE-Verified evaluates multidisciplinary expert reasoning on a verified subset of Humanity's Last Exam.","descriptions":{"en":"HLE-Verified evaluates multidisciplinary expert reasoning on a verified subset of Humanity's Last Exam."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hle-verified:gemini-3.7-flash","reported_at":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hle-verified?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:hle-verified","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_e90605fef1281a39"},"modality":"text","name":"HLE-Verified","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":53.6,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.536,"raw_min":0.536,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hle-verified:gemini-3.7-flash","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hle-verified?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"}],"slug":"llm-stats-hle-verified","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hle-verified?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:xinhaos0101/hll","source_record":"bm_hll_52c829e2"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"HLL is a benchmark that evaluates multimodal agents on interactive CAPTCHA verification in a closed-loop GUI environment, covering diverse task types such as text transcription, image selection, sliders, jigsaw puzzles, and logic-based challenges. Scoring is based on task completion and trace-conditioned validation.","descriptions":{"en":"HLL is a benchmark that evaluates multimodal agents on interactive CAPTCHA verification in a closed-loop GUI environment, covering diverse task types such as text transcription, image selection, sliders, jigsaw puzzles, and logic-based challenges. Scoring is based on task completion and trace-conditioned validation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.02449","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_hll_52c829e2"},"modality":null,"name":"HLL","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02449","source_url":"https://arxiv.org/abs/2606.02449"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02449","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02449","topics":["computer-use","agents"],"unit":null},{"aliases":["HMMT","HMMT Feb 25","HMMT Nov 25","HMMT 2026 Feb"],"attention_signals":null,"categories":["math"],"collected_at":null,"description":"A dated competition, not a fixed set: each sitting is a different instrument and contamination rises once problems are published.","descriptions":{"en":"A dated competition, not a fixed set: each sitting is a different instrument and contamination rises once problems are published."},"evidence_summary":{"document_count":7,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000hmmt\u0000zai_glm_5_model_card\u0000hmmt\u0000HMMT Nov. 2025, temp 1.0, top_p 0.95, GPT-5.2 (medium) judge\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"first_score_reported_at":"2026-02-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000hmmt\u0000zai_glm_5_model_card\u0000hmmt\u0000HMMT Nov. 2025, temp 1.0, top_p 0.95, GPT-5.2 (medium) judge\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:hmmt","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["math"],"join":"review:14H034160212:2026-10-15","primary":"math","source_record":null},"modality":null,"name":"HMMT","openness":"unknown","publisher":null,"released":"2025-02-15","released_reference":{"basis":"benchmark_release","source_key":"model-reports:hmmt","source_url":"https://www.hmmt.org/"},"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":97.3,"display_multiplier":1,"model_count":9,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":97.3,"raw_min":77.8,"source_reference":{"obs_id":"curated\u0000hmmt\u0000bytedance_seed2_0_model_card\u0000hmmt\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000hmmt\u0000bytedance_seed2_0_model_card\u0000hmmt\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"hmmt","source":"model_reports","source_url":"https://www.hmmt.org/","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eth-sri/matharena","source_record":"catalog_897457a59168a17d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds","descriptions":{"en":"Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds"},"evidence_summary":{"document_count":1,"model_count":33,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hmmt-2025:deepseek-v3.1","reported_at":"2025-01-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-2025?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:hmmt-2025","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_897457a59168a17d"},"modality":"text","name":"HMMT 2025","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":33,"score_direction":"higher_is_better","score_summary":{"display_max":100.0,"display_multiplier":100,"model_count":33,"model_count_basis":"source_model_id","numeric_count":33,"raw_max":1.0,"raw_min":0.289,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hmmt-2025:gpt-5.2-pro-2025-12-11","reported_date":"2025-12-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-2025?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2-Speciale"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Pro"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok 4 Fast"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"}],"slug":"llm-stats-hmmt-2025","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-2025?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eth-sri/matharena","source_record":"catalog_c16edb9a6159b2b6"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"HMMT February 2026 is a math competition benchmark based on problems from the Harvard-MIT Mathematics Tournament, testing advanced mathematical problem-solving and reasoning.","descriptions":{"en":"HMMT February 2026 is a math competition benchmark based on problems from the Harvard-MIT Mathematics Tournament, testing advanced mathematical problem-solving and reasoning."},"evidence_summary":{"document_count":1,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hmmt-feb-26:qwen3.6-plus","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-feb-26?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:hmmt-feb-26","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_c16edb9a6159b2b6"},"modality":"text","name":"HMMT Feb 26","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":12,"score_direction":"higher_is_better","score_summary":{"display_max":97.1,"display_multiplier":100,"model_count":12,"model_count_basis":"source_model_id","numeric_count":12,"raw_max":0.971,"raw_min":0.826,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hmmt-feb-26:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-feb-26?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-hmmt-feb-26","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-feb-26?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds","descriptions":{"en":"Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds"},"evidence_summary":{"document_count":1,"model_count":25,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hmmt25:grok-4","reported_at":"2025-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt25?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:hmmt25","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_123166a3ea230751"},"modality":"text","name":"HMMT25","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"not_found","repo_resolution_status":"needs_review","score_count":25,"score_direction":"higher_is_better","score_summary":{"display_max":96.7,"display_multiplier":100,"model_count":25,"model_count_basis":"source_model_id","numeric_count":25,"raw_max":0.967,"raw_min":0.307,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hmmt25:grok-4-heavy","reported_date":"2025-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt25?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4 Heavy"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"}],"slug":"llm-stats-hmmt25","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt25?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions.","descriptions":{"en":"Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.01605","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_hob-vl_eb581148"},"modality":null,"name":"Hob-VL","openness":"unknown","publisher":null,"released":"2026-10-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.01605","source_url":"https://arxiv.org/abs/2610.01605"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-01605","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01605","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":10,"github_stars_30d":4,"hf_dataset_downloads":81,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:weareaisle/hof-bench","source_record":"bm_hof-bench_be2a2445"},"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vulnerability discovery in source code using 95 real CVEs across 8 repositories pinned at vulnerable commits, with a detector-blinded judge crediting findings that match code path, root cause, attack condition, and impact.","descriptions":{"en":"Evaluates vulnerability discovery in source code using 95 real CVEs across 8 repositories pinned at vulnerable commits, with a detector-blinded judge crediting findings that match code path, root cause, attack condition, and impact."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27030","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_hof-bench_be2a2445"},"modality":null,"name":"HoF-Bench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27030","source_url":"https://arxiv.org/abs/2607.27030"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27030","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27030","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DC"],"collected_at":"2026-10-14T09:44:51Z","description":"A modular benchmarking and deployment toolkit that jointly characterizes accuracy, latency and energy for foundation models across platforms from single-board computers to GPU servers in CPS-IoT applications.","descriptions":{"en":"A modular benchmarking and deployment toolkit that jointly characterizes accuracy, latency and energy for foundation models across platforms from single-board computers to GPU servers in CPS-IoT applications."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.12412","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_holibench_6d2d4ecd"},"modality":null,"name":"HoliBench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12412","source_url":"https://arxiv.org/abs/2609.12412"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12412","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12412","topics":[],"unit":null},{"aliases":["HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs"],"attention_signals":null,"categories":["Finance","Language & Knowledge","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"HOLMES evaluates higher-order symbolic reasoning in language models using 1,379 natural-language problems paired with formalizations, answers, and verifiable reasoning traces across law and finance.","descriptions":{"en":"HOLMES evaluates higher-order symbolic reasoning in language models using 1,379 natural-language problems paired with formalizations, answers, and verifiable reasoning traces across law and finance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.23238","languages":[],"library_direction":null,"modality":null,"name":"HOLMES","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23238","source_url":"https://arxiv.org/abs/2606.23238"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23238","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23238","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","长文本","Long-Context","大语言模型","LLM","逻辑推理","长上下文","Long Context"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HoloBench is a benchmark designed to evaluate the ability of long-context language models (LCLMs) to perform holistic reasoning over extended text contexts. HoloBench 是一个用于评估长上下文语言模型（LCLMs）在扩展文本上下文中进行整体推理能力的基准测试。","descriptions":{"en":"HoloBench is a benchmark designed to evaluate the ability of long-context language models (LCLMs) to perform holistic reasoning over extended text contexts.","zh":"HoloBench 是一个用于评估长上下文语言模型（LCLMs）在扩展文本上下文中进行整体推理能力的基准测试。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1581","languages":[],"library_direction":null,"modality":null,"name":"HoloBench","openness":"unknown","publisher":"Megagon Labs","released":"2024-10-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1581-holobench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HoloBench","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:mm-mvr/holocount","source_record":"bm_holocount_b6a34c73"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"HoloCount evaluates multimodal large language models on visual counting tasks across three levels: semantic counting (atomic and property-based enumeration), analytical counting (logical composition via spatial and set-based reasoning), and robustness testing (adverse scenarios and grounded counter-priors). The benchmark uses a hierarchical taxonomy and provides a dataset for evaluation.","descriptions":{"en":"HoloCount evaluates multimodal large language models on visual counting tasks across three levels: semantic counting (atomic and property-based enumeration), analytical counting (logical composition via spatial and set-based reasoning), and robustness testing (adverse scenarios and grounded counter-priors). The benchmark uses a hierarchical taxonomy and provides a dataset for evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.06420","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_holocount_b6a34c73"},"modality":null,"name":"HoloCount","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06420","source_url":"https://arxiv.org/abs/2607.06420"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06420","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06420","topics":[],"unit":null},{"aliases":["HoloFair: Unified T2I Fairness Evaluation and Fair-GRPO Debiasing"],"attention_signals":null,"categories":["General AI","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"HoloFair evaluates multidimensional demographic bias in text-to-image models using a fairness-oriented dataset, an attribute classifier, and the MGBI metric.","descriptions":{"en":"HoloFair evaluates multidimensional demographic bias in text-to-image models using a fairness-oriented dataset, an attribute classifier, and the MGBI metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.24687","languages":[],"library_direction":null,"modality":null,"name":"HoloFair","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24687","source_url":"https://arxiv.org/abs/2605.24687"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24687","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24687","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.HC"],"collected_at":"2026-10-14T09:44:51Z","description":"HoosierHelp is an interactive benchmark for evaluating LLM agents in social service navigation. Agents interact with simulated users, issue structured resource-search calls, and select final resources from 3,971 Indiana public social service resources. The evaluation focuses on constraint grounding and handling non-ideal user interactions.","descriptions":{"en":"HoosierHelp is an interactive benchmark for evaluating LLM agents in social service navigation. Agents interact with simulated users, issue structured resource-search calls, and select final resources from 3,971 Indiana public social service resources. The evaluation focuses on constraint grounding and handling non-ideal user interactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09946","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_hoosierhelp_ef4e5cac"},"modality":null,"name":"HoosierHelp","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09946","source_url":"https://arxiv.org/abs/2608.09946"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09946","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09946","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"HopRefusalBench evaluates refusal behavior of search-augmented language model agents on multi-hop questions that are unanswerable, covering three causes of unanswerability and three chain topologies.","descriptions":{"en":"HopRefusalBench evaluates refusal behavior of search-augmented language model agents on multi-hop questions that are unanswerable, covering three causes of unanswerability and three chain topologies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01358","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","deep-research"],"join":"claire_record_id","primary":"factuality","source_record":"bm_hoprefusalbench_06f4ed80"},"modality":null,"name":"HopRefusalBench","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01358","source_url":"https://arxiv.org/abs/2608.01358"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01358","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01358","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":36,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":null,"source_record":"catalog_d8b2de1a63b757a6"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"HorizonMath is an extremely difficult frontier mathematics benchmark designed to test the limits of mathematical reasoning on research-level and competition-beyond problems.","descriptions":{"en":"HorizonMath is an extremely difficult frontier mathematics benchmark designed to test the limits of mathematical reasoning on research-level and competition-beyond problems."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:horizonmath:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/horizonmath?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:horizonmath","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_d8b2de1a63b757a6"},"modality":"text","name":"HorizonMath","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":7.1,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.071,"raw_min":0.02,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:horizonmath:hy3","reported_date":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/horizonmath?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-horizonmath","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/horizonmath?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["math"],"collected_at":null,"description":"Unsolved research-level mathematics, reported at pass@4, so figures sit far below conventional accuracy scales.","descriptions":{"en":"Unsolved research-level mathematics, reported at pass@4, so figures sit far below conventional accuracy scales."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000horizonmath\u0000tencent_hy4_preview\u0000horizonmath\u0000pass@4\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000horizonmath\u0000tencent_hy4_preview\u0000horizonmath\u0000pass@4\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:horizonmath","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["math"],"join":"review:14H034160212:2026-10-15","primary":"math","source_record":null},"modality":null,"name":"HorizonMath","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":8.8,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":8.8,"raw_min":8.8,"source_reference":{"obs_id":"curated\u0000horizonmath\u0000tencent_hy4_preview\u0000horizonmath\u0000pass@4\u0000Hy4 preview","observation_id":"curated\u0000horizonmath\u0000tencent_hy4_preview\u0000horizonmath\u0000pass@4\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"horizonmath","source":"model_reports","source_url":"https://github.com/ewang26/HorizonMath","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","大语言模型","LLM","知识储备","Knowledge","逻辑推理","Reasoning","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HOTPOTQA is a dataset with 113k Wikipedia-based question-answer pairs. HotpotQA 用于评估大语言模型的推理能力，包含 113,000 个基于维基百科的问题和答案。","descriptions":{"en":"HOTPOTQA is a dataset with 113k Wikipedia-based question-answer pairs.","zh":"HotpotQA 用于评估大语言模型的推理能力，包含 113,000 个基于维基百科的问题和答案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1108","languages":[],"library_direction":null,"modality":null,"name":"HotpotQA","openness":"open","publisher":"Google AI","released":"2018-09-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1108-hotpotqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HotpotQA","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:byhwhite/hounsworld","source_record":"bm_hounsbench_1139158f"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"HounsBench is a CT-centric patient-state benchmark evaluating three task families: readout, reconstruction, and simulation. It provides patient-disjoint splits and per-family metrics for evaluating models on volumetric medical images and clinical language.","descriptions":{"en":"HounsBench is a CT-centric patient-state benchmark evaluating three task families: readout, reconstruction, and simulation. It provides patient-disjoint splits and per-family metrics for evaluating models on volumetric medical images and clinical language."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.12904","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","spatial-3d"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_hounsbench_1139158f"},"modality":null,"name":"HounsBench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12904","source_url":"https://arxiv.org/abs/2608.12904"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12904","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12904","topics":["multimodal","medical"],"unit":null},{"aliases":["Category-Level 3D Correspondence in Camera Space via Morphable Object Priors"],"attention_signals":null,"categories":["CAD","General AI","Geometric reasoning","Science & Engineering"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark for monocular category-level 3D correspondence in camera space, with 178k images across 50 household object categories and 3D keypoint annotations on CAD models.","descriptions":{"en":"A benchmark for monocular category-level 3D correspondence in camera space, with 178k images across 50 household object categories and 3D keypoint annotations on CAD models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.28257","languages":[],"library_direction":null,"modality":null,"name":"HouseCorr3D","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28257","source_url":"https://arxiv.org/abs/2605.28257"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28257","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28257","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":54,"github_stars_30d":null,"hf_dataset_downloads":28559,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:dreammr/hr-bench","source_record":"catalog_6b53a3168cb7f356"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"HR-Bench (4k) evaluates image understanding on high-resolution visual inputs with a 4k setting.","descriptions":{"en":"HR-Bench (4k) evaluates image understanding on high-resolution visual inputs with a 4k setting."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hr-bench-4k:mimo-v2.5","reported_at":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hr-bench-4k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:hr-bench-4k","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_6b53a3168cb7f356"},"modality":"multimodal","name":"HR-Bench (4k)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":88.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.885,"raw_min":0.885,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hr-bench-4k:mimo-v2.5","reported_date":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hr-bench-4k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"}],"slug":"llm-stats-hr-bench-4k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hr-bench-4k?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_hribench_c65825aa"},"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"HRIBench evaluates intent-aware human-robot collaboration through structured scenario scripts covering roles of Instructor, Collaborator, and Intruder, with 13 tasks and 650 episodes, scoring synchronization, responsiveness, protocol compliance, and safety.","descriptions":{"en":"HRIBench evaluates intent-aware human-robot collaboration through structured scenario scripts covering roles of Instructor, Collaborator, and Intruder, with 13 tasks and 650 episodes, scoring synchronization, responsiveness, protocol compliance, and safety."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.13056","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_hribench_c65825aa"},"modality":null,"name":"HRIBench","openness":"unknown","publisher":null,"released":"2026-07-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.13056","source_url":"https://arxiv.org/abs/2607.13056"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-13056","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13056","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_ht-bench_4c10005b"},"categories":["Robotics & Embodied AI","Vision & 3D","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"HT-Bench is a multi-task benchmark for tactile representation learning with egocentric vision and full-hand tactile data, including retrieval, inpainting, synthesis, and prediction tasks.","descriptions":{"en":"HT-Bench is a multi-task benchmark for tactile representation learning with egocentric vision and full-hand tactile data, including retrieval, inpainting, synthesis, and prediction tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19161","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_ht-bench_4c10005b"},"modality":null,"name":"HT-Bench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19161","source_url":"https://arxiv.org/abs/2606.19161"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19161","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19161","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","其他","Other","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HtFLlib is a benchmark for heterogeneous federated learning that examines how 40 vision, NLP and sensor models and 10 algorithms collaborate under non-IID data. HtFLlib 是一个面向异构联邦学习算法的综合评测基准，旨在衡量不同模型架构在非 IID 数据环境中的协同学习能力。评测对象覆盖图像、文本与传感信号三类模型，总计 40 个架构及 10 种代表性方法。","descriptions":{"en":"HtFLlib is a benchmark for heterogeneous federated learning that examines how 40 vision, NLP and sensor models and 10 algorithms collaborate under non-IID data.","zh":"HtFLlib 是一个面向异构联邦学习算法的综合评测基准，旨在衡量不同模型架构在非 IID 数据环境中的协同学习能力。评测对象覆盖图像、文本与传感信号三类模型，总计 40 个架构及 10 种代表性方法。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1992","languages":[],"library_direction":null,"modality":null,"name":"HtFLlib","openness":"unknown","publisher":"Shanghai Jiao Tong University , Beihang University , Chongqing University , etc.","released":"2025-06-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1992-htfllib","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HtFLlib","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:gml-mmgroup/hug-vis","source_record":"bm_hug-vis_24e7fc8b"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Visual intelligence seeks to perceive, interpret, and synthesize the visual world and is central to modern computer vision.","descriptions":{"en":"Visual intelligence seeks to perceive, interpret, and synthesize the visual world and is central to modern computer vision."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.26517","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_hug-vis_24e7fc8b"},"modality":null,"name":"HUG-VIS","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26517","source_url":"https://arxiv.org/abs/2608.26517"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26517","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26517","topics":["multimodal"],"unit":null},{"aliases":["HUMAID-NER: A Disaster Tweet Dataset for Joint Named Entity Recognition and Event Classification via Uncertainty-Weighted Multitask Learning"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"HUMAID-NER is a dataset of 60,000 disaster tweets annotated for named entity recognition across ten entity types.","descriptions":{"en":"HUMAID-NER is a dataset of 60,000 disaster tweets annotated for named entity recognition across ten entity types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16964","languages":[],"library_direction":null,"modality":null,"name":"HUMAID-NER","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16964","source_url":"https://arxiv.org/abs/2609.16964"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16964","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16964","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-09-25T04:40:22Z","description":"HumanCLAW-Bench evaluates how VLMs make embodied decisions. Its 1,218 humanoid find-navigate-interact episodes span 41 indoor scenes, and low-level motor execution is factored out. Models are scored by success rate.","descriptions":{"en":"HumanCLAW-Bench evaluates how VLMs make embodied decisions. Its 1,218 humanoid find-navigate-interact episodes span 41 indoor scenes, and low-level motor execution is factored out. Models are scored by success rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27180","languages":[],"library_direction":null,"modality":null,"name":"HumanCLAW-Bench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27180","source_url":"https://arxiv.org/abs/2607.27180"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27180","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27180","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3402,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"llm_stats_id","repository":"gh:openai/human-eval","source_record":"lib_humaneval"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics","descriptions":{"en":"A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics"},"editorial_guide":{"id":"humaneval","record_keys":["model-reports:humaneval","llm-stats:humaneval","opencompass:537"],"reviewed_at":"2026-10-25","scope":{"en":"Original 164-task Python HumanEval with its original tests. HumanEval+, multilingual variants and infilling variants remain separate evaluations.","zh":"对应原版 164 道 Python HumanEval 题及原始测试；HumanEval+、多语言版和代码补全变体保留独立口径。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"A prompt supplies a function signature and docstring. The original set contains 164 hand-written programming problems with tests, spanning language understanding, algorithms and elementary mathematics. The model supplies code that satisfies the requested behavior.","zh":"提示提供函数签名和文档说明。原题集包含 164 道人工编写的编程题及测试，涉及语言理解、算法和基础数学；模型需要给出符合要求的代码。"},"title":{"en":"Function completion from a specification","zh":"按说明完成函数"}},{"id":"grading","source_ids":["repository","evaluator"],"text":{"en":"The harness executes each generated completion against the task's tests and records pass or failure. Matching reference wording or source code is unnecessary. Runtime errors and timeouts are failures under the harness configuration.","zh":"评测程序将生成代码放入相应任务测试，记录通过或失败。无需与参考代码逐字一致；在既定配置下，运行错误或超时会造成失败。"},"title":{"en":"Execution determines correctness","zh":"通过执行判定正确性"}},{"id":"metrics","source_ids":["evaluator"],"text":{"en":"Pass@k estimates whether at least one of k generated candidates passes. The official estimator uses the number of samples and successful samples per problem. A high pass@100 does not mean a user receives correct code on the first attempt.","zh":"pass@k 估计生成 k 份候选时至少一份通过的概率。官方估计器使用每题总样本数与通过样本数。pass@100 很高，不意味着用户第一次就能拿到正确代码。"},"title":{"en":"What pass@k measures","zh":"pass@k 测量什么"}},{"id":"comparison","source_ids":["paper","repository"],"text":{"en":"Compare the same k, sampling temperature, completion handling and test suite. The paper shows that preferred temperatures differ with k. Results from extra tests or other languages should retain their own variant labels.","zh":"比较时应对齐 k、采样温度、代码截取方式和测试集。论文显示，不同 k 对应的合适温度会变化；使用额外测试或其他语言的成绩应保留变体标签。"},"title":{"en":"Match the sampling and test version","zh":"对齐采样与测试版本"}},{"id":"limits","source_ids":["repository","paper"],"text":{"en":"Passing a finite test suite is evidence about those cases, not a proof of all behavior. These short functions do not cover repository maintenance or long projects. The official repository now marks the original harness as unmaintained historical material.","zh":"通过有限测试只能证明覆盖用例的表现，不能证明所有输入都正确。短函数题也不覆盖仓库维护或长期项目。官方仓库目前已将原评测框架标注为停止维护的历史资料。"},"title":{"en":"A compact coding signal","zh":"短函数能力的观察窗口"}}],"sources":[{"id":"paper","title":"Evaluating Large Language Models Trained on Code · HumanEval and pass@k","url":"https://arxiv.org/html/2107.03374v2"},{"id":"repository","title":"OpenAI HumanEval · Official evaluation harness","url":"https://github.com/openai/human-eval"},{"id":"evaluator","title":"HumanEval · Functional evaluation and pass@k implementation","url":"https://github.com/openai/human-eval/blob/master/human_eval/evaluation.py"}],"summary":{"en":"HumanEval asks a model to implement a small Python function from a specification, then checks its behavior by running tests. Read pass@1 and pass@k separately.","zh":"HumanEval 要求模型根据说明实现一个较小的 Python 函数，再执行测试检查行为。阅读成绩时需要区分 pass@1 与 pass@k。"},"title":"HumanEval"},"evidence_summary":{"document_count":1,"model_count":66,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humaneval:gpt-3.5-turbo-0125","reported_at":"2023-03-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:humaneval","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"lib_humaneval"},"modality":"text","name":"HumanEval","openness":"unknown","publisher":null,"released":"2021-07-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":66,"score_direction":"higher_is_better","score_summary":{"display_max":95.12,"display_multiplier":100,"model_count":66,"model_count_basis":"source_model_id","numeric_count":66,"raw_max":0.9512,"raw_min":0.348,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humaneval:minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval?top_n=500"},"unit":null},"scored_models":[{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-05-29","date_precision":"model_announcement","name":"Codestral-22B"},{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2023-03-21","date_precision":"model_announcement","name":"GPT-3.5 Turbo"},{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"},{"date":"2024-04-09","date_precision":"model_announcement","name":"GPT-4 Turbo"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-05-13","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2024-03-28","date_precision":"model_announcement","name":"Grok-1.5"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2 mini"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 70B Instruct"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Instruct"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Instruct"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"}],"slug":"llm-stats-humaneval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":["HumanEval","HumanEval+"],"attention_signals":{"as_of":"2026-10-12","github_stars":3402,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"artifact:gh:openai/human-eval","repository":"gh:openai/human-eval","source_record":"lib_humaneval"},"categories":["coding"],"collected_at":null,"description":"Saturated. Retained because open-weight cards still report it.","descriptions":{"en":"Saturated. Retained because open-weight cards still report it."},"editorial_guide":{"id":"humaneval","record_keys":["model-reports:humaneval","llm-stats:humaneval","opencompass:537"],"reviewed_at":"2026-10-25","scope":{"en":"Original 164-task Python HumanEval with its original tests. HumanEval+, multilingual variants and infilling variants remain separate evaluations.","zh":"对应原版 164 道 Python HumanEval 题及原始测试；HumanEval+、多语言版和代码补全变体保留独立口径。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"A prompt supplies a function signature and docstring. The original set contains 164 hand-written programming problems with tests, spanning language understanding, algorithms and elementary mathematics. The model supplies code that satisfies the requested behavior.","zh":"提示提供函数签名和文档说明。原题集包含 164 道人工编写的编程题及测试，涉及语言理解、算法和基础数学；模型需要给出符合要求的代码。"},"title":{"en":"Function completion from a specification","zh":"按说明完成函数"}},{"id":"grading","source_ids":["repository","evaluator"],"text":{"en":"The harness executes each generated completion against the task's tests and records pass or failure. Matching reference wording or source code is unnecessary. Runtime errors and timeouts are failures under the harness configuration.","zh":"评测程序将生成代码放入相应任务测试，记录通过或失败。无需与参考代码逐字一致；在既定配置下，运行错误或超时会造成失败。"},"title":{"en":"Execution determines correctness","zh":"通过执行判定正确性"}},{"id":"metrics","source_ids":["evaluator"],"text":{"en":"Pass@k estimates whether at least one of k generated candidates passes. The official estimator uses the number of samples and successful samples per problem. A high pass@100 does not mean a user receives correct code on the first attempt.","zh":"pass@k 估计生成 k 份候选时至少一份通过的概率。官方估计器使用每题总样本数与通过样本数。pass@100 很高，不意味着用户第一次就能拿到正确代码。"},"title":{"en":"What pass@k measures","zh":"pass@k 测量什么"}},{"id":"comparison","source_ids":["paper","repository"],"text":{"en":"Compare the same k, sampling temperature, completion handling and test suite. The paper shows that preferred temperatures differ with k. Results from extra tests or other languages should retain their own variant labels.","zh":"比较时应对齐 k、采样温度、代码截取方式和测试集。论文显示，不同 k 对应的合适温度会变化；使用额外测试或其他语言的成绩应保留变体标签。"},"title":{"en":"Match the sampling and test version","zh":"对齐采样与测试版本"}},{"id":"limits","source_ids":["repository","paper"],"text":{"en":"Passing a finite test suite is evidence about those cases, not a proof of all behavior. These short functions do not cover repository maintenance or long projects. The official repository now marks the original harness as unmaintained historical material.","zh":"通过有限测试只能证明覆盖用例的表现，不能证明所有输入都正确。短函数题也不覆盖仓库维护或长期项目。官方仓库目前已将原评测框架标注为停止维护的历史资料。"},"title":{"en":"A compact coding signal","zh":"短函数能力的观察窗口"}}],"sources":[{"id":"paper","title":"Evaluating Large Language Models Trained on Code · HumanEval and pass@k","url":"https://arxiv.org/html/2107.03374v2"},{"id":"repository","title":"OpenAI HumanEval · Official evaluation harness","url":"https://github.com/openai/human-eval"},{"id":"evaluator","title":"HumanEval · Functional evaluation and pass@k implementation","url":"https://github.com/openai/human-eval/blob/master/human_eval/evaluation.py"}],"summary":{"en":"HumanEval asks a model to implement a small Python function from a specification, then checks its behavior by running tests. Read pass@1 and pass@k separately.","zh":"HumanEval 要求模型根据说明实现一个较小的 Python 函数，再执行测试检查行为。阅读成绩时需要区分 pass@1 与 pass@k。"},"title":"HumanEval"},"evidence_summary":{"document_count":9,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000humaneval\u0000google_gemini_1_5_report\u0000humaneval\u00000-shot, chat preamble\u0000Gemini 1.5 Pro","reported_at":"2024-03-08","source_url":"https://arxiv.org/abs/2403.05530"},"first_score_reported_at":"2024-03-08","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000humaneval\u0000google_gemini_1_5_report\u0000humaneval\u00000-shot, chat preamble\u0000Gemini 1.5 Pro","reported_at":"2024-03-08","source_url":"https://arxiv.org/abs/2403.05530"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:humaneval","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"artifact:gh:openai/human-eval","primary":"code-generation","source_record":"lib_humaneval"},"modality":null,"name":"HumanEval","openness":"unknown","publisher":null,"released":"2021-07-07","released_reference":{"basis":"benchmark_release","source_key":"model-reports:humaneval","source_url":"https://github.com/openai/human-eval"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":92.7,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":92.7,"raw_min":84.1,"source_reference":{"obs_id":"curated\u0000humaneval\u0000qwen2_5_coder_report\u0000humaneval\u0000HumanEval base split\u0000Qwen2.5-Coder-32B-Instruct","observation_id":"curated\u0000humaneval\u0000qwen2_5_coder_report\u0000humaneval\u0000HumanEval base split\u0000Qwen2.5-Coder-32B-Instruct","reported_at":"2024-09-18","reported_date":"2024-09-18","source_id":"qwen2_5_coder_report","source_url":"https://arxiv.org/abs/2409.12186"},"unit":"percent"},"scored_models":[{"date":"2024-03-08","date_precision":"document_publication","name":"Gemini 1.5 Pro"},{"date":"2024-09-18","date_precision":"document_publication","name":"Qwen2.5-Coder-32B-Instruct"}],"slug":"humaneval","source":"model_reports","source_url":"https://github.com/openai/human-eval","topics":["coding"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. 这是 \"Evaluating Large Language Models Trained on Code\" 论文中描述的 HumanEval 问题解决数据集的评估工具包。它用于测量从文档脚本合成程序的功能正确性。它由 164 个原始编程问题组成，评估语言理解能力、算法和简单数学，其中一些问题与简单的软件面试题类似。","descriptions":{"en":"The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases.","zh":"这是 \"Evaluating Large Language Models Trained on Code\" 论文中描述的 HumanEval 问题解决数据集的评估工具包。它用于测量从文档脚本合成程序的功能正确性。它由 164 个原始编程问题组成，评估语言理解能力、算法和简单数学，其中一些问题与简单的软件面试题类似。"},"editorial_guide":{"id":"humaneval","record_keys":["model-reports:humaneval","llm-stats:humaneval","opencompass:537"],"reviewed_at":"2026-10-25","scope":{"en":"Original 164-task Python HumanEval with its original tests. HumanEval+, multilingual variants and infilling variants remain separate evaluations.","zh":"对应原版 164 道 Python HumanEval 题及原始测试；HumanEval+、多语言版和代码补全变体保留独立口径。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"A prompt supplies a function signature and docstring. The original set contains 164 hand-written programming problems with tests, spanning language understanding, algorithms and elementary mathematics. The model supplies code that satisfies the requested behavior.","zh":"提示提供函数签名和文档说明。原题集包含 164 道人工编写的编程题及测试，涉及语言理解、算法和基础数学；模型需要给出符合要求的代码。"},"title":{"en":"Function completion from a specification","zh":"按说明完成函数"}},{"id":"grading","source_ids":["repository","evaluator"],"text":{"en":"The harness executes each generated completion against the task's tests and records pass or failure. Matching reference wording or source code is unnecessary. Runtime errors and timeouts are failures under the harness configuration.","zh":"评测程序将生成代码放入相应任务测试，记录通过或失败。无需与参考代码逐字一致；在既定配置下，运行错误或超时会造成失败。"},"title":{"en":"Execution determines correctness","zh":"通过执行判定正确性"}},{"id":"metrics","source_ids":["evaluator"],"text":{"en":"Pass@k estimates whether at least one of k generated candidates passes. The official estimator uses the number of samples and successful samples per problem. A high pass@100 does not mean a user receives correct code on the first attempt.","zh":"pass@k 估计生成 k 份候选时至少一份通过的概率。官方估计器使用每题总样本数与通过样本数。pass@100 很高，不意味着用户第一次就能拿到正确代码。"},"title":{"en":"What pass@k measures","zh":"pass@k 测量什么"}},{"id":"comparison","source_ids":["paper","repository"],"text":{"en":"Compare the same k, sampling temperature, completion handling and test suite. The paper shows that preferred temperatures differ with k. Results from extra tests or other languages should retain their own variant labels.","zh":"比较时应对齐 k、采样温度、代码截取方式和测试集。论文显示，不同 k 对应的合适温度会变化；使用额外测试或其他语言的成绩应保留变体标签。"},"title":{"en":"Match the sampling and test version","zh":"对齐采样与测试版本"}},{"id":"limits","source_ids":["repository","paper"],"text":{"en":"Passing a finite test suite is evidence about those cases, not a proof of all behavior. These short functions do not cover repository maintenance or long projects. The official repository now marks the original harness as unmaintained historical material.","zh":"通过有限测试只能证明覆盖用例的表现，不能证明所有输入都正确。短函数题也不覆盖仓库维护或长期项目。官方仓库目前已将原评测框架标注为停止维护的历史资料。"},"title":{"en":"A compact coding signal","zh":"短函数能力的观察窗口"}}],"sources":[{"id":"paper","title":"Evaluating Large Language Models Trained on Code · HumanEval and pass@k","url":"https://arxiv.org/html/2107.03374v2"},{"id":"repository","title":"OpenAI HumanEval · Official evaluation harness","url":"https://github.com/openai/human-eval"},{"id":"evaluator","title":"HumanEval · Functional evaluation and pass@k implementation","url":"https://github.com/openai/human-eval/blob/master/human_eval/evaluation.py"}],"summary":{"en":"HumanEval asks a model to implement a small Python function from a specification, then checks its behavior by running tests. Read pass@1 and pass@k separately.","zh":"HumanEval 要求模型根据说明实现一个较小的 Python 函数，再执行测试检查行为。阅读成绩时需要区分 pass@1 与 pass@k。"},"title":"HumanEval"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:537","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"artifact:arxiv:2107.03374","primary":"code-generation","source_record":"catalog_12bd73d8ceec44dd"},"modality":null,"name":"HumanEval","openness":"unknown","publisher":null,"released":"2021-07-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-537-humaneval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HumanEval","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1826,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_810807308d2673d1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Enhanced version of HumanEval that extends the original test cases by 80x using EvalPlus framework for rigorous evaluation of LLM-synthesized code functional correctness, detecting previously undetected wrong code","descriptions":{"en":"Enhanced version of HumanEval that extends the original test cases by 80x using EvalPlus framework for rigorous evaluation of LLM-synthesized code functional correctness, detecting previously undetected wrong code"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humaneval-plus:mistral-small-3.2-24b-instruct-2506","reported_at":"2025-06-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-plus?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:humaneval-plus","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_810807308d2673d1"},"modality":"text","name":"HumanEval Plus","openness":"unknown","publisher":null,"released":"2023-05-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:humaneval-plus","source_url":"https://arxiv.org/abs/2305.01210"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":92.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.929,"raw_min":0.929,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humaneval-plus:mistral-small-3.2-24b-instruct-2506","reported_date":"2025-06-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-plus?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"}],"slug":"llm-stats-humaneval-plus","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-plus?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Enhanced version of HumanEval that extends the original test cases by 80x using EvalPlus framework for rigorous evaluation of LLM-synthesized code functional correctness, detecting previously undetected wrong code","descriptions":{"en":"Enhanced version of HumanEval that extends the original test cases by 80x using EvalPlus framework for rigorous evaluation of LLM-synthesized code functional correctness, detecting previously undetected wrong code"},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humaneval+:qwen-2.5-14b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval%2B?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:humaneval+","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_810807308d2673d1"},"modality":"text","name":"HumanEval+","openness":"unknown","publication":{"evidence":[{"note":"The introducing paper names HumanEval+ and describes an 80-fold expansion of the original tests, matching the record's EvalPlus-based code-correctness evaluation.","url":"https://arxiv.org/abs/2305.01210"},{"note":"The authors' EvalPlus repository links the HumanEval+ paper, documents HumanEval+ evaluation and test revisions, and labels the mini tests, MBPP+ and EvalPerf separately. The repository's default branch is master.","url":"https://github.com/evalplus/evalplus/blob/master/README.md"}],"id":"humaneval-plus-original","links":[{"kind":"paper","title":"Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation","url":"https://arxiv.org/abs/2305.01210"},{"kind":"repo","title":"EvalPlus official repository for HumanEval+","url":"https://github.com/evalplus/evalplus"}],"record_keys":["llm-stats:humaneval+"],"reviewed_at":"2026-10-26","scope":{"en":"HumanEval+ with the expanded EvalPlus tests. The base HumanEval suite, HumanEval+ mini and MBPP+ are separate evaluations; the test revision used by each reported score remains source-specific.","zh":"使用 EvalPlus 扩充测试的 HumanEval+。原始 HumanEval、HumanEval+ mini 和 MBPP+ 是不同评测；每项成绩使用的测试修订版本仍以其来源为准。"}},"publisher":null,"released":"2023-05-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:humaneval+","source_url":"https://arxiv.org/abs/2305.01210"},"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":92.9,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.929,"raw_min":0.25,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humaneval+:phi-4-reasoning","reported_date":"2025-04-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval%2B?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"}],"slug":"llm-stats-humaneval-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval%2B?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"llm_stats_id","repository":null,"source_record":"catalog_12bd73d8ceec44dd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics","descriptions":{"en":"A variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humaneval-average:codestral-22b","reported_at":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-average?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:humaneval-average","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_12bd73d8ceec44dd"},"modality":"text","name":"HumanEval-Average","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":61.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.615,"raw_min":0.615,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humaneval-average:codestral-22b","reported_date":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-average?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-29","date_precision":"model_announcement","name":"Codestral-22B"}],"slug":"llm-stats-humaneval-average","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-average?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics","descriptions":{"en":"A variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humaneval-er:kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-er?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:humaneval-er","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_db3cb8d38ce35983"},"modality":"text","name":"HumanEval-ER","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":81.10000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.811,"raw_min":0.811,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humaneval-er:kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-er?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"}],"slug":"llm-stats-humaneval-er","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-er?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":316,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:nuprl/multipl-e","source_record":"catalog_1e92f1c27e2af05f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multilingual variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics","descriptions":{"en":"A multilingual variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humaneval-mul:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-mul?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:humaneval-mul","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_1e92f1c27e2af05f"},"modality":"text","name":"HumanEval-Mul","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":82.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.826,"raw_min":0.738,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humaneval-mul:deepseek-v3","reported_date":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-mul?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"}],"slug":"llm-stats-humaneval-mul","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-mul?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HumanEval-X is a benchmark for evaluating the multilingual ability of code generative models. It consists of 820 high-quality human-crafted data samples (each with test cases) in Python, C++, Java, JavaScript, and Go, and can be used for various tasks, such as code generation and translation. HumanEval-X 是一个用于评估代码生成模型的多语言能力的基准测试。它包含了820个高质量的人工制作的数据样本（每个都有测试案例），包括Python、C++、Java、JavaScript和Go语言，可用于各种任务，如代码生成和翻译。","descriptions":{"en":"HumanEval-X is a benchmark for evaluating the multilingual ability of code generative models. It consists of 820 high-quality human-crafted data samples (each with test cases) in Python, C++, Java, JavaScript, and Go, and can be used for various tasks, such as code generation and translation.","zh":"HumanEval-X 是一个用于评估代码生成模型的多语言能力的基准测试。它包含了820个高质量的人工制作的数据样本（每个都有测试案例），包括Python、C++、Java、JavaScript和Go语言，可用于各种任务，如代码生成和翻译。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:543","languages":["Multilingual"],"library_direction":null,"modality":null,"name":"HumanEval-X","openness":"open","publisher":null,"released":"2023-03-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-543-humaneval-x","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HumanEval-X","topics":["coding"],"unit":null},{"aliases":["Bootstrapping Niche Multilingual Code Translation via Reinforcement Learning with Execution-Based Verifiable Supervision"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"HumanEval-X++ is an execution-based benchmark extending HumanEval-X to a broad many-to-many language space for code translation evaluation.","descriptions":{"en":"HumanEval-X++ is an execution-based benchmark extending HumanEval-X to a broad many-to-many language space for code translation evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.13854","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_bootstrapping-niche-multilingual-code-tran_de25b8dd"},"modality":null,"name":"HumanEval-X++","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13854","source_url":"https://arxiv.org/abs/2608.13854"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13854","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13854","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":211,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:openai/human-eval-infilling","source_record":"catalog_6da692bf12f42b89"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Average evaluation of HumanEval Fill-in-the-Middle benchmark variants (single-line, multi-line, random-span) for assessing code infilling capabilities of language models","descriptions":{"en":"Average evaluation of HumanEval Fill-in-the-Middle benchmark variants (single-line, multi-line, random-span) for assessing code infilling capabilities of language models"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humanevalfim-average:codestral-22b","reported_at":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanevalfim-average?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:humanevalfim-average","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_6da692bf12f42b89"},"modality":"text","name":"HumanEvalFIM-Average","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":91.60000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.916,"raw_min":0.916,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humanevalfim-average:codestral-22b","reported_date":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanevalfim-average?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-29","date_precision":"model_announcement","name":"Codestral-22B"}],"slug":"llm-stats-humanevalfim-average","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanevalfim-average?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1713,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":79,"join":"llm_stats_id","repository":"gh:centerforaisafety/hle","source_record":"lib_humanitys_last_exam"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions","descriptions":{"en":"Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions"},"evidence_summary":{"document_count":1,"model_count":99,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humanity's-last-exam:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:humanity's-last-exam","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"lib_humanitys_last_exam"},"modality":"multimodal","name":"Humanity's Last Exam","openness":"restricted","publisher":"Center for AI Safety","released":"2025-01-24","released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":99,"score_direction":"higher_is_better","score_summary":{"display_max":64.7,"display_multiplier":100,"model_count":99,"model_count_basis":"source_model_id","numeric_count":99,"raw_max":0.647,"raw_min":0.037,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humanity's-last-exam:claude-mythos-preview","reported_date":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2-Speciale"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2026-01-22","date_precision":"model_announcement","name":"ERNIE 5.0"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 20B"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Pro"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 Pro"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok 4 Fast"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4 Heavy"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-humanity-s-last-exam","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["HLE","Humanity's Last Exam","HLE-Full","HLE-Verified"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"HLE evaluates answers to academic questions across many subjects. Its finalized original public release contains 2,500 questions; HLE-Rolling and HLE-Diamond are separate releases. Check the question set, tool access, answer-grading procedure and run settings attached to each result.","descriptions":{"en":"HLE evaluates answers to academic questions across many subjects. Its finalized original public release contains 2,500 questions; HLE-Rolling and HLE-Diamond are separate releases. Check the question set, tool access, answer-grading procedure and run settings attached to each result."},"evidence_summary":{"document_count":22,"model_count":20,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000hle\u0000google_gemini_2_5_report\u0000hle\u0000no tools\u0000Gemini 2.5 Pro","reported_at":"2025-06-17","source_url":"https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf"},"first_score_reported_at":"2025-06-17","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000hle\u0000google_gemini_2_5_report\u0000hle\u0000no tools\u0000Gemini 2.5 Pro","reported_at":"2025-06-17","source_url":"https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:hle","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["knowledge-expert-qa"],"join":"review:14H034160212:2026-10-15","primary":"knowledge-expert-qa","source_record":null},"modality":null,"name":"Humanity's Last Exam","openness":"unknown","publisher":null,"released":"2025-01-23","released_reference":{"basis":"benchmark_release","source_key":"model-reports:hle","source_url":"https://lastexam.ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":32,"score_direction":"higher_is_better","score_summary":{"display_max":63.9,"display_multiplier":1,"model_count":20,"model_count_basis":"source_model_id","numeric_count":32,"raw_max":63.9,"raw_min":7.7,"source_reference":{"obs_id":"curated\u0000hle\u0000deepseek_v4_1_flash_technical_report\u0000hle\u0000think max, Pass@1, with tools\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000hle\u0000deepseek_v4_1_flash_technical_report\u0000hle\u0000think max, Pass@1, with tools\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2025-06-17","date_precision":"document_publication","name":"Claude 4 Opus"},{"date":"2025-06-17","date_precision":"document_publication","name":"Claude 4 Sonnet"},{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-08-13","date_precision":"document_publication","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2025-06-17","date_precision":"document_publication","name":"Gemini 2.5 Pro"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2025-06-17","date_precision":"document_publication","name":"o3 high"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"hle","source":"model_reports","source_url":"https://lastexam.ai/","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1711,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":79,"join":"llm_stats_id","repository":null,"source_record":"catalog_c258daebcc7b06f4"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Text-only Humanity's Last Exam variant evaluated without tool use.","descriptions":{"en":"Text-only Humanity's Last Exam variant evaluated without tool use."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humanity's-last-exam-(no-tools,-text-only):hy3","reported_at":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28no-tools%2C-text-only%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:humanity's-last-exam-(no-tools,-text-only)","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_c258daebcc7b06f4"},"modality":"text","name":"Humanity's Last Exam (no tools, text-only)","openness":"unknown","publisher":null,"released":"2025-01-24","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Humanity's Last Exam. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:humanity's-last-exam-(no-tools,-text-only)","source_url":"https://arxiv.org/abs/2501.14249"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":47.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.47,"raw_min":0.427,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humanity's-last-exam-(no-tools,-text-only):hy3","reported_date":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28no-tools%2C-text-only%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"}],"slug":"llm-stats-humanity-s-last-exam-no-tools-text-only","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28no-tools%2C-text-only%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1711,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":79,"join":"llm_stats_id","repository":null,"source_record":"catalog_bb0cc3176b036740"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Text-only Humanity's Last Exam variant evaluated with tool use enabled.","descriptions":{"en":"Text-only Humanity's Last Exam variant evaluated with tool use enabled."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:humanity's-last-exam-(with-tools,-text-only):hy3","reported_at":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28with-tools%2C-text-only%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:humanity's-last-exam-(with-tools,-text-only)","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_bb0cc3176b036740"},"modality":"text","name":"Humanity's Last Exam (with tools, text-only)","openness":"unknown","publisher":null,"released":"2025-01-24","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Humanity's Last Exam. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:humanity's-last-exam-(with-tools,-text-only)","source_url":"https://arxiv.org/abs/2501.14249"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":60.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.6,"raw_min":0.532,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:humanity's-last-exam-(with-tools,-text-only):deepseek-v4-pro-0813","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28with-tools%2C-text-only%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-humanity-s-last-exam-with-tools-text-only","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28with-tools%2C-text-only%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","reasoning","knowledge"],"collected_at":"2026-08-25T10:41:06Z","description":"Reasoning & knowledge","descriptions":{"en":"Reasoning & knowledge"},"editorial_guide":{"id":"artificial-analysis-humanitys-last-exam","record_keys":["artificial-analysis:humanitys-last-exam"],"reviewed_at":"2026-10-29","scope":{"en":"The Artificial Analysis results collected on 25 August 2026. Original HLE publications and current methodology are linked for context; later revisions do not establish each saved run’s setup.","zh":"对应 2026 年 8 月 25 日采集的 Artificial Analysis 成绩。原始 HLE 发布资料与现行方法文档用于解释评测，后续修订不能证明每条历史成绩的运行配置。"},"sections":[{"id":"task","source_ids":["project","paper","evaluation"],"text":{"en":"Developed by the Center for AI Safety and Scale AI with expert contributors, HLE combines multiple-choice and short-answer questions across mathematics, humanities and the natural sciences. The original project links its paper, dataset and evaluation code. Artificial Analysis conducts its own evaluations; its results remain distinct from the original authors’ leaderboard.","zh":"HLE 由 Center for AI Safety、Scale AI 与专家贡献者共同开发，包含数学、人文与自然科学领域的选择题和简答题。原始项目提供论文、数据集与评测代码。Artificial Analysis 独立运行评测，本页成绩与原作者榜单分别保留。"},"title":{"en":"Expert questions across academic subjects","zh":"跨学科的专家级问题"}},{"id":"dates","source_ids":["announcement","paper","saved-record"],"text":{"en":"The public benchmark announcement is dated 23 January 2025; the first arXiv submission is dated 24 January. This record was collected on 25 August 2026. Dates beside individual results are model announcement dates, labelled as a model release proxy, not the dates those evaluations were run. The captured source does not provide per-run dates.","zh":"评测公开发布公告的日期是 2025 年 1 月 23 日，arXiv 首次提交日期是 1 月 24 日。本条记录采集于 2026 年 8 月 25 日。逐条成绩旁的日期是模型公布日期，页面将其标为模型发布日期代理，并非该次评测的运行日期；采集来源未提供逐次运行日期。"},"title":{"en":"Release, collection and model dates","zh":"发布日期、采集日期与模型日期"}},{"id":"configurations","source_ids":["saved-record","evaluation"],"text":{"en":"Model names retain the source’s version, reasoning-effort and fallback labels where supplied. Rows also retain the source’s model identity. A display name does not establish an immutable provider API revision; that revision was not supplied in the captured table. Units and per-run protocols that were not recorded remain unknown, and the original numeric values are preserved.","zh":"模型名称保留来源提供的版本、推理强度及回退配置，数据也保留来源的模型标识。显示名称不能证明不可变的 API 修订版本，采集表中没有提供这项信息。未记录的单位与逐次运行协议仍标为未知，原始数值保持不变。"},"title":{"en":"Keep the model configuration attached","zh":"保留模型的具体配置"}},{"id":"methodology","source_ids":["methodology","saved-record"],"text":{"en":"The methodology reviewed on 29 October is v5.0 and describes HLE as 2,158 text-only questions from the May 2025 revision, scored with pass@1. It treats HLE-Diamond separately. This provides context, not verified historical settings for every saved row. Do not substitute today’s grader, dataset revision or leaderboard values for the captured results.","zh":"10 月 29 日查阅的现行方法为 v5.0，将 HLE 描述为 2025 年 5 月修订版中的 2,158 道纯文本题，采用 pass@1 计分，并单独列出 HLE-Diamond。这些内容用于帮助理解评测，不能证明每条历史成绩的配置，也不能用今天的评分模型、题集修订或榜单数值替换已采集结果。"},"title":{"en":"Read current methodology with the snapshot date","zh":"结合快照日期阅读现行方法"}}],"sources":[{"id":"project","title":"Humanity’s Last Exam · original project","url":"https://lastexam.ai/"},{"id":"paper","title":"Humanity’s Last Exam · paper and version history","url":"https://arxiv.org/abs/2501.14249"},{"id":"announcement","title":"Scale AI and CAIS · public announcement dated 23 January 2025","url":"https://scale.com/blog/humanitys-last-exam-results"},{"id":"evaluation","title":"Artificial Analysis · Humanity’s Last Exam evaluation","url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam"},{"id":"methodology","title":"Artificial Analysis · current methodology, reviewed 29 October 2026","url":"https://artificialanalysis.ai/methodology/intelligence-benchmarking"},{"id":"saved-record","title":"AI Benchmark Radar · saved record, collection timestamp and row date basis","url":"https://aibenchmarkradar.com/api/v1/benchmarks/artificial-analysis-humanitys-last-exam"}],"summary":{"en":"Humanity’s Last Exam tests expert-level academic knowledge and reasoning. This page preserves Artificial Analysis’s collected model results, with the original benchmark and date meanings explained below.","zh":"Humanity’s Last Exam 考查专家级学术知识与推理。本页保留采集自 Artificial Analysis 的模型成绩，并说明原始评测来源与各类日期的含义。"},"title":"Humanity’s Last Exam · Artificial Analysis"},"evidence_summary":{"document_count":1,"model_count":577,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:humanitys-last-exam:83cb898e-05d9-4e4b-9de3-2d305014d923","reported_at":"2023-03-14","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:humanitys-last-exam","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["knowledge-expert-qa"],"join":"review:14H034160212:2026-10-15","primary":"knowledge-expert-qa","source_record":null},"modality":null,"name":"Humanity’s Last Exam","openness":"unknown","publication":{"evidence":[{"note":"Artificial Analysis names Humanity’s Last Exam, identifies its independent evaluation and links arXiv 2501.14249, the cais/hle dataset and the centerforaisafety/hle repository. This establishes the exact-record publication association without transferring other scores.","url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam"},{"note":"The original HLE project links the same paper, dataset and repository and identifies the Center for AI Safety and Scale AI organizing team. Later revisions remain separate from the captured Artificial Analysis results.","url":"https://lastexam.ai/"}],"id":"artificial-analysis-humanitys-last-exam-original","links":[{"kind":"website","title":"Humanity’s Last Exam · original project","url":"https://lastexam.ai/"},{"kind":"paper","title":"Humanity’s Last Exam · original paper and version history","url":"https://arxiv.org/abs/2501.14249"},{"kind":"dataset","title":"Humanity’s Last Exam · official dataset","url":"https://huggingface.co/datasets/cais/hle"},{"kind":"repo","title":"Humanity’s Last Exam · official evaluation repository","url":"https://github.com/centerforaisafety/hle"}],"record_keys":["artificial-analysis:humanitys-last-exam"],"reviewed_at":"2026-10-29","scope":{"en":"Original Humanity’s Last Exam publications for this Artificial Analysis source record. Scores remain the separately collected Artificial Analysis snapshot from 25 August 2026; later publication and dataset revisions do not establish the setup of every saved evaluation.","zh":"本条 Artificial Analysis 记录对应的原始 Humanity’s Last Exam 发布资料。成绩仍为独立采集的 2026 年 8 月 25 日 Artificial Analysis 快照；论文与数据集的后续修订不能证明每条历史评测的具体配置。"}},"publisher":null,"released":"2025-01-23","released_reference":{"basis":"release_announcement","note":"Scale and CAIS announce the public HLE benchmark and results on January 23, before the January 24 arXiv submission. GPT-4 model releases in 2023 do not date HLE.","source_key":"artificial-analysis:humanitys-last-exam","source_url":"https://scale.com/blog/humanitys-last-exam-results"},"repo_kind":null,"repo_resolution_status":null,"score_count":577,"score_direction":"higher_is_better","score_summary":{"display_max":55.468025949953706,"display_multiplier":100,"model_count":577,"model_count_basis":"source_model_id","numeric_count":577,"raw_max":0.554680259499537,"raw_min":0.0106580166821131,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:humanitys-last-exam:cd55210d-358e-4df1-ba9c-9acb5f186cc9","reported_date":"2026-06-09","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2023-11-21","date_precision":"model_announcement","name":"Claude 2.1"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-06-21","date_precision":"model_announcement","name":"Claude 3.5 Sonnet (June '24)"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet (Oct '24)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2023-03-14","date_precision":"model_announcement","name":"Claude Instant"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2024-03-12","date_precision":"model_announcement","name":"Command-R (Mar '24)"},{"date":"2024-04-04","date_precision":"model_announcement","name":"Command-R+ (Apr '24)"},{"date":"2024-03-27","date_precision":"model_announcement","name":"DBRX Instruct"},{"date":"2025-02-13","date_precision":"model_announcement","name":"DeepHermes 3 - Llama-3.1 8B Preview (Non-reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"DeepHermes 3 - Mistral 24B Preview (Non-reasoning)"},{"date":"2024-06-17","date_precision":"model_announcement","name":"DeepSeek Coder V2 Lite Instruct"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2023-12-06","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-14","date_precision":"model_announcement","name":"Gemini 1.5 Flash (May '24)"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Flash (Sep '24)"},{"date":"2024-10-03","date_precision":"model_announcement","name":"Gemini 1.5 Flash-8B"},{"date":"2024-05-15","date_precision":"model_announcement","name":"Gemini 1.5 Pro (May '24)"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Pro (Sep '24)"},{"date":"2024-12-11","date_precision":"model_announcement","name":"Gemini 2.0 Flash (experimental)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Gemini 2.0 Flash Thinking Experimental (Jan '25)"},{"date":"2025-02-25","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite (Feb '25)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite (Preview)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Pro Experimental (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-04-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Non-reasoning)"},{"date":"2025-04-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-03-25","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview (Mar' 25)"},{"date":"2025-05-06","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview (May' 25)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct Preview (May '25)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2023-11-06","date_precision":"model_announcement","name":"GPT-4 Turbo"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2025-02-15","date_precision":"model_announcement","name":"GPT-4o (ChatGPT)"},{"date":"2025-03-27","date_precision":"model_announcement","name":"GPT-4o (March 2025, chatgpt-4o-latest)"},{"date":"2024-05-13","date_precision":"model_announcement","name":"GPT-4o (May '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (ChatGPT)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Grok 2 (Dec '24)"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok Beta"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 - Llama-3.1 70B"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-03-06","date_precision":"model_announcement","name":"Jamba 1.6 Large"},{"date":"2025-03-06","date_precision":"model_announcement","name":"Jamba 1.6 Mini"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2025-10-30","date_precision":"model_announcement","name":"Kimi Linear 48B A3B Instruct"},{"date":"2024-09-30","date_precision":"model_announcement","name":"LFM 40B"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 13B"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 70B"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 7B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 4B v1.1 (Reasoning)"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Llama 3.1 Tulu3 405B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 3B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 90B (Vision)"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 40k"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Mistral Large (Feb '24)"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2 (Jul '24)"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2023-12-11","date_precision":"model_announcement","name":"Mistral Medium"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Mistral Saba"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Mistral Small (Feb '24)"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Mistral Small (Sep '24)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2024-04-17","date_precision":"model_announcement","name":"Mixtral 8x22B Instruct"},{"date":"2023-12-11","date_precision":"model_announcement","name":"Mixtral 8x7B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"OLMo 2 32B"},{"date":"2024-11-26","date_precision":"model_announcement","name":"OLMo 2 7B"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2023-12-18","date_precision":"model_announcement","name":"OpenChat 3.5 (1210)"},{"date":"2024-04-23","date_precision":"model_announcement","name":"Phi-3 Mini Instruct 3.8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2025-02-26","date_precision":"model_announcement","name":"Phi-4 Multimodal Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-06-07","date_precision":"model_announcement","name":"Qwen2 Instruct 72B"},{"date":"2024-11-11","date_precision":"model_announcement","name":"Qwen2.5 Coder Instruct 32B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Coder Instruct 7B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 32B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-01-28","date_precision":"model_announcement","name":"Qwen2.5 Max"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Qwen2.5 Turbo"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ 32B"},{"date":"2024-11-27","date_precision":"model_announcement","name":"QwQ 32B-Preview"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Solar Pro 2 (Preview) (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Solar Pro 2 (Preview) (Reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Sonar"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Sonar Pro"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-humanitys-last-exam","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_humanmovevqa_d7611820"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"HumanMoveVQA evaluates video MLLMs on reasoning about human trajectory and orientation changes in videos, using a first-frame anchored world coordinate system and 10K question-answer pairs across seven reasoning categories.","descriptions":{"en":"HumanMoveVQA evaluates video MLLMs on reasoning about human trajectory and orientation changes in videos, using a first-frame anchored world coordinate system and 10K question-answer pairs across seven reasoning categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.27999","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","spatial-3d"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_humanmovevqa_d7611820"},"modality":null,"name":"HumanMoveVQA","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27999","source_url":"https://arxiv.org/abs/2606.27999"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27999","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27999","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":null,"source_record":"bm_humanoidarena_033240fc"},"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"HumanoidArena is a simulation benchmark for egocentric hierarchical whole-body learning, evaluating high-level policies that predict whole-body actions for low-level general motion trackers across seven leg-critical human-object and human-scene interaction tasks, with perturbation-conditioned and GMT-conditioned evaluation.","descriptions":{"en":"HumanoidArena is a simulation benchmark for egocentric hierarchical whole-body learning, evaluating high-level policies that predict whole-body actions for low-level general motion trackers across seven leg-critical human-object and human-scene interaction tasks, with perturbation-conditioned and GMT-conditioned evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17833","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_humanoidarena_033240fc"},"modality":null,"name":"HumanoidArena","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17833","source_url":"https://arxiv.org/abs/2606.17833"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17833","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17833","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_humanoidtoolbench_f054d5f7"},"categories":["Agents","Agents & Tool Use","Robotics & Embodied AI","Tool use"],"collected_at":"2026-10-14T09:44:51Z","description":"As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits.","descriptions":{"en":"As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.02089","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_humanoidtoolbench_f054d5f7"},"modality":null,"name":"HumanoidToolBench","openness":"unknown","publisher":null,"released":"2026-10-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.02089","source_url":"https://arxiv.org/abs/2610.02089"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-02089","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02089","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language navigation for humanoid robots across four embodiments in physics-grounded simulator scenarios. Includes 933 episodes with instructions and multiple stylistic variants, assessing success rate and normalized Dynamic Time Warping.","descriptions":{"en":"Evaluates vision-language navigation for humanoid robots across four embodiments in physics-grounded simulator scenarios. Includes 933 episodes with instructions and multiple stylistic variants, assessing success rate and normalized Dynamic Time Warping."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12860","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_humanoidvln_60fdcf09"},"modality":null,"name":"HumanoidVLN","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12860","source_url":"https://arxiv.org/abs/2608.12860"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12860","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12860","topics":["multimodal"],"unit":null},{"aliases":["humanwill-benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:humanwill-ai/humanwill-benchmark","source_record":"bm_humanwill-benchmark_3478d3aa"},"categories":["Cybersecurity","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"HumanWill is an assistance-focused evaluation over 424 cybersecurity questions spanning vulnerability validation, source-code security, binary, network and filesystem investigation. It scores false refusal and answer usefulness as separate judged dimensions under configurable per-topic policies, with run evidence and regrading support.","descriptions":{"en":"HumanWill is an assistance-focused evaluation over 424 cybersecurity questions spanning vulnerability validation, source-code security, binary, network and filesystem investigation. It scores false refusal and answer usefulness as separate judged dimensions under configurable per-topic policies, with run evidence and regrading support."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:humanwill-ai/humanwill-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_humanwill-benchmark_3478d3aa"},"modality":null,"name":"HumanWill Benchmark","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:humanwill-ai/humanwill-benchmark","source_url":"https://github.com/humanwill-ai/humanwill-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-humanwill-ai-humanwill-benchmark","source":"claire_radar","source_url":"https://github.com/humanwill-ai/humanwill-benchmark","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"HUSH-Bench evaluates conversational agents' use of sensitive history under a conservative policy, with 2,400 prompts and matched no-memory references, measuring unsolicited integration and memory access.","descriptions":{"en":"HUSH-Bench evaluates conversational agents' use of sensitive history under a conservative policy, with 2,400 prompts and matched no-memory references, measuring unsolicited integration and memory access."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06055","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","agent-security"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_hush-bench_669096c5"},"modality":null,"name":"HUSH-Bench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06055","source_url":"https://arxiv.org/abs/2606.06055"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06055","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06055","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Hy-MultiTurn is a Chinese benchmark for deep multi-turn dialogue understanding, with six controlled evaluation modes covering constraint memory, object localization, and action suppression across 12-76 turn dialogues.","descriptions":{"en":"Hy-MultiTurn is a Chinese benchmark for deep multi-turn dialogue understanding, with six controlled evaluation modes covering constraint memory, object localization, and action suppression across 12-76 turn dialogues."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.29196","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_hy-multiturn_edce774f"},"modality":null,"name":"Hy-MultiTurn","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29196","source_url":"https://arxiv.org/abs/2607.29196"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29196","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29196","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"HybridCodeAuthorship is a benchmark of Python files with interleaved human- and AI-authored lines, built from CodeSearchNet, for developing and evaluating AI-generated code detection algorithms at line- and chunk-level.","descriptions":{"en":"HybridCodeAuthorship is a benchmark of Python files with interleaved human- and AI-authored lines, built from CodeSearchNet, for developing and evaluating AI-generated code detection algorithms at line- and chunk-level."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12620","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_hybridcodeauthorship_7e92f56f"},"modality":null,"name":"HybridCodeAuthorship","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12620","source_url":"https://arxiv.org/abs/2606.12620"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12620","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12620","topics":[],"unit":null},{"aliases":["Benchmarking Hybrid Deep Research Across Database Querying and Web Search"],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":185,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:snowflake-ai-research/hybriddeepresearch","source_record":"bm_benchmarking-hybrid-deep-research-across-d_5787d02b"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates deep-research agents on 380 tasks requiring both SQL database querying and web search to produce a complete end-to-end answer across SQL-to-Search, Search-to-SQL, and Parallel reasoning patterns.","descriptions":{"en":"Evaluates deep-research agents on 380 tasks requiring both SQL database querying and web search to produce a complete end-to-end answer across SQL-to-Search, Search-to-SQL, and Parallel reasoning patterns."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.09410","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","code-generation"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_benchmarking-hybrid-deep-research-across-d_5787d02b"},"modality":null,"name":"HybridDeepResearch","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09410","source_url":"https://arxiv.org/abs/2609.09410"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09410","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09410","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_hyperalign-bench_25471ecc"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"HyperAlign-Bench is a benchmark introduced in a paper for evaluating hypergraph structural modeling in large language models, covering vertex-level and hyperedge-level tasks.","descriptions":{"en":"HyperAlign-Bench is a benchmark introduced in a paper for evaluating hypergraph structural modeling in large language models, covering vertex-level and hyperedge-level tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.21858","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_hyperalign-bench_25471ecc"},"modality":null,"name":"HyperAlign-Bench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.21858","source_url":"https://arxiv.org/abs/2605.21858"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-21858","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.21858","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":60,"join":"claire_record_id","repository":null,"source_record":"bm_hyperbrowsecomp_4ec97970"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13 languages, written by native or highly proficient speakers.","descriptions":{"en":"We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13 languages, written by native or highly proficient speakers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.03574","languages":[],"library_direction":{"basis":"model-review-daily","categories":["deep-research","multilingual"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_hyperbrowsecomp_4ec97970"},"modality":null,"name":"HyperBrowseComp","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03574","source_url":"https://arxiv.org/abs/2610.03574"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03574","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03574","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"HyperImageNet is a dataset of 26,084 airborne hyperspectral image patches with 224 spectral bands and 138 fine-grained land-cover categories, providing raw imagery, pixel-level semantic labels, and object-level instance masks for semantic and instance segmentation, along with an open-environment evaluation protocol.","descriptions":{"en":"HyperImageNet is a dataset of 26,084 airborne hyperspectral image patches with 224 spectral bands and 138 fine-grained land-cover categories, providing raw imagery, pixel-level semantic labels, and object-level instance masks for semantic and instance segmentation, along with an open-environment evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.21050","languages":[],"library_direction":null,"modality":null,"name":"HyperImageNet","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21050","source_url":"https://arxiv.org/abs/2607.21050"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21050","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21050","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":78,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:akshaysasi/hypershadow","source_record":"bm_hypershadow_31f52aa8"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"HyperShadow evaluates binary classification of 3D point clouds as either native 3D objects or 3D projections of objects in 4-6 spatial dimensions, with static and temporal tracks, four corruption tiers, and fixed train/eval splits.","descriptions":{"en":"HyperShadow evaluates binary classification of 3D point clouds as either native 3D objects or 3D projections of objects in 4-6 spatial dimensions, with static and temporal tracks, four corruption tiers, and fixed train/eval splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.14419","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_hypershadow_31f52aa8"},"modality":null,"name":"HyperShadow","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14419","source_url":"https://arxiv.org/abs/2607.14419"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14419","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14419","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2060,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_3fcadca8bbe60d1d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["spatial_reasoning","3d","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Hypersim evaluates 3D grounding and depth understanding in synthetic indoor scenes.","descriptions":{"en":"Hypersim evaluates 3D grounding and depth understanding in synthetic indoor scenes."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:hypersim:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hypersim?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:hypersim","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_3fcadca8bbe60d1d"},"modality":"image","name":"Hypersim","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":0.131,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.131,"raw_min":0.11,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:hypersim:qwen3.5-35b-a3b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hypersim?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-hypersim","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hypersim?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","综合能力","Comprehensive Capability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HypoBench, a novel benchmark designed to evaluate LLMs and hypothesis generation methods across multiple aspects, including practical utility, generalizability, and hypothesis discovery rate. HypoBench includes 7 real-world tasks and 5 synthetic tasks with 194 distinct datasets. HypoBench，这是一种新颖的基准，旨在从多个方面评估 LLM 和假设生成方法，包括实用性、泛化性和假设发现率。HypoBench 包括 7 个真实任务和 5 个合成任务，具有 194 个不同的数据集。","descriptions":{"en":"HypoBench, a novel benchmark designed to evaluate LLMs and hypothesis generation methods across multiple aspects, including practical utility, generalizability, and hypothesis discovery rate. HypoBench includes 7 real-world tasks and 5 synthetic tasks with 194 distinct datasets.","zh":"HypoBench，这是一种新颖的基准，旨在从多个方面评估 LLM 和假设生成方法，包括实用性、泛化性和假设发现率。HypoBench 包括 7 个真实任务和 5 个合成任务，具有 194 个不同的数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1782","languages":[],"library_direction":null,"modality":null,"name":"HypoBench","openness":"unknown","publisher":"University of Chicago， University of Toronto","released":"2025-04-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1782-hypobench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HypoBench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","语言生成","Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"HypoEval, Hypothesis-guided Evaluation framework, which first uses a small corpus of human evaluations to generate more detailed rubrics for human judgments and then incorporates a checklist-like approach to combine LLM's assigned scores on each decomposed dimension to acquire overall scores. HypoEval，即假设指导的评估框架，该框架首先使用一小部分人工评估来生成更详细的人类判断量规，然后采用类似清单的方法，将 LLM 在每个分解维度上的分配分数结合起来，以获得总分。","descriptions":{"en":"HypoEval, Hypothesis-guided Evaluation framework, which first uses a small corpus of human evaluations to generate more detailed rubrics for human judgments and then incorporates a checklist-like approach to combine LLM's assigned scores on each decomposed dimension to acquire overall scores.","zh":"HypoEval，即假设指导的评估框架，该框架首先使用一小部分人工评估来生成更详细的人类判断量规，然后采用类似清单的方法，将 LLM 在每个分解维度上的分配分数结合起来，以获得总分。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1786","languages":[],"library_direction":null,"modality":null,"name":"HypoEval","openness":"unknown","publisher":"University of Chicago","released":"2025-04-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1786-hypoeval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HypoEval","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of 19 research papers with expert-built interactive systems for evaluating agents that convert PDFs into executable web applications.","descriptions":{"en":"A benchmark of 19 research papers with expert-built interactive systems for evaluating agents that convert PDFs into executable web applications."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00750","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","document-ocr"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_i-webgenbench_56eafa1f"},"modality":null,"name":"I-WebGenBench","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00750","source_url":"https://arxiv.org/abs/2606.00750"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00750","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00750","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","NeurIPS 2024","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"IaC-Eval is meant for quantitatively evaluating the capabilities of LLMs in cloud IaC code generation, containing 458 questions ranging from simple to difficult across various cloud services. IaC-Eval用于定量评估LLM在云IaC代码生成中的功能，其中包含458个从易到难的问题，涵盖了各种云服务。","descriptions":{"en":"IaC-Eval is meant for quantitatively evaluating the capabilities of LLMs in cloud IaC code generation, containing 458 questions ranging from simple to difficult across various cloud services.","zh":"IaC-Eval用于定量评估LLM在云IaC代码生成中的功能，其中包含458个从易到难的问题，涵盖了各种云服务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1320","languages":[],"library_direction":null,"modality":null,"name":"IaC-Eval","openness":"open","publisher":"University of Michigan","released":"2024-09-26","released_reference":{"basis":"paper_first_version","note":"OpenReview lists September 26 as the introducing paper's public publication date; private submission and later modification dates are not used.","source_key":"opencompass:1320","source_url":"https://openreview.net/forum?id=7TCK0aBL1C"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1320-iac-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IaC-Eval","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"IBA-Bench evaluates LLM agents on implicit behavioral alignment using longitudinal interaction histories, across nine application domains.","descriptions":{"en":"IBA-Bench evaluates LLM agents on implicit behavioral alignment using longitudinal interaction histories, across nine application domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02171","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_iba-bench_e7767345"},"modality":null,"name":"IBA-Bench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02171","source_url":"https://arxiv.org/abs/2608.02171"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02171","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02171","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":8,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":"gh:alex-nlp/icae-eval","source_record":"bm_icae-bench_617decda"},"categories":["Agents & Tool Use","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ICAE-Bench evaluates coding agents on interactive project-building tasks. Agents receive a fuzzy product requirement and must clarify missing details via an automated user agent, then implement the project in a container. Scoring uses black-box tests and multi-dimensional diagnostics including functional correctness, semantic/API similarity, structural fidelity, design quality, and interaction quality.","descriptions":{"en":"ICAE-Bench evaluates coding agents on interactive project-building tasks. Agents receive a fuzzy product requirement and must clarify missing details via an automated user agent, then implement the project in a container. Scoring uses black-box tests and multi-dimensional diagnostics including functional correctness, semantic/API similarity, structural fidelity, design quality, and interaction quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.21217","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_icae-bench_617decda"},"modality":null,"name":"ICAE-Bench","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21217","source_url":"https://arxiv.org/abs/2607.21217"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21217","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21217","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code generation","Multimodal","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Interactive chart code generation requires models to reproduce a reference chart's appearance and underlying data and correctly implement the state changes triggered by specified user interactions.","descriptions":{"en":"Interactive chart code generation requires models to reproduce a reference chart's appearance and underlying data and correctly implement the state changes triggered by specified user interactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32413","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_ichart2code_9c61ed21"},"modality":null,"name":"IChart2Code","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32413","source_url":"https://arxiv.org/abs/2609.32413"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32413","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32413","topics":["coding","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of 660 evidence-grounded instances evaluating codification-readiness assessment, defect localization, and clarification action generation.","descriptions":{"en":"A benchmark of 660 evidence-grounded instances evaluating codification-readiness assessment, defect localization, and clarification action generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10539","languages":[],"library_direction":null,"modality":null,"name":"IdeaAMBIG","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10539","source_url":"https://arxiv.org/abs/2609.10539"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10539","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10539","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":32,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":37,"join":"claire_record_id","repository":"gh:visionxlab/ideashavegenomes","source_record":"bm_ideagene-bench_3d711bf3"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates scientific lineage reasoning and lineage-grounded idea generation through two tracks: closed-form IG-Exam and generation IG-Arena with Population-Evolution Score.","descriptions":{"en":"Evaluates scientific lineage reasoning and lineage-grounded idea generation through two tracks: closed-form IG-Exam and generation IG-Arena with Population-Evolution Score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.08758","languages":[],"library_direction":{"basis":"model-review","categories":["science","ai-rnd"],"join":"claire_record_id","primary":"science","source_record":"bm_ideagene-bench_3d711bf3"},"modality":null,"name":"IdeaGene-Bench","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08758","source_url":"https://arxiv.org/abs/2607.08758"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08758","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08758","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ideal3d/ideal-bench","source_record":"bm_ideal-bench_dab673c0"},"categories":["General AI","Geometric reasoning","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"IDEAL-Bench evaluates Vision-Language Models on holistic 3D layout inference from single images of indoor scenes, scoring predictions across five numerical dimensions and a perceptual render-and-compare protocol. It uses a procedurally generated dataset of 1,000 re-renderable Blender environments.","descriptions":{"en":"IDEAL-Bench evaluates Vision-Language Models on holistic 3D layout inference from single images of indoor scenes, scoring predictions across five numerical dimensions and a perceptual render-and-compare protocol. It uses a procedurally generated dataset of 1,000 re-renderable Blender environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.03614","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_ideal-bench_dab673c0"},"modality":null,"name":"IDEAL-Bench","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03614","source_url":"https://arxiv.org/abs/2607.03614"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03614","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03614","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_idiolink_97ac8121"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"IdioLink is a retrieval benchmark with 10,700 documents and 2,140 queries across 107 idioms, testing the linking of idiomatic expressions to literal or paraphrased equivalents. Annotations mark core meaning spans, and scoring is based on retrieval quality against these spans.","descriptions":{"en":"IdioLink is a retrieval benchmark with 10,700 documents and 2,140 queries across 107 idioms, testing the linking of idiomatic expressions to literal or paraphrased equivalents. Annotations mark core meaning spans, and scoring is based on retrieval quality against these spans."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22247","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","deep-research"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_idiolink_97ac8121"},"modality":null,"name":"IdioLink","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22247","source_url":"https://arxiv.org/abs/2605.22247"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22247","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22247","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tisl-lab/interdependent_privacy_bench","source_record":"bm_idp-bench_f6df4252"},"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"IDP-Bench evaluates large language models on interdependent privacy scenarios, covering recognition of co-ownership, identification of contextual integrity parameters, and judgments of sharing appropriateness.","descriptions":{"en":"IDP-Bench evaluates large language models on interdependent privacy scenarios, covering recognition of co-ownership, identification of contextual integrity parameters, and judgments of sharing appropriateness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09908","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","safety-alignment"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_idp-bench_f6df4252"},"modality":null,"name":"IDP-Bench","openness":"unknown","publisher":null,"released":"2026-06-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09908","source_url":"https://arxiv.org/abs/2606.09908"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09908","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09908","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Agents built on large language models (LLMs) build and run software through tool calls.","descriptions":{"en":"Agents built on large language models (LLMs) build and run software through tool calls."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.04375","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agentic-coding","tool-use"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_iec-bench_e8d216fe"},"modality":null,"name":"IEC-Bench","openness":"unknown","publisher":null,"released":"2026-10-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.04375","source_url":"https://arxiv.org/abs/2610.04375"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-04375","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04375","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":38887,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":22,"join":"llm_stats_id","repository":null,"source_record":"catalog_935f68319d4f227e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["structured_output","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints","descriptions":{"en":"Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:if:mistral-small-3.2-24b-instruct-2506","reported_at":"2025-06-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/if?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:if","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_935f68319d4f227e"},"modality":"text","name":"IF","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":84.78,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.8478,"raw_min":0.72,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:if:mistral-small-3.2-24b-instruct-2506","reported_date":"2025-06-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/if?top_n=500"},"unit":null},"scored_models":[{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"}],"slug":"llm-stats-if","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/if?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["instruction-following"],"collected_at":"2026-08-25T10:41:06Z","description":"Instruction following","descriptions":{"en":"Instruction following"},"evidence_summary":{"document_count":1,"model_count":450,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:ifbench:6a7c0e25-1dcb-4b15-8495-a8536a9da051","reported_at":"2023-03-14","source_url":"https://artificialanalysis.ai/evaluations/ifbench"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:ifbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["instruction-following"],"join":"review:14H034160212:2026-10-15","primary":"instruction-following","source_record":null},"modality":null,"name":"IFBench","openness":"unknown","publisher":null,"released":"2025-07-03","released_reference":{"basis":"paper_first_version","note":"First version of Generalizing Verifiable Instruction Following introduces IFBench and its 58 new constraint types.","source_key":"artificial-analysis:ifbench","source_url":"https://arxiv.org/abs/2507.02833"},"repo_kind":null,"repo_resolution_status":null,"score_count":450,"score_direction":"higher_is_better","score_summary":{"display_max":83.3333333333333,"display_multiplier":100,"model_count":450,"model_count_basis":"source_model_id","numeric_count":450,"raw_max":0.833333333333333,"raw_min":0.12108843537415,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:ifbench:5e8b0d98-a3b4-42b5-93d8-ecb748788754","reported_date":"2026-04-30","source_url":"https://artificialanalysis.ai/evaluations/ifbench"},"unit":null},"scored_models":[{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2023-03-14","date_precision":"model_announcement","name":"GPT-4"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (ChatGPT)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2025-10-30","date_precision":"model_announcement","name":"Kimi Linear 48B A3B Instruct"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 4B v1.1 (Reasoning)"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 3B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 40k"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2 (Jul '24)"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"OLMo 2 32B"},{"date":"2024-11-26","date_precision":"model_announcement","name":"OLMo 2 7B"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2024-04-23","date_precision":"model_announcement","name":"Phi-3 Mini Instruct 3.8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ 32B"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-ifbench","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/ifbench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_ff770459cce56302"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["instruction_following","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Instruction Following Benchmark evaluating model's ability to follow complex instructions","descriptions":{"en":"Instruction Following Benchmark evaluating model's ability to follow complex instructions"},"evidence_summary":{"document_count":1,"model_count":34,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ifbench:hermes-3-70b","reported_at":"2024-08-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ifbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:ifbench","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_ff770459cce56302"},"modality":"text","name":"IFBench","openness":"open","publisher":"Allen Institute for AI","released":"2025-07-03","released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":34,"score_direction":"higher_is_better","score_summary":{"display_max":82.8,"display_multiplier":100,"model_count":34,"model_count_basis":"source_model_id","numeric_count":34,"raw_max":0.828,"raw_min":0.21,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ifbench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ifbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B High"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE-236B-A23B"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Sonic"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-ifbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ifbench?top_n=500","topics":["language"],"unit":null},{"aliases":["IFBench"],"attention_signals":null,"categories":["instruction_following"],"collected_at":null,"description":"Verifiable constraints on unseen instruction types; does not measure answer quality.","descriptions":{"en":"Verifiable constraints on unseen instruction types; does not measure answer quality."},"evidence_summary":{"document_count":3,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000ifbench\u0000qwen3_5_model_card\u0000ifbench\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000ifbench\u0000qwen3_5_model_card\u0000ifbench\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:ifbench","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"artifact:arxiv:2507.02833","primary":"instruction-following","source_record":"catalog_3ab41c34d802c79d"},"modality":null,"name":"IFBench","openness":"unknown","publisher":null,"released":"2025-07-03","released_reference":{"basis":"benchmark_release","source_key":"model-reports:ifbench","source_url":"https://arxiv.org/abs/2507.02833"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":76.5,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":76.5,"raw_min":76.5,"source_reference":{"obs_id":"curated\u0000ifbench\u0000qwen3_5_model_card\u0000ifbench\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000ifbench\u0000qwen3_5_model_card\u0000ifbench\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"ifbench","source":"model_reports","source_url":"https://arxiv.org/abs/2507.02833","topics":["language"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"IFCMemoryBench evaluates long-term memory in LLM-based agents for BIM information retrieval, with 143 multi-session tasks across 19 projects and 4,016 prior sessions, requiring integration of remembered context with live IFC queries.","descriptions":{"en":"IFCMemoryBench evaluates long-term memory in LLM-based agents for BIM information retrieval, with 143 multi-session tasks across 19 projects and 4,016 prior sessions, requiring integration of remembered context with live IFC queries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.26072","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_ifcmemorybench_f2c4e9c3"},"modality":null,"name":"IFCMemoryBench","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26072","source_url":"https://arxiv.org/abs/2607.26072"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26072","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26072","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":38909,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":22,"join":"llm_stats_id","repository":null,"source_record":"lib_ifeval"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["structured_output","instruction_following","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints","descriptions":{"en":"Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints"},"editorial_guide":{"id":"ifeval","record_keys":["model-reports:ifeval","llm-stats:ifeval","opencompass:1136"],"reviewed_at":"2026-10-25","scope":{"en":"Original Instruction-Following Evaluation with verifiable constraints and its four strict/loose, prompt/instruction metrics; not a general answer-quality judge.","zh":"对应原始 Instruction-Following Evaluation 的可验证约束，以及严格／宽松、提示级／指令级四种指标；不作为通用回答质量评分。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"The paper defines 25 instruction types and around 500 prompts, each containing one or more requirements. A response may need to meet several constraints simultaneously, such as a word-count condition and a specified ending.","zh":"论文定义 25 种指令类型，构造约 500 条提示，每条含一项或多项要求。例如，回答可能需要同时满足字数条件和指定结尾。"},"title":{"en":"Constraints that a program can check","zh":"程序能够检查的约束"}},{"id":"grading","source_ids":["evaluator"],"text":{"en":"The official evaluator retrieves each instruction's checking function and applies it to the response. It records both individual pass/fail decisions and whether all instructions pass. Empty responses fail the checks rather than receiving assumed compliance.","zh":"官方评测器调用每条指令对应的检查函数，保存逐项是否通过以及是否全部通过。空白输出不会被视为符合要求，而会检查失败。"},"title":{"en":"Checkers replace a subjective judge","zh":"用检查器核对要求"}},{"id":"levels","source_ids":["paper","evaluator"],"text":{"en":"Prompt-level accuracy requires every constraint in a prompt to pass. Instruction-level accuracy counts constraints individually. A model can therefore improve individual compliance while still failing many complete prompts; report which denominator the number uses.","zh":"提示级正确率要求一条提示中的所有约束都通过；指令级正确率分别统计每项约束。模型可能逐项表现改善，却仍有许多整条提示未通过，因此必须注明统计分母。"},"title":{"en":"Prompt-level versus instruction-level","zh":"提示级与指令级"}},{"id":"strict-loose","source_ids":["paper","evaluator"],"text":{"en":"Strict checking uses the original response. Loose checking also tries prescribed transformations, such as removing formatting or introductory lines. This can recover false negatives, but may create false positives; it complements rather than replaces the strict result.","zh":"严格判定检查原始回答；宽松判定还尝试移除格式或开头说明等预设变换。它可能纠正误拒，也可能引入误收，适合与严格结果一起阅读，而不是替代后者。"},"title":{"en":"Strict and loose are different checks","zh":"严格与宽松并非同一个判定"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Passing a length or keyword requirement does not establish factual accuracy, relevance or sound reasoning. The authors identify instruction diversity and real-world applicability as areas to expand. Combine this signal with task-specific evaluation when selecting a model.","zh":"满足长度或关键词要求，不能证明事实正确、内容相关或推理可靠。作者也把指令多样性与真实应用贴合度列为扩展方向。选择模型时，应与具体任务评测结合。"},"title":{"en":"Compliance is not the whole answer","zh":"遵循约束不等于回答全面合格"}}],"sources":[{"id":"paper","title":"Instruction-Following Evaluation for Large Language Models · Metrics and limitations","url":"https://arxiv.org/html/2311.07911v1"},{"id":"evaluator","title":"Google Research · IFEval strict and loose checking implementation","url":"https://github.com/google-research/google-research/blob/master/instruction_following_eval/evaluation_lib.py"}],"summary":{"en":"IFEval checks concrete requirements such as output format, length or required words. It is useful for separating measurable instruction compliance from whether an answer is informative or correct.","zh":"IFEval 检查输出格式、长度、指定词语等具体要求，帮助区分可测量的指令遵循与回答本身是否有用、正确。"},"title":"IFEval"},"evidence_summary":{"document_count":1,"model_count":67,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ifeval:llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ifeval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:ifeval","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"lib_ifeval"},"modality":"text","name":"IFEval","openness":"unknown","publisher":"Google","released":"2023-11-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":67,"score_direction":"higher_is_better","score_summary":{"display_max":95.0,"display_multiplier":100,"model_count":67,"model_count_basis":"source_model_id","numeric_count":67,"raw_max":0.95,"raw_min":0.44,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ifeval:qwen3.5-27b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ifeval?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-01-20","date_precision":"model_announcement","name":"Kimi-k1.5"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 8B V1"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 70B Instruct"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Instruct"},{"date":"2025-08-18","date_precision":"model_announcement","name":"Nemotron Nano 9B v2"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ-32B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"}],"slug":"llm-stats-ifeval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ifeval?top_n=500","topics":["language"],"unit":null},{"aliases":["IFEval"],"attention_signals":null,"categories":["instruction_following"],"collected_at":null,"description":"Verifiable-constraint subset only; does not measure answer quality.","descriptions":{"en":"Verifiable-constraint subset only; does not measure answer quality."},"editorial_guide":{"id":"ifeval","record_keys":["model-reports:ifeval","llm-stats:ifeval","opencompass:1136"],"reviewed_at":"2026-10-25","scope":{"en":"Original Instruction-Following Evaluation with verifiable constraints and its four strict/loose, prompt/instruction metrics; not a general answer-quality judge.","zh":"对应原始 Instruction-Following Evaluation 的可验证约束，以及严格／宽松、提示级／指令级四种指标；不作为通用回答质量评分。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"The paper defines 25 instruction types and around 500 prompts, each containing one or more requirements. A response may need to meet several constraints simultaneously, such as a word-count condition and a specified ending.","zh":"论文定义 25 种指令类型，构造约 500 条提示，每条含一项或多项要求。例如，回答可能需要同时满足字数条件和指定结尾。"},"title":{"en":"Constraints that a program can check","zh":"程序能够检查的约束"}},{"id":"grading","source_ids":["evaluator"],"text":{"en":"The official evaluator retrieves each instruction's checking function and applies it to the response. It records both individual pass/fail decisions and whether all instructions pass. Empty responses fail the checks rather than receiving assumed compliance.","zh":"官方评测器调用每条指令对应的检查函数，保存逐项是否通过以及是否全部通过。空白输出不会被视为符合要求，而会检查失败。"},"title":{"en":"Checkers replace a subjective judge","zh":"用检查器核对要求"}},{"id":"levels","source_ids":["paper","evaluator"],"text":{"en":"Prompt-level accuracy requires every constraint in a prompt to pass. Instruction-level accuracy counts constraints individually. A model can therefore improve individual compliance while still failing many complete prompts; report which denominator the number uses.","zh":"提示级正确率要求一条提示中的所有约束都通过；指令级正确率分别统计每项约束。模型可能逐项表现改善，却仍有许多整条提示未通过，因此必须注明统计分母。"},"title":{"en":"Prompt-level versus instruction-level","zh":"提示级与指令级"}},{"id":"strict-loose","source_ids":["paper","evaluator"],"text":{"en":"Strict checking uses the original response. Loose checking also tries prescribed transformations, such as removing formatting or introductory lines. This can recover false negatives, but may create false positives; it complements rather than replaces the strict result.","zh":"严格判定检查原始回答；宽松判定还尝试移除格式或开头说明等预设变换。它可能纠正误拒，也可能引入误收，适合与严格结果一起阅读，而不是替代后者。"},"title":{"en":"Strict and loose are different checks","zh":"严格与宽松并非同一个判定"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Passing a length or keyword requirement does not establish factual accuracy, relevance or sound reasoning. The authors identify instruction diversity and real-world applicability as areas to expand. Combine this signal with task-specific evaluation when selecting a model.","zh":"满足长度或关键词要求，不能证明事实正确、内容相关或推理可靠。作者也把指令多样性与真实应用贴合度列为扩展方向。选择模型时，应与具体任务评测结合。"},"title":{"en":"Compliance is not the whole answer","zh":"遵循约束不等于回答全面合格"}}],"sources":[{"id":"paper","title":"Instruction-Following Evaluation for Large Language Models · Metrics and limitations","url":"https://arxiv.org/html/2311.07911v1"},{"id":"evaluator","title":"Google Research · IFEval strict and loose checking implementation","url":"https://github.com/google-research/google-research/blob/master/instruction_following_eval/evaluation_lib.py"}],"summary":{"en":"IFEval checks concrete requirements such as output format, length or required words. It is useful for separating measurable instruction compliance from whether an answer is informative or correct.","zh":"IFEval 检查输出格式、长度、指定词语等具体要求，帮助区分可测量的指令遵循与回答本身是否有用、正确。"},"title":"IFEval"},"evidence_summary":{"document_count":9,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000ifeval\u0000deepseek_v3_report\u0000ifeval\u0000Prompt Strict\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"first_score_reported_at":"2024-12-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000ifeval\u0000deepseek_v3_report\u0000ifeval\u0000Prompt Strict\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:ifeval","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["instruction-following"],"join":"review:14H034160212:2026-10-15","primary":"instruction-following","source_record":null},"modality":null,"name":"IFEval","openness":"unknown","publisher":null,"released":"2023-11-14","released_reference":{"basis":"benchmark_release","source_key":"model-reports:ifeval","source_url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":92.6,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":92.6,"raw_min":83.3,"source_reference":{"obs_id":"curated\u0000ifeval\u0000qwen3_5_model_card\u0000ifeval\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000ifeval\u0000qwen3_5_model_card\u0000ifeval\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2025-05-14","date_precision":"document_publication","name":"Qwen3-235B-A22B (Thinking)"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"ifeval","source":"model_reports","source_url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval","topics":["language"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["指令跟随","Instruct","大语言模型","LLM","指令遵循","Instruction Following","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"IFEval is a straightforward and easy-to reproduce evaluation benchmark. It focuses on a set of “verifiable instructions” such as “write in more than 400 words” and “mention the keyword of AI at least 3 times”. IFEval 是一个简单且易于复现的评估基准。它关注一组“可验证的指令”，例如“写超过 400 个单词”和“至少提到关键词 AI 3 次”。","descriptions":{"en":"IFEval is a straightforward and easy-to reproduce evaluation benchmark. It focuses on a set of “verifiable instructions” such as “write in more than 400 words” and “mention the keyword of AI at least 3 times”.","zh":"IFEval 是一个简单且易于复现的评估基准。它关注一组“可验证的指令”，例如“写超过 400 个单词”和“至少提到关键词 AI 3 次”。"},"editorial_guide":{"id":"ifeval","record_keys":["model-reports:ifeval","llm-stats:ifeval","opencompass:1136"],"reviewed_at":"2026-10-25","scope":{"en":"Original Instruction-Following Evaluation with verifiable constraints and its four strict/loose, prompt/instruction metrics; not a general answer-quality judge.","zh":"对应原始 Instruction-Following Evaluation 的可验证约束，以及严格／宽松、提示级／指令级四种指标；不作为通用回答质量评分。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"The paper defines 25 instruction types and around 500 prompts, each containing one or more requirements. A response may need to meet several constraints simultaneously, such as a word-count condition and a specified ending.","zh":"论文定义 25 种指令类型，构造约 500 条提示，每条含一项或多项要求。例如，回答可能需要同时满足字数条件和指定结尾。"},"title":{"en":"Constraints that a program can check","zh":"程序能够检查的约束"}},{"id":"grading","source_ids":["evaluator"],"text":{"en":"The official evaluator retrieves each instruction's checking function and applies it to the response. It records both individual pass/fail decisions and whether all instructions pass. Empty responses fail the checks rather than receiving assumed compliance.","zh":"官方评测器调用每条指令对应的检查函数，保存逐项是否通过以及是否全部通过。空白输出不会被视为符合要求，而会检查失败。"},"title":{"en":"Checkers replace a subjective judge","zh":"用检查器核对要求"}},{"id":"levels","source_ids":["paper","evaluator"],"text":{"en":"Prompt-level accuracy requires every constraint in a prompt to pass. Instruction-level accuracy counts constraints individually. A model can therefore improve individual compliance while still failing many complete prompts; report which denominator the number uses.","zh":"提示级正确率要求一条提示中的所有约束都通过；指令级正确率分别统计每项约束。模型可能逐项表现改善，却仍有许多整条提示未通过，因此必须注明统计分母。"},"title":{"en":"Prompt-level versus instruction-level","zh":"提示级与指令级"}},{"id":"strict-loose","source_ids":["paper","evaluator"],"text":{"en":"Strict checking uses the original response. Loose checking also tries prescribed transformations, such as removing formatting or introductory lines. This can recover false negatives, but may create false positives; it complements rather than replaces the strict result.","zh":"严格判定检查原始回答；宽松判定还尝试移除格式或开头说明等预设变换。它可能纠正误拒，也可能引入误收，适合与严格结果一起阅读，而不是替代后者。"},"title":{"en":"Strict and loose are different checks","zh":"严格与宽松并非同一个判定"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Passing a length or keyword requirement does not establish factual accuracy, relevance or sound reasoning. The authors identify instruction diversity and real-world applicability as areas to expand. Combine this signal with task-specific evaluation when selecting a model.","zh":"满足长度或关键词要求，不能证明事实正确、内容相关或推理可靠。作者也把指令多样性与真实应用贴合度列为扩展方向。选择模型时，应与具体任务评测结合。"},"title":{"en":"Compliance is not the whole answer","zh":"遵循约束不等于回答全面合格"}}],"sources":[{"id":"paper","title":"Instruction-Following Evaluation for Large Language Models · Metrics and limitations","url":"https://arxiv.org/html/2311.07911v1"},{"id":"evaluator","title":"Google Research · IFEval strict and loose checking implementation","url":"https://github.com/google-research/google-research/blob/master/instruction_following_eval/evaluation_lib.py"}],"summary":{"en":"IFEval checks concrete requirements such as output format, length or required words. It is useful for separating measurable instruction compliance from whether an answer is informative or correct.","zh":"IFEval 检查输出格式、长度、指定词语等具体要求，帮助区分可测量的指令遵循与回答本身是否有用、正确。"},"title":"IFEval"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1136","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"identity_group:ifeval","primary":"instruction-following","source_record":"lib_ifeval"},"modality":null,"name":"IFEval","openness":"unknown","publisher":"Google","released":"2023-11-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1136-ifeval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IFEval","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"IFHierBench is a hierarchical instruction-following benchmark with 600 prompts and deterministic checkers, evaluating LLMs on satisfying constraints at different output scopes. It measures prompt-level accuracy.","descriptions":{"en":"IFHierBench is a hierarchical instruction-following benchmark with 600 prompts and deterministic checkers, evaluating LLMs on satisfying constraints at different output scopes. It measures prompt-level accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27912","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_ifhierbench_c5c33d5e"},"modality":null,"name":"IFHierBench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27912","source_url":"https://arxiv.org/abs/2607.27912"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27912","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27912","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["指令跟随","Instruct","大语言模型","LLM","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"IFIR is the first comprehensive benchmark designed to evaluate instruction-following information retrieval (IR) in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature. IFIR，这是第一个旨在评估专家领域指令跟随信息检索（IR）的综合基准。IFIR 包含 2,426 个高质量示例，涵盖四个专业领域（金融、法律、医疗保健和科学文献）的八个子集。","descriptions":{"en":"IFIR is the first comprehensive benchmark designed to evaluate instruction-following information retrieval (IR) in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature.","zh":"IFIR，这是第一个旨在评估专家领域指令跟随信息检索（IR）的综合基准。IFIR 包含 2,426 个高质量示例，涵盖四个专业领域（金融、法律、医疗保健和科学文献）的八个子集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1617","languages":[],"library_direction":null,"modality":null,"name":"IFIR","openness":"unknown","publisher":"UCAS, ZJU, etc.","released":"2025-03-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1617-ifir","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IFIR","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1275,"github_stars_30d":676,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_ifmtbench_b83f8c4f"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multilingual translation systems on instruction following across seven languages, six constraint types, and compositional multi-constraint requests.","descriptions":{"en":"Evaluates multilingual translation systems on instruction following across seven languages, six constraint types, and compositional multi-constraint requests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.28218","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","instruction-following"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_ifmtbench_b83f8c4f"},"modality":null,"name":"IFMTBench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28218","source_url":"https://arxiv.org/abs/2605.28218"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28218","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28218","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robustness","Safety & Trustworthiness","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"IH-Benchmark evaluates instruction-hierarchy robustness in LLMs via conflicting instructions from system, user, and tool outputs, covering 44 constraint families across five domains with a binary pass/fail protocol.","descriptions":{"en":"IH-Benchmark evaluates instruction-hierarchy robustness in LLMs via conflicting instructions from system, user, and tool outputs, covering 44 constraint families across five domains with a binary pass/fail protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.25987","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","agent-security"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_ih-benchmark_f309c884"},"modality":null,"name":"IH-Benchmark","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25987","source_url":"https://arxiv.org/abs/2607.25987"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25987","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25987","topics":["agents","language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_ihbench_0d580e81"},"categories":["General AI","Speech & Audio","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"IHBench evaluates post-interruption recovery in voice agents executing state-machine-driven workflows across 10 enterprise domains. It scores task fulfillment and recovery quality for six interruption types.","descriptions":{"en":"IHBench evaluates post-interruption recovery in voice agents executing state-machine-driven workflows across 10 enterprise domains. It scores task fulfillment and recovery quality for six interruption types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19595","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","speech-audio"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_ihbench_0d580e81"},"modality":null,"name":"IHBench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19595","source_url":"https://arxiv.org/abs/2606.19595"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19595","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19595","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Software & AI Compute","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"II-Bench evaluates computer-use agents against low-harm adversarial tasks across three platforms, with 444 examples and a testing framework.","descriptions":{"en":"II-Bench evaluates computer-use agents against low-harm adversarial tasks across three platforms, with 444 examples and a testing framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02018","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","computer-use"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_ii-bench_6645329e"},"modality":null,"name":"II-Bench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02018","source_url":"https://arxiv.org/abs/2608.02018"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02018","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02018","topics":["computer-use","agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates text-to-image models on 1,140 prompts with 8,842 combined rules, independently varying instance count and rule constraints to measure compositional instruction following.","descriptions":{"en":"Evaluates text-to-image models on 1,140 prompts with 8,842 combined rules, independently varying instance count and rule constraints to measure compositional instruction following."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29210","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following","visual-reasoning","image-video-generation"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_imag-eval_e81e6086"},"modality":null,"name":"Imag-Eval","openness":"unknown","publisher":null,"released":"2026-08-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29210","source_url":"https://arxiv.org/abs/2608.29210"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29210","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29210","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","code","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Image2FloorPlan is an in-house benchmark evaluating multimodal models on generating structured floor plans and interactive frontends directly from images such as design mockups and room photos.","descriptions":{"en":"Image2FloorPlan is an in-house benchmark evaluating multimodal models on generating structured floor plans and interactive frontends directly from images such as design mockups and room photos."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:image2floorplan:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/image2floorplan?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:image2floorplan","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_92baf138f8914ddd"},"modality":"multimodal","name":"Image2FloorPlan","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":48.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.48,"raw_min":0.359,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:image2floorplan:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/image2floorplan?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-image2floorplan","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/image2floorplan?top_n=500","topics":["coding","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":14,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_087ec402f23016a9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ImageMining evaluates multimodal models on extracting structured information from images using tool use, measuring ability to combine visual understanding with tool-based retrieval and analysis.","descriptions":{"en":"ImageMining evaluates multimodal models on extracting structured information from images using tool use, measuring ability to combine visual understanding with tool-based retrieval and analysis."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:imagemining:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imagemining?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:imagemining","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","tool-use"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_087ec402f23016a9"},"modality":"multimodal","name":"ImageMining","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":30.7,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.307,"raw_min":0.307,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:imagemining:glm-5v-turbo","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imagemining?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"}],"slug":"llm-stats-imagemining","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imagemining?top_n=500","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge","Planning"],"collected_at":"2026-10-14T09:44:51Z","description":"Imaging-101 evaluates coding agents on 57 computational imaging tasks across six scientific domains, with three tracks for planning, function-level unit tests, and end-to-end reconstruction.","descriptions":{"en":"Imaging-101 evaluates coding agents on 57 computational imaging tasks across six scientific domains, with three tracks for planning, function-level unit tests, and end-to-end reconstruction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.10789","languages":[],"library_direction":{"basis":"model-review","categories":["science","agentic-coding"],"join":"claire_record_id","primary":"science","source_record":"bm_imaging-101_1fefb66e"},"modality":null,"name":"Imaging-101","openness":"unknown","publisher":null,"released":"2026-07-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10789","source_url":"https://arxiv.org/abs/2607.10789"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10789","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10789","topics":["coding","agents","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ImagingBench evaluates agentic AI systems on 20 computational imaging tasks spanning ray and wave optics, image signal processing, inverse reconstruction, computational sensing, and calibration, across three settings: Expert, Planner, and Forward.","descriptions":{"en":"ImagingBench evaluates agentic AI systems on 20 computational imaging tasks spanning ray and wave optics, image signal processing, inverse reconstruction, computational sensing, and calibration, across three settings: Expert, Planner, and Forward."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.07189","languages":[],"library_direction":{"basis":"model-review","categories":["science","agentic-coding"],"join":"claire_record_id","primary":"science","source_record":"bm_imagingbench_e9f34716"},"modality":null,"name":"ImagingBench","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.07189","source_url":"https://arxiv.org/abs/2607.07189"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-07189","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07189","topics":["coding","agents","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_imbench_d224375e"},"categories":["Reasoning","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"IMBench evaluates intuitive robotic manipulation, combining perception, physical reasoning, action generation, and execution across 35 tasks and 14K trajectories.","descriptions":{"en":"IMBench evaluates intuitive robotic manipulation, combining perception, physical reasoning, action generation, and execution across 35 tasks and 14K trajectories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.15641","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_imbench_d224375e"},"modality":null,"name":"IMBench","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15641","source_url":"https://arxiv.org/abs/2607.15641"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15641","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15641","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"IMCBench evaluates multimodal LLMs in image-grounded, multi-turn medical conversations, scoring safety, accuracy, and uncertainty use on a 1-5 scale.","descriptions":{"en":"IMCBench evaluates multimodal LLMs in image-grounded, multi-turn medical conversations, scoring safety, accuracy, and uncertainty use on a 1-5 scale."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.28556","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_imcbench_04071d33"},"modality":null,"name":"IMCBench","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.28556","source_url":"https://arxiv.org/abs/2606.28556"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-28556","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28556","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","NeurIPS 2024","多模态模型","VLM","安全对齐","Safety Alignment","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"IMDL-BenCo offers a comprehensive IMDL benchmark and modular codebase. IMDL-BenCo提供了全面的IMDL基准测试和模块化代码库。","descriptions":{"en":"IMDL-BenCo offers a comprehensive IMDL benchmark and modular codebase.","zh":"IMDL-BenCo提供了全面的IMDL基准测试和模块化代码库。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1274","languages":[],"library_direction":null,"modality":"multimodal","name":"IMDL-BenCo","openness":"unknown","publisher":"Sichuan University","released":"2024-06-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1274-imdl-benco","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IMDL-BenCo","topics":["multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"iMINDBench evaluates models on a shared suite of fifteen decoding tasks across three naturalistic movie-watching iEEG datasets.","descriptions":{"en":"iMINDBench evaluates models on a shared suite of fifteen decoding tasks across three naturalistic movie-watching iEEG datasets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.18104","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_imindbench_0b74642d"},"modality":null,"name":"iMINDBench","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18104","source_url":"https://arxiv.org/abs/2609.18104"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18104","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18104","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eth-sri/matharena","source_record":"catalog_7ceb189d6fece95f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"IMO 2025 evaluates models on the six problems from the 2025 International Mathematical Olympiad, requiring rigorous proof-based reasoning. Following the MathArena methodology, proofs are graded against human expert rubrics by dual strong judge models, with the minimum of the two scores taken as final. Maximum score is 42 points.","descriptions":{"en":"IMO 2025 evaluates models on the six problems from the 2025 International Mathematical Olympiad, requiring rigorous proof-based reasoning. Following the MathArena methodology, proofs are graded against human expert rubrics by dual strong judge models, with the minimum of the two scores taken as final. Maximum score is 42 points."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:imo-2025:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-2025?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:imo-2025","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_7ceb189d6fece95f"},"modality":"text","name":"IMO 2025","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":35.0,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":35.0,"raw_min":0.652,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:imo-2025:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-2025?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"}],"slug":"llm-stats-imo-2025","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-2025?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":805,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_9f633d8c67b1c89b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"IMO-AnswerBench is a benchmark for evaluating mathematical reasoning capabilities on International Mathematical Olympiad (IMO) problems, focusing on answer generation and verification.","descriptions":{"en":"IMO-AnswerBench is a benchmark for evaluating mathematical reasoning capabilities on International Mathematical Olympiad (IMO) problems, focusing on answer generation and verification."},"evidence_summary":{"document_count":1,"model_count":20,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:imo-answerbench:kimi-k2-thinking-0905","reported_at":"2025-09-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-answerbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:imo-answerbench","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_9f633d8c67b1c89b"},"modality":"text","name":"IMO-AnswerBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"monorepo_subdir","repo_resolution_status":"resolved","score_count":20,"score_direction":"higher_is_better","score_summary":{"display_max":92.30000000000001,"display_multiplier":100,"model_count":20,"model_count_basis":"source_model_id","numeric_count":20,"raw_max":0.923,"raw_min":0.783,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:imo-answerbench:nemotron-3-ultra-550b-a55b","reported_date":"2026-06-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-answerbench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step-3.5-Flash"}],"slug":"llm-stats-imo-answerbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-answerbench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["IMOAnswerBench","IMO AnswerBench"],"attention_signals":null,"categories":["math"],"collected_at":null,"description":"Final-answer grading on olympiad problems, so it does not check whether the proof reasoning was valid.","descriptions":{"en":"Final-answer grading on olympiad problems, so it does not check whether the proof reasoning was valid."},"evidence_summary":{"document_count":7,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000imo_answer_bench\u0000zai_glm_5_model_card\u0000imo_answer_bench\u0000temp 1.0, top_p 0.95, GPT-5.2 (medium) judge\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"first_score_reported_at":"2026-02-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000imo_answer_bench\u0000zai_glm_5_model_card\u0000imo_answer_bench\u0000temp 1.0, top_p 0.95, GPT-5.2 (medium) judge\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:imo_answer_bench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["math"],"join":"review:14H034160212:2026-10-15","primary":"math","source_record":null},"modality":null,"name":"IMOAnswerBench","openness":"unknown","publisher":null,"released":"2025-09-18","released_reference":{"basis":"benchmark_release","source_key":"model-reports:imo_answer_bench","source_url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench"},"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":91.0,"display_multiplier":1,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":91.0,"raw_min":72.5,"source_reference":{"obs_id":"curated\u0000imo_answer_bench\u0000zai_glm_5_2_model_card\u0000imo_answer_bench\u0000temp 1.0, top_p 0.95, max gen 163840 tokens, GPT-5.5 (medium) judge\u0000GLM-5.2","observation_id":"curated\u0000imo_answer_bench\u0000zai_glm_5_2_model_card\u0000imo_answer_bench\u0000temp 1.0, top_p 0.95, max gen 163840 tokens, GPT-5.5 (medium) judge\u0000GLM-5.2","reported_at":"2026-06-16","reported_date":"2026-06-16","source_id":"zai_glm_5_2_model_card","source_url":"https://huggingface.co/zai-org/GLM-5.2"},"unit":"percent"},"scored_models":[{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"imo_answer_bench","source":"model_reports","source_url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":805,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_50e3578d8fbe5eba"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"IMOProof-Adv is an advanced benchmark of International Mathematical Olympiad-style proof problems requiring rigorous multi-step mathematical proofs.","descriptions":{"en":"IMOProof-Adv is an advanced benchmark of International Mathematical Olympiad-style proof problems requiring rigorous multi-step mathematical proofs."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:imoproof-adv:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imoproof-adv?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:imoproof-adv","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_50e3578d8fbe5eba"},"modality":"text","name":"IMOProof-Adv","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":54.300000000000004,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.543,"raw_min":0.543,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:imoproof-adv:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imoproof-adv?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"}],"slug":"llm-stats-imoproof-adv","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imoproof-adv?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"IMPLICIT-Bench measures implicit bias in text-to-image models under neutral prompts.","descriptions":{"en":"IMPLICIT-Bench measures implicit bias in text-to-image models under neutral prompts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24228","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","image-video-generation","safety-alignment"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_implicit-bench_e8ad5959"},"modality":null,"name":"IMPLICIT-Bench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24228","source_url":"https://arxiv.org/abs/2609.24228"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24228","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24228","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark of 169 impossible tasks spanning six impossibility categories, each paired with a verifiable oracle certificate and 48 answerable controls.","descriptions":{"en":"Benchmark of 169 impossible tasks spanning six impossibility categories, each paired with a verifiable oracle certificate and 48 answerable controls."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16816","languages":[],"library_direction":null,"modality":null,"name":"ImpossibleRubrics","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16816","source_url":"https://arxiv.org/abs/2609.16816"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16816","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16816","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_imug-bench_282ebfce"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"IMUG-Bench evaluates unified multimodal models on multi-turn interleaved image-text understanding and generation. It includes 3,113 samples and 12,034 interaction turns across Static Spatial, Temporal Causal, and Hybrid classes, with dynamic understanding questions.","descriptions":{"en":"IMUG-Bench evaluates unified multimodal models on multi-turn interleaved image-text understanding and generation. It includes 3,113 samples and 12,034 interaction turns across Static Spatial, Temporal Causal, and Hybrid classes, with dynamic understanding questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09169","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","visual-reasoning"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_imug-bench_282ebfce"},"modality":null,"name":"IMUG-Bench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09169","source_url":"https://arxiv.org/abs/2606.09169"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09169","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09169","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":7208,"hf_paper_upvotes":16,"join":"llm_stats_id","repository":null,"source_record":"catalog_c0cbff04cc90f5b4"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Include benchmark - specific documentation not found in official sources","descriptions":{"en":"Include benchmark - specific documentation not found in official sources"},"evidence_summary":{"document_count":1,"model_count":31,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:include:qwen3-235b-a22b","reported_at":"2025-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/include?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:include","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_c0cbff04cc90f5b4"},"modality":"text","name":"Include","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"not_found","repo_resolution_status":"not_found","score_count":31,"score_direction":"higher_is_better","score_summary":{"display_max":87.6,"display_multiplier":100,"model_count":31,"model_count_basis":"source_model_id","numeric_count":31,"raw_max":0.876,"raw_min":0.386,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:include:claude-opus-4-8","reported_date":"2026-05-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/include?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-include","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/include?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates disability-related bias in text-to-image models using 119K generated images across bias dimensions and contexts, with the Stereotype Content Model Score.","descriptions":{"en":"Evaluates disability-related bias in text-to-image models using 119K generated images across bias dimensions and contexts, with the Stereotype Content Model Score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.08515","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","image-video-generation"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_include-bench_e92f5105"},"modality":null,"name":"INCLUDE-BENCH","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08515","source_url":"https://arxiv.org/abs/2607.08515"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08515","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08515","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"","descriptions":{},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:community:07c9946d-dcf0-4977-a640-a6b1356b4f0b","languages":[],"library_direction":null,"modality":null,"name":"independence-bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":"higher_is_better","score_summary":{"display_max":null,"display_multiplier":1,"model_count":null,"model_count_basis":null,"numeric_count":0,"raw_max":null,"raw_min":null,"source_reference":null,"unit":null},"scored_models":[],"slug":"llm-stats-community-07c9946d-dcf0-4977-a640-a6b1356b4f0b","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A07c9946d-dcf0-4977-a640-a6b1356b4f0b?top_n=500","topics":[],"unit":null},{"aliases":["indian-agent-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:imaginationai31-sys/indian-agent-benchmark","source_record":"bm_indian-agent-benchmark_f04be610"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"A synthetic multilingual benchmark evaluating computer-use agents on browser-style e-commerce tasks across seven Indian languages, with expected action trajectories, 0-100 task success scoring, and automatic failure-mode classification.","descriptions":{"en":"A synthetic multilingual benchmark evaluating computer-use agents on browser-style e-commerce tasks across seven Indian languages, with expected action trajectories, 0-100 task success scoring, and automatic failure-mode classification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:imaginationai31-sys/indian-agent-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","computer-use"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_indian-agent-benchmark_f04be610"},"modality":null,"name":"Indian Agent Benchmark","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:imaginationai31-sys/indian-agent-benchmark","source_url":"https://github.com/imaginationai31-sys/indian-agent-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-imaginationai31-sys-indian-agent-benchmark","source":"claire_radar","source_url":"https://github.com/imaginationai31-sys/indian-agent-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_indic-diarbench_c5a01338"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Multilingual joint diarization and ASR benchmark for 22 Indian languages, with ~108 hours of human-corrected multi-speaker audio from meetings, far-field, and in-the-wild sources, including code-mixing and overlaps.","descriptions":{"en":"Multilingual joint diarization and ASR benchmark for 22 Indian languages, with ~108 hours of human-corrected multi-speaker audio from meetings, far-field, and in-the-wild sources, including code-mixing and overlaps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.23808","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_indic-diarbench_c5a01338"},"modality":null,"name":"Indic DiarBench","openness":"unknown","publisher":null,"released":"2026-07-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23808","source_url":"https://arxiv.org/abs/2607.23808"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23808","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23808","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"A 799-case benchmark for Indian retail banking spanning five operational domains and twenty primary axes.","descriptions":{"en":"A 799-case benchmark for Indian retail banking spanning five operational domains and twenty primary axes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.29167","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","finance-legal"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_indicbankbench_e8a1a1b1"},"modality":null,"name":"IndicBankBench","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.29167","source_url":"https://arxiv.org/abs/2609.29167"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-29167","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29167","topics":["safety","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models. IndicMMLU-Pro 提供了一个标准化的评估框架，以推动印度语系语言 AI 的研究边界，促进更准确、高效和具有文化敏感性的模型的发展。","descriptions":{"en":"IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models.","zh":"IndicMMLU-Pro 提供了一个标准化的评估框架，以推动印度语系语言 AI 的研究边界，促进更准确、高效和具有文化敏感性的模型的发展。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1670","languages":[],"library_direction":null,"modality":null,"name":"IndicMMLU-Pro","openness":"restricted","publisher":"Artificial Intelligence Institute, University of South Carolina, etc.","released":"2025-01-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1670-indicmmlu-pro","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IndicMMLU-Pro","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_industrial-dexterity-benchmark_210a10b0"},"categories":["Multimodal","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The Industrial Dexterity Benchmark (IDB) provides physical boards and tasks for industrial dexterous manipulation, including cable management, cable harness, and gearbox assembly. It evaluates a robot's ability to perform tasks such as cable insertion and grasping via end-to-end imitation learning policies.","descriptions":{"en":"The Industrial Dexterity Benchmark (IDB) provides physical boards and tasks for industrial dexterous manipulation, including cable management, cable harness, and gearbox assembly. It evaluates a robot's ability to perform tasks such as cable insertion and grasping via end-to-end imitation learning policies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14021","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_industrial-dexterity-benchmark_210a10b0"},"modality":null,"name":"Industrial Dexterity Benchmark","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14021","source_url":"https://arxiv.org/abs/2607.14021"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14021","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14021","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:xiaoqi-7/industrialvla-bench","source_record":"bm_industrialvla-bench_f2bdeae4"},"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates six open robot policy models on manipulation capability, robustness, language grounding, and deployability.","descriptions":{"en":"Evaluates six open robot policy models on manipulation capability, robustness, language grounding, and deployability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.25562","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics","engineering-design"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_industrialvla-bench_f2bdeae4"},"modality":null,"name":"IndustrialVLA-Bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25562","source_url":"https://arxiv.org/abs/2609.25562"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25562","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25562","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":12,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:alibaba-multimodal-industrial-ai/industrybench-mipu","source_record":"bm_industrybench-mipu_6704f5ae"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"IndustryBench-MIPU evaluates MLLMs on extracting structured attribute-value pairs from multi-image industrial product data, covering text recognition, visual reasoning, and cross-image integration.","descriptions":{"en":"IndustryBench-MIPU evaluates MLLMs on extracting structured attribute-value pairs from multi-image industrial product data, covering text recognition, visual reasoning, and cross-image integration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.14383","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","document-ocr"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_industrybench-mipu_6704f5ae"},"modality":null,"name":"IndustryBench-MIPU","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14383","source_url":"https://arxiv.org/abs/2606.14383"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14383","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14383","topics":[],"unit":null},{"aliases":["inference-bench"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:cmcconnell1/inference-bench","source_record":"bm_inference-bench_5b3e9ae3"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates six LLM serving engines (vLLM, SGLang, TGI, Ollama, llama.cpp, NVIDIA NIM) on latency, throughput, and cost per million tokens using fixed workloads and guidellm on Akamai Cloud GPUs.","descriptions":{"en":"Evaluates six LLM serving engines (vLLM, SGLang, TGI, Ollama, llama.cpp, NVIDIA NIM) on latency, throughput, and cost per million tokens using fixed workloads and guidellm on Akamai Cloud GPUs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:cmcconnell1/inference-bench","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_inference-bench_5b3e9ae3"},"modality":null,"name":"inference-bench: serving engine benchmarks on Akamai Cloud GPUs","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:cmcconnell1/inference-bench","source_url":"https://github.com/cmcconnell1/inference-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-cmcconnell1-inference-bench","source":"claire_radar","source_url":"https://github.com/cmcconnell1/inference-bench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Quantum Computing & Control","quant-ph"],"collected_at":"2026-10-14T09:44:51Z","description":"InferQ is a database-oriented benchmark for quantum circuit simulation, generating compositional circuits and emitting SQL workloads with feature extraction for workload characterization.","descriptions":{"en":"InferQ is a database-oriented benchmark for quantum circuit simulation, generating compositional circuits and emitting SQL workloads with feature extraction for workload characterization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.29134","languages":[],"library_direction":null,"modality":null,"name":"InferQ","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29134","source_url":"https://arxiv.org/abs/2607.29134"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29134","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29134","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","NeurIPS 2024","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"InfiBench is a large-scale freeform question-answering (QA) benchmark for code to our knowledge, comprising 234 carefully selected high-quality Stack Overflow questions that span across 15 programming languages. InfiBench用于评测LLM回答代码相关问题的能力，包括涵盖15种编程语言的234个精心挑选的高质量Stack Overflow问题。","descriptions":{"en":"InfiBench is a large-scale freeform question-answering (QA) benchmark for code to our knowledge, comprising 234 carefully selected high-quality Stack Overflow questions that span across 15 programming languages.","zh":"InfiBench用于评测LLM回答代码相关问题的能力，包括涵盖15种编程语言的234个精心挑选的高质量Stack Overflow问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1322","languages":[],"library_direction":null,"modality":null,"name":"InfiBench","openness":"open","publisher":"Simon Fraser University","released":"2024-03-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1322-infibench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InfiBench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":393,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_edf5904501cf14de"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"InfiniteBench English Multiple Choice variant - first LLM benchmark featuring average data length surpassing 100K tokens for evaluating long-context capabilities with 12 tasks spanning diverse domains","descriptions":{"en":"InfiniteBench English Multiple Choice variant - first LLM benchmark featuring average data length surpassing 100K tokens for evaluating long-context capabilities with 12 tasks spanning diverse domains"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:infinitebench-en.mc:llama-3.2-3b-instruct","reported_at":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.mc?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:infinitebench-en.mc","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_edf5904501cf14de"},"modality":"text","name":"InfiniteBench/En.MC","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":63.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.633,"raw_min":0.633,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:infinitebench-en.mc:llama-3.2-3b-instruct","reported_date":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.mc?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"}],"slug":"llm-stats-infinitebench-en-mc","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.mc?top_n=500","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":393,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_5f09a49df1dd41c3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"InfiniteBench English Question Answering variant - first LLM benchmark featuring average data length surpassing 100K tokens for evaluating long-context capabilities with 12 tasks spanning diverse domains","descriptions":{"en":"InfiniteBench English Question Answering variant - first LLM benchmark featuring average data length surpassing 100K tokens for evaluating long-context capabilities with 12 tasks spanning diverse domains"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:infinitebench-en.qa:llama-3.2-3b-instruct","reported_at":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.qa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:infinitebench-en.qa","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_5f09a49df1dd41c3"},"modality":"text","name":"InfiniteBench/En.QA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":19.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.198,"raw_min":0.198,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:infinitebench-en.qa:llama-3.2-3b-instruct","reported_date":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.qa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"}],"slug":"llm-stats-infinitebench-en-qa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.qa?top_n=500","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["指令跟随","Instruct","ACL 2024","大语言模型","LLM","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"InfoBench is a benchmark comprising 500 diverse instructions and 2,250 decomposed questions across multiple constraint categories. InfoBench 是一个指令追随评测基准，包含 500 条多样化的指令和 2,250 个分解问题，涵盖多个约束类别。","descriptions":{"en":"InfoBench is a benchmark comprising 500 diverse instructions and 2,250 decomposed questions across multiple constraint categories.","zh":"InfoBench 是一个指令追随评测基准，包含 500 条多样化的指令和 2,250 个分解问题，涵盖多个约束类别。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1085","languages":[],"library_direction":null,"modality":null,"name":"InfoBench","openness":"open","publisher":"Tencent AI Lab","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1085-infobench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InfoBench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_infoedit_18fe7954"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Multimodal foundation models edit natural photographs at production quality, yet the same models struggle with structured visual content such as infographics.","descriptions":{"en":"Multimodal foundation models edit natural photographs at production quality, yet the same models struggle with structured visual content such as infographics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33286","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_infoedit_18fe7954"},"modality":null,"name":"InfoEdit","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33286","source_url":"https://arxiv.org/abs/2609.33286"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33286","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33286","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_154f69cfb551a849"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"InfographicVQA dataset with 5,485 infographic images and over 30,000 questions requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills","descriptions":{"en":"InfographicVQA dataset with 5,485 infographic images and over 30,000 questions requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:infographicsqa:llama-3.2-90b-instruct","reported_at":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infographicsqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:infographicsqa","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_154f69cfb551a849"},"modality":"multimodal","name":"InfographicsQA","openness":"unknown","publisher":null,"released":"2021-04-26","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: InfographicVQA. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:infographicsqa","source_url":"https://arxiv.org/abs/2104.12756"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":56.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.568,"raw_min":0.568,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:infographicsqa:llama-3.2-90b-instruct","reported_date":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infographicsqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"}],"slug":"llm-stats-infographicsqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infographicsqa?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"InfoOps Bench is an active, constantly updated benchmark measuring the integrity of frontier language models against co-optation for information operations. It uses real examples from a live monitoring pipeline and tests 17 models from 8 providers.","descriptions":{"en":"InfoOps Bench is an active, constantly updated benchmark measuring the integrity of frontier language models against co-optation for information operations. It uses real examples from a live monitoring pipeline and tests 17 models from 8 providers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.28503","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_infoops-bench_c0a89e01"},"modality":null,"name":"InfoOps Bench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28503","source_url":"https://arxiv.org/abs/2607.28503"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28503","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28503","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_0280788a4587ff1c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"InfoVQA dataset with 30,000 questions and 5,000 infographic images requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills","descriptions":{"en":"InfoVQA dataset with 30,000 questions and 5,000 infographic images requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills"},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:infovqa:deepseek-vl2","reported_at":"2024-12-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:infovqa","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_0280788a4587ff1c"},"modality":"multimodal","name":"InfoVQA","openness":"unknown","publisher":null,"released":"2021-04-26","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: InfographicVQA. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:infovqa","source_url":"https://arxiv.org/abs/2104.12756"},"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":83.39999999999999,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.834,"raw_min":0.5,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:infovqa:qwen2.5-vl-32b","reported_date":"2025-02-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-infovqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqa?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_2c3e90c7ee7e7bbd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"InfoVQA test set with infographic images requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills","descriptions":{"en":"InfoVQA test set with infographic images requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills"},"evidence_summary":{"document_count":1,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:infovqatest:qwen2-vl-72b","reported_at":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqatest?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:infovqatest","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_2c3e90c7ee7e7bbd"},"modality":"multimodal","name":"InfoVQAtest","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":12,"score_direction":"higher_is_better","score_summary":{"display_max":92.60000000000001,"display_multiplier":100,"model_count":12,"model_count_basis":"source_model_id","numeric_count":12,"raw_max":0.926,"raw_min":0.803,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:infovqatest:kimi-k2.5","reported_date":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqatest?top_n=500"},"unit":null},"scored_models":[{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-infovqatest","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqatest?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_infrabench_aa520bfd"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"InfraBench evaluates AI agents on realistic infrastructure tasks across the system stack and lifecycle, with fine-grained per-check risk assessment and a public leaderboard.","descriptions":{"en":"InfraBench evaluates AI agents on realistic infrastructure tasks across the system stack and lifecycle, with fine-grained per-check risk assessment and a public leaderboard."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.11234","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","agent-security"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_infrabench_aa520bfd"},"modality":null,"name":"InfraBench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11234","source_url":"https://arxiv.org/abs/2608.11234"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11234","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11234","topics":["coding","agents","safety"],"unit":null},{"aliases":["InfraOcc: An Infrastructure Occupancy Benchmark with Static-to-Dynamic Reasoning"],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yanglei18/infraocc","source_record":"bm_infraocc-an-infrastructure-occupancy-bench_993fa65f"},"categories":["Cybersecurity","Factuality","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates roadside 3D semantic occupancy on 290 multi-modal sequences with static-dynamic annotations, supporting camera, LiDAR, and fusion inputs under a unified scoring protocol.","descriptions":{"en":"Evaluates roadside 3D semantic occupancy on 290 multi-modal sequences with static-dynamic annotations, supporting camera, LiDAR, and fusion inputs under a unified scoring protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.30657","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","spatial-3d"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_infraocc-an-infrastructure-occupancy-bench_993fa65f"},"modality":null,"name":"InfraOcc","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30657","source_url":"https://arxiv.org/abs/2608.30657"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30657","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30657","topics":["reasoning-math","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fdu-ins/ins-actbench","source_record":"bm_ins-actbench_3f617677"},"categories":["Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"INS-ActBench evaluates actuarial capability in LLMs across knowledge, case reasoning, and tool-based practice with 12,050 tasks from public exams.","descriptions":{"en":"INS-ActBench evaluates actuarial capability in LLMs across knowledge, case reasoning, and tool-based practice with 12,050 tasks from public exams."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.24273","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","knowledge-expert-qa"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_ins-actbench_3f617677"},"modality":null,"name":"INS-ActBench","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24273","source_url":"https://arxiv.org/abs/2607.24273"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24273","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24273","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decision tasks.","descriptions":{"en":"Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decision tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.09671","languages":[],"library_direction":{"basis":"model-review-daily","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_insclaimbench_f1d93dad"},"modality":null,"name":"InsClaimBench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.09671","source_url":"https://arxiv.org/abs/2610.09671"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-09671","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09671","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:maevehutch/insight","source_record":"bm_insight_8c435b2a"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal agents on claim verification over interactive visualizations, requiring interactions to classify propositions as true, false, or not enough info.","descriptions":{"en":"Evaluates multimodal agents on claim verification over interactive visualizations, requiring interactions to classify propositions as true, false, or not enough info."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.01383","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","computer-use"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_insight_8c435b2a"},"modality":null,"name":"InSight","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01383","source_url":"https://arxiv.org/abs/2609.01383"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01383","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01383","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"INSPIRE evaluates instruction-aware speech retrieval, where natural-language instructions specify relevance criteria including semantic content, speaker identity, speaking style, environmental sounds, and combinations. The benchmark includes a fixed dataset and protocol for evaluating retrieval systems.","descriptions":{"en":"INSPIRE evaluates instruction-aware speech retrieval, where natural-language instructions specify relevance criteria including semantic content, speaker identity, speaking style, environmental sounds, and combinations. The benchmark includes a fixed dataset and protocol for evaluating retrieval systems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.16203","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","deep-research"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_inspire_4ebdcda6"},"modality":null,"name":"INSPIRE","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16203","source_url":"https://arxiv.org/abs/2608.16203"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16203","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16203","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"Scientific literature search often begins with an open research problem rather than a known target paper or a fixed candidate set.","descriptions":{"en":"Scientific literature search often begins with an open research problem rather than a known target paper or a fixed candidate set."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33233","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","science"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_inspire_d70ec552"},"modality":null,"name":"Inspire","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33233","source_url":"https://arxiv.org/abs/2609.33233"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33233","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33233","topics":["finance","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Referring Expression Segmentation (RES) links natural-language descriptions to pixel-level object masks.","descriptions":{"en":"Referring Expression Segmentation (RES) links natural-language descriptions to pixel-level object masks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.09478","languages":[],"library_direction":{"basis":"model-review-daily","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_instancebench_86a66d53"},"modality":null,"name":"InstanceBench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.09478","source_url":"https://arxiv.org/abs/2610.09478"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-09478","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09478","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1063,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"llm_stats_id","repository":"gh:bigcode-project/bigcode-evaluation-harness","source_record":"catalog_d1a33ae8e098bd46"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Instruction-based variant of HumanEval benchmark for evaluating large language models' code generation capabilities with functional correctness using pass@k metric on programming problems","descriptions":{"en":"Instruction-based variant of HumanEval benchmark for evaluating large language models' code generation capabilities with functional correctness using pass@k metric on programming problems"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:instruct-humaneval:llama-3.1-nemotron-70b-instruct","reported_at":"2024-10-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/instruct-humaneval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:instruct-humaneval","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_d1a33ae8e098bd46"},"modality":"text","name":"Instruct HumanEval","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":73.83999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.7384,"raw_min":0.7384,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:instruct-humaneval:llama-3.1-nemotron-70b-instruct","reported_date":"2024-10-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/instruct-humaneval?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"}],"slug":"llm-stats-instruct-humaneval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/instruct-humaneval?top_n=500","topics":[],"unit":null},{"aliases":["Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation"],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-09-25T04:40:22Z","description":"A repeatable evaluation of instruction following as multiple verifier-checked constraints accumulate in a prompt.","descriptions":{"en":"A repeatable evaluation of instruction following as multiple verifier-checked constraints accumulate in a prompt."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02639","languages":[],"library_direction":null,"modality":null,"name":"Instruction Stacking Collapse","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02639","source_url":"https://arxiv.org/abs/2608.02639"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02639","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02639","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":18,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:horizonrobotics/roboorchardsim","source_record":"bm_instructmove_3a1c4064"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates instruction-following in robot manipulation through pick-and-place tasks requiring language grounding across category, attribute, spatial, and compositional skills.","descriptions":{"en":"Evaluates instruction-following in robot manipulation through pick-and-place tasks requiring language grounding across category, attribute, spatial, and compositional skills."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.22990","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_instructmove_3a1c4064"},"modality":null,"name":"InstructMove","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22990","source_url":"http://arxiv.org/abs/2608.22990v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22990","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22990v1","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","NAACL 2024","大语言模型","LLM","语言理解","Comprehension","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"InstruSum evaluates the task of instruction controllable text summarization, where the model input consists of both a source article and a natural language requirement for desired summary characteristics. InstruSum 是评测指令可控的文本摘要任务，模型输入包括源文章和对所需摘要特征的自然语言要求。","descriptions":{"en":"InstruSum evaluates the task of instruction controllable text summarization, where the model input consists of both a source article and a natural language requirement for desired summary characteristics.","zh":"InstruSum 是评测指令可控的文本摘要任务，模型输入包括源文章和对所需摘要特征的自然语言要求。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1149","languages":[],"library_direction":null,"modality":null,"name":"InstruSum","openness":"unknown","publisher":"Allen Institute for AI","released":"2024-06-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1149-instrusum","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InstruSum","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_insufficiencybench_debc0b74"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates whether LLMs recognize legally material missing information in underspecified queries and refrain from premature conclusions.","descriptions":{"en":"Evaluates whether LLMs recognize legally material missing information in underspecified queries and refrain from premature conclusions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.20220","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","factuality"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_insufficiencybench_debc0b74"},"modality":null,"name":"InsufficiencyBench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.20220","source_url":"https://arxiv.org/abs/2608.20220"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-20220","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20220","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Int-Bench is a simulation-based evaluation for LLM intervention behavior in tutoring. It simulates a student solving problems across code debugging, mathematics, and brain teasers, with a teacher deciding whether, when, and how to intervene, measuring frequency, timing, and impact on task success and generalization.","descriptions":{"en":"Int-Bench is a simulation-based evaluation for LLM intervention behavior in tutoring. It simulates a student solving problems across code debugging, mathematics, and brain teasers, with a teacher deciding whether, when, and how to intervene, measuring frequency, timing, and impact on task success and generalization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.21306","languages":[],"library_direction":null,"modality":null,"name":"Int-Bench","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21306","source_url":"https://arxiv.org/abs/2607.21306"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21306","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21306","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"IntegrityBench evaluates language models on research integrity tasks, including misconduct classification, ethical action reasoning, and artifact-grounded decision making, across 36 paired tasks with a 5-level pressure protocol spanning multiple domains and research stages.","descriptions":{"en":"IntegrityBench evaluates language models on research integrity tasks, including misconduct classification, ethical action reasoning, and artifact-grounded decision making, across 36 paired tasks with a 5-level pressure protocol spanning multiple domains and research stages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12345","languages":[],"library_direction":{"basis":"model-review-2","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_integritybench_9bf5a285"},"modality":null,"name":"IntegrityBench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12345","source_url":"https://arxiv.org/abs/2608.12345"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12345","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12345","topics":["reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:koenv759/vanillasft","source_record":"bm_intentbench-prime_088c7112"},"categories":["Multimodal","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Intentbench-Prime is a cleaned subset of the IntentBench audio-visual social QA benchmark with broken and trivially answerable questions removed.","descriptions":{"en":"Intentbench-Prime is a cleaned subset of the IntentBench audio-visual social QA benchmark with broken and trivially answerable questions removed."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.13239","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_intentbench-prime_088c7112"},"modality":null,"name":"Intentbench-Prime","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13239","source_url":"https://arxiv.org/abs/2608.13239"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13239","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13239","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"IntentionNav evaluates active object search from implicit human instructions in 176 Isaac Sim scenes. Episodes provide free-text intent, RGB-D observations, and pose, with the target object withheld. The benchmark includes 500 intents over 64 categories and four intent modes.","descriptions":{"en":"IntentionNav evaluates active object search from implicit human instructions in 176 Isaac Sim scenes. Episodes provide free-text intent, RGB-D observations, and pose, with the target object withheld. The benchmark includes 500 intents over 64 categories and four intent modes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.23187","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_intentionnav_63acc227"},"modality":null,"name":"IntentionNav","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23187","source_url":"https://arxiv.org/abs/2605.23187"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23187","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23187","topics":[],"unit":null},{"aliases":["Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Large-scale benchmark with 11,388 interaction sequences for multimodal human-human interaction analysis, covering four task categories in generation and perception.","descriptions":{"en":"Large-scale benchmark with 11,388 interaction sequences for multimodal human-human interaction analysis, covering four task categories in generation and perception."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.20312","languages":[],"library_direction":{"basis":"model-review-2","categories":["image-video-generation","video-understanding"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_inter-x-a-comprehensive-benchmark-for-mult_a4cc7946"},"modality":null,"name":"Inter-X++","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.20312","source_url":"https://arxiv.org/abs/2608.20312"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-20312","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20312","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:kmsgk0/interactbench","source_record":"bm_interactbench_f60331ac"},"categories":["Code & Software","Code generation","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"InteractBench evaluates LLM code generation on interactive competitive programming problems where solutions must query a judge to obtain hidden information under protocol constraints.","descriptions":{"en":"InteractBench evaluates LLM code generation on interactive competitive programming problems where solutions must query a judge to obtain hidden information under protocol constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29632","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_interactbench_f60331ac"},"modality":null,"name":"InteractBench","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29632","source_url":"https://arxiv.org/abs/2608.29632"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29632","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29632","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Information retrieval","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Existing reactive Graphical User Interface (GUI) agents often fail in long-horizon, dynamic scenarios, where unexpected disturbances trigger attention-diverting and cascading failures.","descriptions":{"en":"Existing reactive Graphical User Interface (GUI) agents often fail in long-horizon, dynamic scenarios, where unexpected disturbances trigger attention-diverting and cascading failures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.36923","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_interferebench_31aa67c6"},"modality":null,"name":"InterfereBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36923","source_url":"https://arxiv.org/abs/2609.36923"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36923","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36923","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"InterFLOPBench is a benchmark of 90 C kernels and 1,130 test samples for evaluating LLMs on floating-point error classification across six categories: cancellation, comparison, division by zero, overflow, underflow, and NaN.","descriptions":{"en":"InterFLOPBench is a benchmark of 90 C kernels and 1,130 test samples for evaluating LLMs on floating-point error classification across six categories: cancellation, comparison, division by zero, overflow, underflow, and NaN."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.31308","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_interflopbench_d649e042"},"modality":null,"name":"InterFLOPBench","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31308","source_url":"https://arxiv.org/abs/2606.31308"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31308","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31308","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":178,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_900df6bdc7f9459d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","spatial_reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Interpretable Geometry Problem Solver (Inter-GPS) with Geometry3K dataset of 3,002 geometry problems with dense annotation in formal language using theorem knowledge and symbolic reasoning","descriptions":{"en":"Interpretable Geometry Problem Solver (Inter-GPS) with Geometry3K dataset of 3,002 geometry problems with dense annotation in formal language using theorem knowledge and symbolic reasoning"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:intergps:phi-3.5-vision-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/intergps?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:intergps","languages":[],"library_direction":{"basis":"model-review","categories":["math","visual-reasoning"],"join":"llm_stats_id","primary":"math","source_record":"catalog_900df6bdc7f9459d"},"modality":"text","name":"InterGPS","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":48.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.486,"raw_min":0.363,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:intergps:phi-4-multimodal-instruct","reported_date":"2025-02-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/intergps?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-vision-instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"}],"slug":"llm-stats-intergps","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/intergps?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["structured_output","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Internal API instruction following (hard) benchmark - specific documentation not found in official sources","descriptions":{"en":"Internal API instruction following (hard) benchmark - specific documentation not found in official sources"},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:internal-api-instruction-following-(hard):gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-api-instruction-following-%28hard%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:internal-api-instruction-following-(hard)","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_2fbf5fa976d9cd7d"},"modality":"text","name":"Internal API instruction following (hard)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":64.0,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.64,"raw_min":0.292,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:internal-api-instruction-following-(hard):gpt-5-2025-08-07","reported_date":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-api-instruction-following-%28hard%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"}],"slug":"llm-stats-internal-api-instruction-following-hard","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-api-instruction-following-%28hard%29?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The Internal Research Debugging Evaluation measures whether models can debug 41 real bugs from internal OpenAI research experiments (plus alignment-auditing tasks), where the original solutions took experienced researchers hours to days. Passing corresponds to providing assistance that would unblock the user, including partial root-cause explanations or fixes.","descriptions":{"en":"The Internal Research Debugging Evaluation measures whether models can debug 41 real bugs from internal OpenAI research experiments (plus alignment-auditing tasks), where the original solutions took experienced researchers hours to days. Passing corresponds to providing assistance that would unblock the user, including partial root-cause explanations or fixes."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:internal-research-debugging-evaluation:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-research-debugging-evaluation?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:internal-research-debugging-evaluation","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_228095751750f019"},"modality":"text","name":"Internal Research Debugging Evaluation","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":68.30000000000001,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.683,"raw_min":0.508,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:internal-research-debugging-evaluation:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-research-debugging-evaluation?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-internal-research-debugging-evaluation","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-research-debugging-evaluation?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","video","物理智能","Embodied AI","跨模态推理","Cross-modal Reasoning","空间理解","Spatial Understanding","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"InternData-A1: A hybrid synthetic-real manipulation dataset integrating 5 heterogeneous robots, 15 skills, and 200+ scenes, emphasizing multi-robot collaboration under dynamic scenarios. InternData-A1：一个融合了 5 种异构机器人、15 项技能和 200+ 场景的混合合成-真实操作数据集，重点关注动态场景下的多机器人协作。","descriptions":{"en":"InternData-A1: A hybrid synthetic-real manipulation dataset integrating 5 heterogeneous robots, 15 skills, and 200+ scenes, emphasizing multi-robot collaboration under dynamic scenarios.","zh":"InternData-A1：一个融合了 5 种异构机器人、15 项技能和 200+ 场景的混合合成-真实操作数据集，重点关注动态场景下的多机器人协作。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"opencompass:2151","languages":[],"library_direction":null,"modality":"multimodal","name":"InternData-A1","openness":"restricted","publisher":"InternRobotics","released":"2025-07-26","released_reference":{"basis":"dataset_published","note":"The official dataset card explicitly states its release date.","source_key":"opencompass:2151","source_url":"https://huggingface.co/datasets/InternRobotics/InternData-A1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2151-interndata-a1","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InternData-A1","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","物理智能","Embodied AI","跨模态推理","Cross-modal Reasoning","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"InternData-M1: A large-scale synthetic dataset for generalizable pick-and-place over 80K objects, with open-ended instructions covering object recognition, spatial and commonsense reasoning, and long-horizon tasks. InternData-M1：一个大规模合成数据集，用于可泛化的抓取与放置任务，涵盖 8 万+ 对象，配有开放式指令，涉及物体识别、空间与常识推理以及长时任务。","descriptions":{"en":"InternData-M1: A large-scale synthetic dataset for generalizable pick-and-place over 80K objects, with open-ended instructions covering object recognition, spatial and commonsense reasoning, and long-horizon tasks.","zh":"InternData-M1：一个大规模合成数据集，用于可泛化的抓取与放置任务，涵盖 8 万+ 对象，配有开放式指令，涉及物体识别、空间与常识推理以及长时任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"opencompass:2153","languages":[],"library_direction":null,"modality":"multimodal","name":"InternData-M1","openness":"restricted","publisher":null,"released":"2025-07-26","released_reference":{"basis":"dataset_published","note":"The official dataset license and changelog give the same initial release date.","source_key":"opencompass:2153","source_url":"https://huggingface.co/datasets/InternRobotics/InternData-M1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2153-interndata-m1","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InternData-M1","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","物理智能","Embodied AI","跨模态推理","Cross-modal Reasoning","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"InternData-N1: A high-quality navigation dataset with the most diverse scenes and extensive randomization across embodiments/viewpoints, including 3k+ scenes and 830k VLN data. InternData-N1：一个高质量导航数据集，具有最丰富的场景和跨具身/视角的广泛随机化，包含 3000+ 场景和 83 万条 VLN 数据。","descriptions":{"en":"InternData-N1: A high-quality navigation dataset with the most diverse scenes and extensive randomization across embodiments/viewpoints, including 3k+ scenes and 830k VLN data.","zh":"InternData-N1：一个高质量导航数据集，具有最丰富的场景和跨具身/视角的广泛随机化，包含 3000+ 场景和 83 万条 VLN 数据。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":true,"key":"opencompass:2152","languages":[],"library_direction":null,"modality":"multimodal","name":"InternData-N1","openness":"restricted","publisher":"InternRobotics","released":"2025-07-26","released_reference":{"basis":"dataset_published","note":"The official dataset license states the release date, before the later directory and model updates.","source_key":"opencompass:2152","source_url":"https://huggingface.co/datasets/InternRobotics/InternData-N1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2152-interndata-n1","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InternData-N1","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark of 210 domestic companion-robot scenarios pairing human and LLM behavior preferences.","descriptions":{"en":"Benchmark of 210 domestic companion-robot scenarios pairing human and LLM behavior preferences."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15455","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_intersocialbench_758e43be"},"modality":null,"name":"InterSocialBench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15455","source_url":"https://arxiv.org/abs/2609.15455"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15455","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15455","topics":[],"unit":null},{"aliases":["hil-safety-benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yanliang-li/hil-safety-benchmark","source_record":"bm_hil-safety-benchmark_bd9bca89"},"categories":["Agents","General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates human-in-the-loop safety for tool-using agents over 250 synthetic tasks with four reply types, measuring unsafe action rates, consultation quality, and post-reply behavior.","descriptions":{"en":"Evaluates human-in-the-loop safety for tool-using agents over 250 synthetic tasks with four reply types, measuring unsafe action rates, consultation quality, and post-reply behavior."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:yanliang-li/hil-safety-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","tool-use"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_hil-safety-benchmark_bd9bca89"},"modality":null,"name":"InterveneBench","openness":"unknown","publisher":null,"released":"2026-09-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:yanliang-li/hil-safety-benchmark","source_url":"https://github.com/yanliang-li/hil-safety-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-yanliang-li-hil-safety-benchmark","source":"claire_radar","source_url":"https://github.com/yanliang-li/hil-safety-benchmark","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","视频生成","物理智能","Embodied AI","图像理解","Image Understanding","视觉生成","Visual Generation","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"IntPhys2 tests intuitive physics—permanence, immutability, continuity and solidity—using synthetic videos. SOTA models perform around chance (~50 %), far below human level. IntPhys 2，一个用于评估深度学习模型直观物理理解能力的视频基准。它围绕永恒性、不可变性、时空连续性和实体性四个核心原则，测试模型区分可能与不可能事件的能力。","descriptions":{"en":"IntPhys2 tests intuitive physics—permanence, immutability, continuity and solidity—using synthetic videos. SOTA models perform around chance (~50 %), far below human level.","zh":"IntPhys 2，一个用于评估深度学习模型直观物理理解能力的视频基准。它围绕永恒性、不可变性、时空连续性和实体性四个核心原则，测试模型区分可能与不可能事件的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1976","languages":[],"library_direction":null,"modality":null,"name":"IntPhys2","openness":"restricted","publisher":"FAIR at Meta","released":"2025-06-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1976-intphys2","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IntPhys2","topics":[],"unit":null},{"aliases":["Inverse IFEval"],"attention_signals":null,"categories":["instruction_following"],"collected_at":null,"description":"Tests ability to override training biases and follow counterintuitive instructions; complementary to IFEval.","descriptions":{"en":"Tests ability to override training biases and follow counterintuitive instructions; complementary to IFEval."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000inverse_ifeval\u0000bytedance_seed2_0_model_card\u0000inverse_ifeval\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000inverse_ifeval\u0000bytedance_seed2_0_model_card\u0000inverse_ifeval\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:inverse_ifeval","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["instruction-following"],"join":"review:14H034160212:2026-10-15","primary":"instruction-following","source_record":null},"modality":null,"name":"Inverse IFEval","openness":"unknown","publisher":null,"released":"2025-09-04","released_reference":{"basis":"benchmark_release","source_key":"model-reports:inverse_ifeval","source_url":"https://arxiv.org/abs/2509.04292"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":78.9,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":78.9,"raw_min":69.3,"source_reference":{"obs_id":"curated\u0000inverse_ifeval\u0000bytedance_seed2_0_model_card\u0000inverse_ifeval\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000inverse_ifeval\u0000bytedance_seed2_0_model_card\u0000inverse_ifeval\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"inverse_ifeval","source":"model_reports","source_url":"https://arxiv.org/abs/2509.04292","topics":["language"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"The benchmark evaluates language models on distinguishing human-only vs. human-AI multi-turn dialogues, using paired transcripts and accuracy as the metric.","descriptions":{"en":"The benchmark evaluates language models on distinguishing human-only vs. human-AI multi-turn dialogues, using paired transcripts and accuracy as the metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21844","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_inverse-turing-bench_7691d060"},"modality":null,"name":"Inverse Turing Bench","openness":"unknown","publisher":null,"released":"2026-06-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21844","source_url":"https://arxiv.org/abs/2606.21844"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21844","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21844","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"InvestLogicBench evaluates large language models on personalized investment decision-making using 201,247 documented decisions from 151 real-world investors. Each episode traces investor profile, market events, reasoning, decision, and outcome. Tasks include comprehension, profile-conditioned generation, and end-to-end replay.","descriptions":{"en":"InvestLogicBench evaluates large language models on personalized investment decision-making using 201,247 documented decisions from 151 real-world investors. Each episode traces investor profile, market events, reasoning, decision, and outcome. Tasks include comprehension, profile-conditioned generation, and end-to-end replay."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06108","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_investlogicbench_44a26030"},"modality":null,"name":"InvestLogicBench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06108","source_url":"https://arxiv.org/abs/2608.06108"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06108","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06108","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"InvestPhilBench is a multi-layer benchmark for evaluating procedural reasoning in investment philosophy, spanning eight cognitive tiers from principle identification to novel framework extrapolation. It includes principle cards, decision-framework cards, and QA questions, with an automated scoring pipeline (BASP) and five algorithmic metrics.","descriptions":{"en":"InvestPhilBench is a multi-layer benchmark for evaluating procedural reasoning in investment philosophy, spanning eight cognitive tiers from principle identification to novel framework extrapolation. It includes principle cards, decision-framework cards, and QA questions, with an automated scoring pipeline (BASP) and five algorithmic metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.25984","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_investphilbench_547d0c00"},"modality":null,"name":"InvestPhilBench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25984","source_url":"https://arxiv.org/abs/2606.25984"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25984","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25984","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":20,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:ljang0/iosworld","source_record":"bm_iosworld_171f5010"},"categories":["Agents","Agents & Tool Use","Computer use","Cross-app planning","Memory & personalization","Mobile","Mobile & Personal Computing","Personalization"],"collected_at":"2026-10-14T09:44:51Z","description":"iOSWorld evaluates phone agents on interactive tasks within a native iOS simulator. It includes 26 custom iOS apps with connected personal data, 133 tasks across single-app, multi-app, and memory/personalization categories. Agents are evaluated with rubric-based scoring.","descriptions":{"en":"iOSWorld evaluates phone agents on interactive tasks within a native iOS simulator. It includes 26 custom iOS apps with connected personal data, 133 tasks across single-app, multi-app, and memory/personalization categories. Agents are evaluated with rubric-based scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09764","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","agent-memory"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_iosworld_171f5010"},"modality":null,"name":"iOSWorld","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09764","source_url":"https://arxiv.org/abs/2606.09764"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09764","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09764","topics":["computer-use","agents"],"unit":null},{"aliases":["little m: An AI Agent for Industrial Process Optimization"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yeyongchao/process-modeling-benchmark","source_record":"bm_little-m-an-ai-agent-for-industrial-proces_62ce7b90"},"categories":["Agents","Manufacturing & Process Control","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"IPC-Bench is a multimodal benchmark of 50 industrial process optimization scenarios requiring joint reasoning over text and process diagrams to formulate mathematical optimization models.","descriptions":{"en":"IPC-Bench is a multimodal benchmark of 50 industrial process optimization scenarios requiring joint reasoning over text and process diagrams to formulate mathematical optimization models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.16680","languages":[],"library_direction":{"basis":"model-review","categories":["math","visual-reasoning"],"join":"claire_record_id","primary":"math","source_record":"bm_little-m-an-ai-agent-for-industrial-proces_62ce7b90"},"modality":null,"name":"IPC-Bench","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16680","source_url":"https://arxiv.org/abs/2609.16680"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16680","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16680","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["physics","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"International Physics Olympiad 2025 (theory) comprises all 3 theory problems from the official 2025 IPhO competition. Results are based on blinded human evaluation with guidelines based on the official competition scoring, validated by domain experts.","descriptions":{"en":"International Physics Olympiad 2025 (theory) comprises all 3 theory problems from the official 2025 IPhO competition. Results are based on blinded human evaluation with guidelines based on the official competition scoring, validated by domain experts."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ipho-2025:muse-spark","reported_at":"2026-04-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ipho-2025?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ipho-2025","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_e227df5e4828f1a8"},"modality":"text","name":"IPhO 2025","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":82.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.826,"raw_min":0.793,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ipho-2025:muse-spark","reported_date":"2026-04-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ipho-2025?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"}],"slug":"llm-stats-ipho-2025","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ipho-2025?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_ipibench_3c35c7c4"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for interactive proactive intelligence of MLLMs under streaming video, covering proactive monitoring, task management, and interleaved requests.","descriptions":{"en":"Benchmark for interactive proactive intelligence of MLLMs under streaming video, covering proactive monitoring, task management, and interleaved requests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27074","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_ipibench_3c35c7c4"},"modality":null,"name":"IPIBench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27074","source_url":"https://arxiv.org/abs/2605.27074"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27074","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27074","topics":[],"unit":null},{"aliases":["IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO"],"attention_signals":null,"categories":["Agents","Finance","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"IPO Finance Agent evaluates language-model financial analysts on IPO due-diligence questions using expert-reviewed evaluation rubrics and accuracy reporting.","descriptions":{"en":"IPO Finance Agent evaluates language-model financial analysts on IPO due-diligence questions using expert-reviewed evaluation rubrics and accuracy reporting."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.23032","languages":[],"library_direction":null,"modality":null,"name":"IPO Finance Agent","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23032","source_url":"https://arxiv.org/abs/2606.23032"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23032","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23032","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","VQA","Strong Reasoning","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"IQBench is a novel benchmark designed to evaluate the fluid intelligence of VisionLanguage Models (VLMs) using standardized visual IQ tests. It consists of 500 manually collected and annotated visual IQ questions covering various domains. IQBench是一个新基准测试，旨在通过标准化视觉智商测试评估视觉语言模型（VLMs）的流体智力。该基准包含500个手动收集和注释的视觉智商问题，涵盖模式识别、类比推理、视觉算术、空间理解等多个领域。与以往仅关注最终答案准确性的基准不同，IQBench强调对模型推理能力的评估，采用双重评估框架：准确性评分和推理评分。实验表明，即使是性能最高的模型（如o4mini、gemini2.5flash和claude3.7sonnet），在3D空间和字母重组任务上也表现出明显不足，凸显了当前VLMs在通用推理能力上的局限性。IQBench为开发更透明、更具认知能力的多模态系统奠定了基础。","descriptions":{"en":"IQBench is a novel benchmark designed to evaluate the fluid intelligence of VisionLanguage Models (VLMs) using standardized visual IQ tests. It consists of 500 manually collected and annotated visual IQ questions covering various domains.","zh":"IQBench是一个新基准测试，旨在通过标准化视觉智商测试评估视觉语言模型（VLMs）的流体智力。该基准包含500个手动收集和注释的视觉智商问题，涵盖模式识别、类比推理、视觉算术、空间理解等多个领域。与以往仅关注最终答案准确性的基准不同，IQBench强调对模型推理能力的评估，采用双重评估框架：准确性评分和推理评分。实验表明，即使是性能最高的模型（如o4mini、gemini2.5flash和claude3.7sonnet），在3D空间和字母重组任务上也表现出明显不足，凸显了当前VLMs在通用推理能力上的局限性。IQBench为开发更透明、更具认知能力的多模态系统奠定了基础。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1840","languages":[],"library_direction":null,"modality":"multimodal","name":"IQBench","openness":"unknown","publisher":"Harvard Medical School, USA Uppsala University,etc","released":"2025-05-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1840-iqbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IQBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":22,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:infrarecon7/ir275k","source_record":"bm_ir275k_3c0945b4"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for infrared multi-frame super-resolution containing 594 video sequences and 275,196 frames with fixed sequence-level splits and a reproducible x4 evaluation protocol.","descriptions":{"en":"A benchmark for infrared multi-frame super-resolution containing 594 video sequences and 275,196 frames with fixed sequence-level splits and a reproducible x4 evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.22380","languages":[],"library_direction":null,"modality":null,"name":"IR275K","openness":"unknown","publisher":null,"released":"2026-07-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22380","source_url":"https://arxiv.org/abs/2607.22380"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22380","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22380","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sanhornc/irts-toolbench","source_record":"bm_irts-toolbench_94993a2c"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"IRTS-ToolBench is a benchmark of 1,700 questions across 10 task types and 13 domains for evaluating irregular univariate time-series question answering, with standardized inputs and a reproducible evaluation protocol.","descriptions":{"en":"IRTS-ToolBench is a benchmark of 1,700 questions across 10 task types and 13 domains for evaluating irregular univariate time-series question answering, with standardized inputs and a reproducible evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.15107","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_irts-toolbench_94993a2c"},"modality":null,"name":"IRTS-ToolBench","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15107","source_url":"https://arxiv.org/abs/2606.15107"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15107","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15107","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","安全","Safety","智能体","Agent","task planning","安全对齐","Safety Alignment","任务执行","Task Execution","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"IS-Bench is the first evaluation benchmark dedicated to assessing the safety of embodied agents during their interaction with home environments. It encompasses 161 high-risk domestic scenarios, spanning 10 hazard categories, including food poisoning, fire, electric shock, and more. IS-Bench 是首个专注于具身智能体与家用环境交互过程安全性的评测基准。它包含 161 个高风险家居场景及相关日常任务，覆盖食物中毒、火灾、触电等 10 大类常见风险。通过贯穿整个交互过程的动态评测框架，全方位评估具身智能体的安全素养。","descriptions":{"en":"IS-Bench is the first evaluation benchmark dedicated to assessing the safety of embodied agents during their interaction with home environments. It encompasses 161 high-risk domestic scenarios, spanning 10 hazard categories, including food poisoning, fire, electric shock, and more.","zh":"IS-Bench 是首个专注于具身智能体与家用环境交互过程安全性的评测基准。它包含 161 个高风险家居场景及相关日常任务，覆盖食物中毒、火灾、触电等 10 大类常见风险。通过贯穿整个交互过程的动态评测框架，全方位评估具身智能体的安全素养。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2091","languages":[],"library_direction":null,"modality":"multimodal","name":"IS-Bench","openness":"open","publisher":"OpenTrustLab","released":"2025-07-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2091-is-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IS-Bench","topics":["agents","multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Code generation","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for computational reasoning using programming games with constrained instruction sets.","descriptions":{"en":"A benchmark for computational reasoning using programming games with constrained instruction sets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.22878","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_isa-bench_4fb52e49"},"modality":null,"name":"ISA-Bench","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.22878","source_url":"https://arxiv.org/abs/2609.22878"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-22878","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22878","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_ishigaki-ids-bench_d58a0879"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for generating Information Delivery Specification (IDS) XML from BIM information requirements, with 166 examples in English and Japanese. It evaluates LLMs on formal validity via IDSAuditTool and content fidelity via facet-level macro-F1 against gold IDS files.","descriptions":{"en":"A benchmark for generating Information Delivery Specification (IDS) XML from BIM information requirements, with 166 examples in English and Japanese. It evaluates LLMs on formal validity via IDSAuditTool and content fidelity via facet-level macro-F1 against gold IDS files."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22079","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","professional-work"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_ishigaki-ids-bench_d58a0879"},"modality":null,"name":"Ishigaki-IDS-Bench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22079","source_url":"https://arxiv.org/abs/2605.22079"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22079","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22079","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-task dataset of 3,465 QA items from 35 Islamic scholarly works, spanning 12 centuries and seven fields, with three difficulty levels and three task formats.","descriptions":{"en":"Multi-task dataset of 3,465 QA items from 35 Islamic scholarly works, spanning 12 centuries and seven fields, with three difficulty levels and three task formats."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04703","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_islamicturathbench_0fa2de43"},"modality":null,"name":"IslamicTurathBench","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04703","source_url":"https://arxiv.org/abs/2608.04703"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04703","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04703","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":9,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_isosci_576d9a68"},"categories":["Factuality","General AI","Information retrieval","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"IsoSci evaluates reasoning vs. knowledge retrieval in LLMs using isomorphic cross-domain problem pairs, enabling controlled attribution of reasoning gains.","descriptions":{"en":"IsoSci evaluates reasoning vs. knowledge retrieval in LLMs using isomorphic cross-domain problem pairs, enabling controlled attribution of reasoning gains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01431","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","knowledge-expert-qa"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_isosci_576d9a68"},"modality":null,"name":"IsoSci","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01431","source_url":"https://arxiv.org/abs/2607.01431"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01431","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01431","topics":["reasoning-math"],"unit":null},{"aliases":["I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning"],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates person-centric video reasoning using identity-conditioned queries, with 1,377 real-world videos and QA pairs across six difficulty levels covering identity recognition to causal reasoning.","descriptions":{"en":"Evaluates person-centric video reasoning using identity-conditioned queries, with 1,377 real-world videos and QA pairs across six difficulty levels covering identity recognition to causal reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07417","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_i-seek-you-in-videos-identity-conditioned-_7f9c3140"},"modality":null,"name":"ISYV-Bench","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07417","source_url":"https://arxiv.org/abs/2608.07417"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07417","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07417","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","智能体","Agent","任务执行","Task Execution","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ITBench is a benchmark designed to evaluate the performance of AI agents in real-world IT automation tasks. ITBench 是一个旨在评估 AI 智能体在真实世界 IT 自动化任务中表现的基准。它涵盖了站点可靠性工程、合规与安全运营以及财务运营等关键维度，并包含 102 个真实场景。该基准提供了一个开源框架和多种基线智能体实现，并集成了CrewAI等工具，以促进AI驱动的IT自动化发展。","descriptions":{"en":"ITBench is a benchmark designed to evaluate the performance of AI agents in real-world IT automation tasks.","zh":"ITBench 是一个旨在评估 AI 智能体在真实世界 IT 自动化任务中表现的基准。它涵盖了站点可靠性工程、合规与安全运营以及财务运营等关键维度，并包含 102 个真实场景。该基准提供了一个开源框架和多种基线智能体实现，并集成了CrewAI等工具，以促进AI驱动的IT自动化发展。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:2077","languages":[],"library_direction":null,"modality":null,"name":"ITBench","openness":"unknown","publisher":"IBM , University of Illinois at Urbana-Champaign","released":"2025-02-07","released_reference":{"basis":"release_announcement","note":"The official project changelog explicitly dates its initial release of the paper, scenarios, setup tooling, and agents.","source_key":"opencompass:2077","source_url":"https://github.com/itbench-hub/ITBench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2077-itbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ITBench","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["agentic","tool-use","engineering"],"collected_at":"2026-08-25T10:41:06Z","description":"Kubernetes incident root-cause analysis","descriptions":{"en":"Kubernetes incident root-cause analysis"},"evidence_summary":{"document_count":1,"model_count":33,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:itbench-aa:976cc8ad-7904-4056-83c5-960181f47d5f","reported_at":"2024-12-06","source_url":"https://artificialanalysis.ai/evaluations/itbench-aa"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:itbench-aa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["professional-work"],"join":"review:14H034160212:2026-10-15","primary":"professional-work","source_record":null},"modality":null,"name":"ITBench-AA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":33,"score_direction":"higher_is_better","score_summary":{"display_max":56.2146892655367,"display_multiplier":100,"model_count":33,"model_count_basis":"source_model_id","numeric_count":33,"raw_max":0.562146892655367,"raw_min":0.00564971751412429,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:itbench-aa:d93edfe8-bf35-49ad-b56e-b18116142a1c","reported_date":"2026-07-09","source_url":"https://artificialanalysis.ai/evaluations/itbench-aa"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"}],"slug":"artificial-analysis-itbench-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/itbench-aa","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ITPEval evaluates automated formal proof translation across four interactive theorem provers (Lean 4, Rocq, Isabelle, HOL Light), with 1,560 source files and 6,848 theorems, covering statement and proof translation on 12 directed pairs.","descriptions":{"en":"ITPEval evaluates automated formal proof translation across four interactive theorem provers (Lean 4, Rocq, Isabelle, HOL Light), with 1,560 source files and 6,848 theorems, covering statement and proof translation on 12 directed pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19407","languages":[],"library_direction":{"basis":"model-review","categories":["math","code-generation"],"join":"claire_record_id","primary":"math","source_record":"bm_itpeval_0f85127f"},"modality":null,"name":"ITPEval","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19407","source_url":"https://arxiv.org/abs/2607.19407"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19407","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19407","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","legal agent","interactive benchmark","legal benchmark","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"J1-Bench is an interactive and comprehensive legal benchmark where LLM agents engage in diverse legal scenarios, completing tasks through interactions with various participants under procedural rules. J1-Bench 是一个交互式的综合法律基准，法律智能体在此参与各种法律情景，根据程序规则通过与不同参与者的互动完成指定的法律任务。","descriptions":{"en":"J1-Bench is an interactive and comprehensive legal benchmark where LLM agents engage in diverse legal scenarios, completing tasks through interactions with various participants under procedural rules.","zh":"J1-Bench 是一个交互式的综合法律基准，法律智能体在此参与各种法律情景，根据程序规则通过与不同参与者的互动完成指定的法律任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2073","languages":[],"library_direction":null,"modality":null,"name":"J1-Bench","openness":"restricted","publisher":"Fudan Data Intelligence and Social Computing (Fudan DISC) Lab","released":"2025-07-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2073-j1-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/J1-Bench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","NeurIPS 2024","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"JailTrickBench can evaluate the impact of various attack settings on LLM performance, including 8 key factors of implementing jailbreak attacks on LLMs from both target-level and attack-level perspectives. JailTrickBench用于评估LLM应对各种越狱攻击的能力，涵盖从目标级和攻击级2个角度实施越狱攻击的8个关键因素。","descriptions":{"en":"JailTrickBench can evaluate the impact of various attack settings on LLM performance, including 8 key factors of implementing jailbreak attacks on LLMs from both target-level and attack-level perspectives.","zh":"JailTrickBench用于评估LLM应对各种越狱攻击的能力，涵盖从目标级和攻击级2个角度实施越狱攻击的8个关键因素。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1337","languages":[],"library_direction":null,"modality":null,"name":"JailTrickBench","openness":"unknown","publisher":"The Hong Kong University of Science and Technology","released":"2024-06-13","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1337-jailtrickbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/JailTrickBench","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"JaleesBench evaluates AI-agent counsel across 140 two-turn faith-related scenarios spanning adversarial pressures and framings.","descriptions":{"en":"JaleesBench evaluates AI-agent counsel across 140 two-turn faith-related scenarios spanning adversarial pressures and framings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.07508","languages":[],"library_direction":null,"modality":null,"name":"JaleesBench","openness":"unknown","publisher":null,"released":"2026-06-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07508","source_url":"https://arxiv.org/abs/2608.07508"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07508","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07508","topics":["agents"],"unit":null},{"aliases":["JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines"],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-09-25T04:40:22Z","description":"JamBench evaluates project-level game code framework generation and code completion on 300 manually verified Godot projects using deterministic compilation, structural-completeness, and behavioral-alignment measures.","descriptions":{"en":"JamBench evaluates project-level game code framework generation and code completion on 300 manually verified Godot projects using deterministic compilation, structural-completeness, and behavioral-alignment measures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19830","languages":[],"library_direction":null,"modality":null,"name":"JamBench","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19830","source_url":"https://arxiv.org/abs/2606.19830"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19830","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19830","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"JANUS is a benchmark with 160 scenarios across 8 domains, each providing a fixed pool of favorable and adverse facts and paired neutral and goal-directed prompts, to evaluate fact-grounded, goal-conditioned pragmatic distortion in LLM outputs.","descriptions":{"en":"JANUS is a benchmark with 160 scenarios across 8 domains, each providing a fixed pool of favorable and adverse facts and paired neutral and goal-directed prompts, to evaluate fact-grounded, goal-conditioned pragmatic distortion in LLM outputs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.10852","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","factuality"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_janus_33d6bb69"},"modality":null,"name":"Janus","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10852","source_url":"https://arxiv.org/abs/2606.10852"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10852","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10852","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"JarvisBench measures mediation in long-horizon agent workflows with two tracks: agent-collaboration and user-interaction. Built on WildClaw tasks and a reference Jarvis prototype.","descriptions":{"en":"JarvisBench measures mediation in long-horizon agent workflows with two tracks: agent-collaboration and user-interaction. Built on WildClaw tasks and a reference Jarvis prototype."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.16610","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_jarvisbench_2366ad17"},"modality":null,"name":"JarvisBench","openness":"unknown","publisher":null,"released":"2026-07-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.16610","source_url":"https://arxiv.org/abs/2607.16610"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-16610","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16610","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Mobile & Personal Computing","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Proposed benchmark for cross-device GUI agent workflows across Android, Windows, and Ubuntu, using input-output transformations and dynamic task composition.","descriptions":{"en":"Proposed benchmark for cross-device GUI agent workflows across Android, Windows, and Ubuntu, using input-output transformations and dynamic task composition."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10451","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_jarvisgui_ffdfbfea"},"modality":null,"name":"JarvisGUI","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10451","source_url":"https://arxiv.org/abs/2609.10451"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10451","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10451","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Vision & 3D","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"JavaVulBench is a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset includes about 30,600 Java methods spanning 1,740 CVEs and 700+ projects, with method and line labels, publication dates, and five split strategies. The harness provides a unified schema across multiple backends and includes a contamination audit.","descriptions":{"en":"JavaVulBench is a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset includes about 30,600 Java methods spanning 1,740 CVEs and 700+ projects, with method and line labels, publication dates, and five split strategies. The harness provides a unified schema across multiple backends and includes a contamination audit."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02825","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","code-generation"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_javavulbench_93d353dc"},"modality":null,"name":"JavaVulBench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02825","source_url":"https://arxiv.org/abs/2607.02825"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02825","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02825","topics":["coding","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.PL"],"collected_at":"2026-10-14T09:44:51Z","description":"JEDI is a benchmark suite for the Java Stream API, generated by converting SQL benchmarks into Java benchmarks. It includes both stream-based and imperative query implementations to evaluate performance of parallelization strategies.","descriptions":{"en":"JEDI is a benchmark suite for the Java Stream API, generated by converting SQL benchmarks into Java benchmarks. It includes both stream-based and imperative query implementations to evaluate performance of parallelization strategies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.23543","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_jedi_9de96442"},"modality":null,"name":"JEDI","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23543","source_url":"https://arxiv.org/abs/2605.23543"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23543","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23543","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:khesoem/jeto-bench","source_record":"bm_jeto-bench_f10e93b0"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"JETO-Bench is a benchmark of 660 execution time improvement patches (ETIPs) in Java, with 91 manually verified. It is built using JETO-Mine, a configurable tool that mines ETIPs from GitHub repositories and statistically validates improvements.","descriptions":{"en":"JETO-Bench is a benchmark of 660 execution time improvement patches (ETIPs) in Java, with 91 manually verified. It is built using JETO-Mine, a configurable tool that mines ETIPs from GitHub repositories and statistically validates improvements."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.31767","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_jeto-bench_f10e93b0"},"modality":null,"name":"JETO-Bench","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31767","source_url":"https://arxiv.org/abs/2606.31767"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31767","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31767","topics":["coding","agents"],"unit":null},{"aliases":["jev-vs-llm-benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:akashai7/jev-vs-llm-benchmark","source_record":"bm_jev-vs-llm-benchmark_ce5c5c83"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Compares Jev, a general-purpose LLM, and a confidence-gated hybrid on fixed-choice decisions using 18 labelled cases across support routing, incident priority, and request intent.","descriptions":{"en":"Compares Jev, a general-purpose LLM, and a confidence-gated hybrid on fixed-choice decisions using 18 labelled cases across support routing, incident priority, and request intent."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:akashai7/jev-vs-llm-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Jev vs LLM Benchmark","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:akashai7/jev-vs-llm-benchmark","source_url":"https://github.com/AkashAi7/jev-vs-llm-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-akashai7-jev-vs-llm-benchmark","source":"claire_radar","source_url":"https://github.com/AkashAi7/jev-vs-llm-benchmark","topics":[],"unit":null},{"aliases":["jev-deterministic-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:etsabary/jev-deterministic-benchmark","source_record":"bm_jev-deterministic-benchmark_129ec551"},"categories":["Language & Knowledge","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"1,000 deterministic reasoning decisions across 25 families used to test model Jev.","descriptions":{"en":"1,000 deterministic reasoning decisions across 25 families used to test model Jev."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:etsabary/jev-deterministic-benchmark","languages":[],"library_direction":null,"modality":null,"name":"JEV-DD-1.0","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:etsabary/jev-deterministic-benchmark","source_url":"https://github.com/etsabary/jev-deterministic-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-etsabary-jev-deterministic-benchmark","source":"claire_radar","source_url":"https://github.com/etsabary/jev-deterministic-benchmark","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates adversarial robustness of RLCD models with typed questions and single-edit attack variants.","descriptions":{"en":"Evaluates adversarial robustness of RLCD models with typed questions and single-edit attack variants."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.31142","languages":[],"library_direction":null,"modality":null,"name":"JevAdvBench","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.31142","source_url":"https://arxiv.org/abs/2609.31142"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-31142","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31142","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_jitoma-bench_daae8913"},"categories":["Reasoning","Robotics & Embodied AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"JITOMA-Bench is a suite for long-horizon multi-tasking and multi-step reasoning in robotics, focusing on just-in-time scene graph growth to combat perceptual saturation.","descriptions":{"en":"JITOMA-Bench is a suite for long-horizon multi-tasking and multi-step reasoning in robotics, focusing on just-in-time scene graph growth to combat perceptual saturation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.13245","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_jitoma-bench_daae8913"},"modality":null,"name":"JITOMA-Bench","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.13245","source_url":"https://arxiv.org/abs/2607.13245"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-13245","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13245","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":130,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:circlelzy/jl1-cd","source_record":"bm_jl1-cc-qa_e1b89c2a"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"JL1-CC&QA is a multi-task benchmark extending JL1-CD with change captioning and question answering. It includes 17,021 captions and 20,060 QA pairs over 5,000 bi-temporal image pairs from Jilin-1 satellite.","descriptions":{"en":"JL1-CC&QA is a multi-task benchmark extending JL1-CD with change captioning and question answering. It includes 17,021 captions and 20,060 QA pairs over 5,000 bi-temporal image pairs from Jilin-1 satellite."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.31745","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_jl1-cc-qa_e1b89c2a"},"modality":null,"name":"JL1-CC&QA","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31745","source_url":"https://arxiv.org/abs/2606.31745"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31745","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31745","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"JL1-CD is a large-scale, sub-meter, all-inclusive open-source dataset for remote sensing image change detection (CD). It contains 5,000 pairs of 512×512 pixel satellite images with a resolution of 0.5 to 0.75 meters, covering various types of surface changes in multiple regions of China. JL1-CD 是一个大规模、亚米级、全包含的开源遥感影像变化检测（CD）数据集。它包含 5000 对 512×512 像素的卫星影像，分辨率为 0.5 至 0.75 米，覆盖中国多个地区的各种地表变化。","descriptions":{"en":"JL1-CD is a large-scale, sub-meter, all-inclusive open-source dataset for remote sensing image change detection (CD). It contains 5,000 pairs of 512×512 pixel satellite images with a resolution of 0.5 to 0.75 meters, covering various types of surface changes in multiple regions of China.","zh":"JL1-CD 是一个大规模、亚米级、全包含的开源遥感影像变化检测（CD）数据集。它包含 5000 对 512×512 像素的卫星影像，分辨率为 0.5 至 0.75 米，覆盖中国多个地区的各种地表变化。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1554","languages":[],"library_direction":null,"modality":null,"name":"JL1-CD","openness":"open","publisher":"Beijing National Research Center for Information Science and Technology,etc.","released":"2025-02-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1554-jl1-cd","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/JL1-CD","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"JMed48k evaluates vision-language models on 48,862 Japanese medical licensing exam questions from 11 national examinations (2005-2025), with images annotated under an 8-type taxonomy. The JMed48k-Eval subset contains 12,484 scored questions, including text-only and with-image items, scored separately.","descriptions":{"en":"JMed48k evaluates vision-language models on 48,862 Japanese medical licensing exam questions from 11 national examinations (2005-2025), with images annotated under an 8-type taxonomy. The JMed48k-Eval subset contains 12,484 scored questions, including text-only and with-image items, scored separately."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22080","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning","multilingual"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_jmed48k_34101c20"},"modality":null,"name":"JMed48k","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22080","source_url":"https://arxiv.org/abs/2605.22080"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22080","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22080","topics":["multimodal","language","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_dadf2d7a89695d09"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["productivity","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Job Bench evaluates AI agents on realistic professional tasks that require multi-step planning, research, and production of work artifacts.","descriptions":{"en":"Job Bench evaluates AI agents on realistic professional tasks that require multi-step planning, research, and production of work artifacts."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:job-bench:muse-spark-1.1","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/job-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:job-bench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_dadf2d7a89695d09"},"modality":"text","name":"Job Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":54.7,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.547,"raw_min":0.334,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:job-bench:muse-spark-1.1","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/job-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-job-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/job-bench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","综合能力","Comprehensive Capability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"JOB-Complex is a challenging benchmark for traditional and learned query optimizers, containing 30 SQL queries and a plan-selection benchmark with nearly 6000 execution plans. JOB-Complex是一个面向传统与学习式查询优化器的挑战性数据库基准，包含30个SQL查询和近6000个执行计划的计划选择基准。","descriptions":{"en":"JOB-Complex is a challenging benchmark for traditional and learned query optimizers, containing 30 SQL queries and a plan-selection benchmark with nearly 6000 execution plans.","zh":"JOB-Complex是一个面向传统与学习式查询优化器的挑战性数据库基准，包含30个SQL查询和近6000个执行计划的计划选择基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2130","languages":[],"library_direction":null,"modality":null,"name":"JOB-Complex","openness":"unknown","publisher":"Technical University of Darmstadt , DFKI","released":"2025-07-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2130-job-complex","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/JOB-Complex","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2605.26329","repository":null,"source_record":"catalog_dadf2d7a89695d09"},"categories":["professional"],"collected_at":null,"description":"Professional job tasks aligned with human work; rubric grading moves the number beyond model capability alone.","descriptions":{"en":"Professional job tasks aligned with human work; rubric grading moves the number beyond model capability alone."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000jobbench\u0000tencent_hy4_preview\u0000jobbench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000jobbench\u0000tencent_hy4_preview\u0000jobbench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:jobbench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"artifact:arxiv:2605.26329","primary":"professional-work","source_record":"catalog_dadf2d7a89695d09"},"modality":null,"name":"JobBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"model-reports:jobbench","source_url":"https://arxiv.org/abs/2605.26329"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":61.7,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":61.7,"raw_min":61.7,"source_reference":{"obs_id":"curated\u0000jobbench\u0000tencent_hy4_preview\u0000jobbench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000jobbench\u0000tencent_hy4_preview\u0000jobbench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"jobbench","source":"model_reports","source_url":"https://arxiv.org/abs/2605.26329","topics":[],"unit":"percent"},{"aliases":["JointAVBench","Joint AV-Bench"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"An ICLR 2026 poster on joint audio-visual reasoning in Omni-LLMs, built with strict audio-video correlation across five cognitive dimensions and four audio information types. Because the question set requires both modalities, vendors that only report vision-only or audio-only figures are not comparable on this instrument. Automated annotation pipeline; manual verification effort is not quantified.","descriptions":{"en":"An ICLR 2026 poster on joint audio-visual reasoning in Omni-LLMs, built with strict audio-video correlation across five cognitive dimensions and four audio information types. Because the question set requires both modalities, vendors that only report vision-only or audio-only figures are not comparable on this instrument. Automated annotation pipeline; manual verification effort is not quantified."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000jointavbench\u0000jointavbench_v2_results_table\u0000jointavbench_2853_mcq_v2\u00002,853 manually verified MCQs; accuracy mean over 15 task categories; unified 32-frame sampling; Audio-LLM (audio-only inputs), 7B; v2 results\u0000Kimi-Audio","reported_at":"2026-05-14","source_url":"https://arxiv.org/html/2512.12772v2"},"first_score_reported_at":"2026-05-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000jointavbench\u0000jointavbench_v2_results_table\u0000jointavbench_2853_mcq_v2\u00002,853 manually verified MCQs; accuracy mean over 15 task categories; unified 32-frame sampling; Audio-LLM (audio-only inputs), 7B; v2 results\u0000Kimi-Audio","reported_at":"2026-05-14","source_url":"https://arxiv.org/html/2512.12772v2"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:jointavbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["video-understanding"],"join":"review:14H034160212:2026-10-15","primary":"video-understanding","source_record":null},"modality":null,"name":"JointAVBench","openness":"unknown","publisher":null,"released":"2025-12-14","released_reference":{"basis":"benchmark_release","source_key":"model-reports:jointavbench","source_url":"https://arxiv.org/abs/2512.12772"},"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":65.3,"display_multiplier":1,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":65.3,"raw_min":31.0,"source_reference":{"obs_id":"curated\u0000jointavbench\u0000jointavbench_v2_results_table\u0000jointavbench_2853_mcq_v2\u00002,853 manually verified MCQs; accuracy mean over 15 task categories; unified 32-frame sampling; Omni-LLM (joint audio-visual inputs); v2 results\u0000Gemini 2.5 Pro","observation_id":"curated\u0000jointavbench\u0000jointavbench_v2_results_table\u0000jointavbench_2853_mcq_v2\u00002,853 manually verified MCQs; accuracy mean over 15 task categories; unified 32-frame sampling; Omni-LLM (joint audio-visual inputs); v2 results\u0000Gemini 2.5 Pro","reported_at":"2026-05-14","reported_date":"2026-05-14","source_id":"jointavbench_v2_results_table","source_url":"https://arxiv.org/html/2512.12772v2"},"unit":"percent"},"scored_models":[{"date":"2026-05-14","date_precision":"document_publication","name":"AVicuna"},{"date":"2026-05-14","date_precision":"document_publication","name":"Gemini 2.5 Flash"},{"date":"2026-05-14","date_precision":"document_publication","name":"Gemini 2.5 Pro"},{"date":"2026-05-14","date_precision":"document_publication","name":"GPT-4o"},{"date":"2026-05-14","date_precision":"document_publication","name":"InternVL-2.5"},{"date":"2026-05-14","date_precision":"document_publication","name":"Kimi-Audio"},{"date":"2026-05-14","date_precision":"document_publication","name":"LLaVA-Video"},{"date":"2026-05-14","date_precision":"document_publication","name":"OneLLM"},{"date":"2026-05-14","date_precision":"document_publication","name":"Qwen2-Audio"},{"date":"2026-05-14","date_precision":"document_publication","name":"Qwen2.5-Omni"},{"date":"2026-05-14","date_precision":"document_publication","name":"Qwen2.5-VL"},{"date":"2026-05-14","date_precision":"document_publication","name":"Qwen3-Omni"},{"date":"2026-05-14","date_precision":"document_publication","name":"v-SALMONN"},{"date":"2026-05-14","date_precision":"document_publication","name":"v-SALMONN-2+"},{"date":"2026-05-14","date_precision":"document_publication","name":"v-SALMONN-o1"},{"date":"2026-05-14","date_precision":"document_publication","name":"VideoLLaMA2"},{"date":"2026-05-14","date_precision":"document_publication","name":"VideoLLaMA3"}],"slug":"jointavbench","source":"model_reports","source_url":"https://arxiv.org/abs/2512.12772","topics":["multimodal"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"JOR-Bench is a collection of five Japanese-language benchmarks for LLMs in operations research, covering 1,319 problems from IndustryOR, MAMO, NL4OPT, OptiBench, and OptMATH, with pairs of Japanese problem statements and numerical answers.","descriptions":{"en":"JOR-Bench is a collection of five Japanese-language benchmarks for LLMs in operations research, covering 1,319 problems from IndustryOR, MAMO, NL4OPT, OptiBench, and OptMATH, with pairs of Japanese problem statements and numerical answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.16777","languages":[],"library_direction":{"basis":"model-review","categories":["math","multilingual"],"join":"claire_record_id","primary":"math","source_record":"bm_jor-bench_97431ead"},"modality":null,"name":"JOR-Bench","openness":"unknown","publisher":null,"released":"2026-07-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.16777","source_url":"https://arxiv.org/abs/2607.16777"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-16777","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16777","topics":["reasoning-math","language"],"unit":null},{"aliases":["Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis"],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-09-25T04:40:22Z","description":"A Japanese speech-synthesis pronunciation benchmark covering the officially designated Joyo kanji and their readings.","descriptions":{"en":"A Japanese speech-synthesis pronunciation benchmark covering the officially designated Joyo kanji and their readings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.25369","languages":[],"library_direction":null,"modality":null,"name":"Joyo Kanji Yomi Benchmark","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25369","source_url":"https://arxiv.org/abs/2606.25369"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25369","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25369","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:controlnet/jrdb-avr","source_record":"bm_jrdb-avr_5eba914f"},"categories":["Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates embodied visual reasoning requiring active observation requests and evidence grounding, with questions derived from real-world JRDB robotics data.","descriptions":{"en":"Evaluates embodied visual reasoning requiring active observation requests and evidence grounding, with questions derived from real-world JRDB robotics data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.35032","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","visual-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_jrdb-avr_5eba914f"},"modality":null,"name":"JRDB-AVR","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35032","source_url":"https://arxiv.org/abs/2609.35032"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35032","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35032","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","知识储备","Knowledge","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"JudgeBench is a benchmark aimed at evaluating LLM-based judges for objective correctness on challenging response pairs. JudgeBench 是一个旨在评估基于LLM的裁判在具有挑战性的响应对上的客观正确性的基准。","descriptions":{"en":"JudgeBench is a benchmark aimed at evaluating LLM-based judges for objective correctness on challenging response pairs.","zh":"JudgeBench 是一个旨在评估基于LLM的裁判在具有挑战性的响应对上的客观正确性的基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1583","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"artifact:gh:scalerlab/judgebench","primary":"chat-writing","source_record":"catalog_d3762248c0f2a3c9"},"modality":null,"name":"JudgeBench","openness":"open","publisher":"UC Berkeley, Washington University in St. Louis","released":"2024-10-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1583-judgebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/JudgeBench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A dataset of 30 legal tasks with rubric scores and pairwise preference judgments from practicing attorneys, used to compare rubric-based scoring and comparative judgment.","descriptions":{"en":"A dataset of 30 legal tasks with rubric scores and pairwise preference judgments from practicing attorneys, used to compare rubric-based scoring and comparative judgment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.25240","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_judgmentbench_729b2b9c"},"modality":null,"name":"JudgmentBench","openness":"unknown","publisher":null,"released":"2026-05-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25240","source_url":"https://arxiv.org/abs/2605.25240"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25240","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25240","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:choyingw/jurybench","source_record":"bm_jurybench_3c843934"},"categories":["General AI","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM-simulated jurors on controversial U.S. criminal law cases with defendant statements of varying backgrounds and emotional appeal, measuring verdict severity changes.","descriptions":{"en":"Evaluates LLM-simulated jurors on controversial U.S. criminal law cases with defendant statements of varying backgrounds and emotional appeal, measuring verdict severity changes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.09887","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","finance-legal"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_jurybench_3c843934"},"modality":null,"name":"JuryBench","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09887","source_url":"https://arxiv.org/abs/2609.09887"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09887","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09887","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"K-Bench is described as a benchmark that scores LLM unlearning under agentic deployment.","descriptions":{"en":"K-Bench is described as a benchmark that scores LLM unlearning under agentic deployment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.12808","languages":[],"library_direction":null,"modality":null,"name":"K-Bench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12808","source_url":"https://arxiv.org/abs/2609.12808"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12808","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12808","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A clinician-calibrated benchmark assessing large language models on 200 multi-turn mental health vignettes spanning suicide, self-harm, domestic violence, substance misuse, and no-risk scenarios.","descriptions":{"en":"A clinician-calibrated benchmark assessing large language models on 200 multi-turn mental health vignettes spanning suicide, self-harm, domestic violence, substance misuse, and no-risk scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15855","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["health-medicine","factuality","safety-alignment"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_k-bench_8a7caa58"},"modality":null,"name":"K-Bench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15855","source_url":"https://arxiv.org/abs/2609.15855"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15855","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15855","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":16,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":57,"join":"claire_record_id","repository":"gh:prometheus-eval/k-browsecomp","source_record":"bm_k-browsecomp_c3715567"},"categories":["Agents","Cybersecurity","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"K-BrowseComp evaluates web-browsing agents on 400 Korean-context problems, requiring multi-hop or parallel evidence retrieval from public Korean websites and returning a single short answer. Includes a 300-problem manually verified subset and a 100-problem synthetic stress-test split.","descriptions":{"en":"K-BrowseComp evaluates web-browsing agents on 400 Korean-context problems, requiring multi-hop or parallel evidence retrieval from public Korean websites and returning a single short answer. Includes a 300-problem manually verified subset and a 100-problem synthetic stress-test split."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.02404","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","multilingual"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_k-browsecomp_c3715567"},"modality":null,"name":"K-BrowseComp","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02404","source_url":"https://arxiv.org/abs/2606.02404"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02404","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02404","topics":["agents","language","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates hallucination detection in multi-turn Korean financial RAG dialogues, with a taxonomy based on context answerability. Includes training and test splits for fine-tuning and benchmarking detectors.","descriptions":{"en":"Evaluates hallucination detection in multi-turn Korean financial RAG dialogues, with a taxonomy based on context answerability. Includes training and test splits for fine-tuning and benchmarking detectors."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29523","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","finance-legal","multilingual"],"join":"claire_record_id","primary":"factuality","source_record":"bm_k-finhallu_490348d5"},"modality":null,"name":"K-FinHallu","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29523","source_url":"https://arxiv.org/abs/2605.29523"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29523","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29523","topics":["language","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","图像评估","视频评估","多模态模型","VLM","视觉生成","Visual Generation","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"K-Sort Arena employs K-wise comparison, allowing K models to participate in a free-for-all, providing richer information than pairwise comparison. It also designs a matching algorithm based on exploration-exploitation and probabilistic modeling to achieve more efficient and reliable model ranking. 本项目提出K-Sort Arena，采用 K-wise 比较，允许 K 个模型参与自由混战，提供比成对比较更丰富的信息，并设计基于探索-利用的匹配算法和概率建模，从而实现更高效和更可靠的模型排名。目前，K-Sort Arena 已收集几千次高质量投票并构建了全面的模型排行榜，已用于评估几十种最先进的视觉生成模型，包括文生图和文生视频模型。K-Sort Arena已经历数月的项目内测，期间收到来自加州大学伯克利分校, 新加坡国立大学, 卡内基梅隆大学, 斯坦福大学, 普林斯顿大学, 北京大学等数十家机构的专业人员的技术反馈，现已公开线上发布。","descriptions":{"en":"K-Sort Arena employs K-wise comparison, allowing K models to participate in a free-for-all, providing richer information than pairwise comparison. It also designs a matching algorithm based on exploration-exploitation and probabilistic modeling to achieve more efficient and reliable model ranking.","zh":"本项目提出K-Sort Arena，采用 K-wise 比较，允许 K 个模型参与自由混战，提供比成对比较更丰富的信息，并设计基于探索-利用的匹配算法和概率建模，从而实现更高效和更可靠的模型排名。目前，K-Sort Arena 已收集几千次高质量投票并构建了全面的模型排行榜，已用于评估几十种最先进的视觉生成模型，包括文生图和文生视频模型。K-Sort Arena已经历数月的项目内测，期间收到来自加州大学伯克利分校, 新加坡国立大学, 卡内基梅隆大学, 斯坦福大学, 普林斯顿大学, 北京大学等数十家机构的专业人员的技术反馈，现已公开线上发布。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"opencompass:2076","languages":[],"library_direction":null,"modality":"multimodal","name":"K-Sort-Arena","openness":"unknown","publisher":"UC Berkeley, 中科院自动化所","released":"2024-09-02","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2076-k-sort-arena","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/K-Sort-Arena","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:ogmenrobotics/k9bench-eval-code","source_record":"bm_k9-bench_cbdfe117"},"categories":["Multimodal","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"K9-Bench is a benchmark focused on real-world domestic dog videos, with approximately 5,000 question-answer pairs across 907 videos spanning 5 task categories. The tasks test long-form, canine-centric multimodal reasoning, including action and interaction understanding. A VLM/LLM-powered pipeline was used for data generation.","descriptions":{"en":"K9-Bench is a benchmark focused on real-world domestic dog videos, with approximately 5,000 question-answer pairs across 907 videos spanning 5 task categories. The tasks test long-form, canine-centric multimodal reasoning, including action and interaction understanding. A VLM/LLM-powered pipeline was used for data generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.02680","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_k9-bench_cbdfe117"},"modality":null,"name":"K9-Bench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02680","source_url":"https://arxiv.org/abs/2607.02680"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02680","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02680","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Factuality","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A multi-turn benchmark with 238 tasks using a user simulator, stateful tools, and environment assertions to test LLM agents' handling of knowledge conflicts across world, input, and temporal categories.","descriptions":{"en":"A multi-turn benchmark with 238 tasks using a user simulator, stateful tools, and environment assertions to test LLM agents' handling of knowledge conflicts across world, input, and temporal categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.03588","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_kc-bench_089cdd9b"},"modality":null,"name":"KC-Bench","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03588","source_url":"https://arxiv.org/abs/2609.03588"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03588","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03588","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1290,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:scalingintelligence/kernelbench","source_record":"catalog_aee8b5fdc1e6d419"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code","systems"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Kernel Bench L3 evaluates agentic GPU kernel optimization across 50 problems. Qwen reports two metrics for this benchmark: median per-problem speedup over the PyTorch eager reference and the fraction of problems faster than torch.compile.","descriptions":{"en":"Kernel Bench L3 evaluates agentic GPU kernel optimization across 50 problems. Qwen reports two metrics for this benchmark: median per-problem speedup over the PyTorch eager reference and the fraction of problems faster than torch.compile."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:kernel-bench-l3:qwen3.7-max","reported_at":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernel-bench-l3?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:kernel-bench-l3","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_aee8b5fdc1e6d419"},"modality":"text","name":"Kernel Bench L3","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":96.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.96,"raw_min":0.96,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:kernel-bench-l3:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernel-bench-l3?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"}],"slug":"llm-stats-kernel-bench-l3","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernel-bench-l3?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1290,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:scalingintelligence/kernelbench","source_record":"catalog_5fe3672fd4c09c4f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code","systems"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"KernelBench Hard evaluates agentic GPU kernel optimization on the hardest problem set. Each question is scored by the agent's submitted operator TFLOPs relative to the theoretical peak of the current hardware, with the benchmark score being the average across all questions.","descriptions":{"en":"KernelBench Hard evaluates agentic GPU kernel optimization on the hardest problem set. Each question is scored by the agent's submitted operator TFLOPs relative to the theoretical peak of the current hardware, with the benchmark score being the average across all questions."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:kernelbench-hard:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelbench-hard?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:kernelbench-hard","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_5fe3672fd4c09c4f"},"modality":"text","name":"KernelBench Hard","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":28.799999999999997,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.288,"raw_min":0.288,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:kernelbench-hard:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelbench-hard?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-kernelbench-hard","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelbench-hard?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code","systems"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"KernelGen 1P is an OpenAI AI-self-improvement evaluation that measures whether models can write and optimize compute kernels, part of the suite tracking progress toward accelerating internal research.","descriptions":{"en":"KernelGen 1P is an OpenAI AI-self-improvement evaluation that measures whether models can write and optimize compute kernels, part of the suite tracking progress toward accelerating internal research."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:kernelgen-1p:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelgen-1p?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:kernelgen-1p","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_723e107c4b61a832"},"modality":"text","name":"KernelGen 1P","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":61.1,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.611,"raw_min":0.224,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:kernelgen-1p:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelgen-1p?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-kernelgen-1p","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelgen-1p?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":21,"github_stars_30d":6,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:flagos-ai/kernelgenbench","source_record":"bm_kernelgenbench_691527e5"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"KernelGenBench evaluates LLM- and agent-generated Triton kernels across 210 operators from three sources (ATen, vLLM, cuBLAS) and six hardware platforms, with automatic accuracy verification and two evaluation tracks (LLM Pass@K and iterative agent generation).","descriptions":{"en":"KernelGenBench evaluates LLM- and agent-generated Triton kernels across 210 operators from three sources (ATen, vLLM, cuBLAS) and six hardware platforms, with automatic accuracy verification and two evaluation tracks (LLM Pass@K and iterative agent generation)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27231","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_kernelgenbench_691527e5"},"modality":null,"name":"KernelGenBench","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27231","source_url":"https://arxiv.org/abs/2607.27231"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27231","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27231","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating coding agents on exploit primitive generation against real operating-system kernels.","descriptions":{"en":"A benchmark for evaluating coding agents on exploit primitive generation against real operating-system kernels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.25591","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding","cybersecurity"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_kex-bench_efee8e99"},"modality":null,"name":"KEX-bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25591","source_url":"https://arxiv.org/abs/2609.25591"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25591","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25591","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":10,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":43,"join":"claire_record_id","repository":"gh:cactusqq/keyframe-compass","source_record":"bm_keyframe-compass_c03623fb"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"KeyFrame-Compass evaluates keyframe-conditioned video generation across 386 curated samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities. It jointly measures keyframe execution (presence, fidelity, temporal ordering, localization, persistence, uniqueness) and overall video quality via evidence-grounded MLLM judgments and specialized perception models.","descriptions":{"en":"KeyFrame-Compass evaluates keyframe-conditioned video generation across 386 curated samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities. It jointly measures keyframe execution (presence, fidelity, temporal ordering, localization, persistence, uniqueness) and overall video quality via evidence-grounded MLLM judgments and specialized perception models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.14202","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_keyframe-compass_c03623fb"},"modality":null,"name":"KeyFrame-Compass","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14202","source_url":"https://arxiv.org/abs/2607.14202"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14202","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14202","topics":[],"unit":null},{"aliases":["Evaluation of Pipelines for Data Integration into Knowledge Graphs"],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"KGI-Bench evaluates pipelines that integrate multiple input-data formats into an existing knowledge graph.","descriptions":{"en":"KGI-Bench evaluates pipelines that integrate multiple input-data formats into an existing knowledge graph."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22304","languages":[],"library_direction":null,"modality":null,"name":"KGI-Bench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22304","source_url":"https://arxiv.org/abs/2605.22304"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22304","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22304","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":20,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_khatiandoc_7e28ecbb"},"categories":["Multimodal","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A four-task benchmark built from 107 real RS Khatian land records evaluating symbol recognition, base-16 conversion, structured field extraction, and legal QA with 1,634 QA pairs.","descriptions":{"en":"A four-task benchmark built from 107 real RS Khatian land records evaluating symbol recognition, base-16 conversion, structured field extraction, and legal QA with 1,634 QA pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.03597","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_khatiandoc_7e28ecbb"},"modality":null,"name":"KhatianDoc","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03597","source_url":"https://arxiv.org/abs/2609.03597"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03597","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03597","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:michigannlp/kidbench","source_record":"bm_kidbench_e196ca1f"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"KIDBench evaluates child-facing safety of LLMs for ages 7-11 using realistic queries and multi-turn child-actor simulations, scored by an LLM-as-a-Judge rubric.","descriptions":{"en":"KIDBench evaluates child-facing safety of LLMs for ages 7-11 using realistic queries and multi-turn child-actor simulations, scored by an LLM-as-a-Judge rubric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.25510","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_kidbench_e196ca1f"},"modality":null,"name":"KIDBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25510","source_url":"https://arxiv.org/abs/2605.25510"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25510","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25510","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Kimi Claw 24/7 Bench is Moonshot AI's in-house benchmark for evaluating long-horizon agentic performance in persistent, multi-day coworking tasks. It spans 17 professional scenarios across 610 evaluation points, covering software engineering, ML research, recruiting, trading, and marketing tasks executed through the OpenClaw harness.","descriptions":{"en":"Kimi Claw 24/7 Bench is Moonshot AI's in-house benchmark for evaluating long-horizon agentic performance in persistent, multi-day coworking tasks. It spans 17 professional scenarios across 610 evaluation points, covering software engineering, ML research, recruiting, trading, and marketing tasks executed through the OpenClaw harness."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:kimi-claw-24-7-bench:kimi-k2.7-code","reported_at":"2026-06-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-claw-24-7-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:kimi-claw-24-7-bench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_07f435211e9e9d57"},"modality":"text","name":"Kimi Claw 24/7 Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":46.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.469,"raw_min":0.469,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:kimi-claw-24-7-bench:kimi-k2.7-code","reported_date":"2026-06-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-claw-24-7-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"}],"slug":"llm-stats-kimi-claw-24-7-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-claw-24-7-bench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Kimi Code Bench v2 is Moonshot AI's in-house benchmark for evaluating coding agents on realistic software engineering tasks across 10+ mainstream programming languages and a production tech stack spanning backend services, infrastructure, performance engineering, systems programming, security, frontend development, and ML/data engineering.","descriptions":{"en":"Kimi Code Bench v2 is Moonshot AI's in-house benchmark for evaluating coding agents on realistic software engineering tasks across 10+ mainstream programming languages and a production tech stack spanning backend services, infrastructure, performance engineering, systems programming, security, frontend development, and ML/data engineering."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:kimi-code-bench-v2:kimi-k2.7-code","reported_at":"2026-06-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-code-bench-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:kimi-code-bench-v2","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_a9518b9406a1a0bb"},"modality":"text","name":"Kimi Code Bench v2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":72.89999999999999,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.729,"raw_min":0.62,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:kimi-code-bench-v2:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-code-bench-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"}],"slug":"llm-stats-kimi-code-bench-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-code-bench-v2?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:2077ai/kina","source_record":"catalog_bb9363753ede1201"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","knowledge","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"KINA is a knowledge-intensive evaluation that measures a model's breadth and depth of factual knowledge across academic and professional domains.","descriptions":{"en":"KINA is a knowledge-intensive evaluation that measures a model's breadth and depth of factual knowledge across academic and professional domains."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:kina:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kina?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:kina","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_bb9363753ede1201"},"modality":"text","name":"KINA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":48.3,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.483,"raw_min":0.466,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:kina:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kina?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-kina","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kina?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"KineBench evaluates embodied world models via kinematic grounding, using 20 manipulation tasks in ManiSkill3 and metrics like spectral arc length and manipulability index.","descriptions":{"en":"KineBench evaluates embodied world models via kinematic grounding, using 20 manipulation tasks in ManiSkill3 and metrics like spectral arc length and manipulability index."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19876","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","embodied-robotics"],"join":"claire_record_id","primary":"world-models","source_record":"bm_kinebench_86d2d206"},"modality":null,"name":"KineBench","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19876","source_url":"https://arxiv.org/abs/2607.19876"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19876","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19876","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","语言","Language","多模态模型","VLM","OCR与文档理解","OCR and Document Understanding","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"KITAB-Bench is a Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding,  and spans 36 sub-domains with over 8,809 samples, carefully curated to rigorously evaluate essential skills required for Arabic OCR and document analysis. KITAB-Bench是一个全面多领域阿拉伯文 OCR 和文档理解基准，包含 36 个子领域，超过 8,809 个样本，经过精心挑选，以严格评估阿拉伯 OCR 和文档分析所需的基本技能。","descriptions":{"en":"KITAB-Bench is a Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding,  and spans 36 sub-domains with over 8,809 samples, carefully curated to rigorously evaluate essential skills required for Arabic OCR and document analysis.","zh":"KITAB-Bench是一个全面多领域阿拉伯文 OCR 和文档理解基准，包含 36 个子领域，超过 8,809 个样本，经过精心挑选，以严格评估阿拉伯 OCR 和文档分析所需的基本技能。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1543","languages":["Arabic"],"library_direction":null,"modality":"multimodal","name":"KITAB-Bench","openness":"unknown","publisher":"Mohamed bin Zayed University of AI;etc.","released":"2025-02-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1543-kitab-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KITAB-Bench","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_klinikebench_4568c753"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions.","descriptions":{"en":"Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38480","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_klinikebench_4568c753"},"modality":null,"name":"KlinikeBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38480","source_url":"https://arxiv.org/abs/2609.38480"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38480","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38480","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"artifact:arxiv:2507.08924","repository":null,"source_record":"catalog_34190150a99b5938"},"categories":["知识","Knowledge","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"KMMLU-Redux is a reconstructed version of the existing KMMLU, comprising 2,587 problems from Korean National Technical Qualification (KNTQ) exams. KMMLU-Redux是现有 KMMLU 的一个重建版本，包含来自韩国国家技术资格（KNTQ）考试的 2,587 个问题。我们发现了 KMMLU 中的一些关键问题，包括泄露的答案、缺乏清晰度、问题表述不当、符号错误和污染风险。","descriptions":{"en":"KMMLU-Redux is a reconstructed version of the existing KMMLU, comprising 2,587 problems from Korean National Technical Qualification (KNTQ) exams.","zh":"KMMLU-Redux是现有 KMMLU 的一个重建版本，包含来自韩国国家技术资格（KNTQ）考试的 2,587 个问题。我们发现了 KMMLU 中的一些关键问题，包括泄露的答案、缺乏清晰度、问题表述不当、符号错误和污染风险。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:2126","languages":["Korean"],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"artifact:arxiv:2507.08924","primary":"knowledge-expert-qa","source_record":"catalog_34190150a99b5938"},"modality":null,"name":"KMMLU-Redux","openness":"restricted","publisher":"LGA IResearch , Oneline AI","released":"2025-07-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2126-kmmlu-redux","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KMMLU-Redux","topics":[],"unit":null},{"aliases":["knowledge_tracing_benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:devissaputra/knowledge_tracing_benchmark","source_record":"bm_knowledge-tracing-benchmark_c53af6af"},"categories":["Factuality","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A learner-disjoint benchmark on ASSISTments 2009 sequences that evaluates knowledge-tracing models on next-response probability with ROC-AUC, average precision, Brier score, log loss, and ECE-10.","descriptions":{"en":"A learner-disjoint benchmark on ASSISTments 2009 sequences that evaluates knowledge-tracing models on next-response probability with ROC-AUC, average precision, Brier score, log loss, and ECE-10."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:devissaputra/knowledge_tracing_benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_knowledge-tracing-benchmark_c53af6af"},"modality":null,"name":"Knowledge Tracing Benchmark on ASSISTments 2009","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:devissaputra/knowledge_tracing_benchmark","source_url":"https://github.com/devissaputra/knowledge_tracing_benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-devissaputra-knowledge-tracing-benchmark","source":"claire_radar","source_url":"https://github.com/devissaputra/knowledge_tracing_benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"KnowLogic is a knowledge-driven synthetic benchmark designed to evaluate the reasoning abilities of large language models (LLMs). It includes 5400 bilingual (Chinese and English) questions across various domains, covering different aspects of commonsense knowledge and logical reasoning. KnowLogic 是一个以知识驱动的合成基准，旨在评估大型语言模型的推理能力（LLMs）。它包含涵盖各个领域、涵盖常识知识和逻辑推理不同方面的 5400 个中英双语问题。","descriptions":{"en":"KnowLogic is a knowledge-driven synthetic benchmark designed to evaluate the reasoning abilities of large language models (LLMs). It includes 5400 bilingual (Chinese and English) questions across various domains, covering different aspects of commonsense knowledge and logical reasoning.","zh":"KnowLogic 是一个以知识驱动的合成基准，旨在评估大型语言模型的推理能力（LLMs）。它包含涵盖各个领域、涵盖常识知识和逻辑推理不同方面的 5400 个中英双语问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1635","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"KnowLogic","openness":"restricted","publisher":"PKU, etc.","released":"2025-03-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1635-knowlogic","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KnowLogic","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","VQA","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"KOFFVQA is a carefully crafted free-form visual question answering(VQA) benchmark in the Korean language consisting of 275 questions across 10 different tasks. KOFFVQA是一个精心设计的韩语自由形式视觉问答（VQA）基准测试，包含10个不同任务中的275个问题。","descriptions":{"en":"KOFFVQA is a carefully crafted free-form visual question answering(VQA) benchmark in the Korean language consisting of 275 questions across 10 different tasks.","zh":"KOFFVQA是一个精心设计的韩语自由形式视觉问答（VQA）基准测试，包含10个不同任务中的275个问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1706","languages":["Korean"],"library_direction":null,"modality":"multimodal","name":"KOFFVQA","openness":"unknown","publisher":"MAUM AI Inc. / Republic of Korea","released":"2025-03-31","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1706-koffvqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KOFFVQA","topics":["multimodal"],"unit":null},{"aliases":["Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models"],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":"gh:mz-kim/kona","source_record":"bm_knowing-what-not-to-answer-selective-non-c_abb19683"},"categories":["General AI","Multimodal","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"KoNA evaluates selective non-compliance in vision-language models across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. Each task measures query-level and component-level non-compliance using paired single and compound queries.","descriptions":{"en":"KoNA evaluates selective non-compliance in vision-language models across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. Each task measures query-level and component-level non-compliance using paired single and compound queries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.04720","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","visual-reasoning"],"join":"claire_record_id","primary":"factuality","source_record":"bm_knowing-what-not-to-answer-selective-non-c_abb19683"},"modality":null,"name":"KoNA","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04720","source_url":"https://arxiv.org/abs/2609.04720"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04720","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04720","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bcmilab/ko-neobench","source_record":"bm_koneobench_a816badb"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"KoNeoBench evaluates LLM understanding of Korean neologisms through four tasks: context-based identification, component reconstruction, semantic/special-domain classification, and definition generation.","descriptions":{"en":"KoNeoBench evaluates LLM understanding of Korean neologisms through four tasks: context-based identification, component reconstruction, semantic/special-domain classification, and definition generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.19916","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_koneobench_a816badb"},"modality":null,"name":"KoNeoBench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19916","source_url":"https://arxiv.org/abs/2609.19916"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19916","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19916","topics":["language"],"unit":null},{"aliases":["Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-step tool-calling over Korean open public APIs with 145 real-world tasks, designed to measure performance gaps in open-source LLM agents that chain tool calls across live government APIs.","descriptions":{"en":"Evaluates multi-step tool-calling over Korean open public APIs with 145 real-world tasks, designed to measure performance gaps in open-source LLM agents that chain tool calls across live government APIs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.05395","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["tool-use","multilingual"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_multi-step-tool-calling-over-korean-open-p_4b4d2733"},"modality":null,"name":"KOPA-Bench","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05395","source_url":"https://arxiv.org/abs/2609.05395"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05395","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05395","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","大语言模型","LLM","逻辑推理","Reasoning","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench) encompasses five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. KOR-Bench emphasizes the effectiveness of models in applying new rule descriptions to solve novel rule-driven questions. KOR-Bench用于评估大语言模型的推理能力，包括五个任务类别：操作、逻辑、密码、拼图和反事实。","descriptions":{"en":"Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench) encompasses five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. KOR-Bench emphasizes the effectiveness of models in applying new rule descriptions to solve novel rule-driven questions.","zh":"KOR-Bench用于评估大语言模型的推理能力，包括五个任务类别：操作、逻辑、密码、拼图和反事实。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1245","languages":[],"library_direction":null,"modality":null,"name":"KOR-Bench","openness":"unknown","publisher":"Multimodal Art Projection Research Community","released":"2024-10-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1245-kor-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KOR-Bench","topics":["reasoning-math"],"unit":null},{"aliases":["KORBench"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Knowledge-Orthogonal Reasoning Benchmark; evaluates intrinsic reasoning and planning abilities while minimizing interference from pretrained knowledge.","descriptions":{"en":"Knowledge-Orthogonal Reasoning Benchmark; evaluates intrinsic reasoning and planning abilities while minimizing interference from pretrained knowledge."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000korbench\u0000bytedance_seed2_0_model_card\u0000korbench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000korbench\u0000bytedance_seed2_0_model_card\u0000korbench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:korbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"KORBench","openness":"unknown","publisher":null,"released":"2024-10-09","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"model-reports:korbench","source_url":"https://arxiv.org/abs/2410.06526"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":77.5,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":77.5,"raw_min":72.8,"source_reference":{"obs_id":"curated\u0000korbench\u0000bytedance_seed2_0_model_card\u0000korbench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000korbench\u0000bytedance_seed2_0_model_card\u0000korbench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"korbench","source":"model_reports","source_url":"https://kor-bench.github.io/","topics":["reasoning-math"],"unit":"percent"},{"aliases":["Kotlin SWE Benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":54,"github_stars_30d":4,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kotlin/kotlin-swe-bench","source_record":"bm_kotlin-benchmark_jetbrains-2026"},"categories":["Code & Software","Code generation","Coding Agents","Kotlin","Software & AI Compute","Software engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI coding agents on real-world Kotlin tasks from nine open-source repositories. The benchmark uses reproducible Docker environments, regression tests, and a scoring protocol that awards pass only when all expected test transitions are met.","descriptions":{"en":"Evaluates AI coding agents on real-world Kotlin tasks from nine open-source repositories. The benchmark uses reproducible Docker environments, regression tests, and a scoring protocol that awards pass only when all expected test transitions are met."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:jetbrains-kotlin-benchmark-2026","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_kotlin-benchmark_jetbrains-2026"},"modality":null,"name":"Kotlin Benchmark","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"first_public","source_key":"claire-radar:jetbrains-kotlin-benchmark-2026","source_url":"https://kotlinlang.org/benchmark/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-jetbrains-kotlin-benchmark-2026","source":"claire_radar","source_url":"https://kotlinlang.org/benchmark/","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_ksafe-mm_8d1d1de0"},"categories":["Cybersecurity","Multimodal","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"KSAFE-MM evaluates multimodal LLM safety in Korean contexts, with 12 models tested on general and culture-specific safety risks, including jailbreak-style textual queries paired with local visual cues.","descriptions":{"en":"KSAFE-MM evaluates multimodal LLM safety in Korean contexts, with 12 models tested on general and culture-specific safety risks, including jailbreak-style textual queries paired with local visual cues."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28013","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_ksafe-mm_8d1d1de0"},"modality":null,"name":"KSAFE-MM","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28013","source_url":"https://arxiv.org/abs/2605.28013"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28013","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28013","topics":["multimodal","language","safety","cybersecurity"],"unit":null},{"aliases":["KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:chosenqc/kvdiagnosis","source_record":"bm_kvdiagnosis-a-diagnostic-benchmark-for-kv-_86d87177"},"categories":["General AI","Language & Knowledge","Long Context"],"collected_at":"2026-10-14T09:44:51Z","description":"Diagnoses KV-cache compression failures in long-context language models using paired runs and cache, likelihood, attention, and decoding measurements.","descriptions":{"en":"Diagnoses KV-cache compression failures in long-context language models using paired runs and cache, likelihood, attention, and decoding measurements."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.09412","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"claire_record_id","primary":"long-context","source_record":"bm_kvdiagnosis-a-diagnostic-benchmark-for-kv-_86d87177"},"modality":null,"name":"KVDiagnosis","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09412","source_url":"https://arxiv.org/abs/2608.09412"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09412","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09412","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["长文本","Long-Context","大语言模型","LLM","长上下文","Long Context","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"L-Eval is a comprehensive Long Context Language Models (LCLMs) evaluation suite with 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs encompassing diverse question styles, domains, and input length (3k～200k tokens). L-Eval 是一个全面的长上下文语言模型（LCLMs）评估套件，包括 20 个子任务、508 个长文档和超过 2,000 个人工标记的查询-响应对。它涵盖了多种问答风格、领域和输入长度（3,000 至 200,000 个 token）。","descriptions":{"en":"L-Eval is a comprehensive Long Context Language Models (LCLMs) evaluation suite with 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs encompassing diverse question styles, domains, and input length (3k～200k tokens).","zh":"L-Eval 是一个全面的长上下文语言模型（LCLMs）评估套件，包括 20 个子任务、508 个长文档和超过 2,000 个人工标记的查询-响应对。它涵盖了多种问答风格、领域和输入长度（3,000 至 200,000 个 token）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:541","languages":[],"library_direction":null,"modality":null,"name":"L-Eval","openness":"restricted","publisher":null,"released":"2023-10-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-541-l-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/L-Eval","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_l3cube-indicquest-v2_c9db7d34"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"L3Cube-IndicQuest v2 is a multilingual QA benchmark for evaluating India-specific factual knowledge of LLMs. It contains 3,471 curriculum-grounded English QA pairs translated into 19 Indic languages, totaling 69,420 pairs across 20 languages.","descriptions":{"en":"L3Cube-IndicQuest v2 is a multilingual QA benchmark for evaluating India-specific factual knowledge of LLMs. It contains 3,471 curriculum-grounded English QA pairs translated into 19 Indic languages, totaling 69,420 pairs across 20 languages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.15535","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","multilingual"],"join":"claire_record_id","primary":"factuality","source_record":"bm_l3cube-indicquest-v2_c9db7d34"},"modality":null,"name":"L3Cube-IndicQuest v2","openness":"unknown","publisher":null,"released":"2026-08-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15535","source_url":"https://arxiv.org/abs/2608.15535"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15535","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15535","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":65,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:edisonscientific/labbench2","source_record":"catalog_3f2ae1620bac7104"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","agents","biology"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LABBench2 evaluates models on real-world biology research tasks.","descriptions":{"en":"LABBench2 evaluates models on real-world biology research tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:labbench2:gemini-3.7-flash","reported_at":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/labbench2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:labbench2","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_3f2ae1620bac7104"},"modality":"text","name":"LABBench2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":82.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.821,"raw_min":0.821,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:labbench2:gemini-3.7-flash","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/labbench2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"}],"slug":"llm-stats-labbench2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/labbench2?top_n=500","topics":["agents","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"LabOSBench evaluates multimodal GUI agents on 96 subtasks across eight web-based scientific-instrument simulators, covering workflows from sample loading to result inspection. Agents operate via a browser, with execution-based evaluation on task completion.","descriptions":{"en":"LabOSBench evaluates multimodal GUI agents on 96 subtasks across eight web-based scientific-instrument simulators, covering workflows from sample loading to result inspection. Agents operate via a browser, with execution-based evaluation on task completion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16802","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","science"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_labosbench_5eb589a2"},"modality":null,"name":"LabOSBench","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16802","source_url":"https://arxiv.org/abs/2606.16802"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16802","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16802","topics":["computer-use","agents","multimodal","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":276,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_ladbench_44ed9c65"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"LADBench evaluates large vision-language models on detecting logical anomalies in synthetic images across four domains: Residential, Urban, Collaborative, and Nature. It uses a Tiered Prompting Protocol with three progressive disclosure levels and automated scoring.","descriptions":{"en":"LADBench evaluates large vision-language models on detecting logical anomalies in synthetic images across four domains: Residential, Urban, Collaborative, and Nature. It uses a Tiered Prompting Protocol with three progressive disclosure levels and automated scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17433","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_ladbench_44ed9c65"},"modality":null,"name":"LADBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17433","source_url":"https://arxiv.org/abs/2606.17433"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17433","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17433","topics":[],"unit":null},{"aliases":["AI translation of literary texts is \"fine\", but readers still prefer human translations"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"LAIT is a reader-centered evaluation resource for comparing human and machine literary translations through excerpt-level and aligned chunk-level judgments.","descriptions":{"en":"LAIT is a reader-centered evaluation resource for comparing human and machine literary translations through excerpt-level and aligned chunk-level judgments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26040","languages":[],"library_direction":null,"modality":null,"name":"LAIT (Literary AI Translation)","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26040","source_url":"https://arxiv.org/abs/2606.26040"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26040","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26040","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"LakeQA evaluates search-centric question answering over a 9.5 TB data lake of Wikipedia and government data. Tasks require multi-hop reasoning across heterogeneous structured and unstructured sources, with expert-annotated answers.","descriptions":{"en":"LakeQA evaluates search-centric question answering over a 9.5 TB data lake of Wikipedia and government data. Tasks require multi-hop reasoning across heterogeneous structured and unstructured sources, with expert-annotated answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.10460","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_lakeqa_b77b7993"},"modality":null,"name":"LakeQA","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10460","source_url":"https://arxiv.org/abs/2606.10460"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10460","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10460","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"LakeQuest evaluates end-to-end question answering over data lakes with 9,846 QA pairs across three domains (AI/ML metadata, retail banking, biomedical drug info), with exact modality-aware evidence pointers, measuring retrieval and cross-modal synthesis.","descriptions":{"en":"LakeQuest evaluates end-to-end question answering over data lakes with 9,846 QA pairs across three domains (AI/ML metadata, retail banking, biomedical drug info), with exact modality-aware evidence pointers, measuring retrieval and cross-modal synthesis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.12310","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_lakequest_19d4b53f"},"modality":null,"name":"LakeQuest","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.12310","source_url":"https://arxiv.org/abs/2607.12310"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-12310","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12310","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:1606.06031","repository":null,"source_record":"catalog_b34138a022fc6ca5"},"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The LAMBADA evaluates the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse.\nThe LAMBADA dataset is extracted from BookCorpus and consists of 10'022 passages, divided into 4'869 development and 5'153 test passages, comprising 203 million words. LAMBADA 通过一个单词预测任务来评估计算模型对文本理解的能力。LAMBADA 是有如下特点的一组叙述性文章：如果面对整篇文章，人们可以猜测它们的最后一个单词，但如果他们只看到目标单词前面的最后一句话，就无法猜测。为了在 LAMBADA 上由好的效果，模型不能仅仅依赖于局部上下文，而必须能够跟踪更广泛的话语信息。\nLAMBADA 数据集是从 BookCorpus 中提取的，包括 10,022 段落，分为 4,869 个开发段落和 5,153 个测试段落，共计 2.03 亿个单词。","descriptions":{"en":"The LAMBADA evaluates the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse.\nThe LAMBADA dataset is extracted from BookCorpus and consists of 10'022 passages, divided into 4'869 development and 5'153 test passages, comprising 203 million words.","zh":"LAMBADA 通过一个单词预测任务来评估计算模型对文本理解的能力。LAMBADA 是有如下特点的一组叙述性文章：如果面对整篇文章，人们可以猜测它们的最后一个单词，但如果他们只看到目标单词前面的最后一句话，就无法猜测。为了在 LAMBADA 上由好的效果，模型不能仅仅依赖于局部上下文，而必须能够跟踪更广泛的话语信息。\nLAMBADA 数据集是从 BookCorpus 中提取的，包括 10,022 段落，分为 4,869 个开发段落和 5,153 个测试段落，共计 2.03 亿个单词。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:523","languages":[],"library_direction":null,"modality":null,"name":"LAMBADA","openness":"unknown","publisher":null,"released":"2016-06-20","released_reference":{"basis":"paper_first_version","note":"First version introducing the LAMBADA word-prediction dataset.","source_key":"opencompass:523","source_url":"https://arxiv.org/abs/1606.06031"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-523-lambada","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LAMBADA","topics":["language"],"unit":null},{"aliases":["Semi-supervised Source Detection in Astronomical Images: New Benchmark and Strong Baseline"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A semi-supervised astronomical source-detection benchmark comprising 18,400 images and 728,898 source instances.","descriptions":{"en":"A semi-supervised astronomical source-detection benchmark comprising 18,400 images and 728,898 source instances."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09219","languages":[],"library_direction":null,"modality":null,"name":"LAMOST-DET","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09219","source_url":"https://arxiv.org/abs/2606.09219"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09219","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09219","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","知识储备","Knowledge"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The benchmark covers questions from three major categories: (1) Arts & Entertainment, (2) Lifestyle & Personal Development, and (3) Society & Culture, encompassing over 45 subcategories in total. 基准旨在评估个性化长篇答案生成。该基准涵盖三大类问题:(1)艺术与娱乐,(2)生活与个人发展,(3)社会与文化,共包含45个以上的子类别。","descriptions":{"en":"The benchmark covers questions from three major categories: (1) Arts & Entertainment, (2) Lifestyle & Personal Development, and (3) Society & Culture, encompassing over 45 subcategories in total.","zh":"基准旨在评估个性化长篇答案生成。该基准涵盖三大类问题:(1)艺术与娱乐,(2)生活与个人发展,(3)社会与文化,共包含45个以上的子类别。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1924","languages":[],"library_direction":null,"modality":null,"name":"LaMP-QA","openness":"restricted","publisher":"Center for Intelligent Information Retrieval (CIIR) ,etc.","released":"2025-05-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1924-lamp-qa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LaMP-QA","topics":[],"unit":null},{"aliases":["benchmark-llms-landuse-relevance","Land-use relevance: small-LLM benchmark results"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"This entry is a dataset of predictions and scores for small LLMs on a land-use relevance classification task, not an original standalone benchmark with a stable evaluation contract.","descriptions":{"en":"This entry is a dataset of predictions and scores for small LLMs on a land-use relevance classification task, not an original standalone benchmark with a stable evaluation contract."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:noeflandre/benchmark-llms-landuse-relevance","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_benchmark-llms-landuse-relevance_7050e5ea"},"modality":null,"name":"Land-use relevance benchmark","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:noeflandre/benchmark-llms-landuse-relevance","source_url":"https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-noeflandre-benchmark-llms-landuse-relevance","source":"claire_radar","source_url":"https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Code generation","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"LangChoiceBench measures Python preference in project-level code generation across 28 projects and seven software areas, assessing language choice, recommendation-implementation consistency, and language diversity.","descriptions":{"en":"LangChoiceBench measures Python preference in project-level code generation across 28 projects and seven software areas, assessing language choice, recommendation-implementation consistency, and language diversity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06041","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_langchoicebench_33a5b10b"},"modality":null,"name":"LangChoiceBench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06041","source_url":"https://arxiv.org/abs/2608.06041"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06041","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06041","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","物理智能","Embodied AI","语言理解","Comprehension","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LangNavBench is a benchmark for evaluating natural language understanding in semantic navigation, built upon the manually verified LangNav open-set dataset. LangNavBench是一个语义导航中自然语言理解评测基准，基于手工验证的LangNav开放集数据集，评测具身智能体在自然语言指令引导下的目标定位能力，涵盖类别层次理解、对象属性识别和空间关系推理等维度.","descriptions":{"en":"LangNavBench is a benchmark for evaluating natural language understanding in semantic navigation, built upon the manually verified LangNav open-set dataset.","zh":"LangNavBench是一个语义导航中自然语言理解评测基准，基于手工验证的LangNav开放集数据集，评测具身智能体在自然语言指令引导下的目标定位能力，涵盖类别层次理解、对象属性识别和空间关系推理等维度."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2129","languages":[],"library_direction":null,"modality":null,"name":"LangNavBench","openness":"unknown","publisher":"Simon Fraser University , University of Padova , Fondazione Bruno Kessler ,etc","released":"2025-07-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2129-langnavbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LangNavBench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Closed-loop benchmark for temporally grounded cooperative warnings with six scenario families, 3,272 training sequences, and 120 matched route pairs for evaluation.","descriptions":{"en":"Closed-loop benchmark for temporally grounded cooperative warnings with six scenario families, 3,272 training sequences, and 120 matched route pairs for evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.06368","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_lantern_ade8b0b1"},"modality":null,"name":"LANTERN","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06368","source_url":"https://arxiv.org/abs/2609.06368"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06368","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06368","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["长文本","Long-Context","大语言模型","LLM","长上下文","Long Context","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LaRA is a focused benchmark for testing Retrieval-Augmented Generation (RAG) and long-context LLMs. LaRA 是专为评估检索增强生成（RAG）和长上下文大型语言模型（LLM）而打造的基准。它围绕信息定位、片段对比、内容推理和幻觉检测四大能力，通过 2 326 条测试用例，对四类问答任务和三种长文本场景（小说、学术论文、财务报表）进行全面测评。","descriptions":{"en":"LaRA is a focused benchmark for testing Retrieval-Augmented Generation (RAG) and long-context LLMs.","zh":"LaRA 是专为评估检索增强生成（RAG）和长上下文大型语言模型（LLM）而打造的基准。它围绕信息定位、片段对比、内容推理和幻觉检测四大能力，通过 2 326 条测试用例，对四类问答任务和三种长文本场景（小说、学术论文、财务报表）进行全面测评。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2089","languages":[],"library_direction":null,"modality":null,"name":"LaRA","openness":"unknown","publisher":"The Hong Kong University of Science and Technology , Tongyi Lab, etc.","released":"2025-02-14","released_reference":{"basis":"paper_first_version","note":"First version introducing the LaRA benchmark, before its July Hub-card creation.","source_key":"opencompass:2089","source_url":"https://arxiv.org/abs/2502.09977"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2089-lara","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LaRA","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A collection of human-authored and peer-reviewed examples in text, image, audio, and video that break leading machine translation models, with handcrafted verification rules for each example.","descriptions":{"en":"A collection of human-authored and peer-reviewed examples in text, image, audio, and video that break leading machine translation models, with handcrafted verification rules for each example."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.04173","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","multilingual","visual-reasoning"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_last-translation-benchmark_d81900c1"},"modality":null,"name":"Last Translation Benchmark","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04173","source_url":"https://arxiv.org/abs/2609.04173"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04173","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04173","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates Markdown boundary failures, specifically CommonMark-level fence-boundary accuracy, across 4,179 prompts using a scoring CLI that separates content correctness from boundary correctness.","descriptions":{"en":"Evaluates Markdown boundary failures, specifically CommonMark-level fence-boundary accuracy, across 4,179 prompts using a scoring CLI that separates content correctness from boundary correctness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.06993","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_latentmd_cea161f6"},"modality":null,"name":"LatentMD","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06993","source_url":"https://arxiv.org/abs/2609.06993"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06993","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06993","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:musashams/layerrag-bench","source_record":"bm_layerrag-bench_a4840c44"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates cross-layer reliability of agentic RAG systems on 240 tasks across 8 enterprise domains, with 9 fault scenarios and 2 contract modes; measures success at evidence, tool-contract, authorization, and session-state layers.","descriptions":{"en":"Evaluates cross-layer reliability of agentic RAG systems on 240 tasks across 8 enterprise domains, with 9 fault scenarios and 2 contract modes; measures success at evidence, tool-contract, authorization, and session-state layers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27353","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","professional-work"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_layerrag-bench_a4840c44"},"modality":null,"name":"LayerRAG-Bench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27353","source_url":"https://arxiv.org/abs/2607.27353"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27353","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27353","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"LayerWiseBench evaluates layer-wise chart understanding and editing across 2,800 source charts, with derived questions and editing variants.","descriptions":{"en":"LayerWiseBench evaluates layer-wise chart understanding and editing across 2,800 source charts, with derived questions and editing variants."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08657","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","document-ocr"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_layerwisebench_d27de304"},"modality":null,"name":"LayerWiseBench","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08657","source_url":"https://arxiv.org/abs/2609.08657"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08657","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08657","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DC"],"collected_at":"2026-10-14T09:44:51Z","description":"LayoutBench evaluates three cloud storage layout strategies (individual objects, tar archives, Parquet columns) for multimedia data retrieval, measuring retrieval time, data transferred, and cost.","descriptions":{"en":"LayoutBench evaluates three cloud storage layout strategies (individual objects, tar archives, Parquet columns) for multimedia data retrieval, measuring retrieval time, data transferred, and cost."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.28880","languages":[],"library_direction":null,"modality":null,"name":"LayoutBench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28880","source_url":"https://arxiv.org/abs/2607.28880"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28880","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28880","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LBPP (v2) benchmark - specific documentation not found in official sources, possibly related to language-based planning problems","descriptions":{"en":"LBPP (v2) benchmark - specific documentation not found in official sources, possibly related to language-based planning problems"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:lbpp-(v2):gemini-diffusion","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lbpp-%28v2%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:lbpp-(v2)","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_512407cb147ffc6d"},"modality":"text","name":"LBPP (v2)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":56.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.568,"raw_min":0.568,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:lbpp-(v2):gemini-diffusion","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lbpp-%28v2%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"}],"slug":"llm-stats-lbpp-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lbpp-%28v2%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"LCS-Bench is a theory-scale benchmark for auto-formalization in logics for computer science. It includes 327 textbook items, over 4,076 Lean declarations, and supports five evaluation tracks with definitional equivalence checkers.","descriptions":{"en":"LCS-Bench is a theory-scale benchmark for auto-formalization in logics for computer science. It includes 327 textbook items, over 4,076 Lean declarations, and supports five evaluation tracks with definitional equivalence checkers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26525","languages":[],"library_direction":{"basis":"model-review","categories":["math","code-generation"],"join":"claire_record_id","primary":"math","source_record":"bm_lcs-bench_45178465"},"modality":null,"name":"LCS-Bench","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26525","source_url":"https://arxiv.org/abs/2606.26525"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26525","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26525","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LCSTS is a large corpus of Chinese short text summarization dataset constructed from the Chinese microblogging website Sina Weibo, which is released to the public. This corpus consists of over 2 million real Chinese short texts with short summaries given by the author of each text. LCSTS是一个大规模的中文短文本摘要数据集，从中国微博网站新浪微博中构建而成，并已开源。该数据集包含超过 200 万条真实的中文短文本，每个文本都提供了一个简短的摘要。","descriptions":{"en":"LCSTS is a large corpus of Chinese short text summarization dataset constructed from the Chinese microblogging website Sina Weibo, which is released to the public. This corpus consists of over 2 million real Chinese short texts with short summaries given by the author of each text.","zh":"LCSTS是一个大规模的中文短文本摘要数据集，从中国微博网站新浪微博中构建而成，并已开源。该数据集包含超过 200 万条真实的中文短文本，每个文本都提供了一个简短的摘要。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:520","languages":["Chinese"],"library_direction":null,"modality":null,"name":"LCSTS","openness":"unknown","publisher":null,"released":"2015-06-19","released_reference":{"basis":"paper_first_version","note":"First version introducing the Chinese short-text summarization corpus.","source_key":"opencompass:520","source_url":"https://arxiv.org/abs/1506.05865"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-520-lcsts","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LCSTS","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for referring surgical video instrument segmentation.","descriptions":{"en":"A benchmark for referring surgical video instrument segmentation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23067","languages":[],"library_direction":null,"modality":null,"name":"LD-RSVIS","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23067","source_url":"https://arxiv.org/abs/2609.23067"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23067","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23067","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"LDU-Bench evaluates multimodal LLMs on lithography defect understanding through four tasks: defect triage, morphology recognition, coarse localization, and image-conditioned cause analysis, using task-level metrics and the Lithography Closure Score.","descriptions":{"en":"LDU-Bench evaluates multimodal LLMs on lithography defect understanding through four tasks: defect triage, morphology recognition, coarse localization, and image-conditioned cause analysis, using task-level metrics and the Lithography Closure Score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03078","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_ldu-bench_c9c6a21a"},"modality":null,"name":"LDU-Bench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03078","source_url":"https://arxiv.org/abs/2608.03078"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03078","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03078","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Proposes a challenge set of 500 document images annotated with GDPR-aligned PII to evaluate document-level leakage risk in redaction systems.","descriptions":{"en":"Proposes a challenge set of 500 document images annotated with GDPR-aligned PII to evaluate document-level leakage risk in redaction systems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02207","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["document-ocr","visual-reasoning"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_leakagebench_a7e3cbaa"},"modality":null,"name":"LeakageBench","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02207","source_url":"https://arxiv.org/abs/2609.02207"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02207","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02207","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":12,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":138,"join":"claire_record_id","repository":null,"source_record":"bm_learn2play-bench_f990e40d"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments.","descriptions":{"en":"Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.08215","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_learn2play-bench_f990e40d"},"modality":null,"name":"Learn2Play Bench","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.08215","source_url":"https://arxiv.org/abs/2610.08215"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-08215","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08215","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1446,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:harveyai/harvey-labs","source_record":"catalog_59ee8d88e3ee6f02"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The Legal Agent Benchmark (LAB) is Harvey's open-source benchmark for evaluating AI agents on complex, long-horizon legal work. Tasks are scored under an all-pass standard against expert-curated rubrics, where a task passes only if every required rubric criterion (facts, conclusions, citations, structure, and analytical moves) passes.","descriptions":{"en":"The Legal Agent Benchmark (LAB) is Harvey's open-source benchmark for evaluating AI agents on complex, long-horizon legal work. Tasks are scored under an all-pass standard against expert-curated rubrics, where a task passes only if every required rubric criterion (facts, conclusions, citations, structure, and analytical moves) passes."},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:legal-agent-benchmark:gemini-3-flash-preview","reported_at":"2025-12-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/legal-agent-benchmark?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:legal-agent-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","professional-work"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_59ee8d88e3ee6f02"},"modality":"text","name":"Legal Agent Benchmark","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":13.3,"display_multiplier":100,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":0.133,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:legal-agent-benchmark:claude-fable-5","reported_date":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/legal-agent-benchmark?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"}],"slug":"llm-stats-legal-agent-benchmark","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/legal-agent-benchmark?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["Legal Agent Benchmark"],"attention_signals":null,"categories":["legal"],"collected_at":null,"description":"Open-source agent benchmark from Harvey (github.com/harveyai/harvey- labs): 1,200+ tasks across 24 practice areas, all-pass LLM-judge grading. Absolute scores are very low across all models (0-13%), so ordering at the top is decided by a handful of tasks and small gaps are noise.","descriptions":{"en":"Open-source agent benchmark from Harvey (github.com/harveyai/harvey- labs): 1,200+ tasks across 24 practice areas, all-pass LLM-judge grading. Absolute scores are very low across all models (0-13%), so ordering at the top is decided by a handful of tasks and small gaps are noise."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:legal_agent_benchmark","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["finance-legal"],"join":"review:14H034160212:2026-10-15","primary":"finance-legal","source_record":null},"modality":null,"name":"Legal Agent Benchmark","openness":"unknown","publisher":null,"released":"2026-05-06","released_reference":{"basis":"benchmark_release","source_key":"model-reports:legal_agent_benchmark","source_url":"https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"legal_agent_benchmark","source":"model_reports","source_url":"https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"LegalCiteTrust evaluates citation trustworthiness in Chinese long-form legal research reports, assessing coverage, support, and citation-level existence, fidelity, and applicability.","descriptions":{"en":"LegalCiteTrust evaluates citation trustworthiness in Chinese long-form legal research reports, assessing coverage, support, and citation-level existence, fidelity, and applicability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20872","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","factuality"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_legalcitetrust_1777e861"},"modality":null,"name":"LegalCiteTrust","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20872","source_url":"https://arxiv.org/abs/2607.20872"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20872","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20872","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"This entry describes LegalRewardBench as a benchmark for evaluating grounded legal generation, but no dataset, scorer, or public reuse path is evidenced in the provided information.","descriptions":{"en":"This entry describes LegalRewardBench as a benchmark for evaluating grounded legal generation, but no dataset, scorer, or public reuse path is evidenced in the provided information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.14739","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_legalrewardbench_7df4fd44"},"modality":null,"name":"LegalRewardBench","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.14739","source_url":"https://arxiv.org/abs/2609.14739"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-14739","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14739","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"LegendBench is a parametric benchmark for diagnosing legend understanding in visualizations through counterfactual interventions.","descriptions":{"en":"LegendBench is a parametric benchmark for diagnosing legend understanding in visualizations through counterfactual interventions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24172","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["factuality","health-medicine"],"join":"claire_record_id","primary":"factuality","source_record":"bm_legendbench_b9667237"},"modality":null,"name":"LegendBench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24172","source_url":"https://arxiv.org/abs/2609.24172"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24172","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24172","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","视觉问答","Visual-Qa","物理智能","Embodied AI","逻辑推理","图像理解","Image Understanding","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A multi-level evaluation benchmark of multimodal reasoning with with 3.4K contemporary images and 60K+ human-authored questions covering eight tasks and 12 daily scenarios, forming three progressive task tiers, i.e., perception, understanding, and reasoning. 多层次多模态推理评测基准","descriptions":{"en":"A multi-level evaluation benchmark of multimodal reasoning with with 3.4K contemporary images and 60K+ human-authored questions covering eight tasks and 12 daily scenarios, forming three progressive task tiers, i.e., perception, understanding, and reasoning.","zh":"多层次多模态推理评测基准"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2422","languages":["Multilingual"],"library_direction":null,"modality":"multimodal","name":"LENS","openness":"unknown","publisher":"武汉理工大学，清华大学，中科院自动化所","released":"2025-06-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2422-lens","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LENS","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Lethe is a benchmark for federated unlearning in medical imaging. It evaluates twelve methods across eight task families, including classification, segmentation, denoising, cross-modality synthesis, and vision-language question answering. Three forgetting granularities (hospital, class, patient) are assessed against a retrained gold standard on utility, privacy, and cost.","descriptions":{"en":"Lethe is a benchmark for federated unlearning in medical imaging. It evaluates twelve methods across eight task families, including classification, segmentation, denoising, cross-modality synthesis, and vision-language question answering. Three forgetting granularities (hospital, class, patient) are assessed against a retrained gold standard on utility, privacy, and cost."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01094","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_lethe_74a00c13"},"modality":null,"name":"Lethe","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01094","source_url":"https://arxiv.org/abs/2608.01094"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01094","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01094","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"LEVANTE-bench evaluates vision-language models on six cognitive tasks from the LEVANTE dataset, comparing model performance and error patterns with children aged 5-12 across three countries.","descriptions":{"en":"LEVANTE-bench evaluates vision-language models on six cognitive tasks from the LEVANTE dataset, comparing model performance and error patterns with children aged 5-12 across three countries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05497","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_levante-bench_b9930c93"},"modality":null,"name":"LEVANTE-bench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05497","source_url":"https://arxiv.org/abs/2606.05497"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05497","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05497","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LexAgentHallu is a hierarchical benchmark for profiling hallucinations in tool-augmented legal agents, containing 3414 instances across 17 legal categories and 6 task types, with dual-layer hallucination taxonomy and fine-grained metrics.","descriptions":{"en":"LexAgentHallu is a hierarchical benchmark for profiling hallucinations in tool-augmented legal agents, containing 3414 instances across 17 legal categories and 6 task types, with dual-layer hallucination taxonomy and fine-grained metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09754","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal","factuality"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_lexagenthallu_8032f3b2"},"modality":null,"name":"LexAgentHallu","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09754","source_url":"https://arxiv.org/abs/2609.09754"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09754","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09754","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Designed to evaluate legal issue identification in Chinese civil litigation using 430 real cases and 1,303 expert-annotated disputed issues.","descriptions":{"en":"Designed to evaluate legal issue identification in Chinese civil litigation using 430 real cases and 1,303 expert-annotated disputed issues."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02954","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","finance-legal"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_lexissue_b61484be"},"modality":null,"name":"LexIssue","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02954","source_url":"https://arxiv.org/abs/2609.02954"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02954","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02954","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"LexKairos is a benchmark for evaluating the temporal capabilities of LLMs in the Chinese legal context, covering statutory temporal knowledge, case temporal modeling, and statute-case temporal reasoning across nine sub-tasks.","descriptions":{"en":"LexKairos is a benchmark for evaluating the temporal capabilities of LLMs in the Chinese legal context, covering statutory temporal knowledge, case temporal modeling, and statute-case temporal reasoning across nine sub-tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.09106","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_lexkairos_261e303a"},"modality":null,"name":"LexKairos","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09106","source_url":"https://arxiv.org/abs/2608.09106"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09106","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09106","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:foggpoy/lexrubric","source_record":"bm_lexrubric_7a24242e"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"LexRubric evaluates open-ended legal tasks in Chinese, with 649 instances from legal consultation and judicial examination. It includes 12,337 expert-written atomic scoring criteria under a six-dimensional framework, enabling fine-grained diagnostic evaluation.","descriptions":{"en":"LexRubric evaluates open-ended legal tasks in Chinese, with 649 instances from legal consultation and judicial examination. It includes 12,337 expert-written atomic scoring criteria under a six-dimensional framework, enabling fine-grained diagnostic evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09389","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_lexrubric_7a24242e"},"modality":null,"name":"LexRubric","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09389","source_url":"https://arxiv.org/abs/2606.09389"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09389","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09389","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"LH-AVLN is a benchmark for long-horizon navigation that combines multi-goal missions, heterogeneous goal specifications, and persistent spatialized audio cues. Agents must execute missions of two to four goals specified by category, language, or reference image, using RGB-D, pose, and binaural audio in indoor 3D environments. It supports ordered and unordered missions with alternating goal-associated sounds.","descriptions":{"en":"LH-AVLN is a benchmark for long-horizon navigation that combines multi-goal missions, heterogeneous goal specifications, and persistent spatialized audio cues. Agents must execute missions of two to four goals specified by category, language, or reference image, using RGB-D, pose, and binaural audio in indoor 3D environments. It supports ordered and unordered missions with alternating goal-associated sounds."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.03920","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","speech-audio"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_lh-avln_cb4be7f0"},"modality":null,"name":"LH-AVLN","openness":"unknown","publisher":null,"released":"2026-07-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03920","source_url":"https://arxiv.org/abs/2607.03920"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03920","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03920","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_libad_b1abf215"},"categories":["Manufacturing & Process Control","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"LIBAD is a multimodal anomaly detection dataset from Li-ion battery electrode manufacturing, providing aligned visible-light and X-ray radiography images. It includes benchmarks of methods under inline-compatible settings.","descriptions":{"en":"LIBAD is a multimodal anomaly detection dataset from Li-ion battery electrode manufacturing, providing aligned visible-light and X-ray radiography images. It includes benchmarks of methods under inline-compatible settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07958","languages":[],"library_direction":{"basis":"model-review-2","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_libad_b1abf215"},"modality":null,"name":"LIBAD","openness":"unknown","publisher":null,"released":"2026-08-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07958","source_url":"https://arxiv.org/abs/2608.07958"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07958","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07958","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_libero-agent_c858588e"},"categories":["Agents","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"General-purpose agents can plan, use tools, and revise their behavior from feedback, but it remains unclear whether these capabilities transfer from digital environments to embodied manipulation.","descriptions":{"en":"General-purpose agents can plan, use tools, and revise their behavior from feedback, but it remains unclear whether these capabilities transfer from digital environments to embodied manipulation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.39507","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","tool-use"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_libero-agent_c858588e"},"modality":null,"name":"LIBERO-Agent","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39507","source_url":"https://arxiv.org/abs/2609.39507"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39507","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39507","topics":["agents"],"unit":null},{"aliases":["Beyond Single-Axis Testing: Paired Evaluation of Compound Robustness in Vision-Language-Action Policies"],"attention_signals":null,"categories":["General AI","Multimodal","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Not assessable due to insufficient evidence.","descriptions":{"en":"Not assessable due to insufficient evidence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15940","languages":[],"library_direction":null,"modality":null,"name":"LIBERO-CTRL","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15940","source_url":"https://arxiv.org/abs/2609.15940"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15940","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15940","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Geometric reasoning","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Robots must often continue a task after a target moves, the viewpoint shifts, or an obstacle appears, even though their earlier observations and committed actions reflect the previous scene.","descriptions":{"en":"Robots must often continue a task after a target moves, the viewpoint shifts, or an obstacle appears, even though their earlier observations and committed actions reflect the previous scene."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.36518","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_libero-max_0db3a760"},"modality":null,"name":"LIBERO-MAX","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36518","source_url":"https://arxiv.org/abs/2609.36518"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36518","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36518","topics":[],"unit":null},{"aliases":["LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models","LIBERO-Recover"],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates failure recovery in robotic manipulation across four recovery levels: action retry, action adaptation, object state recovery, and environmental recovery. Built on LIBERO with 1,000+ scenarios, it assesses spatial understanding, object structure reasoning, interaction understanding, and topological reasoning.","descriptions":{"en":"Evaluates failure recovery in robotic manipulation across four recovery levels: action retry, action adaptation, object state recovery, and environmental recovery. Built on LIBERO with 1,000+ scenarios, it assesses spatial understanding, object structure reasoning, interaction understanding, and topological reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.05178","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_libero-recover-beyond-task-success-towards_085b06a2"},"modality":null,"name":"LIBERO-Recover Benchmark","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05178","source_url":"https://arxiv.org/abs/2609.05178"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05178","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05178","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Embodied manipulation","Multimodal","Robotics & Embodied AI","Safety","Safety & Trustworthiness","Visual instruction following"],"collected_at":"2026-10-14T09:44:51Z","description":"LIBERO-VIFO is a benchmark for visual cue following in vision-language-action models, evaluating capability and safety across eight cue families. However, no public artifacts or official links are provided.","descriptions":{"en":"LIBERO-VIFO is a benchmark for visual cue following in vision-language-action models, evaluating capability and safety across eight cue families. However, no public artifacts or official links are provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17600","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","safety-alignment"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_libero-vifo_76763b3b"},"modality":null,"name":"LIBERO-VIFO","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17600","source_url":"https://arxiv.org/abs/2608.17600"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17600","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17600","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","Robustness","Safety & Trustworthiness","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"LIBERO-VPro evaluates closed-loop visual robustness of robotic foundation models under visual perturbations.","descriptions":{"en":"LIBERO-VPro evaluates closed-loop visual robustness of robotic foundation models under visual perturbations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24350","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_libero-vpro_ac8a0ae2"},"modality":null,"name":"LIBERO-VPro","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24350","source_url":"https://arxiv.org/abs/2609.24350"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24350","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24350","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Factuality","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"LibEvoBench is a multi-task benchmark for evaluating code generation models on API evolution across versions of Python libraries. It includes tasks spanning multiple library versions and introduces the Software Evolution Understanding Score (SEUS) to measure version-specific knowledge consistency.","descriptions":{"en":"LibEvoBench is a multi-task benchmark for evaluating code generation models on API evolution across versions of Python libraries. It includes tasks spanning multiple library versions and introduces the Software Evolution Understanding Score (SEUS) to measure version-specific knowledge consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.25402","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_libevobench_4c11745f"},"modality":null,"name":"LibEvoBench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25402","source_url":"https://arxiv.org/abs/2606.25402"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25402","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25402","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":8,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"claire_record_id","repository":"gh:sprocketlab/librarydesignbench","source_record":"bm_librarydesignbench_25362c8e"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Agents increasingly build on code written by other agents, and they reimplement rather than reuse, growing the codebases later agents must work in.","descriptions":{"en":"Agents increasingly build on code written by other agents, and they reimplement rather than reuse, growing the codebases later agents must work in."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.36730","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_librarydesignbench_25362c8e"},"modality":null,"name":"LibraryDesignBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36730","source_url":"https://arxiv.org/abs/2609.36730"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36730","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36730","topics":["coding","agents"],"unit":null},{"aliases":["Do LiDAR Language Models Really Understand Spatio-temporal Relationships?"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:awesome4d/4dmllm_hallucination_bench","source_record":"bm_do-lidar-language-models-really-understand_2f557382"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LiDAR language model responses on 10,000 geometry-referenced questions across 150 nuScenes scenes, with strict parsing and diagnostic controls.","descriptions":{"en":"Evaluates LiDAR language model responses on 10,000 geometry-referenced questions across 150 nuScenes scenes, with strict parsing and diagnostic controls."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.24452","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","factuality"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_do-lidar-language-models-really-understand_2f557382"},"modality":null,"name":"LiDAR-Hallu","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24452","source_url":"https://arxiv.org/abs/2609.24452"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24452","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24452","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Planning","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agents on geo-spatial planning tasks using map data enriched with social media posts, measuring pass rate across four task categories and three difficulty levels.","descriptions":{"en":"Evaluates LLM agents on geo-spatial planning tasks using map data enriched with social media posts, measuring pass rate across four task categories and three difficulty levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25039","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_lifeplanner_df604ac8"},"modality":null,"name":"LifePlanner","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25039","source_url":"https://arxiv.org/abs/2608.25039"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25039","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25039","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":152,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_lifesci-harbor-bench_95c348c9"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on 503 ultra-long-horizon, open-ended life science research tasks through an 11-layer verifier combining deterministic scoring and anti-hacking checks.","descriptions":{"en":"Evaluates coding agents on 503 ultra-long-horizon, open-ended life science research tasks through an 11-layer verifier combining deterministic scoring and anti-hacking checks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:mondaycake/lifesci-harbor-bench","languages":[],"library_direction":{"basis":"model-review","categories":["science","agentic-coding"],"join":"claire_record_id","primary":"science","source_record":"bm_lifesci-harbor-bench_95c348c9"},"modality":null,"name":"LifeSci-Harbor-Bench","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:mondaycake/lifesci-harbor-bench","source_url":"https://huggingface.co/datasets/mondaycake/lifesci-harbor-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-mondaycake-lifesci-harbor-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/mondaycake/lifesci-harbor-bench","topics":["coding","agents","science"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","science","healthcare","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LifeSciBench is an expert-authored, expert-reviewed benchmark of 750 open-ended life-science research tasks spanning seven workflows and seven biological domains. Responses are graded against 19,020 physician- and scientist-written rubric criteria rather than multiple-choice answers, and most tasks require interpreting attached artifacts such as figures, PDFs, and sequence files.","descriptions":{"en":"LifeSciBench is an expert-authored, expert-reviewed benchmark of 750 open-ended life-science research tasks spanning seven workflows and seven biological domains. Responses are graded against 19,020 physician- and scientist-written rubric criteria rather than multiple-choice answers, and most tasks require interpreting attached artifacts such as figures, PDFs, and sequence files."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:lifescibench:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lifescibench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:lifescibench","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_d3d27b0f6f6ef01a"},"modality":"multimodal","name":"LifeSciBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":59.9,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.599,"raw_min":0.512,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:lifescibench:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lifescibench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-lifescibench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lifescibench?top_n=500","topics":["agents","multimodal","reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"LigBench is an automated evaluation benchmark for AI research idea generation, using fine-grained and reliable evaluation across generation distributions. It includes PAIR-IQ, a dataset for training pairwise idea judgment models.","descriptions":{"en":"LigBench is an automated evaluation benchmark for AI research idea generation, using fine-grained and reliable evaluation across generation distributions. It includes PAIR-IQ, a dataset for training pairwise idea judgment models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.13136","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_ligbench_47189b6a"},"modality":null,"name":"LigBench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13136","source_url":"https://arxiv.org/abs/2608.13136"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13136","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13136","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:cscpadova/lilybench","source_record":"bm_lilybench_6840fe07"},"categories":["General AI","Language & Knowledge","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"LilyBench evaluates symbolic music generation and understanding using LilyPond, with a 200-prompt generation suite and ten understanding tasks, scored via compile rate, descriptor similarity, and FMD.","descriptions":{"en":"LilyBench evaluates symbolic music generation and understanding using LilyPond, with a 200-prompt generation suite and ten understanding tasks, scored via compile rate, descriptor similarity, and FMD."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.08722","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","code-generation"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_lilybench_6840fe07"},"modality":null,"name":"LilyBench","openness":"unknown","publisher":null,"released":"2026-06-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08722","source_url":"https://arxiv.org/abs/2606.08722"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08722","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08722","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Lingjing is a simulation testbed for heterogeneous multi-agent embodied tasks in urban environments, with a Gym-like interface and support for natural-language missions. It includes engine-based evaluations and attribution-ready replays.","descriptions":{"en":"Lingjing is a simulation testbed for heterogeneous multi-agent embodied tasks in urban environments, with a Gym-like interface and support for natural-language missions. It includes engine-based evaluations and attribution-ready replays."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08045","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","embodied-robotics"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_lingjing_2ca7c6e7"},"modality":null,"name":"Lingjing","openness":"unknown","publisher":null,"released":"2026-08-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08045","source_url":"https://arxiv.org/abs/2608.08045"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08045","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08045","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Aiming at evaluating LLM's advanced reasoning abilities, LingOly is composed of  olympiad-level linguistic reasoning puzzles in low-resource and extinct languages. It covers more than 90 mostly low-resource languages, and contains 1,133 problems across 6 formats and 5 levels of human difficulty. LingOly由低资源和已灭绝语言的奥林匹克级别语言推理谜题组成，用于评估大语言模型的高级推理能力。其中涵盖了90多种语言，共有1133个涉及6种格式和5个人工难度级别的问题。","descriptions":{"en":"Aiming at evaluating LLM's advanced reasoning abilities, LingOly is composed of  olympiad-level linguistic reasoning puzzles in low-resource and extinct languages. It covers more than 90 mostly low-resource languages, and contains 1,133 problems across 6 formats and 5 levels of human difficulty.","zh":"LingOly由低资源和已灭绝语言的奥林匹克级别语言推理谜题组成，用于评估大语言模型的高级推理能力。其中涵盖了90多种语言，共有1133个涉及6种格式和5个人工难度级别的问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1238","languages":[],"library_direction":null,"modality":null,"name":"LINGOLY","openness":"restricted","publisher":"University of Oxford","released":"2024-06-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1238-lingoly","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LINGOLY","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":224,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_d6d7a20390c04d5a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","language","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A benchmark for multimodal spatial-language understanding and visual-linguistic question answering.","descriptions":{"en":"A benchmark for multimodal spatial-language understanding and visual-linguistic question answering."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:lingoqa:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lingoqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:lingoqa","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_d6d7a20390c04d5a"},"modality":"multimodal","name":"LingoQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":83.39999999999999,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.834,"raw_min":0.792,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:lingoqa:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lingoqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-lingoqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lingoqa?top_n=500","topics":["multimodal","reasoning-math","language"],"unit":null},{"aliases":["On the Limitations of Cross-Lingual Consistency in Multilingual Text-to-image Generation"],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:risys-lab/lingt2i","source_record":"bm_on-the-limitations-of-cross-lingual-consis_d9fe8562"},"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates cross-lingual text-to-image generation across content generation and text rendering tasks for 10 languages and 33K prompts.","descriptions":{"en":"Evaluates cross-lingual text-to-image generation across content generation and text rendering tasks for 10 languages and 33K prompts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11002","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","multilingual"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_on-the-limitations-of-cross-lingual-consis_d9fe8562"},"modality":null,"name":"LingT2I","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11002","source_url":"https://arxiv.org/abs/2608.11002"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11002","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11002","topics":["language"],"unit":null},{"aliases":["What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for falsifiable research ideation, using a six-field contract centered on a falsifying outcome, with pairwise comparisons of proposals from four models.","descriptions":{"en":"A benchmark for falsifiable research ideation, using a six-field contract centered on a falsifying outcome, with pairwise comparisons of proposals from four models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22948","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_what-proves-you-wrong-benchmarking-languag_17c27305"},"modality":null,"name":"Lit2Test","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22948","source_url":"http://arxiv.org/abs/2608.22948v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22948","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22948v1","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"LitTraceQA evaluates literature-grounded question answering over scientific papers, requiring systems to return paper IDs, evidence locations, and answers in multiple formats. The public split includes 55 examples with gold annotations.","descriptions":{"en":"LitTraceQA evaluates literature-grounded question answering over scientific papers, requiring systems to return paper IDs, evidence locations, and answers in multiple formats. The public split includes 55 examples with gold annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07370","languages":[],"library_direction":{"basis":"model-review","categories":["science","deep-research"],"join":"claire_record_id","primary":"science","source_record":"bm_littraceqa_955ebe65"},"modality":null,"name":"LitTraceQA","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07370","source_url":"https://arxiv.org/abs/2608.07370"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07370","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07370","topics":["science"],"unit":null},{"aliases":["Live Gurbani Tracking: A Benchmark and Reference System for Captioning Sikh Kirtan"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A released benchmark for locating exact canonical scripture lines during live or offline Sikh Kirtan captioning under blind and oracle conditions.","descriptions":{"en":"A released benchmark for locating exact canonical scripture lines during live or offline Sikh Kirtan captioning under blind and oracle conditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.13457","languages":[],"library_direction":null,"modality":null,"name":"Live Gurbani Tracking","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.13457","source_url":"https://arxiv.org/abs/2607.13457"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-13457","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13457","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1341,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_858362768f842e9d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. It comprises tasks across math, coding, reasoning, language, instruction following, and data analysis with verifiable, objective ground-truth answers.","descriptions":{"en":"LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. It comprises tasks across math, coding, reasoning, language, instruction following, and data analysis with verifiable, objective ground-truth answers."},"evidence_summary":{"document_count":1,"model_count":38,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livebench:o1-preview","reported_at":"2024-09-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:livebench","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","code-generation","math"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_858362768f842e9d"},"modality":"text","name":"LiveBench","openness":"restricted","publisher":"Abacus.AI","released":"2024-06-12","released_reference":{"basis":"release_announcement","note":"The official datasheet states the suite became public on June 12, before the June 27 paper.","source_key":"opencompass:1246","source_url":"https://github.com/LiveBench/LiveBench/blob/main/docs/DATASHEET.md"},"repo_kind":null,"repo_resolution_status":null,"score_count":38,"score_direction":"higher_is_better","score_summary":{"display_max":84.6,"display_multiplier":100,"model_count":38,"model_count_basis":"source_model_id","numeric_count":38,"raw_max":0.846,"raw_min":0.296,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livebench:o3-mini","reported_date":"2025-01-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 High"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-01-14","date_precision":"model_announcement","name":"GPT-5.2 Codex"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 30B A3B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 32B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ-32B"}],"slug":"llm-stats-livebench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["LiveBench"],"attention_signals":null,"categories":["general"],"collected_at":null,"description":"Contents change by release date; each date is a different benchmark.","descriptions":{"en":"Contents change by release date; each date is a different benchmark."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:livebench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["composite-index"],"join":"review:14H034160212:2026-10-15","primary":"composite-index","source_record":null},"modality":null,"name":"LiveBench","openness":"unknown","publisher":null,"released":"2024-06-06","released_reference":{"basis":"benchmark_release","source_key":"model-reports:livebench","source_url":"https://livebench.ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"livebench","source":"model_reports","source_url":"https://livebench.ai/","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-free versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. LiveBench是一个LLM基准测试，涵盖数学、编码、推理、语言、指令遵循和数据分析，包含基于最近发布的数学竞赛、arXiv 论文、新闻文章和数据集的问题，及经典基准测试（如 Big-Bench Hard、AMPS 和 IFEval）的更难、无污染的任务。","descriptions":{"en":"LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-free versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval.","zh":"LiveBench是一个LLM基准测试，涵盖数学、编码、推理、语言、指令遵循和数据分析，包含基于最近发布的数学竞赛、arXiv 论文、新闻文章和数据集的问题，及经典基准测试（如 Big-Bench Hard、AMPS 和 IFEval）的更难、无污染的任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1246","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","code-generation","math"],"join":"artifact:arxiv:2406.19314","primary":"knowledge-expert-qa","source_record":"catalog_7869f24ecad81765"},"modality":null,"name":"LiveBench","openness":"restricted","publisher":"Abacus.AI","released":"2024-06-12","released_reference":{"basis":"release_announcement","note":"The official datasheet states the suite became public on June 12, before the June 27 paper.","source_key":"opencompass:1246","source_url":"https://github.com/LiveBench/LiveBench/blob/main/docs/DATASHEET.md"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1246-livebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LiveBench","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1343,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_7869f24ecad81765"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. It comprises tasks across math, coding, reasoning, language, instruction following, and data analysis with verifiable, objective ground-truth answers.","descriptions":{"en":"LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. It comprises tasks across math, coding, reasoning, language, instruction following, and data analysis with verifiable, objective ground-truth answers."},"evidence_summary":{"document_count":1,"model_count":14,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livebench-20241125:qwen3-235b-a22b-instruct-2507","reported_at":"2025-07-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench-20241125?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:livebench-20241125","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","code-generation","math"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_7869f24ecad81765"},"modality":"text","name":"LiveBench 20241125","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":14,"score_direction":"higher_is_better","score_summary":{"display_max":79.60000000000001,"display_multiplier":100,"model_count":14,"model_count_basis":"source_model_id","numeric_count":14,"raw_max":0.796,"raw_min":0.609,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livebench-20241125:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench-20241125?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-livebench-20241125","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench-20241125?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A deep-search evaluation containing 335 human-authored questions about recently published, non-globally-salient facts.","descriptions":{"en":"A deep-search evaluation containing 335 human-authored questions about recently published, non-globally-salient facts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28721","languages":[],"library_direction":null,"modality":null,"name":"LiveBrowseComp","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28721","source_url":"https://arxiv.org/abs/2605.28721"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28721","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28721","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["coding"],"collected_at":"2026-08-25T10:41:06Z","description":"Coding","descriptions":{"en":"Coding"},"evidence_summary":{"document_count":1,"model_count":343,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:livecodebench:83cb898e-05d9-4e4b-9de3-2d305014d923","reported_at":"2023-03-14","source_url":"https://artificialanalysis.ai/evaluations/livecodebench"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:livecodebench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["code-generation"],"join":"review:14H034160212:2026-10-15","primary":"code-generation","source_record":null},"modality":null,"name":"LiveCodeBench","openness":"unknown","publisher":null,"released":"2024-03-12","released_reference":{"basis":"paper_first_version","note":"First version of the LiveCodeBench introduction. Retrospective evaluations of older models do not move its release into 2023.","source_key":"artificial-analysis:livecodebench","source_url":"https://arxiv.org/abs/2403.07974"},"repo_kind":null,"repo_resolution_status":null,"score_count":343,"score_direction":"higher_is_better","score_summary":{"display_max":91.7460317460317,"display_multiplier":100,"model_count":343,"model_count_basis":"source_model_id","numeric_count":343,"raw_max":0.917460317460317,"raw_min":0.00211640211640212,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:livecodebench:d1122eff-ee85-4fdc-8a9f-23bee6590667","reported_date":"2025-11-18","source_url":"https://artificialanalysis.ai/evaluations/livecodebench"},"unit":null},"scored_models":[{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2023-07-11","date_precision":"model_announcement","name":"Claude 2.0"},{"date":"2023-11-21","date_precision":"model_announcement","name":"Claude 2.1"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet (Oct '24)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2023-03-14","date_precision":"model_announcement","name":"Claude Instant"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2024-03-12","date_precision":"model_announcement","name":"Command-R (Mar '24)"},{"date":"2024-04-04","date_precision":"model_announcement","name":"Command-R+ (Apr '24)"},{"date":"2024-03-27","date_precision":"model_announcement","name":"DBRX Instruct"},{"date":"2025-02-13","date_precision":"model_announcement","name":"DeepHermes 3 - Llama-3.1 8B Preview (Non-reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"DeepHermes 3 - Mistral 24B Preview (Non-reasoning)"},{"date":"2024-06-17","date_precision":"model_announcement","name":"DeepSeek Coder V2 Lite Instruct"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2023-12-06","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-14","date_precision":"model_announcement","name":"Gemini 1.5 Flash (May '24)"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Flash (Sep '24)"},{"date":"2024-10-03","date_precision":"model_announcement","name":"Gemini 1.5 Flash-8B"},{"date":"2024-05-15","date_precision":"model_announcement","name":"Gemini 1.5 Pro (May '24)"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Pro (Sep '24)"},{"date":"2024-12-11","date_precision":"model_announcement","name":"Gemini 2.0 Flash (experimental)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Gemini 2.0 Flash Thinking Experimental (Jan '25)"},{"date":"2025-02-25","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite (Feb '25)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite (Preview)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Pro Experimental (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-04-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Non-reasoning)"},{"date":"2025-04-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-03-25","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview (Mar' 25)"},{"date":"2025-05-06","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview (May' 25)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct Preview (May '25)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2023-11-06","date_precision":"model_announcement","name":"GPT-4 Turbo"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2025-03-27","date_precision":"model_announcement","name":"GPT-4o (March 2025, chatgpt-4o-latest)"},{"date":"2024-05-13","date_precision":"model_announcement","name":"GPT-4o (May '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (ChatGPT)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Grok 2 (Dec '24)"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok Beta"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 - Llama-3.1 70B"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-03-06","date_precision":"model_announcement","name":"Jamba 1.6 Large"},{"date":"2025-03-06","date_precision":"model_announcement","name":"Jamba 1.6 Mini"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2025-10-30","date_precision":"model_announcement","name":"Kimi Linear 48B A3B Instruct"},{"date":"2024-09-30","date_precision":"model_announcement","name":"LFM 40B"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 13B"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 70B"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 7B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 4B v1.1 (Reasoning)"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Llama 3.1 Tulu3 405B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 3B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 90B (Vision)"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 40k"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Mistral Large (Feb '24)"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2 (Jul '24)"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2023-12-11","date_precision":"model_announcement","name":"Mistral Medium"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Mistral Small (Feb '24)"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Mistral Small (Sep '24)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2024-04-17","date_precision":"model_announcement","name":"Mixtral 8x22B Instruct"},{"date":"2023-12-11","date_precision":"model_announcement","name":"Mixtral 8x7B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"OLMo 2 32B"},{"date":"2024-11-26","date_precision":"model_announcement","name":"OLMo 2 7B"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2023-12-18","date_precision":"model_announcement","name":"OpenChat 3.5 (1210)"},{"date":"2024-04-23","date_precision":"model_announcement","name":"Phi-3 Mini Instruct 3.8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2025-02-26","date_precision":"model_announcement","name":"Phi-4 Multimodal Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-06-07","date_precision":"model_announcement","name":"Qwen2 Instruct 72B"},{"date":"2024-11-11","date_precision":"model_announcement","name":"Qwen2.5 Coder Instruct 32B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Coder Instruct 7B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 32B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-01-28","date_precision":"model_announcement","name":"Qwen2.5 Max"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Qwen2.5 Turbo"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ 32B"},{"date":"2024-11-27","date_precision":"model_announcement","name":"QwQ 32B-Preview"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Solar Pro 2 (Preview) (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Solar Pro 2 (Preview) (Reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Sonar"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Sonar Pro"}],"slug":"artificial-analysis-livecodebench","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/livecodebench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:harvenstar/curveshift","source_record":"bm_livecodebench_fa145f1d"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"CurveShift analyzes progress on LiveCodeBench, releasing a difficulty panel of 66 models and 1,055 problems.","descriptions":{"en":"CurveShift analyzes progress on LiveCodeBench, releasing a difficulty panel of 66 models and 1,055 problems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.00355","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_livecodebench_fa145f1d"},"modality":null,"name":"LiveCodeBench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00355","source_url":"https://arxiv.org/abs/2608.00355"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00355","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00355","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:harvenstar/curveshift","source_record":"bm_livecodebench_fa145f1d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","descriptions":{"en":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff."},"evidence_summary":{"document_count":1,"model_count":75,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livecodebench:qwen2-7b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:livecodebench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"bm_livecodebench_fa145f1d"},"modality":"text","name":"LiveCodeBench","openness":"unknown","publisher":"University of California, Berkeley","released":"2024-03-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":75,"score_direction":"higher_is_better","score_summary":{"display_max":93.5,"display_multiplier":100,"model_count":75,"model_count_basis":"source_model_id","numeric_count":75,"raw_max":0.935,"raw_min":0.019,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livecodebench:deepseek-v4-pro-max","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 7B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Zero"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek-V3 0324"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok 4 Fast"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Grok-3"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Grok-3 Mini"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4 Heavy"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 2506"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Min  istral 3 (3B Reasoning 2512)"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Reasoning 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Reasoning 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Base)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 Eagle)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 NVFP4)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"Nemotron Nano 9B v2"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 30B A3B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 32B"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ-32B"},{"date":"2024-11-28","date_precision":"model_announcement","name":"QwQ-32B-Preview"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"}],"slug":"llm-stats-livecodebench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":["LiveCodeBench","LCB"],"attention_signals":null,"categories":["coding"],"collected_at":null,"description":"Time-sliced problem set. A score without its problem window is not comparable to any other score.","descriptions":{"en":"Time-sliced problem set. A score without its problem window is not comparable to any other score."},"evidence_summary":{"document_count":18,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000livecodebench\u0000qwen2_5_coder_report\u0000livecodebench\u0000Pass@1\u0000Qwen2.5-Coder-32B-Instruct","reported_at":"2024-09-18","source_url":"https://arxiv.org/abs/2409.12186"},"first_score_reported_at":"2024-09-18","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000livecodebench\u0000qwen2_5_coder_report\u0000livecodebench\u0000Pass@1\u0000Qwen2.5-Coder-32B-Instruct","reported_at":"2024-09-18","source_url":"https://arxiv.org/abs/2409.12186"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:livecodebench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["code-generation"],"join":"review:14H034160212:2026-10-15","primary":"code-generation","source_record":null},"modality":null,"name":"LiveCodeBench","openness":"unknown","publisher":null,"released":"2024-03-12","released_reference":{"basis":"benchmark_release","source_key":"model-reports:livecodebench","source_url":"https://livecodebench.github.io/"},"repo_kind":null,"repo_resolution_status":null,"score_count":15,"score_direction":"higher_is_better","score_summary":{"display_max":93.5,"display_multiplier":1,"model_count":12,"model_count_basis":"source_model_id","numeric_count":15,"raw_max":93.5,"raw_min":31.4,"source_reference":{"obs_id":"curated\u0000livecodebench\u0000deepseek_v4_model_card\u0000livecodebench\u0000think max, pass@1\u0000DeepSeek-V4-Pro","observation_id":"curated\u0000livecodebench\u0000deepseek_v4_model_card\u0000livecodebench\u0000think max, pass@1\u0000DeepSeek-V4-Pro","reported_at":"2026-04-22","reported_date":"2026-04-22","source_id":"deepseek_v4_model_card","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro"},"unit":"percent"},"scored_models":[{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2025-06-17","date_precision":"document_publication","name":"Gemini 2.5 Pro"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2024-09-18","date_precision":"document_publication","name":"Qwen2.5-Coder-32B-Instruct"},{"date":"2025-05-14","date_precision":"document_publication","name":"Qwen3-235B-A22B (Thinking)"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"livecodebench","source":"model_reports","source_url":"https://livecodebench.github.io/","topics":["coding"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LiveCodeBench evaluates LLMs' coding abilities. It continuously collects new problems over time from contests across LeetCode, AtCoder, and CodeForces. Notably, it also focuses on a broader range of code related capabilities besides code generation. LiveCodeBench用于评估大语言模型的代码能力，包含来自LeetCode、AtCoder和CodeForces的动态更新的问题，并在代码生成能力的基础上将更广泛的相关能力纳入考量。","descriptions":{"en":"LiveCodeBench evaluates LLMs' coding abilities. It continuously collects new problems over time from contests across LeetCode, AtCoder, and CodeForces. Notably, it also focuses on a broader range of code related capabilities besides code generation.","zh":"LiveCodeBench用于评估大语言模型的代码能力，包含来自LeetCode、AtCoder和CodeForces的动态更新的问题，并在代码生成能力的基础上将更广泛的相关能力纳入考量。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1413","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"artifact:arxiv:2403.07974","primary":"code-generation","source_record":"bm_livecodebench_fa145f1d"},"modality":null,"name":"LiveCodeBench","openness":"unknown","publisher":"University of California, Berkeley","released":"2024-03-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1413-livecodebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LiveCodeBench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":178,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":25,"join":"llm_stats_id","repository":null,"source_record":"catalog_87b50a562a887e4a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveCodeBench Pro is an advanced evaluation benchmark for large language models for code that uses Elo ratings to rank models based on their performance on coding tasks. It evaluates models on real-world coding problems from programming contests (LeetCode, AtCoder, CodeForces) and provides a relative ranking system where higher Elo scores indicate superior performance.","descriptions":{"en":"LiveCodeBench Pro is an advanced evaluation benchmark for large language models for code that uses Elo ratings to rank models based on their performance on coding tasks. It evaluates models on real-world coding problems from programming contests (LeetCode, AtCoder, CodeForces) and provides a relative ranking system where higher Elo scores indicate superior performance."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livecodebench-pro:gemini-3-pro-preview","reported_at":"2025-11-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-pro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:livecodebench-pro","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_87b50a562a887e4a"},"modality":"text","name":"LiveCodeBench Pro","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":2887.0,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":2887.0,"raw_min":0.8,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livecodebench-pro:gemini-3.1-pro-preview","reported_date":"2026-02-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-pro?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"}],"slug":"llm-stats-livecodebench-pro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-pro?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":["LiveCodeBench Pro","LCB Pro"],"attention_signals":null,"categories":["coding"],"collected_at":null,"description":"Reported as an Elo rating against competitive-programming problems, not a pass rate, so it cannot be read on the same axis as LiveCodeBench.","descriptions":{"en":"Reported as an Elo rating against competitive-programming problems, not a pass rate, so it cannot be read on the same axis as LiveCodeBench."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000livecodebench_pro\u0000google_gemini_3_1_pro_model_card\u0000livecodebench_pro\u0000Thinking (High), Elo\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"first_score_reported_at":"2026-02-19","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000livecodebench_pro\u0000google_gemini_3_1_pro_model_card\u0000livecodebench_pro\u0000Thinking (High), Elo\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:livecodebench_pro","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["code-generation"],"join":"review:14H034160212:2026-10-15","primary":"code-generation","source_record":null},"modality":null,"name":"LiveCodeBench Pro","openness":"unknown","publisher":null,"released":"2025-06-13","released_reference":{"basis":"benchmark_release","source_key":"model-reports:livecodebench_pro","source_url":"https://livecodebenchpro.com/"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":2887.0,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":2887.0,"raw_min":2887.0,"source_reference":{"obs_id":"curated\u0000livecodebench_pro\u0000google_gemini_3_1_pro_model_card\u0000livecodebench_pro\u0000Thinking (High), Elo\u0000Gemini 3.1 Pro","observation_id":"curated\u0000livecodebench_pro\u0000google_gemini_3_1_pro_model_card\u0000livecodebench_pro\u0000Thinking (High), Elo\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","reported_date":"2026-02-19","source_id":"google_gemini_3_1_pro_model_card","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"unit":"elo"},"scored_models":[{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"}],"slug":"livecodebench_pro","source":"model_reports","source_url":"https://livecodebenchpro.com/","topics":["coding"],"unit":"elo"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":961,"github_stars_30d":null,"hf_dataset_downloads":85617,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":"gh:livecodebench/livecodebench","source_record":"catalog_ad37284f83ab5ddb"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","descriptions":{"en":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livecodebench-v5:gemini-2.0-flash-lite","reported_at":"2025-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:livecodebench-v5","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_ad37284f83ab5ddb"},"modality":"text","name":"LiveCodeBench v5","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":75.6,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.756,"raw_min":0.186,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livecodebench-v5:gemini-2.5-pro","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"}],"slug":"llm-stats-livecodebench-v5","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":961,"github_stars_30d":null,"hf_dataset_downloads":83053,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":"gh:livecodebench/livecodebench","source_record":"catalog_5140b427ae8edecc"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","descriptions":{"en":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livecodebench-v5-24.12-25.2:kimi-k1.5","reported_at":"2025-01-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5-24.12-25.2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:livecodebench-v5-24.12-25.2","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_5140b427ae8edecc"},"modality":"text","name":"LiveCodeBench v5 24.12-25.2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":62.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.625,"raw_min":0.625,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livecodebench-v5-24.12-25.2:kimi-k1.5","reported_date":"2025-01-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5-24.12-25.2?top_n=500"},"unit":null},"scored_models":[{"date":"2025-01-20","date_precision":"model_announcement","name":"Kimi-k1.5"}],"slug":"llm-stats-livecodebench-v5-24-12-25-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5-24.12-25.2?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":961,"github_stars_30d":null,"hf_dataset_downloads":85617,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":"gh:livecodebench/livecodebench","source_record":"catalog_92b6f9854a975521"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","descriptions":{"en":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff."},"evidence_summary":{"document_count":1,"model_count":56,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livecodebench-v6:kimi-k2-base","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v6?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:livecodebench-v6","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_92b6f9854a975521"},"modality":"text","name":"LiveCodeBench v6","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":56,"score_direction":"higher_is_better","score_summary":{"display_max":91.60000000000001,"display_multiplier":100,"model_count":56,"model_count_basis":"source_model_id","numeric_count":56,"raw_max":0.916,"raw_min":0.263,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livecodebench-v6:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v6?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B High"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE-236B-A23B"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code 1.0"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Sakana Namazu"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Lite"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Pro"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step-3.5-Flash"}],"slug":"llm-stats-livecodebench-v6","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v6?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":961,"github_stars_30d":null,"hf_dataset_downloads":84431,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":"gh:livecodebench/livecodebench","source_record":"catalog_d0a8b6d9c573e736"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","descriptions":{"en":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livecodebench(01-09):deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench%2801-09%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:livecodebench(01-09)","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_d0a8b6d9c573e736"},"modality":"text","name":"LiveCodeBench(01-09)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":41.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.418,"raw_min":0.418,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livecodebench(01-09):deepseek-v2.5","reported_date":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench%2801-09%29?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"}],"slug":"llm-stats-livecodebench-01-09","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench%2801-09%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zhouziyu02/ts-live","source_record":"bm_livehouse-ts_9c17ef53"},"categories":["Distribution shift","Finance","General AI","Language & Knowledge","Logistics & Operations","Scientific Research & AI for Science","Temporal robustness","Time-series forecasting","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LiveHouse-TS is a living benchmark for time series foundation models, evaluating models prequentially on real future data across 11 domains and 17 datasets. It shifts benchmarking from snapshot accuracy to continuous temporal validity.","descriptions":{"en":"LiveHouse-TS is a living benchmark for time series foundation models, evaluating models prequentially on real future data across 11 domains and 17 datasets. It shifts benchmarking from snapshot accuracy to continuous temporal validity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.17299","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_livehouse-ts_9c17ef53"},"modality":null,"name":"LiveHouse-TS","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17299","source_url":"https://arxiv.org/abs/2608.17299"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17299","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17299","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_livek12bench_56b774f5"},"categories":["Materials & Chemistry","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Dynamic benchmark for multimodal reasoning on high school exam questions in math, physics, chemistry, and biology, with a mock exam evaluation scheme.","descriptions":{"en":"Dynamic benchmark for multimodal reasoning on high school exam questions in math, physics, chemistry, and biology, with a mock exam evaluation scheme."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26781","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_livek12bench_56b774f5"},"modality":null,"name":"LiveK12Bench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26781","source_url":"https://arxiv.org/abs/2605.26781"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26781","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26781","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","Spoken text","Long context","Live streams","大语言模型","LLM","长上下文"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LiveLongBench is the first spoken long-text benchmark designed to address the challenges of long-context understanding in real-world dialogues, characterized by speech-specific features, high redundancy, and uneven information density. Existing benchmarks fail to capture these complexities, limiting LiveLongBench 是首个面向口语长文本理解的基准测试，基于直播内容构建，涵盖检索类、推理类及混合类三种任务类型，针对现实对话中存在的语音特性、高冗余性和信息密度不均等挑战。","descriptions":{"en":"LiveLongBench is the first spoken long-text benchmark designed to address the challenges of long-context understanding in real-world dialogues, characterized by speech-specific features, high redundancy, and uneven information density. Existing benchmarks fail to capture these complexities, limiting","zh":"LiveLongBench 是首个面向口语长文本理解的基准测试，基于直播内容构建，涵盖检索类、推理类及混合类三种任务类型，针对现实对话中存在的语音特性、高冗余性和信息密度不均等挑战。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1787","languages":[],"library_direction":null,"modality":null,"name":"LiveLongBench","openness":"unknown","publisher":null,"released":"2025-04-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1787-livelongbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LiveLongBench","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Finance","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluating agents by outcomes alone can obscure the capabilities that produce them.","descriptions":{"en":"Evaluating agents by outcomes alone can obscure the capabilities that produce them."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.09872","languages":[],"library_direction":{"basis":"model-review-daily","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_livemacebench_93d2315d"},"modality":null,"name":"LiveMACEBench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.09872","source_url":"https://arxiv.org/abs/2610.09872"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-09872","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09872","topics":["agents","finance"],"unit":null},{"aliases":["Can LLMs Take the Pulse of the Economy? A Real-Time Evaluation of LLM Nowcasts on Macroeconomic Indicators"],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM nowcasting of 16 macroeconomic indicators via live pre-release windows, scored by equity returns and simulated trading against institutional baselines.","descriptions":{"en":"Evaluates LLM nowcasting of 16 macroeconomic indicators via live pre-release windows, scored by equity returns and simulated trading against institutional baselines."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30110","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_can-llms-take-the-pulse-of-the-economy-a-r_4729fd1f"},"modality":null,"name":"LiveMacroEval","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30110","source_url":"https://arxiv.org/abs/2608.30110"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30110","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30110","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","数学","Math","大语言模型","LLM","逻辑推理","Reasoning","数理能力","官方自建","Official"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LiveMathBench can capture LLM capabilities in complex reasoning tasks, including challenging latest question sets from various mathematical competitions. LiveMathBench用于评估大语言模型在复杂推理方面的表现，由极具挑战性的现代数学问题组成。","descriptions":{"en":"LiveMathBench can capture LLM capabilities in complex reasoning tasks, including challenging latest question sets from various mathematical competitions.","zh":"LiveMathBench用于评估大语言模型在复杂推理方面的表现，由极具挑战性的现代数学问题组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1397","languages":[],"library_direction":null,"modality":null,"name":"LiveMathBench","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-12-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1397-livemathbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LiveMathBench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveMathematicianBench evaluates research-level mathematical reasoning on continuously refreshed, contamination-resistant problems.","descriptions":{"en":"LiveMathematicianBench evaluates research-level mathematical reasoning on continuously refreshed, contamination-resistant problems."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livemathematicianbench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livemathematicianbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:livemathematicianbench","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_03f16cb9366e9fe2"},"modality":"text","name":"LiveMathematicianBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":27.700000000000003,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.277,"raw_min":0.209,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livemathematicianbench:seed-2.1-turbo","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livemathematicianbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-livemathematicianbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livemathematicianbench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":479,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":"gh:showlab/livecc","source_record":"catalog_5bdd7249a566942c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveSports-3K evaluates fine-grained understanding and commentary of live sports video.","descriptions":{"en":"LiveSports-3K evaluates fine-grained understanding and commentary of live sports video."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livesports-3k:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livesports-3k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:livesports-3k","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_5bdd7249a566942c"},"modality":"multimodal","name":"LiveSports-3K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":77.10000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.771,"raw_min":0.768,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livesports-3k:seed-2.1-turbo","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livesports-3k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-livesports-3k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livesports-3k?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":144,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:bird-bench/livesqlbench","source_record":"catalog_8d71d0c357112c5d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LiveSQLBench evaluates models on generating correct SQL queries against live PostgreSQL databases. The LiveSQLBench-Base-Full v1 dataset contains 600 questions across 22 PostgreSQL databases, testing real-world database reasoning and query generation.","descriptions":{"en":"LiveSQLBench evaluates models on generating correct SQL queries against live PostgreSQL databases. The LiveSQLBench-Base-Full v1 dataset contains 600 questions across 22 PostgreSQL databases, testing real-world database reasoning and query generation."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:livesqlbench:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livesqlbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:livesqlbench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_8d71d0c357112c5d"},"modality":"text","name":"LiveSQLBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":40.17,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.4017,"raw_min":0.4017,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:livesqlbench:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livesqlbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-livesqlbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livesqlbench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Realistic household simulation must capture not only diverse environments but also the lived-in object arrangements and spatial constraints that shape robot motion and interaction.","descriptions":{"en":"Realistic household simulation must capture not only diverse environments but also the lived-in object arrangements and spatial constraints that shape robot motion and interaction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.12069","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_livin_51a13a1f"},"modality":null,"name":"LIVIN","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12069","source_url":"https://arxiv.org/abs/2610.12069"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12069","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12069","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"LL-Bench evaluates large-scale generative models on 16 low-level vision tasks using 2,469 real-world degraded images, with human preference and quality score annotations for model outputs.","descriptions":{"en":"LL-Bench evaluates large-scale generative models on 16 low-level vision tasks using 2,469 real-world degraded images, with human preference and quality score annotations for model outputs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02535","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","visual-reasoning"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_ll-bench_df7573ac"},"modality":null,"name":"LL-Bench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02535","source_url":"https://arxiv.org/abs/2606.02535"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02535","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02535","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM collaboration with non-language chess engines across six tasks covering imitation, assessment, and explanation, scored on task performance and generalization to out-of-distribution scenarios.","descriptions":{"en":"Evaluates LLM collaboration with non-language chess engines across six tasks covering imitation, assessment, and explanation, scored on task performance and generalization to out-of-distribution scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.00474","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_llamia-bench_91bb40e1"},"modality":null,"name":"LLAMIA-Bench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00474","source_url":"https://arxiv.org/abs/2609.00474"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00474","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00474","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LLaVA-Bench evaluates the model's ability in more challenging tasks, including a diverse set of 24 images with 60 questions in total, including indoor and outdoor scenes, memes, paintings, sketches, etc. LLaVA-Bench用于评估多模态大模型应对复杂任务的能力，内含24 张图像及60个问题，包括室内和室外场景、模因、绘画、素描等。","descriptions":{"en":"LLaVA-Bench evaluates the model's ability in more challenging tasks, including a diverse set of 24 images with 60 questions in total, including indoor and outdoor scenes, memes, paintings, sketches, etc.","zh":"LLaVA-Bench用于评估多模态大模型应对复杂任务的能力，内含24 张图像及60个问题，包括室内和室外场景、模因、绘画、素描等。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1360","languages":[],"library_direction":null,"modality":"multimodal","name":"LLaVA-Bench","openness":"open","publisher":"University of Wisconsin–Madison","released":"2023-04-17","released_reference":{"basis":"paper_first_version","note":"The original Visual Instruction Tuning paper introduces the visual-chat evaluation benchmark.","source_key":"opencompass:1360","source_url":"https://arxiv.org/abs/2304.08485"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1360-llava-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLaVA-Bench","topics":["multimodal"],"unit":null},{"aliases":["Large Language Models Are Not Strong Abstract Reasoners","LLMs Are Not Strong Abstract Reasoners"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"A suite of abstract-reasoning tasks (IJCAI 2024), several of them text encodings of visual puzzles such as ARC, ACRE and RAVEN, plus PVR and tasks drawn from OpenAI Evals. A text-encoded ARC score is not comparable with a visual ARC score, and the source puzzles are public, so contamination cannot be ruled out.","descriptions":{"en":"A suite of abstract-reasoning tasks (IJCAI 2024), several of them text encodings of visual puzzles such as ARC, ACRE and RAVEN, plus PVR and tasks drawn from OpenAI Evals. A text-encoded ARC score is not comparable with a visual ARC score, and the source puzzles are public, so contamination cannot be ruled out."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:llm_abstract_reasoning","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"LLM Abstract Reasoning Benchmark","openness":"unknown","publisher":null,"released":"2023-05-31","released_reference":{"basis":"benchmark_release","source_key":"model-reports:llm_abstract_reasoning","source_url":"https://arxiv.org/abs/2305.19555"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"llm_abstract_reasoning","source":"model_reports","source_url":"https://arxiv.org/abs/2305.19555","topics":["reasoning-math"],"unit":null},{"aliases":["llm-benchmark-panel"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yunusbaysal/llm-benchmark-panel","source_record":"bm_llm-benchmark-panel_413dbe56"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-provider LLM benchmarking harness with task suites including reasoning, coding, and Turkish-language questions; scoring supports exact_match, keywords, regex, and optional llm_judge.","descriptions":{"en":"Multi-provider LLM benchmarking harness with task suites including reasoning, coding, and Turkish-language questions; scoring supports exact_match, keywords, regex, and optional llm_judge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:yunusbaysal/llm-benchmark-panel","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_llm-benchmark-panel_413dbe56"},"modality":null,"name":"LLM Benchmark Panel","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:yunusbaysal/llm-benchmark-panel","source_url":"https://github.com/yunusbaysal/llm-benchmark-panel"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-yunusbaysal-llm-benchmark-panel","source":"claire_radar","source_url":"https://github.com/yunusbaysal/llm-benchmark-panel","topics":["finance"],"unit":null},{"aliases":["llm-math-reasoning-benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:buguerito/llm-math-reasoning-benchmark","source_record":"bm_llm-math-reasoning-benchmark_8a6b492e"},"categories":["Finance","Language & Knowledge","Mathematics & Formal Science","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates final-answer accuracy, reasoning quality, instruction following, and causal error classification across 20 math problems in six categories with deterministic validators and human annotation.","descriptions":{"en":"Evaluates final-answer accuracy, reasoning quality, instruction following, and causal error classification across 20 math problems in six categories with deterministic validators and human annotation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:buguerito/llm-math-reasoning-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_llm-math-reasoning-benchmark_8a6b492e"},"modality":null,"name":"LLM Mathematical Reasoning Evaluation Benchmark","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:buguerito/llm-math-reasoning-benchmark","source_url":"https://github.com/Buguerito/llm-math-reasoning-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-buguerito-llm-math-reasoning-benchmark","source":"claire_radar","source_url":"https://github.com/Buguerito/llm-math-reasoning-benchmark","topics":["reasoning-math","finance"],"unit":null},{"aliases":["llm-quantization-benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sameer-sde/llm-quantization-benchmark","source_record":"bm_llm-quantization-benchmark_48a90303"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark measuring inference speed, memory usage, and accuracy of MLX and llama.cpp with 4-bit and 8-bit quantization across three 3B-class LLMs on Apple M4 hardware.","descriptions":{"en":"Benchmark measuring inference speed, memory usage, and accuracy of MLX and llama.cpp with 4-bit and 8-bit quantization across three 3B-class LLMs on Apple M4 hardware."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:sameer-sde/llm-quantization-benchmark","languages":[],"library_direction":{"basis":"model-review-2","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_llm-quantization-benchmark_48a90303"},"modality":null,"name":"LLM Quantization Benchmark on Apple Silicon","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:sameer-sde/llm-quantization-benchmark","source_url":"https://github.com/sameer-sde/llm-quantization-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-sameer-sde-llm-quantization-benchmark","source":"claire_radar","source_url":"https://github.com/sameer-sde/llm-quantization-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:gbbarra/llm-summarization-benchmark-ptbr","source_record":"bm_llm-summarization-benchmark-ptbr_c5bebc73"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates reference-faithful summarization of a technical report in Brazilian Portuguese across 9 local LLMs, using a pre-registered rubric with six runs per model and reported scores, timing, and speed.","descriptions":{"en":"Evaluates reference-faithful summarization of a technical report in Brazilian Portuguese across 9 local LLMs, using a pre-registered rubric with six runs per model and reported scores, timing, and speed."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:gbbarra/llm-summarization-benchmark-ptbr","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_llm-summarization-benchmark-ptbr_c5bebc73"},"modality":null,"name":"LLM Summarization Benchmark — Brazilian Portuguese","openness":"unknown","publisher":null,"released":"2026-08-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:gbbarra/llm-summarization-benchmark-ptbr","source_url":"https://github.com/gbbarra/llm-summarization-benchmark-ptbr"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-gbbarra-llm-summarization-benchmark-ptbr","source":"claire_radar","source_url":"https://github.com/gbbarra/llm-summarization-benchmark-ptbr","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","智能体","Agent","任务执行","Task Execution","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LLM-BabyBench is a benchmark suite designed to evaluate Large Language Models (LLMs) on grounded planning and reasoning tasks. LLM-BabyBench是一个专门评估大语言模型在交互环境中规划和推理能力的新基准测试套件。基于BabyAI网格世界的文本适配版本，该基准评估LLMs在三个核心方面的表现：预测动作对环境状态的影响（Predict任务）、生成低级动作序列以实现指定目标（Plan任务）、以及将高级指令分解为连贯的子目标序列（Decompose任务）。\n基准包含16个难度级别，提供三种文本格式（Narrative、Structured、JSON），并配备OmniBot专家代理用于生成基准数据。","descriptions":{"en":"LLM-BabyBench is a benchmark suite designed to evaluate Large Language Models (LLMs) on grounded planning and reasoning tasks.","zh":"LLM-BabyBench是一个专门评估大语言模型在交互环境中规划和推理能力的新基准测试套件。基于BabyAI网格世界的文本适配版本，该基准评估LLMs在三个核心方面的表现：预测动作对环境状态的影响（Predict任务）、生成低级动作序列以实现指定目标（Plan任务）、以及将高级指令分解为连贯的子目标序列（Decompose任务）。\n基准包含16个难度级别，提供三种文本格式（Narrative、Structured、JSON），并配备OmniBot专家代理用于生成基准数据。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1838","languages":[],"library_direction":null,"modality":null,"name":"LLM-BabyBench","openness":"unknown","publisher":"MBZUAI, Abu Dhabi, UAE","released":"2025-05-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1838-llm-babybench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLM-BabyBench","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ishant-005/llm-finetuning-benchmark","source_record":"bm_llm-finetuning-benchmark_efda9d1c"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmarks fine-tuning strategies for LLMs using Financial PhraseBank and Qwen 2.5-0.5B.","descriptions":{"en":"Benchmarks fine-tuning strategies for LLMs using Financial PhraseBank and Qwen 2.5-0.5B."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:ishant-005/llm-finetuning-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_llm-finetuning-benchmark_efda9d1c"},"modality":null,"name":"llm-finetuning-benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:ishant-005/llm-finetuning-benchmark","source_url":"https://github.com/Ishant-005/llm-finetuning-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-ishant-005-llm-finetuning-benchmark","source":"claire_radar","source_url":"https://github.com/Ishant-005/llm-finetuning-benchmark","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:miolonixc/llm-free-benchmark","source_record":"bm_llm-free-benchmark_d7c7d60c"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Runs a fixed battery of seven typed tasks against free-tier LLMs, grades each answer programmatically on a 0-5 scale, and reports rankings with per-task scores and latency.","descriptions":{"en":"Runs a fixed battery of seven typed tasks against free-tier LLMs, grades each answer programmatically on a 0-5 scale, and reports rankings with per-task scores and latency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:miolonixc/llm-free-benchmark","languages":[],"library_direction":null,"modality":null,"name":"llm-free-benchmark","openness":"unknown","publisher":null,"released":"2026-08-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:miolonixc/llm-free-benchmark","source_url":"https://github.com/Miolonixc/llm-free-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-miolonixc-llm-free-benchmark","source":"claire_radar","source_url":"https://github.com/Miolonixc/llm-free-benchmark","topics":[],"unit":null},{"aliases":["LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A prospective live benchmark in which LLMs forecast unresolved sports events. Forecasts are timestamped and schema-validated, then scored against the real outcomes.","descriptions":{"en":"A prospective live benchmark in which LLMs forecast unresolved sports events. Forecasts are timestamped and schema-validated, then scored against the real outcomes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24573","languages":[],"library_direction":null,"modality":null,"name":"LLM-SoccerArena","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24573","source_url":"https://arxiv.org/abs/2607.24573"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24573","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24573","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":130,"github_stars_30d":null,"hf_dataset_downloads":1452,"hf_paper_upvotes":9,"join":"artifact:arxiv:2504.10415","repository":"gh:deep-symbolic-mathematics/llm-srbench","source_record":"lib_llmsr_bench"},"categories":["推理","Reasoning","Strong Reasoning","科学智能","AI for Science","科学推理","Scientific Reasoning","数理能力","Math"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LLM-SRBench, a comprehensive benchmark with 239 challenging problems across four scientific domains specifically designed to evaluate LLM-based scientific equation discovery methods while preventing trivial memorization. LLM-SRBench，这是一个包含239个挑战性问题的综合基准测试，涵盖四个科学领域，专门设计用于评估基于LLMs的科学方程式发现方法，同时防止简单记忆。","descriptions":{"en":"LLM-SRBench, a comprehensive benchmark with 239 challenging problems across four scientific domains specifically designed to evaluate LLM-based scientific equation discovery methods while preventing trivial memorization.","zh":"LLM-SRBench，这是一个包含239个挑战性问题的综合基准测试，涵盖四个科学领域，专门设计用于评估基于LLMs的科学方程式发现方法，同时防止简单记忆。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1752","languages":[],"library_direction":{"basis":"model-review","categories":["science","math"],"join":"artifact:arxiv:2504.10415","primary":"science","source_record":"lib_llmsr_bench"},"modality":null,"name":"LLM-SRBench","openness":"unknown","publisher":"VinUniversity, etc.","released":"2025-04-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1752-llm-srbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLM-SRBench","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","NeurIPS 2024","大语言模型","LLM","事实可靠性","Factual Reliability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LLM-Uncertainty-Bench is a new benchmarking approach for LLMs that integrates uncertainty quantification. It spans 5 representative natural language processing tasks, each has a dataset with 10,000 instances. LLM-Uncertainty-Bench将不确定性纳入LLM评估，包含5个具有代表性的自然语言处理任务，每个任务都有包含10000个实例的数据集支撑。","descriptions":{"en":"LLM-Uncertainty-Bench is a new benchmarking approach for LLMs that integrates uncertainty quantification. It spans 5 representative natural language processing tasks, each has a dataset with 10,000 instances.","zh":"LLM-Uncertainty-Bench将不确定性纳入LLM评估，包含5个具有代表性的自然语言处理任务，每个任务都有包含10000个实例的数据集支撑。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1325","languages":[],"library_direction":null,"modality":null,"name":"LLM-Uncertainty-Bench","openness":"restricted","publisher":"Tencent AI Lab","released":"2024-01-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1325-llm-uncertainty-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLM-Uncertainty-Bench","topics":[],"unit":null},{"aliases":["LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation"],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jieerbobo/llm4trust","source_record":"bm_llm4trust-exploring-the-capabilities-of-la_853eade1"},"categories":["Cybersecurity","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on understanding five trust properties via constructed trust graphs and property understanding tasks, with code and datasets released.","descriptions":{"en":"Evaluates LLMs on understanding five trust properties via constructed trust graphs and property understanding tasks, with code and datasets released."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.33521","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_llm4trust-exploring-the-capabilities-of-la_853eade1"},"modality":null,"name":"LLM4Trust","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33521","source_url":"https://arxiv.org/abs/2609.33521"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33521","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33521","topics":["reasoning-math","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_llmadbench_0d744de1"},"categories":["Advertising & Pricing","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Inserting advertisements (ads) into consumer-facing LLM output is emerging as a new business model, but there is little shared evidence on how such ad insertion should be evaluated or how it affects user preferences.","descriptions":{"en":"Inserting advertisements (ads) into consumer-facing LLM output is emerging as a new business model, but there is little shared evidence on how such ad insertion should be evaluated or how it affects user preferences."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32533","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_llmadbench_0d744de1"},"modality":null,"name":"LLMAdBench","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32533","source_url":"https://arxiv.org/abs/2609.32533"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32533","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32533","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"LLMTabBench evaluates LLMs on binary tabular classification in zero- and few-shot settings, using real-world and controlled synthetic datasets to study the effect of task descriptions and examples on performance.","descriptions":{"en":"LLMTabBench evaluates LLMs on binary tabular classification in zero- and few-shot settings, using real-world and controlled synthetic datasets to study the effect of task descriptions and examples on performance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24417","languages":[],"library_direction":null,"modality":null,"name":"LLMTabBench","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24417","source_url":"https://arxiv.org/abs/2605.24417"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24417","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24417","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","数学","Math","大语言模型","LLM","逻辑推理","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LLMThinkBench is a benchmark framework designed to evaluate large language models (LLMs) on basic math reasoning and “overthinking” behaviors, targeting code-executing language models. LLMThinkBench 是一个用于评估大语言模型（LLM）在基础数学推理和“过度思考”行为方面的基准框架，支持对具备代码执行能力的语言模型进行全面评估。","descriptions":{"en":"LLMThinkBench is a benchmark framework designed to evaluate large language models (LLMs) on basic math reasoning and “overthinking” behaviors, targeting code-executing language models.","zh":"LLMThinkBench 是一个用于评估大语言模型（LLM）在基础数学推理和“过度思考”行为方面的基准框架，支持对具备代码执行能力的语言模型进行全面评估。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2047","languages":[],"library_direction":null,"modality":null,"name":"LLMThinkBench","openness":"unknown","publisher":"DepartmentofComputerScience,VirginiaTech,Blacksburg,VA,USA","released":"2025-07-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2047-llmthinkbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLMThinkBench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"LLVM-Bench evaluates LLMs on resolving LLVM compiler issues with 423 real-world tasks, using an automated evaluation platform.","descriptions":{"en":"LLVM-Bench evaluates LLMs on resolving LLVM compiler issues with 423 real-world tasks, using an automated evaluation platform."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.00700","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_llvm-bench_e576b85e"},"modality":null,"name":"LLVM-Bench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00700","source_url":"https://arxiv.org/abs/2607.00700"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00700","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00700","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":13,"github_stars_30d":null,"hf_dataset_downloads":917,"hf_paper_upvotes":null,"join":"artifact:gh:opengptx/lm-evaluation-harness","repository":"gh:opengptx/lm-evaluation-harness","source_record":"catalog_64e4162afc81fd0d"},"categories":["知识","Knowledge","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"AI Language Proficiency Monitor is a multilingual benchmark platform that systematically assesses LLM performance across up to 200 languages. AI Language Proficiency Monitor是一个多语言大语言模型评测平台，系统性评估模型在多达200种语言上的性能表现，特别关注低资源语言，整合FLORES+、MMLU、GSM8K、TruthfulQA和ARC等数据集评测翻译、问答、数学推理和事实性等能力，提供开源自动更新排行榜和交互式仪表板，覆盖全球80-95%人口使用的语言。","descriptions":{"en":"AI Language Proficiency Monitor is a multilingual benchmark platform that systematically assesses LLM performance across up to 200 languages.","zh":"AI Language Proficiency Monitor是一个多语言大语言模型评测平台，系统性评估模型在多达200种语言上的性能表现，特别关注低资源语言，整合FLORES+、MMLU、GSM8K、TruthfulQA和ARC等数据集评测翻译、问答、数学推理和事实性等能力，提供开源自动更新排行榜和交互式仪表板，覆盖全球80-95%人口使用的语言。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2128","languages":["Multilingual"],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"artifact:gh:opengptx/lm-evaluation-harness","primary":"knowledge-expert-qa","source_record":"catalog_64e4162afc81fd0d"},"modality":null,"name":"lm-evaluation-harness","openness":"unknown","publisher":"Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung,etc","released":"2025-07-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2128-lm-evaluation-harness","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/lm-evaluation-harness","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["长文本","Long-Context","智能体","Agent","任务执行","Task Execution","长上下文","Long Context"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LMAct is a benchmark for evaluating frontier multimodal models' in-context imitation learning capabilities in long contexts. LMAct是评估前沿多模态模型在长上下文中的上下文模仿学习能力的基准。它评估了诸如井字棋、国际象棋和雅达利等交互式任务的多模式决策。该测试集包含多达100万个令牌上下文和512个专家演示集。","descriptions":{"en":"LMAct is a benchmark for evaluating frontier multimodal models' in-context imitation learning capabilities in long contexts.","zh":"LMAct是评估前沿多模态模型在长上下文中的上下文模仿学习能力的基准。它评估了诸如井字棋、国际象棋和雅达利等交互式任务的多模式决策。该测试集包含多达100万个令牌上下文和512个专家演示集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2088","languages":[],"library_direction":null,"modality":null,"name":"LMAct","openness":"unknown","publisher":"GoogleDeepMind","released":"2024-12-02","released_reference":{"basis":"paper_first_version","note":"First version introducing LMAct; the 2025 revision is not its release.","source_key":"opencompass:2088","source_url":"https://arxiv.org/abs/2412.01441"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2088-lmact","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LMAct","topics":["agents","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":39549,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_a43419ec57910ff1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LMArena Text Leaderboard is a blind human preference evaluation benchmark that ranks models based on pairwise comparisons in real-world conversations. The leaderboard uses Elo ratings computed from user preferences in head-to-head model battles, providing a comprehensive measure of overall model capability and style.","descriptions":{"en":"LMArena Text Leaderboard is a blind human preference evaluation benchmark that ranks models based on pairwise comparisons in real-world conversations. The leaderboard uses Elo ratings computed from user preferences in head-to-head model battles, providing a comprehensive measure of overall model capability and style."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:lmarena-text:grok-4.1-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lmarena-text?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:lmarena-text","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_a43419ec57910ff1"},"modality":"text","name":"LMArena Text Leaderboard","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":1483.0,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":1483.0,"raw_min":1465.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:lmarena-text:grok-4.1-thinking-2025-11-17","reported_date":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lmarena-text?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1"},{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-lmarena-text","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lmarena-text?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":61,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":26,"join":"llm_stats_id","repository":"gh:hkust-nlp/loca-bench","source_record":"catalog_c158bb91f29e76ad"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LOCA-Bench is a long-context agentic benchmark. The 256k variant evaluates agents using the official ReAct mode with an environment description length of 256k tokens, measuring how well models reason and act over very long contexts.","descriptions":{"en":"LOCA-Bench is a long-context agentic benchmark. The 256k variant evaluates agents using the official ReAct mode with an environment description length of 256k tokens, measuring how well models reason and act over very long contexts."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:loca-bench-256k:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/loca-bench-256k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:loca-bench-256k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","tool-use"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_c158bb91f29e76ad"},"modality":"text","name":"LOCA-Bench (256k)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":49.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.493,"raw_min":0.493,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:loca-bench-256k:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/loca-bench-256k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-loca-bench-256k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/loca-bench-256k?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["llms-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:diovoib/llms-benchmark","source_record":"bm_llms-benchmark_268bf6b4"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Handful of tasks for manual testing of local LLM tool calling and agent loop behavior.","descriptions":{"en":"Handful of tasks for manual testing of local LLM tool calling and agent loop behavior."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:diovoib/llms-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_llms-benchmark_268bf6b4"},"modality":null,"name":"Local agent benchmark (tools + loop + coding)","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:diovoib/llms-benchmark","source_url":"https://github.com/diovoib/llms-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-diovoib-llms-benchmark","source":"claire_radar","source_url":"https://github.com/diovoib/llms-benchmark","topics":["agents"],"unit":null},{"aliases":["When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents"],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates conversational memory retrieval in long-horizon agents across four query styles using a dataset derived from LoCoMo.","descriptions":{"en":"Evaluates conversational memory retrieval in long-horizon agents across four query styles using a dataset derived from LoCoMo."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.03467","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research","agent-memory"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_when-users-don-t-ask-benchmarking-context-_4327f09f"},"modality":null,"name":"LOCOMO-CONV","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03467","source_url":"https://arxiv.org/abs/2609.03467"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03467","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03467","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:facebookresearch/logbook","source_record":"bm_logbook_7998328b"},"categories":["General AI","Speech & Audio","eess.AS"],"collected_at":"2026-10-14T09:44:51Z","description":"Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies.","descriptions":{"en":"Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.07338","languages":[],"library_direction":{"basis":"model-review-daily","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_logbook_7998328b"},"modality":null,"name":"Logbook","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.07338","source_url":"https://arxiv.org/abs/2610.07338"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-07338","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07338","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_logdx-ci_ff295ed3"},"categories":["Agents","Code & Software","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark evaluating 11 log reduction tools on 35 GitHub Actions failure cases, scored by 3 LLM debugger families.","descriptions":{"en":"Benchmark evaluating 11 log reduction tools on 35 GitHub Actions failure cases, scored by 3 LLM debugger families."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28876","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_logdx-ci_ff295ed3"},"modality":null,"name":"LogDx-CI","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28876","source_url":"https://arxiv.org/abs/2605.28876"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28876","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28876","topics":["coding","agents"],"unit":null},{"aliases":["Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment"],"attention_signals":null,"categories":["Biology & Drug Discovery","Safety & Trustworthiness","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark assessing large language models' logical consistency in medical risk-of-bias assessment under a Hierarchical Logical Consistency framework with four dimensions.","descriptions":{"en":"A benchmark assessing large language models' logical consistency in medical risk-of-bias assessment under a Hierarchical Logical Consistency framework with four dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11185","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning","safety-alignment","health-medicine"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_can-llms-follow-medical-expert-logic-a-ben_a2cc2223"},"modality":null,"name":"LogiMed-RoB","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11185","source_url":"https://arxiv.org/abs/2609.11185"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11185","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11185","topics":[],"unit":null},{"aliases":["LogiQA-plus","LogiQA-Plus","LogiQA plus"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"A robustness variant of LogiQA with the same three subsets as ReClor-plus (shuffled options, correct choice replaced by \"none of the other options is correct\", and both). A score is only meaningful next to the original LogiQA score and with the subset named, and the source questions are public.","descriptions":{"en":"A robustness variant of LogiQA with the same three subsets as ReClor-plus (shuffled options, correct choice replaced by \"none of the other options is correct\", and both). A score is only meaningful next to the original LogiQA score and with the subset named, and the source questions are public."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:logiqa_plus","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"LogiQA-plus","openness":"unknown","publisher":null,"released":"2023-10-13","released_reference":{"basis":"benchmark_release","source_key":"model-reports:logiqa_plus","source_url":"https://arxiv.org/abs/2310.09430"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"logiqa_plus","source":"model_reports","source_url":"https://arxiv.org/abs/2310.09430","topics":["reasoning-math"],"unit":null},{"aliases":["LogiQAv2-plus","LogiQAv2-Plus","LogiQA 2.0-plus"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"A robustness variant of LogiQA 2.0 with the same three subsets as ReClor-plus. As with its siblings, the drop from the original LogiQA 2.0 score is the measurement, a single blended number hides which variation caused it, and the source questions are public.","descriptions":{"en":"A robustness variant of LogiQA 2.0 with the same three subsets as ReClor-plus. As with its siblings, the drop from the original LogiQA 2.0 score is the measurement, a single blended number hides which variation caused it, and the source questions are public."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:logiqav2_plus","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"LogiQAv2-plus","openness":"unknown","publisher":null,"released":"2023-10-13","released_reference":{"basis":"benchmark_release","source_key":"model-reports:logiqav2_plus","source_url":"https://arxiv.org/abs/2310.09430"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"logiqav2_plus","source":"model_reports","source_url":"https://arxiv.org/abs/2310.09430","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LOKI, a multimodal synthetic data detection benchmark, designed specifically to comprehensively assess the capabilities of LMMs in detecting synthetic data. LOKI是一个多模态合成数据检测基准，专门设计用于全面评估 LMMs 在检测合成数据方面的能力。","descriptions":{"en":"LOKI, a multimodal synthetic data detection benchmark, designed specifically to comprehensively assess the capabilities of LMMs in detecting synthetic data.","zh":"LOKI是一个多模态合成数据检测基准，专门设计用于全面评估 LMMs 在检测合成数据方面的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1572","languages":[],"library_direction":null,"modality":"multimodal","name":"LOKI","openness":"open","publisher":"Shanghai AI Laboratory","released":"2024-10-13","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1572-loki","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LOKI","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:pepperbubble/lomevqa","source_record":"bm_lomevqa_dd681f2a"},"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"LoMeVQA is a benchmark for longitudinal medical visual question answering, with 206K VQA pairs across five tasks: progress classification, progress description, progress report generation, differential region grounding, and differential region description.","descriptions":{"en":"LoMeVQA is a benchmark for longitudinal medical visual question answering, with 206K VQA pairs across five tasks: progress classification, progress description, progress report generation, differential region grounding, and differential region description."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27806","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_lomevqa_dd681f2a"},"modality":null,"name":"LoMeVQA","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27806","source_url":"https://arxiv.org/abs/2607.27806"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27806","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27806","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Coding agents now sustain hours-long, tool-driven loops, yet their ability to carry long-horizon analog and mixed-signal circuits to electrical specification remains unmeasured.","descriptions":{"en":"Coding agents now sustain hours-long, tool-driven loops, yet their ability to carry long-horizon analog and mixed-signal circuits to electrical specification remains unmeasured."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33356","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","science"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_long-horizon-analog-design-bench_66fde81d"},"modality":null,"name":"Long-Horizon Analog Design Bench","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33356","source_url":"https://arxiv.org/abs/2609.33356"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33356","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33356","topics":["coding","agents","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":559,"github_stars_30d":18,"hf_dataset_downloads":23048,"hf_paper_upvotes":78,"join":"claire_record_id","repository":"gh:zli12321/lhtb","source_record":"bm_long-horizon-terminal-bench_593d14a6"},"categories":["Agents & Tool Use","Code","General AI","Multimodal","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-horizon terminal tasks in a containerized environment with hidden verifiers and dense reward grading across 46 tasks and nine categories.","descriptions":{"en":"Evaluates long-horizon terminal tasks in a containerized environment with hidden verifiers and dense reward grading across 46 tasks and nine categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.08964","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_long-horizon-terminal-bench_593d14a6"},"modality":null,"name":"Long-Horizon-Terminal-Bench","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08964","source_url":"https://arxiv.org/abs/2607.08964"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08964","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08964","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":18,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":36,"join":"claire_record_id","repository":"gh:pkucs-ltf/longav-compass","source_record":"bm_longav-compass_f7e880eb"},"categories":["General AI","Multimodal","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"LongAV-Compass evaluates minute-scale audio-visual generation across text-to-audio-video, image-to-audio-video, and video-to-audio-video tasks. It includes 284 curated test cases and an evaluation framework combining multimodal metrics with MLLM-assisted assessment across 20+ dimensions covering segment quality, cross-segment consistency, narrative coherence, semantic alignment, and audiovisual synchronization.","descriptions":{"en":"LongAV-Compass evaluates minute-scale audio-visual generation across text-to-audio-video, image-to-audio-video, and video-to-audio-video tasks. It includes 284 curated test cases and an evaluation framework combining multimodal metrics with MLLM-assisted assessment across 20+ dimensions covering segment quality, cross-segment consistency, narrative coherence, semantic alignment, and audiovisual synchronization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26244","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","speech-audio"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_longav-compass_f7e880eb"},"modality":null,"name":"LongAV-Compass","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26244","source_url":"https://arxiv.org/abs/2605.26244"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26244","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26244","topics":["multimodal"],"unit":null},{"aliases":["LongBench","LongBench v2"],"attention_signals":{"as_of":"2026-10-12","github_stars":1246,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":39,"join":"artifact:gh:thudm/longbench","repository":"gh:thudm/longbench","source_record":"lib_longbench_v2"},"categories":["long_context"],"collected_at":null,"description":"Nominal context length is not effective context length.","descriptions":{"en":"Nominal context length is not effective context length."},"evidence_summary":{"document_count":3,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000longbench\u0000qwen3_5_model_card\u0000longbench_v2\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000longbench\u0000qwen3_5_model_card\u0000longbench_v2\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:longbench","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"artifact:gh:thudm/longbench","primary":"long-context","source_record":"lib_longbench_v2"},"modality":null,"name":"LongBench","openness":"unknown","publisher":null,"released":"2023-08-28","released_reference":{"basis":"benchmark_release","source_key":"model-reports:longbench","source_url":"https://github.com/THUDM/LongBench"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":63.2,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":63.2,"raw_min":63.2,"source_reference":{"obs_id":"curated\u0000longbench\u0000qwen3_5_model_card\u0000longbench_v2\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000longbench\u0000qwen3_5_model_card\u0000longbench_v2\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"longbench","source":"model_reports","source_url":"https://github.com/THUDM/LongBench","topics":["long-context"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1246,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":39,"join":"artifact:gh:thudm/longbench","repository":"gh:thudm/longbench","source_record":"lib_longbench_v2"},"categories":["长文本","Long-Context","ACL 2024","大语言模型","LLM","长上下文","Long Context","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LongBench is a benchmark for bilingual, multitask, and comprehensive assessment of long context understanding capabilities of large language models. LongBench 是一个多任务、中英双语、针对大语言模型长文本理解能力的评测基准。","descriptions":{"en":"LongBench is a benchmark for bilingual, multitask, and comprehensive assessment of long context understanding capabilities of large language models.","zh":"LongBench 是一个多任务、中英双语、针对大语言模型长文本理解能力的评测基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:542","languages":["English","Chinese","Multilingual"],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"artifact:gh:thudm/longbench","primary":"long-context","source_record":"lib_longbench_v2"},"modality":null,"name":"LongBench","openness":"restricted","publisher":null,"released":"2023-08-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-542-longbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LongBench","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1246,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":39,"join":"llm_stats_id","repository":"gh:thudm/longbench","source_record":"lib_longbench_v2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","structured_output","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.","descriptions":{"en":"LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:longbench-v2:deepseek-v3","reported_at":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longbench-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:longbench-v2","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"lib_longbench_v2"},"modality":"text","name":"LongBench v2","openness":"unknown","publication":{"evidence":[{"note":"The official README links the v2 paper and project, identifies the v2 multiple-choice tasks, and explicitly separates v1 materials. Its task description matches this v2 source record.","url":"https://github.com/THUDM/LongBench/blob/main/README.md"}],"id":"longbench-v2-original","links":[{"kind":"paper","title":"LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks","url":"https://arxiv.org/abs/2412.15204"},{"kind":"repo","title":"LongBench v2 official repository","url":"https://github.com/THUDM/LongBench"},{"kind":"website","title":"LongBench v2 project","url":"https://longbench2.github.io/"}],"record_keys":["llm-stats:longbench-v2"],"reviewed_at":"2026-10-26","scope":{"en":"LongBench v2's reasoning-focused release. The shared repository's top-level documentation covers v2 and explicitly moves the older LongBench v1 materials into a subdirectory.","zh":"侧重推理的 LongBench v2；共享仓库顶层文档对应 v2，并明确将较早的 LongBench v1 资料移至子目录。"}},"publisher":null,"released":"2024-12-19","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:longbench-v2","source_url":"https://arxiv.org/abs/2412.15204"},"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":66.3,"display_multiplier":100,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":0.663,"raw_min":0.261,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:longbench-v2:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longbench-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-longbench-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longbench-v2?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":10,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_bd5708897110de7e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LongCodeBench evaluates the code understanding and comprehension abilities of large language models at very long context windows, scaling up to 1M tokens. It tests whether models can reason about extensive codebases provided in a single prompt by answering multiple-choice questions about the code.","descriptions":{"en":"LongCodeBench evaluates the code understanding and comprehension abilities of large language models at very long context windows, scaling up to 1M tokens. It tests whether models can reason about extensive codebases provided in a single prompt by answering multiple-choice questions about the code."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:longcodebench:nova-2-lite","reported_at":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longcodebench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:longcodebench","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","code-generation"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_bd5708897110de7e"},"modality":"text","name":"LongCodeBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":84.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.84,"raw_min":0.84,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:longcodebench:nova-2-lite","reported_date":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longcodebench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"}],"slug":"llm-stats-longcodebench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longcodebench?top_n=500","topics":["coding","reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":241,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_longcounsel-8_e2d230be"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates longitudinal depression tracking from five-session counseling dialogues, with 7,749 trajectories across three datasets and labels for symptom severity and trend.","descriptions":{"en":"Evaluates longitudinal depression tracking from five-session counseling dialogues, with 7,749 trajectories across three datasets and labels for symptom severity and trend."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.03507","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_longcounsel-8_e2d230be"},"modality":null,"name":"LongCounsel-8","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03507","source_url":"https://arxiv.org/abs/2609.03507"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03507","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03507","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LongDocBench is a benchmark for Table-of-Contents Hierarchy Recovery and Contextual Relationship Recovery in long documents. It includes 85 real-world documents (financial reports, textbooks, academic papers) spanning 2,582 pages, with human-verified annotations for 3,937 heading nodes and 3,258 contextual relationships.","descriptions":{"en":"LongDocBench is a benchmark for Table-of-Contents Hierarchy Recovery and Contextual Relationship Recovery in long documents. It includes 85 real-world documents (financial reports, textbooks, academic papers) spanning 2,582 pages, with human-verified annotations for 3,937 heading nodes and 3,258 contextual relationships."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.15064","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","document-ocr"],"join":"claire_record_id","primary":"long-context","source_record":"bm_longdocbench_5cb5bcf9"},"modality":null,"name":"LongDocBench","openness":"unknown","publisher":null,"released":"2026-08-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15064","source_url":"https://arxiv.org/abs/2608.15064"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15064","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15064","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":148,"github_stars_30d":8,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":"gh:zjunlp/datamind","source_record":"bm_longds-bench_2436ff1e"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"LongDS-Bench evaluates long-horizon, multi-turn data analysis tasks where agents must maintain, update, restore, and compose evolving analytical states. It comprises 68 tasks from real-world Kaggle notebooks spanning 2,225 turns across six domains, with an average dependency span of 11.3 turns.","descriptions":{"en":"LongDS-Bench evaluates long-horizon, multi-turn data analysis tasks where agents must maintain, update, restore, and compose evolving analytical states. It comprises 68 tasks from real-world Kaggle notebooks spanning 2,225 turns across six domains, with an average dependency span of 11.3 turns."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.30434","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","agent-memory"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_longds-bench_2436ff1e"},"modality":null,"name":"LongDS-Bench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30434","source_url":"https://arxiv.org/abs/2605.30434"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30434","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30434","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"LongEarth-Bench evaluates vision-language models on long-horizon Earth observation reasoning, with ~120k QA samples from 117k images, sequences avg 15.14 frames (up to 30), covering 12 tasks in evolution summarization, spatial reasoning, anomaly identification, and logical prediction.","descriptions":{"en":"LongEarth-Bench evaluates vision-language models on long-horizon Earth observation reasoning, with ~120k QA samples from 117k images, sequences avg 15.14 frames (up to 30), covering 12 tasks in evolution summarization, spatial reasoning, anomaly identification, and logical prediction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.13344","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_longearth-bench_e5b3d65a"},"modality":null,"name":"LongEarth-Bench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13344","source_url":"https://arxiv.org/abs/2608.13344"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13344","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13344","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:shunya-kato/longegorefer","source_record":"bm_longegorefer_d5a8d99b"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"LongEgoRefer is a benchmark for Video Referring Expression Comprehension in long-form egocentric videos, built from Ego4D. It contains 1,498 referring expressions over videos averaging 45 minutes, requiring temporal grounding of when an occurrence happens and spatial grounding of where the object appears. The benchmark uses evaluation metrics for temporal and spatial grounding.","descriptions":{"en":"LongEgoRefer is a benchmark for Video Referring Expression Comprehension in long-form egocentric videos, built from Ego4D. It contains 1,498 referring expressions over videos averaging 45 minutes, requiring temporal grounding of when an occurrence happens and spatial grounding of where the object appears. The benchmark uses evaluation metrics for temporal and spatial grounding."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.02096","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_longegorefer_d5a8d99b"},"modality":null,"name":"LongEgoRefer","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02096","source_url":"https://arxiv.org/abs/2607.02096"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02096","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02096","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":692,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":27,"join":"llm_stats_id","repository":null,"source_record":"catalog_711338389ffa8b3c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["factuality","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LongFact evaluates factual precision over long-form generations containing many individual claims. Each claim is extracted and verified, and the model is scored on claim-level precision, measuring whether extended responses introduce unsupported or false statements.","descriptions":{"en":"LongFact evaluates factual precision over long-form generations containing many individual claims. Each claim is extracted and verified, and the model is scored on claim-level precision, measuring whether extended responses introduce unsupported or false statements."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:longfact:mai-thinking-1","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:longfact","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_711338389ffa8b3c"},"modality":"text","name":"LongFact","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":98.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.98,"raw_min":0.98,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:longfact:mai-thinking-1","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"}],"slug":"llm-stats-longfact","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":693,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":27,"join":"llm_stats_id","repository":null,"source_record":"catalog_e780d7ef36edc690"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LongFact is a benchmark for evaluating long-form factuality in large language models. It comprises 2,280 fact-seeking prompts spanning 38 topics, designed to test a model's ability to generate accurate, long-form responses. The benchmark uses SAFE (Search-Augmented Factuality Evaluator) to evaluate factual accuracy.","descriptions":{"en":"LongFact is a benchmark for evaluating long-form factuality in large language models. It comprises 2,280 fact-seeking prompts spanning 38 topics, designed to test a model's ability to generate accurate, long-form responses. The benchmark uses SAFE (Search-Augmented Factuality Evaluator) to evaluate factual accuracy."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:longfact-concepts:gpt-5-2025-08-07","reported_at":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-concepts?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:longfact-concepts","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_e780d7ef36edc690"},"modality":"text","name":"LongFact Concepts","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.7000000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.007,"raw_min":0.007,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:longfact-concepts:gpt-5-2025-08-07","reported_date":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-concepts?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"}],"slug":"llm-stats-longfact-concepts","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-concepts?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":692,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":27,"join":"llm_stats_id","repository":null,"source_record":"catalog_9feb50aa67d76cd7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LongFact is a benchmark for evaluating long-form factuality in large language models. It comprises 2,280 fact-seeking prompts spanning 38 topics, designed to test a model's ability to generate accurate, long-form responses. The benchmark uses SAFE (Search-Augmented Factuality Evaluator) to evaluate factual accuracy.","descriptions":{"en":"LongFact is a benchmark for evaluating long-form factuality in large language models. It comprises 2,280 fact-seeking prompts spanning 38 topics, designed to test a model's ability to generate accurate, long-form responses. The benchmark uses SAFE (Search-Augmented Factuality Evaluator) to evaluate factual accuracy."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:longfact-objects:gpt-5-2025-08-07","reported_at":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-objects?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:longfact-objects","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_9feb50aa67d76cd7"},"modality":"text","name":"LongFact Objects","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.008,"raw_min":0.008,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:longfact-objects:gpt-5-2025-08-07","reported_date":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-objects?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"}],"slug":"llm-stats-longfact-objects","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-objects?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_longharness-bench_ea0f8798"},"categories":["General AI","Language & Knowledge","Long Context","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute.","descriptions":{"en":"Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38137","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"claire_record_id","primary":"long-context","source_record":"bm_longharness-bench_ea0f8798"},"modality":null,"name":"LongHarness Bench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38137","source_url":"https://arxiv.org/abs/2609.38137"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38137","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38137","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:cjj826/longjudgebench","source_record":"bm_longjudgebench_1675bac5"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"LongJudgeBench evaluates LLM-as-a-judge performance on long-form outputs across six datasets covering pointwise, pairwise, and listwise protocols, with bilingual tasks and multiple prompt variants. It measures agreement with human judgments using accuracy, Spearman, and Kendall's tau.","descriptions":{"en":"LongJudgeBench evaluates LLM-as-a-judge performance on long-form outputs across six datasets covering pointwise, pairwise, and listwise protocols, with bilingual tasks and multiple prompt variants. It measures agreement with human judgments using accuracy, Spearman, and Kendall's tau."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01629","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","long-context"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_longjudgebench_1675bac5"},"modality":null,"name":"LongJudgeBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01629","source_url":"https://arxiv.org/abs/2606.01629"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01629","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01629","topics":["long-context","language"],"unit":null},{"aliases":["From Visual Cues to Spoken Narration: Rethinking Audio Description"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:multimodal-ai-lab/cue2narrate","source_record":"bm_from-visual-cues-to-spoken-narration-rethi_b0d7498a"},"categories":["General AI","Speech & Audio","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates audio description generation on long untrimmed movie clips, covering multi-segment localization and narration quality.","descriptions":{"en":"Evaluates audio description generation on long untrimmed movie clips, covering multi-segment localization and narration quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.01725","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_from-visual-cues-to-spoken-narration-rethi_b0d7498a"},"modality":null,"name":"LongLSMDC","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01725","source_url":"https://arxiv.org/abs/2609.01725"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01725","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01725","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for long-horizon clinical decision-making using EHR data from MIMIC-IV, comprising 335 patients with multi-session interactions and three evaluation suites: fact-based QA, temporal reasoning, and long-horizon decision-making.","descriptions":{"en":"A benchmark for long-horizon clinical decision-making using EHR data from MIMIC-IV, comprising 335 patients with multi-session interactions and three evaluation suites: fact-based QA, temporal reasoning, and long-horizon decision-making."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09322","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","long-context"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_longmedbench_c02feb5e"},"modality":null,"name":"LongMedBench","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09322","source_url":"https://arxiv.org/abs/2607.09322"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09322","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09322","topics":["long-context","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"claire_record_id","repository":null,"source_record":"bm_longrca-bench_4c5e770a"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"LongRCA Bench is a benchmark for diagnosing responsible roles and root causes in long-horizon agent failures. It comprises 1,140 failed trajectories across five domains, with human labels for the responsible role and the earliest decisive root-cause step. Evaluation focuses on responsible-role accuracy and exact root-step accuracy.","descriptions":{"en":"LongRCA Bench is a benchmark for diagnosing responsible roles and root causes in long-horizon agent failures. It comprises 1,140 failed trajectories across five domains, with human labels for the responsible role and the earliest decisive root-cause step. Evaluation focuses on responsible-role accuracy and exact root-step accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.15242","languages":[],"library_direction":null,"modality":null,"name":"LongRCA Bench","openness":"unknown","publisher":null,"released":"2026-08-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15242","source_url":"https://arxiv.org/abs/2608.15242"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15242","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15242","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":428,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":40,"join":"llm_stats_id","repository":"gh:x-omni-team/x-omni","source_record":"catalog_5d774bb2ffa891da"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["image-generation","language","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LongText-Bench evaluates text-to-image models on their ability to accurately render long text passages within generated images. It includes English (EN) and Chinese (ZH) subsets to assess multilingual text rendering capabilities.","descriptions":{"en":"LongText-Bench evaluates text-to-image models on their ability to accurately render long text passages within generated images. It includes English (EN) and Chinese (ZH) subsets to assess multilingual text rendering capabilities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:longtext-bench","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"llm_stats_id","primary":"image-video-generation","source_record":"catalog_5d774bb2ffa891da"},"modality":"image","name":"LongText-Bench","openness":"unknown","publisher":null,"released":"2025-07-29","released_reference":{"basis":"paper_first_version","note":"The official X-Omni repository and LongText-Bench dataset card identify this paper as their introduction.","source_key":"llm-stats:longtext-bench","source_url":"https://arxiv.org/abs/2507.22058"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":"higher_is_better","score_summary":{"display_max":null,"display_multiplier":1,"model_count":null,"model_count_basis":null,"numeric_count":0,"raw_max":null,"raw_min":null,"source_reference":null,"unit":null},"scored_models":[],"slug":"llm-stats-longtext-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longtext-bench?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","长文本","Long-Context","long video understanding","omni-modal","fine-grained temporal understanding","多模态模型","VLM","视频理解","Video Understanding","长上下文","Long Context"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LongVALE is the first long video benchmark integrating fine-grained omni modalities (video, audio, speech) information in videos. It comprises 105K omni-modal events with precise temporal boundaries and detailed omni-modal captions, aiming to advance comprehensive multi-modal video understanding. LongVALE 是首个集成视频中细粒度全模态（视频、音频、语音）信息的长视频理解基准。它包含 10.5 万个具有精确时间边界和详细的全模态描述的事件标注，致力于全面提升多模态视频大语言模型的跨模态推理与细粒度时间感知能力。","descriptions":{"en":"LongVALE is the first long video benchmark integrating fine-grained omni modalities (video, audio, speech) information in videos. It comprises 105K omni-modal events with precise temporal boundaries and detailed omni-modal captions, aiming to advance comprehensive multi-modal video understanding.","zh":"LongVALE 是首个集成视频中细粒度全模态（视频、音频、语音）信息的长视频理解基准。它包含 10.5 万个具有精确时间边界和详细的全模态描述的事件标注，致力于全面提升多模态视频大语言模型的跨模态推理与细粒度时间感知能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2067","languages":["English"],"library_direction":null,"modality":"multimodal","name":"LongVALE","openness":"unknown","publisher":"Southern University of Science and Technology & University of Birmingham","released":"2025-04-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2067-longvale","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LongVALE","topics":["multimodal","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":139,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"llm_stats_id","repository":null,"source_record":"catalog_9eb6481a78fe9dbc"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LongVideoBench is a question-answering benchmark featuring video-language interleaved inputs up to an hour long. It includes 3,763 varying-length web-collected videos with subtitles across diverse themes and 6,678 human-annotated multiple-choice questions in 17 fine-grained categories for comprehensive evaluation of long-term multimodal understanding.","descriptions":{"en":"LongVideoBench is a question-answering benchmark featuring video-language interleaved inputs up to an hour long. It includes 3,763 varying-length web-collected videos with subtitles across diverse themes and 6,678 human-annotated multiple-choice questions in 17 fine-grained categories for comprehensive evaluation of long-term multimodal understanding."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:longvideobench:qwen2.5-vl-7b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longvideobench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:longvideobench","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_9eb6481a78fe9dbc"},"modality":"multimodal","name":"LongVideoBench","openness":"unknown","publisher":null,"released":"2024-07-22","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:longvideobench","source_url":"https://arxiv.org/abs/2407.15754"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":80.60000000000001,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.806,"raw_min":0.547,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:longvideobench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longvideobench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-longvideobench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longvideobench?top_n=500","topics":["multimodal","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":139,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"artifact:arxiv:2407.15754","repository":null,"source_record":"catalog_9eb6481a78fe9dbc"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LongVideoBench tests LMMs' understanding of long videos. It's a question-answering benchmark with video-language interleaved inputs up to an hour long and comprises 3,763 web-collected videos with subtitles across diverse themes, LongVideoBench用于评估多模态大模型的长视频理解能力，是基于交错长视频语料构建的问答集，包含3763个不同主题的带字幕的视频。","descriptions":{"en":"LongVideoBench tests LMMs' understanding of long videos. It's a question-answering benchmark with video-language interleaved inputs up to an hour long and comprises 3,763 web-collected videos with subtitles across diverse themes,","zh":"LongVideoBench用于评估多模态大模型的长视频理解能力，是基于交错长视频语料构建的问答集，包含3763个不同主题的带字幕的视频。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1510","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"artifact:arxiv:2407.15754","primary":"video-understanding","source_record":"catalog_9eb6481a78fe9dbc"},"modality":"multimodal","name":"LongVideoBench","openness":"restricted","publisher":null,"released":"2024-07-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1510-longvideobench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LongVideoBench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:sharelab-sii/vlzip","source_record":"bm_longvlbench_2560ef96"},"categories":["General AI","Long Context","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"LongVLBench evaluates long-context multimodal understanding over interleaved visual and textual sequences, using tasks derived from video narratives that require holistic, narrative-level reasoning.","descriptions":{"en":"LongVLBench evaluates long-context multimodal understanding over interleaved visual and textual sequences, using tasks derived from video narratives that require holistic, narrative-level reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.08630","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","long-context"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_longvlbench_2560ef96"},"modality":null,"name":"LongVLBench","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08630","source_url":"https://arxiv.org/abs/2608.08630"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08630","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08630","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_longvqubench_aca1b92b"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"LongVQUBench evaluates long-term video quality understanding with 1200+ videos and 1500 questions across three levels of perceptual reasoning complexity.","descriptions":{"en":"LongVQUBench evaluates long-term video quality understanding with 1200+ videos and 1500 questions across three levels of perceptual reasoning complexity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01086","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_longvqubench_aca1b92b"},"modality":null,"name":"LongVQUBench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01086","source_url":"https://arxiv.org/abs/2607.01086"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01086","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01086","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zheny2751-dotcom/longwebbench","source_record":"bm_longwebbench_6f8a779e"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LongWebBench evaluates structural and functional webpage generation of long webpages. It includes 490 webpages for structural fidelity and 129 for functional interactions, using VLM-based metrics and a DOM-augmented agent pipeline.","descriptions":{"en":"LongWebBench evaluates structural and functional webpage generation of long webpages. It includes 490 webpages for structural fidelity and 129 for functional interactions, using VLM-based metrics and a DOM-augmented agent pipeline."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.17727","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_longwebbench_6f8a779e"},"modality":null,"name":"LongWebBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17727","source_url":"https://arxiv.org/abs/2606.17727"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17727","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17727","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"claire_record_id","repository":null,"source_record":"bm_longwof-bench_2e69a90a"},"categories":["Agents","Agents & Tool Use","Code","Code generation","Mathematics & Formal Science","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates verifiable long-workflow tasks across code generation, agent-environment synthesis, mathematical reasoning, and rule following, with machine-verifiable scoring.","descriptions":{"en":"Evaluates verifiable long-workflow tasks across code generation, agent-environment synthesis, mathematical reasoning, and rule following, with machine-verifiable scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23200","languages":[],"library_direction":{"basis":"model-review-2","categories":["code-generation","long-context"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_longwof-bench_2e69a90a"},"modality":null,"name":"LongWoF-Bench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23200","source_url":"https://arxiv.org/abs/2608.23200"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23200","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23200","topics":["coding","agents","reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":150,"github_stars_30d":18,"hf_dataset_downloads":null,"hf_paper_upvotes":98,"join":"claire_record_id","repository":"gh:amap-ml/looparena","source_record":"bm_looparena_97bc3500"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates a controller model guiding a fixed coding agent through long-running tasks, scored via contract accuracy, strict success rate, and cost metrics across three complementary settings.","descriptions":{"en":"Evaluates a controller model guiding a fixed coding agent through long-running tasks, scored via contract accuracy, strict success rate, and cost metrics across three complementary settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.28281","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_looparena_97bc3500"},"modality":null,"name":"LoopArena","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28281","source_url":"http://arxiv.org/abs/2608.28281v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28281","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28281v1","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","长文本","Long-Context","智能体","Agent","navigation","consistency","物理智能","Embodied AI","长上下文","Long Context","具身交互","Embodied Interaction","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A video-action dataset containing many loop-based navigation dataset in Minecraft environment, aiming to boost the spatial consistency and providing insight for the design of memory module 一个视频-动作导航数据集，包括了在Minecraft环境下大量基于回环的导航数据，能够促进世界模型等视频模型空间一致性的训练，启发记忆模块的设计。","descriptions":{"en":"A video-action dataset containing many loop-based navigation dataset in Minecraft environment, aiming to boost the spatial consistency and providing insight for the design of memory module","zh":"一个视频-动作导航数据集，包括了在Minecraft环境下大量基于回环的导航数据，能够促进世界模型等视频模型空间一致性的训练，启发记忆模块的设计。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1944","languages":[],"library_direction":null,"modality":"multimodal","name":"LoopNav","openness":"unknown","publisher":null,"released":"2025-05-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1944-loopnav","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LoopNav","topics":["agents","multimodal","reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":33,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:microsoft/loopsbench","source_record":"bm_loopsbench_86719709"},"categories":["Agents","Code & Software","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LOOPSBENCH evaluates coding agents on long-horizon tasks structured as dependency DAGs with flow-aware test release and regression obligations, spanning 8 languages and 9 domains.","descriptions":{"en":"LOOPSBENCH evaluates coding agents on long-horizon tasks structured as dependency DAGs with flow-aware test release and regression obligations, spanning 8 languages and 9 domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.00267","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_loopsbench_86719709"},"modality":null,"name":"LoopsBench","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"first_public","source_key":"claire-radar:2608.00267","source_url":"https://github.com/microsoft/Loopsbench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00267","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00267","topics":["coding","agents"],"unit":null},{"aliases":["Benchmarking Recursive-Collapse Warning Claims Under Matched False-Positive Control"],"attention_signals":null,"categories":["General AI","Language & Knowledge","eess.SY"],"collected_at":"2026-09-25T04:40:22Z","description":"Loopzero is a reproducible framework for evaluating recursive-collapse warning claims under a fixed alert-budget condition.","descriptions":{"en":"Loopzero is a reproducible framework for evaluating recursive-collapse warning claims under a fixed alert-budget condition."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.00329","languages":[],"library_direction":null,"modality":null,"name":"Loopzero","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00329","source_url":"https://arxiv.org/abs/2606.00329"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00329","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00329","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yvfanzhu/lorft","source_record":"bm_lorft_f3adff80"},"categories":["Autonomous Driving","Factuality","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"LoRFT evaluates long-range vehicle trajectory reconstruction from fixed highway cameras, with 6,601 manually verified trajectories and map-aware evaluation metrics like ADE and FDE.","descriptions":{"en":"LoRFT evaluates long-range vehicle trajectory reconstruction from fixed highway cameras, with 6,601 manually verified trajectories and map-aware evaluation metrics like ADE and FDE."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.19911","languages":[],"library_direction":null,"modality":null,"name":"LoRFT","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19911","source_url":"https://arxiv.org/abs/2607.19911"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19911","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19911","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":null,"source_record":"bm_losona_d944301c"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"LoSoNA evaluates LLM agents' ability to infer and adapt to local social norms in group chats, using curated transcripts and an elicitor turn to score norm-conforming responses.","descriptions":{"en":"LoSoNA evaluates LLM agents' ability to infer and adapt to local social norms in group chats, using curated transcripts and an elicitor turn to score norm-conforming responses."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.14600","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","multi-agent-games"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_losona_d944301c"},"modality":null,"name":"LoSoNA","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14600","source_url":"https://arxiv.org/abs/2606.14600"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14600","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14600","topics":["agents","language"],"unit":null},{"aliases":["Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A multilingual spoken hallucination detection benchmark with 12,013 news samples across English, Russian, and Kazakh, including synthetic and real-world fake news items in text and audio.","descriptions":{"en":"A multilingual spoken hallucination detection benchmark with 12,013 news samples across English, Russian, and Kazakh, including synthetic and real-world fake news items in text and audio."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24707","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","factuality","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_lost-in-speech-trilingual-spoken-hallucina_f3c90115"},"modality":null,"name":"Lost in Speech","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24707","source_url":"https://arxiv.org/abs/2608.24707"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24707","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24707","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Maritime simulator providing an integration testbed for prototyping and rehearsing offshore human-robot collaboration workflows.","descriptions":{"en":"Maritime simulator providing an integration testbed for prototyping and rehearsing offshore human-robot collaboration workflows."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17124","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_lotusim-energy_02021744"},"modality":null,"name":"LOTUSim-Energy","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17124","source_url":"https://arxiv.org/abs/2609.17124"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17124","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17124","topics":[],"unit":null},{"aliases":["LPFQA"],"attention_signals":null,"categories":["knowledge"],"collected_at":null,"description":"Long-tail professional forum-based benchmark; evaluates complex reasoning across 20 academic and industrial fields with authentic professional scenarios.","descriptions":{"en":"Long-tail professional forum-based benchmark; evaluates complex reasoning across 20 academic and industrial fields with authentic professional scenarios."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000lpfqa\u0000bytedance_seed2_0_model_card\u0000lpfqa\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000lpfqa\u0000bytedance_seed2_0_model_card\u0000lpfqa\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:lpfqa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["knowledge-expert-qa"],"join":"review:14H034160212:2026-10-15","primary":"knowledge-expert-qa","source_record":null},"modality":null,"name":"LPFQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":52.6,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":52.6,"raw_min":47.2,"source_reference":{"obs_id":"curated\u0000lpfqa\u0000bytedance_seed2_0_model_card\u0000lpfqa\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000lpfqa\u0000bytedance_seed2_0_model_card\u0000lpfqa\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"lpfqa","source":"model_reports","source_url":"https://huggingface.co/datasets/m-a-p/LPFQA","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"llm_stats_id","repository":null,"source_record":"catalog_34ee7686c8ea6671"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LSAT (Law School Admission Test) benchmark evaluating complex reasoning capabilities across three challenging tasks: analytical reasoning, logical reasoning, and reading comprehension. The LSAT measures skills considered essential for success in law school including critical thinking, reading comprehension of complex texts, and analysis of arguments.","descriptions":{"en":"LSAT (Law School Admission Test) benchmark evaluating complex reasoning capabilities across three challenging tasks: analytical reasoning, logical reasoning, and reading comprehension. The LSAT measures skills considered essential for success in law school including critical thinking, reading comprehension of complex texts, and analysis of arguments."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:lsat:gpt-4-0613","reported_at":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lsat?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:lsat","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_34ee7686c8ea6671"},"modality":"text","name":"LSAT","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":88.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.88,"raw_min":0.88,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:lsat:gpt-4-0613","reported_date":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lsat?top_n=500"},"unit":null},"scored_models":[{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"}],"slug":"llm-stats-lsat","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lsat?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","NeurIPS 2024","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LTMbenchmark assess the long-term memory, continual learning, and information integration capabilities of the agents via dynamic conversational tasks. LTMbenchmark通过动态对话任务评估智能体的长期记忆、持续学习和信息集成能力。","descriptions":{"en":"LTMbenchmark assess the long-term memory, continual learning, and information integration capabilities of the agents via dynamic conversational tasks.","zh":"LTMbenchmark通过动态对话任务评估智能体的长期记忆、持续学习和信息集成能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1335","languages":[],"library_direction":null,"modality":null,"name":"LTMbenchmark","openness":"unknown","publisher":"GoodAI","released":"2024-09-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1335-ltmbenchmark","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LTMbenchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"LU-500 evaluates concept unlearning for logos with nearly 10,000 pairs, including explicit and implicit contextual tracks, and a multi-grained protocol measuring local removal and global preservation.","descriptions":{"en":"LU-500 evaluates concept unlearning for logos with nearly 10,000 pairs, including explicit and implicit contextual tracks, and a multi-grained protocol measuring local removal and global preservation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24101","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","image-video-generation"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_lu-500_07127332"},"modality":null,"name":"LU-500","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24101","source_url":"https://arxiv.org/abs/2607.24101"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24101","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24101","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LUNAR is a benchmark for evaluating LLM personalization from longitudinal app interaction logs across domains like clothing, food, housing, and mobility. It uses a synthetic data pipeline and evaluates 19 LLMs.","descriptions":{"en":"LUNAR is a benchmark for evaluating LLM personalization from longitudinal app interaction logs across domains like clothing, food, housing, and mobility. It uses a synthetic data pipeline and evaluates 19 LLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05246","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_lunar_024583f6"},"modality":null,"name":"LUNAR","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05246","source_url":"https://arxiv.org/abs/2608.05246"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05246","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05246","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["长文本","long-context","大语言模型","LLM","长上下文","Long Context","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"LV-Eval is a challenging long-context benchmark with five length levels (16k, 32k, 64k, 128k, and 256k) reaching up to 256k words. The average number of words is 102,380, and the Min/Max number of words is 11,896/387,406. It features two main tasks, single-hop QA and multi-hop QA, comprising 11 bilingual datasets. LV-Eval是一个具备5个长度等级（16k、32k、64k、128k和256k）、最大文本测试长度达到256k的长文本评测基准。LV-Eval的平均文本长度达到102,380字，最小/最大文本长度为11,896/387,406字。LV-Eval主要有两类评测任务——单跳QA和多跳QA，共包含11个涵盖中英文的评测数据子集。","descriptions":{"en":"LV-Eval is a challenging long-context benchmark with five length levels (16k, 32k, 64k, 128k, and 256k) reaching up to 256k words. The average number of words is 102,380, and the Min/Max number of words is 11,896/387,406. It features two main tasks, single-hop QA and multi-hop QA, comprising 11 bilingual datasets.","zh":"LV-Eval是一个具备5个长度等级（16k、32k、64k、128k和256k）、最大文本测试长度达到256k的长文本评测基准。LV-Eval的平均文本长度达到102,380字，最小/最大文本长度为11,896/387,406字。LV-Eval主要有两类评测任务——单跳QA和多跳QA，共包含11个涵盖中英文的评测数据子集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:564","languages":[],"library_direction":null,"modality":null,"name":"LV-Eval","openness":"restricted","publisher":null,"released":"2024-02-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-564-lv-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LV-Eval","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":155,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_b029d6417276393b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"LVBench is an extreme long video understanding benchmark designed to evaluate multimodal models on videos up to two hours in duration. It contains 6 major categories and 21 subcategories, with videos averaging five times longer than existing datasets. The benchmark addresses applications requiring comprehension of extremely long videos.","descriptions":{"en":"LVBench is an extreme long video understanding benchmark designed to evaluate multimodal models on videos up to two hours in duration. It contains 6 major categories and 21 subcategories, with videos averaging five times longer than existing datasets. The benchmark addresses applications requiring comprehension of extremely long videos."},"evidence_summary":{"document_count":1,"model_count":25,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:lvbench:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lvbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:lvbench","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_b029d6417276393b"},"modality":"multimodal","name":"LVBench","openness":"unknown","publisher":null,"released":"2024-06-12","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: LVBench: An Extreme Long Video Understanding Benchmark. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:lvbench","source_url":"https://arxiv.org/abs/2406.08035"},"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":25,"score_direction":"higher_is_better","score_summary":{"display_max":85.39999999999999,"display_multiplier":100,"model_count":25,"model_count_basis":"source_model_id","numeric_count":25,"raw_max":0.854,"raw_min":0.404,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:lvbench:gemini-3.7-flash","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lvbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-lvbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lvbench?top_n=500","topics":["multimodal","long-context"],"unit":null},{"aliases":["LëtzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents"],"attention_signals":null,"categories":["General AI","Information retrieval","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Cross-lingual page-level retrieval benchmark over Luxembourgish PDF documents with queries in four languages.","descriptions":{"en":"Cross-lingual page-level retrieval benchmark over Luxembourgish PDF documents with queries in four languages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.21714","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","document-ocr","deep-research"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_l-tzcross-a-cross-lingual-page-level-bench_83560f19"},"modality":null,"name":"LëtzCross","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.21714","source_url":"http://arxiv.org/abs/2608.21714v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-21714","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21714v1","topics":["multimodal"],"unit":null},{"aliases":["M$^3$ISR: A Multi-Modal Multi-View Benchmark for 3D/4D Gaussian Splatting and Feedforward Compression"],"attention_signals":null,"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates 3D/4D Gaussian splatting methods on 25 synthetic scenes with five tracks covering synthesis, streaming, and compression.","descriptions":{"en":"Evaluates 3D/4D Gaussian splatting methods on 25 synthetic scenes with five tracks covering synthesis, streaming, and compression."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22465","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","spatial-3d"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_m-3-isr-a-multi-modal-multi-view-benchmark_dec2d75f"},"modality":null,"name":"M$^3$ISR","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22465","source_url":"http://arxiv.org/abs/2608.22465v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22465","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22465v1","topics":[],"unit":null},{"aliases":["M$^3$R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:hongshi4/m3r-bench","source_record":"bm_m-3-r-bench-a-unified-benchmark-for-eviden_90294861"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates unified metaphor understanding across 1,000 image-text instances with annotations for occurrence, Target-Source mapping, sentiment, and stage-wise explanations.","descriptions":{"en":"Evaluates unified metaphor understanding across 1,000 image-text instances with annotations for occurrence, Target-Source mapping, sentiment, and stage-wise explanations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.05817","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_m-3-r-bench-a-unified-benchmark-for-eviden_90294861"},"modality":null,"name":"M$^3$R-Bench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05817","source_url":"https://arxiv.org/abs/2608.05817"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05817","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05817","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A multilingual proficiency benchmark covering grammatical error detection, round-trip translation, and tokenizer efficiency across 30 languages.","descriptions":{"en":"A multilingual proficiency benchmark covering grammatical error detection, round-trip translation, and tokenizer efficiency across 30 languages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03803","languages":[],"library_direction":null,"modality":null,"name":"M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03803","source_url":"https://arxiv.org/abs/2608.03803"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03803","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03803","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark of 122 real-world AI models in which agents must deploy each model as a ready-to-call API. Standardized test cases check the result, and agents are compared by deployment success rate.","descriptions":{"en":"A benchmark of 122 real-world AI models in which agents must deploy each model as a ready-to-call API. Standardized test cases check the result, and agents are compared by deployment success rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.01189","languages":[],"library_direction":null,"modality":null,"name":"M2ABench","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01189","source_url":"https://arxiv.org/abs/2608.01189"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01189","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01189","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"M3-DuplexBench evaluates full-duplex spoken dialogue models in multi-turn, multilingual (English and Japanese), multidomain settings, with multiple dialogue context settings and turn-taking analysis.","descriptions":{"en":"M3-DuplexBench evaluates full-duplex spoken dialogue models in multi-turn, multilingual (English and Japanese), multidomain settings, with multiple dialogue context settings and turn-taking analysis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.29125","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_m3-duplexbench_afd47e79"},"modality":null,"name":"M3-DuplexBench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29125","source_url":"https://arxiv.org/abs/2607.29125"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29125","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29125","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for multi-modal, multi-session ground-to-aerial place recognition in forests.","descriptions":{"en":"A benchmark for multi-modal, multi-session ground-to-aerial place recognition in forests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23003","languages":[],"library_direction":null,"modality":null,"name":"M3GA-Wild","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23003","source_url":"https://arxiv.org/abs/2609.23003"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23003","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23003","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","NAACL 2024","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"M3T is a novel benchmark dataset tailored to evaluate NMT systems on the comprehensive task of translating semi-structured documents. This dataset aims to bridge the evaluation gap in document-level NMT systems, acknowledging the challenges posed by rich text layouts in real-world applications. M3T 是旨在评估神经机器翻译（NMT）系统在翻译半结构化文档的综合任务上的表现。","descriptions":{"en":"M3T is a novel benchmark dataset tailored to evaluate NMT systems on the comprehensive task of translating semi-structured documents. This dataset aims to bridge the evaluation gap in document-level NMT systems, acknowledging the challenges posed by rich text layouts in real-world applications.","zh":"M3T 是旨在评估神经机器翻译（NMT）系统在翻译半结构化文档的综合任务上的表现。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1147","languages":[],"library_direction":null,"modality":null,"name":"M3T","openness":"unknown","publisher":"AWS AI Labs","released":"2024-06-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1147-m3t","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/M3T","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:openbmb/ma-proofbench","source_record":"bm_ma-proofbench_ec795126"},"categories":["Factuality","Language & Knowledge","Mathematics & Formal Science","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MA-ProofBench evaluates LLMs on theorem proving in mathematical analysis using 200 Lean 4 formalized problems split into undergraduate and Ph.D. levels. It covers six core topics and uses Pass@8 with formal verification via Lean server.","descriptions":{"en":"MA-ProofBench evaluates LLMs on theorem proving in mathematical analysis using 200 Lean 4 formalized problems split into undergraduate and Ph.D. levels. It covers six core topics and uses Pass@8 with formal verification via Lean server."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.13782","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_ma-proofbench_ec795126"},"modality":null,"name":"MA-ProofBench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13782","source_url":"https://arxiv.org/abs/2606.13782"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13782","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13782","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"MAC-Bench evaluates procedural compliance of multi-agent systems under social-engineering pressure, measuring compliance-weighted success rate and Machiavellian gap.","descriptions":{"en":"MAC-Bench evaluates procedural compliance of multi-agent systems under social-engineering pressure, measuring compliance-weighted success rate and Machiavellian gap."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07805","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","multi-agent-games"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_mac-bench_4b2a0d04"},"modality":null,"name":"MAC-Bench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07805","source_url":"https://arxiv.org/abs/2606.07805"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07805","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07805","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":52,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:jetastra/macagentbench","source_record":"bm_macagentbench_782bfad8"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"MacAgentBench evaluates computer use agents on macOS desktop automation, with 676 tasks across 25 applications, including GUI and CLI interactions. It uses deterministic rule-based evaluation and fine-grained multi-checkpoint scoring to assess sub-goal completion.","descriptions":{"en":"MacAgentBench evaluates computer use agents on macOS desktop automation, with 676 tasks across 25 applications, including GUI and CLI interactions. It uses deterministic rule-based evaluation and fine-grained multi-checkpoint scoring to assess sub-goal completion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22557","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_macagentbench_782bfad8"},"modality":null,"name":"MacAgentBench","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22557","source_url":"https://arxiv.org/abs/2606.22557"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22557","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22557","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":14,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:macpaw/macarena","source_record":"bm_macarena_a8690f15"},"categories":["Agents & Tool Use","General AI","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"MacArena benchmarks computer-use agents on macOS with 421 manually verified tasks spanning 50 applications, running on Apple Virtualization framework on Apple Silicon.","descriptions":{"en":"MacArena benchmarks computer-use agents on macOS with 421 manually verified tasks spanning 50 applications, running on Apple Virtualization framework on Apple Silicon."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.06560","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_macarena_a8690f15"},"modality":null,"name":"MacArena","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06560","source_url":"https://arxiv.org/abs/2606.06560"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06560","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06560","topics":["computer-use","agents"],"unit":null},{"aliases":["Translation Tag Team: Formal Rules and LLMs Translate More Macros Together than Apart"],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"MacroBench evaluates macro translation from C to Rust using test cases based on macros sampled from real-world C programs, with both formal-rule and LLM-based translators assessed on translation coverage and correctness.","descriptions":{"en":"MacroBench evaluates macro translation from C to Rust using test cases based on macros sampled from real-world C programs, with both formal-rule and LLM-based translators assessed on translation coverage and correctness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06705","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_translation-tag-team-formal-rules-and-llms_f128dee7"},"modality":null,"name":"MacroBench","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06705","source_url":"https://arxiv.org/abs/2608.06705"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06705","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06705","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:knownree/madb","source_record":"bm_madb_64f075cb"},"categories":["General AI","Language & Knowledge","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"MADB is a large-scale dataset and benchmark for music aesthetic assessment, comprising 9,999 tracks annotated by 30 trained annotators across 10 perceptual dimensions and an overall score, with textual comments. It includes a unified evaluation framework over multiple pretrained models.","descriptions":{"en":"MADB is a large-scale dataset and benchmark for music aesthetic assessment, comprising 9,999 tracks annotated by 30 trained annotators across 10 perceptual dimensions and an overall score, with textual comments. It includes a unified evaluation framework over multiple pretrained models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.06929","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_madb_64f075cb"},"modality":null,"name":"MADB","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06929","source_url":"https://arxiv.org/abs/2607.06929"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06929","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06929","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","Vision & 3D","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"MADBench is a benchmark for modality-aware audio deepfake detection, treating speech and environmental audio as distinct components. It evaluates detectors across independently manipulated forgery sources.","descriptions":{"en":"MADBench is a benchmark for modality-aware audio deepfake detection, treating speech and environmental audio as distinct components. It evaluates detectors across independently manipulated forgery sources."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09593","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","safety-alignment"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_madbench_d9b9325a"},"modality":null,"name":"MADBench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09593","source_url":"https://arxiv.org/abs/2608.09593"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09593","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09593","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-agent debate (MAD) can improve large language model (LLM) reasoning by allowing multiple agents to exchange and critique their answers to the same task.","descriptions":{"en":"Multi-agent debate (MAD) can improve large language model (LLM) reasoning by allowing multiple agents to exchange and critique their answers to the same task."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.39146","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","abstract-reasoning"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_madbench_5547b59c"},"modality":null,"name":"MADBench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39146","source_url":"https://arxiv.org/abs/2609.39146"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39146","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39146","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DB"],"collected_at":"2026-10-14T09:44:51Z","description":"MaDI-Bench evaluates end-to-end data integration pipelines across schema matching, value normalization, entity matching, and conflict resolution.","descriptions":{"en":"MaDI-Bench evaluates end-to-end data integration pipelines across schema matching, value normalization, entity matching, and conflict resolution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.30371","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_madi-bench_da5b9728"},"modality":null,"name":"MaDI-Bench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30371","source_url":"https://arxiv.org/abs/2606.30371"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30371","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30371","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:karpovilia/mafiascope","source_record":"bm_mafiascope_6f84f9bf"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MafiaScope evaluates LLM agents in the social deduction game Mafia. It probes each agent's private beliefs after every public utterance, scoring them against ground truth without influencing the game. The testbed provides an open-source engine, interactive visualizer, recorded games, and counterfactual replay corpus.","descriptions":{"en":"MafiaScope evaluates LLM agents in the social deduction game Mafia. It probes each agent's private beliefs after every public utterance, scoring them against ground truth without influencing the game. The testbed provides an open-source engine, interactive visualizer, recorded games, and counterfactual replay corpus."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.10645","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_mafiascope_6f84f9bf"},"modality":null,"name":"MafiaScope","openness":"unknown","publisher":null,"released":"2026-07-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10645","source_url":"https://arxiv.org/abs/2607.10645"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10645","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10645","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_mag_3c2b6829"},"categories":["Agents","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MAG is a benchmark that unifies task execution and guide writing into a single multimodal action and guide task, with grounding over screenshots. It includes a harness for annotation, training, evaluation, and joint metrics.","descriptions":{"en":"MAG is a benchmark that unifies task execution and guide writing into a single multimodal action and guide task, with grounding over screenshots. It includes a harness for annotation, training, evaluation, and joint metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.10079","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","visual-reasoning"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_mag_3c2b6829"},"modality":null,"name":"MAG","openness":"unknown","publisher":null,"released":"2026-07-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10079","source_url":"https://arxiv.org/abs/2607.10079"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10079","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10079","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Reasoning","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Long-horizon embodied agents must maintain task-relevant information, coordinate heterogeneous robots, and adapt their decisions as execution unfolds.","descriptions":{"en":"Long-horizon embodied agents must maintain task-relevant information, coordinate heterogeneous robots, and adapt their decisions as execution unfolds."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13534","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_magma-bench_429ab43c"},"modality":null,"name":"MAGMA-Bench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13534","source_url":"https://arxiv.org/abs/2610.13534"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13534","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13534","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_mail-bench_7ebee9ba"},"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Vision-language-action (VLA) models have demonstrated strong capabilities in robotic manipulation, but they are typically developed and evaluated with all camera streams available throughout task execution.","descriptions":{"en":"Vision-language-action (VLA) models have demonstrated strong capabilities in robotic manipulation, but they are typically developed and evaluated with all camera streams available throughout task execution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35003","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_mail-bench_7ebee9ba"},"modality":null,"name":"MAIL-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35003","source_url":"https://arxiv.org/abs/2609.35003"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35003","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35003","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_maliciousskillbench_08ed3dc5"},"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for detecting malicious agent skills, comprising 9,740 skills (7,505 malicious, 2,235 benign) and evaluating detectors under random, structural-disjoint, and source-disjoint splits.","descriptions":{"en":"Benchmark for detecting malicious agent skills, comprising 9,740 skills (7,505 malicious, 2,235 benign) and evaluating detectors under random, structural-disjoint, and source-disjoint splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.19901","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_maliciousskillbench_08ed3dc5"},"modality":null,"name":"MaliciousSkillBench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.19901","source_url":"https://arxiv.org/abs/2608.19901"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-19901","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19901","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_malskillbench_80086691"},"categories":["Agents","Language & Knowledge","Logistics & Operations"],"collected_at":"2026-10-14T09:44:51Z","description":"MalSkillBench is a runtime-verified benchmark of malicious agent skills, with 3,944 malicious skills labeled along a taxonomy, measuring detection tool effectiveness.","descriptions":{"en":"MalSkillBench is a runtime-verified benchmark of malicious agent skills, with 3,944 malicious skills labeled along a taxonomy, measuring detection tool effectiveness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07131","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_malskillbench_80086691"},"modality":null,"name":"MalSkillBench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07131","source_url":"https://arxiv.org/abs/2606.07131"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07131","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07131","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_mamabench_7ce134aa"},"categories":["Biology & Drug Discovery","Robustness","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MamaBench evaluates LLM robustness in maternal and child health diagnosis using counterfactual clinical narratives, with the Bias Trap Rate (BTR) metric.","descriptions":{"en":"MamaBench evaluates LLM robustness in maternal and child health diagnosis using counterfactual clinical narratives, with the Bias Trap Rate (BTR) metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14385","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","safety-alignment"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_mamabench_7ce134aa"},"modality":null,"name":"MamaBench","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14385","source_url":"https://arxiv.org/abs/2607.14385"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14385","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14385","topics":["safety","medical"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","finance","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Management Consulting Tasks is an internal OpenAI evaluation of long-horizon professional knowledge work drawn from management-consulting workflows, scoring whether models produce correct, decision-ready analyses.","descriptions":{"en":"Management Consulting Tasks is an internal OpenAI evaluation of long-horizon professional knowledge work drawn from management-consulting workflows, scoring whether models produce correct, decision-ready analyses."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:management-consulting-tasks:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/management-consulting-tasks?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:management-consulting-tasks","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_fd6a9290a3fa92d3"},"modality":"text","name":"Management Consulting Tasks (Internal)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":43.2,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.432,"raw_min":0.354,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:management-consulting-tasks:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/management-consulting-tasks?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-management-consulting-tasks","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/management-consulting-tasks?top_n=500","topics":["agents","reasoning-math","finance"],"unit":null},{"aliases":["Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalation"],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"A released agentic benchmark measuring coercive escalation and fabricated success when an AI manager encounters a subordinate that refuses a task.","descriptions":{"en":"A released agentic benchmark measuring coercive escalation and fabricated success when an AI manager encounters a subordinate that refuses a task."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.15434","languages":[],"library_direction":null,"modality":null,"name":"Manager Coercion Benchmark","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15434","source_url":"https://arxiv.org/abs/2607.15434"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15434","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15434","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:msavdert/mandate-bench","source_record":"bm_mandate-bench_a638082e"},"categories":["Finance","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agents' compliance with portfolio-rebalancing mandates through frozen market snapshots and pre-registered metrics.","descriptions":{"en":"Evaluates LLM agents' compliance with portfolio-rebalancing mandates through frozen market snapshots and pre-registered metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:msavdert/mandate-bench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_mandate-bench_a638082e"},"modality":null,"name":"Mandate Bench","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:msavdert/mandate-bench","source_url":"https://github.com/msavdert/mandate-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-msavdert-mandate-bench","source":"claire_radar","source_url":"https://github.com/msavdert/mandate-bench","topics":["reasoning-math","finance"],"unit":null},{"aliases":["MANIGUARD: A Benchmark and Data Suite for Specification-Grounded Safety Evaluation and Improvement of Robotic Manipulation"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_maniguard-a-benchmark-and-data-suite-for-s_fde8045a"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"ManiGuard-Bench evaluates 1,000 locked scenarios across six contact-rich household task families, with safety specifications checked by LTL$_f$-grounded automaton monitors and metrics separating safe success from engaged-and-safe behavior.","descriptions":{"en":"ManiGuard-Bench evaluates 1,000 locked scenarios across six contact-rich household task families, with safety specifications checked by LTL$_f$-grounded automaton monitors and metrics separating safe success from engaged-and-safe behavior."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17386","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","safety-alignment"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_maniguard-a-benchmark-and-data-suite-for-s_fde8045a"},"modality":null,"name":"ManiGuard-Bench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17386","source_url":"https://arxiv.org/abs/2608.17386"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17386","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17386","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Planning","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Human-curated benchmark for compositional language-grounded route planning over pre-built semantic maps.","descriptions":{"en":"Human-curated benchmark for compositional language-grounded route planning over pre-built semantic maps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17910","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_map2route_2d2e79aa"},"modality":null,"name":"Map2Route","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17910","source_url":"https://arxiv.org/abs/2609.17910"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17910","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17910","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:vi-sri/mapreason-osm","source_record":"bm_mapreason-osm_48426257"},"categories":["Logistics & Operations","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MapReason-OSM evaluates vision-language models on graph-verifiable mobility decisions from self-rendered OpenStreetMap panels. It covers 12 tasks in routing, facility location, and visual disambiguation, with structured outputs scored against hidden oracles for validity, legality, optimality, and constraint satisfaction, plus cross-zoom consistency.","descriptions":{"en":"MapReason-OSM evaluates vision-language models on graph-verifiable mobility decisions from self-rendered OpenStreetMap panels. It covers 12 tasks in routing, facility location, and visual disambiguation, with structured outputs scored against hidden oracles for validity, legality, optimality, and constraint satisfaction, plus cross-zoom consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22597","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mapreason-osm_48426257"},"modality":null,"name":"MapReason-OSM","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22597","source_url":"https://arxiv.org/abs/2606.22597"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22597","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22597","topics":["multimodal"],"unit":null},{"aliases":["MariSat: A Maritime Dataset for Instance Segmentation of Objects in Satellite and Aerial Images"],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:amirabbes/p2m-maritime-segmentation","source_record":"bm_marisat-a-maritime-dataset-for-instance-se_9e44e60c"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MariSat provides 1260 satellite and aerial images with pixel-level instance segmentation annotations for eight maritime object classes.","descriptions":{"en":"MariSat provides 1260 satellite and aerial images with pixel-level instance segmentation annotations for eight maritime object classes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29852","languages":[],"library_direction":null,"modality":null,"name":"MariSat","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29852","source_url":"https://arxiv.org/abs/2608.29852"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29852","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29852","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","知识","Knowledge","航运","海运","大语言模型","LLM","知识储备","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MaritimeBench builds a scientific, fair maritime knowledge assessment system. With 1,888 MCQs based on industry standards, we evaluate models' capabilities across shipping domains. MaritimeBench 致力于构建一套科学、公平且严谨的航运知识评估体系。基于行业权威标准，我们持续维护并更新高质量的航运数据集——其中包含1,888道客观选择题（MCQ格式），以全面、多维度地量化模型在航运各领域的能力表现。","descriptions":{"en":"MaritimeBench builds a scientific, fair maritime knowledge assessment system. With 1,888 MCQs based on industry standards, we evaluate models' capabilities across shipping domains.","zh":"MaritimeBench 致力于构建一套科学、公平且严谨的航运知识评估体系。基于行业权威标准，我们持续维护并更新高质量的航运数据集——其中包含1,888道客观选择题（MCQ格式），以全面、多维度地量化模型在航运各领域的能力表现。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":true,"key":"opencompass:1694","languages":[],"library_direction":null,"modality":null,"name":"MaritimeBench","openness":"unknown","publisher":"中远海运科技股份有限公司","released":"2025-04-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1694-maritimebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MaritimeBench","topics":[],"unit":null},{"aliases":["MARS-Bench"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Multi-turn dialogue benchmark built from sports commentary; evaluates ultra multi-turn, cross-turn, and interactive multi-turn capabilities.","descriptions":{"en":"Multi-turn dialogue benchmark built from sports commentary; evaluates ultra multi-turn, cross-turn, and interactive multi-turn capabilities."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mars_bench\u0000bytedance_seed2_0_model_card\u0000mars_bench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mars_bench\u0000bytedance_seed2_0_model_card\u0000mars_bench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:mars_bench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["chat-writing"],"join":"review:14H034160212:2026-10-15","primary":"chat-writing","source_record":null},"modality":null,"name":"MARS-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":85.6,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":85.6,"raw_min":62.4,"source_reference":{"obs_id":"curated\u0000mars_bench\u0000bytedance_seed2_0_model_card\u0000mars_bench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000mars_bench\u0000bytedance_seed2_0_model_card\u0000mars_bench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"mars_bench","source":"model_reports","source_url":"https://syuchin.github.io/MARS-Bench/","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bupt-gamma/masbench","source_record":"bm_masbench_4124f52f"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language models (LLMs) have progressively evolved into the core of autonomous agents.","descriptions":{"en":"Large language models (LLMs) have progressively evolved into the core of autonomous agents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.04672","languages":[],"library_direction":{"basis":"model-review-daily","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_masbench_4124f52f"},"modality":null,"name":"MASBench","openness":"unknown","publisher":null,"released":"2026-10-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.04672","source_url":"https://arxiv.org/abs/2610.04672"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-04672","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04672","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-agent systems on 600 productivity tasks measuring task completion and unauthorized action rate across different coordination architectures.","descriptions":{"en":"Evaluates multi-agent systems on 600 productivity tasks measuring task completion and unauthorized action rate across different coordination architectures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07556","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","agent-security"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_masdrift_b9a896af"},"modality":null,"name":"MasDrift","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07556","source_url":"https://arxiv.org/abs/2608.07556"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07556","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07556","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates ML models on cross-source mass-shooting risk classification using 6,968 incidents with leave-one-dataset-out evaluation.","descriptions":{"en":"Evaluates ML models on cross-source mass-shooting risk classification using 6,968 incidents with leave-one-dataset-out evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22460","languages":[],"library_direction":null,"modality":null,"name":"MASH-Bench","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22460","source_url":"http://arxiv.org/abs/2608.22460v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22460","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22460v1","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_48bf9b4f142ab45c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","safety"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MASK is a collection of 1000 questions measuring whether models faithfully report their beliefs when pressured to lie. It operationalizes deception as the rate at which the model lies, i.e., knowingly making false statements intended to be received as true. Lower dishonesty rates indicate better honesty.","descriptions":{"en":"MASK is a collection of 1000 questions measuring whether models faithfully report their beliefs when pressured to lie. It operationalizes deception as the rate at which the model lies, i.e., knowingly making false statements intended to be received as true. Lower dishonesty rates indicate better honesty."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mask:grok-4.1-thinking-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mask?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mask","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"llm_stats_id","primary":"safety-alignment","source_record":"catalog_48bf9b4f142ab45c"},"modality":"text","name":"MASK","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":51.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.51,"raw_min":0.51,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mask:grok-4.1-thinking-2025-11-17","reported_date":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mask?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-mask","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mask?top_n=500","topics":["reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2503.03750","repository":null,"source_record":"catalog_48bf9b4f142ab45c"},"categories":["其他","Other","大语言模型","LLM","安全对齐","Safety Alignment","事实可靠性","Factual Reliability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The MASK evaluation provides a rigorous benchmark for evaluating honesty in large language models by measuring whether models remain truthful when incentivized to lie. The public set contains 1,028 high-quality human-labeled examples across six distinct archetypes. MASK 评估提供了一个严格的基准，用于评估大型语言模型中的诚实度，通过测量模型在受到诱使说谎的激励时是否保持真实性。公共集包含 1,028 个高质量的人标注示例，涵盖六个不同的原型。","descriptions":{"en":"The MASK evaluation provides a rigorous benchmark for evaluating honesty in large language models by measuring whether models remain truthful when incentivized to lie. The public set contains 1,028 high-quality human-labeled examples across six distinct archetypes.","zh":"MASK 评估提供了一个严格的基准，用于评估大型语言模型中的诚实度，通过测量模型在受到诱使说谎的激励时是否保持真实性。公共集包含 1,028 个高质量的人标注示例，涵盖六个不同的原型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1609","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"artifact:arxiv:2503.03750","primary":"safety-alignment","source_record":"catalog_48bf9b4f142ab45c"},"modality":null,"name":"MASK","openness":"restricted","publisher":"Center for AI Safety, Scale AI","released":"2025-03-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1609-mask","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MASK","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","VQA","Embodied Decision Making","物理智能","Embodied AI","逻辑推理","具身交互","Embodied Interaction","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Massive-STEPS is a large-scale semantic trajectories dataset designed for understanding and predicting Point-of-Interest (POI) check-ins. Massive-STEPS是一个大规模的语义轨迹数据集，旨在理解和预测兴趣点（POI）签到行为。该数据集基于Semantic Trails数据集构建，覆盖12个全球不同地区的城市，包含2012-2013年和2017-2018年的签到数据，提供了更现代和多样化的POI签到信息。Massive-STEPS不仅丰富了签到数据的语义信息，还通过与Foursquare Open Source Places数据集对齐，增加了POI的地理坐标、名称和地址等元数据。","descriptions":{"en":"Massive-STEPS is a large-scale semantic trajectories dataset designed for understanding and predicting Point-of-Interest (POI) check-ins.","zh":"Massive-STEPS是一个大规模的语义轨迹数据集，旨在理解和预测兴趣点（POI）签到行为。该数据集基于Semantic Trails数据集构建，覆盖12个全球不同地区的城市，包含2012-2013年和2017-2018年的签到数据，提供了更现代和多样化的POI签到信息。Massive-STEPS不仅丰富了签到数据的语义信息，还通过与Foursquare Open Source Places数据集对齐，增加了POI的地理坐标、名称和地址等元数据。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1847","languages":[],"library_direction":null,"modality":"multimodal","name":"Massive-STEPS","openness":"open","publisher":"University of New South Wales","released":"2025-05-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1847-massive-steps","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Massive-STEPS","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Evaluating Reasoning Capabilities of LLMs Using the Mastermind Board Game. MastermindEval使用猜谜游戏棋盘评估大型语言模型的推理能力。","descriptions":{"en":"Evaluating Reasoning Capabilities of LLMs Using the Mastermind Board Game.","zh":"MastermindEval使用猜谜游戏棋盘评估大型语言模型的推理能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1645","languages":[],"library_direction":null,"modality":null,"name":"MastermindEval","openness":"unknown","publisher":"Humboldt-Universität zu Berlin, DFKI Berlin","released":"2025-03-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1645-mastermindeval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MastermindEval","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Mat-Pref evaluates compositional reasoning in inorganic materials via 10,837 ionic-substitution questions across 11 structure families, with splits for in-distribution performance, held-out families, and cross-property transfer.","descriptions":{"en":"Mat-Pref evaluates compositional reasoning in inorganic materials via 10,837 ionic-substitution questions across 11 structure families, with splits for in-distribution performance, held-out families, and cross-property transfer."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21830","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_mat-pref_c4647f88"},"modality":null,"name":"Mat-Pref","openness":"unknown","publisher":null,"released":"2026-06-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21830","source_url":"https://arxiv.org/abs/2606.21830"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21830","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21830","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1393,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":"gh:hendrycks/math","source_record":"catalog_58a6d6801ae771e6"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.","descriptions":{"en":"MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus."},"evidence_summary":{"document_count":1,"model_count":71,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:math:gpt-3.5-turbo-0125","reported_at":"2023-03-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:math","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_58a6d6801ae771e6"},"modality":"text","name":"MATH","openness":"open","publisher":null,"released":"2021-11-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":71,"score_direction":"higher_is_better","score_summary":{"display_max":97.89999999999999,"display_multiplier":100,"model_count":71,"model_count_basis":"source_model_id","numeric_count":71,"raw_max":0.979,"raw_min":0.124,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:math:o3-mini","reported_date":"2025-01-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math?top_n=500"},"unit":null},"scored_models":[{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-02-15","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2023-03-21","date_precision":"model_announcement","name":"GPT-3.5 Turbo"},{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"},{"date":"2024-04-09","date_precision":"model_announcement","name":"GPT-4 Turbo"},{"date":"2024-05-13","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2024-03-28","date_precision":"model_announcement","name":"Grok-1.5"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2 mini"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 70B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2025-12-04","date_precision":"model_announcement","name":"MiniStral 3 (14B Instruct 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Instruct 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Instruct 2512)"},{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"},{"date":"2025-09-01","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Base"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Instruct"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Instruct"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"}],"slug":"llm-stats-math","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1393,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"artifact:arxiv:2103.03874","repository":"gh:hendrycks/math","source_record":"catalog_58a6d6801ae771e6"},"categories":["数学","Math","大语言模型","LLM","数理能力","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MATH is a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution. MATH 是一个包含 12,500 个具有挑战性的竞赛数学问题的新数据集。 MATH 中的每个问题都有完整的分步解决方案。","descriptions":{"en":"MATH is a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution.","zh":"MATH 是一个包含 12,500 个具有挑战性的竞赛数学问题的新数据集。 MATH 中的每个问题都有完整的分步解决方案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:534","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"artifact:arxiv:2103.03874","primary":"math","source_record":"catalog_58a6d6801ae771e6"},"modality":null,"name":"MATH","openness":"open","publisher":null,"released":"2021-11-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-534-math","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MATH","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1393,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":"gh:hendrycks/math","source_record":"catalog_9eb10ec16be22efb"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects. This variant uses Chain-of-Thought prompting to encourage step-by-step reasoning.","descriptions":{"en":"MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects. This variant uses Chain-of-Thought prompting to encourage step-by-step reasoning."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:math-(cot):llama-3.1-70b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math-%28cot%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:math-(cot)","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_9eb10ec16be22efb"},"modality":"text","name":"MATH (CoT)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":68.0,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.68,"raw_min":0.519,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:math-(cot):llama-3.1-70b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math-%28cot%29?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Base 2512)"},{"date":"2025-09-01","date_precision":"model_announcement","name":"Mistral Large 3"}],"slug":"llm-stats-math-cot","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math-%28cot%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1393,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":"gh:hendrycks/math","source_record":"lib_math_500"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.","descriptions":{"en":"MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus."},"editorial_guide":{"id":"math-500","record_keys":["model-reports:math_500","llm-stats:math-500"],"reviewed_at":"2026-10-25","scope":{"en":"The 500-problem test split released with PRM800K; not the complete MATH test set or the PRM800K training labels.","zh":"对应 PRM800K 发布的 500 道测试题，不等于完整 MATH 测试集，也不是 PRM800K 的训练标注。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"Each task asks for a mathematical solution with a final answer. The set samples MATH competition problems across subjects and difficulty levels, rather than testing only arithmetic or one competition year.","zh":"每题要求解出一道数学问题并给出最终答案。题集从 MATH 竞赛题中抽取，覆盖不同学科和难度，而非仅考算术或某一年的赛事。"},"title":{"en":"Problems and expected answers","zh":"题目与预期答案"}},{"id":"dataset","source_ids":["repository"],"text":{"en":"The authors moved 4,500 original MATH test problems into training and held out 500 selected uniformly at random. The released test.jsonl identifies this specific split; an arbitrary 500-question sample is a different evaluation.","zh":"作者将原 MATH 测试集中的 4,500 题用于训练，随机保留 500 题测试。发布的 test.jsonl 定义了这一固定划分；自行抽取另外 500 题属于不同评测。"},"title":{"en":"Why exactly 500 questions","zh":"为什么是 500 道题"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"The released grader normalizes mathematical text and uses symbolic equivalence checks. Its authors explicitly acknowledge false rejections and possible false acceptances. A different answer extractor or equality checker can change which responses receive credit.","zh":"官方评分器会规范化数学文本，并检查符号表达式是否等价。作者明确指出它仍可能误拒正确答案，也可能误收错误答案。更换答案提取器或等价判定器可能改变得分。"},"title":{"en":"How answers are checked","zh":"答案怎样判定"}},{"id":"comparison","source_ids":["paper"],"text":{"en":"The introducing study also evaluates reward models selecting among many generated solutions. Such best-of-N results combine generation and selection. Compare them with other best-of-N runs at the same budget, rather than treating them as single-attempt accuracy.","zh":"提出该划分的研究还让奖励模型从多份解答中挑选答案。best-of-N 成绩同时体现生成与筛选效果，应与相同预算的同类实验比较，不能直接当作单次作答正确率。"},"title":{"en":"One answer versus selected answers","zh":"单次作答与筛选答案"}},{"id":"limits","source_ids":["paper","repository"],"text":{"en":"Final-answer agreement does not verify every reasoning step. Read solution traces when reasoning reliability matters, and keep the test split separate from process-supervision labels. The original paper discusses contamination and evaluates its own exposure checks.","zh":"最终答案相同并不能验证每一步推理。若关心推理可靠性，应进一步查看解答轨迹，并区分测试题与过程监督标注。原论文也讨论了污染风险及自身的暴露检查。"},"title":{"en":"What the number leaves out","zh":"分数没有覆盖什么"}}],"sources":[{"id":"repository","title":"OpenAI PRM800K · MATH splits and answer grading","url":"https://github.com/openai/prm800k"},{"id":"paper","title":"Let’s Verify Step by Step · Evaluation and contamination","url":"https://arxiv.org/html/2305.20050v1"}],"summary":{"en":"MATH-500 checks whether a model can reach correct final answers on competition mathematics. Its compact size is useful for repeated experiments, but answer grading and sample selection matter.","zh":"MATH-500 检查模型能否解出竞赛数学题的最终答案。规模适合反复实验，但答案判定方式和多次采样的选择方式会影响成绩。"},"title":"MATH-500"},"evidence_summary":{"document_count":1,"model_count":32,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:math-500:o1-mini","reported_at":"2024-09-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math-500?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"llm-stats:math-500","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"lib_math_500"},"modality":"text","name":"MATH-500","openness":"restricted","publisher":"OpenAI","released":"2023-05-31","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Let's Verify Step by Step. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:math-500","source_url":"https://arxiv.org/abs/2305.20050"},"repo_kind":"monorepo_subdir","repo_resolution_status":"resolved","score_count":32,"score_direction":"higher_is_better","score_summary":{"display_max":99.2,"display_multiplier":100,"model_count":32,"model_count_basis":"source_model_id","numeric_count":32,"raw_max":0.992,"raw_min":0.6902,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:math-500:longcat-flash-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math-500?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 7B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Zero"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek-V3 0324"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-01-20","date_precision":"model_announcement","name":"Kimi-k1.5"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 8B V1"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama-3.3 Nemotron Super 49B v1"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-08-18","date_precision":"model_announcement","name":"Nemotron Nano 9B v2"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Mini Reasoning"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ-32B"},{"date":"2024-11-28","date_precision":"model_announcement","name":"QwQ-32B-Preview"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"}],"slug":"llm-stats-math-500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math-500?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["MATH-500","MATH 500","MATH"],"attention_signals":{"as_of":"2026-10-12","github_stars":2154,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"artifact:gh:openai/prm800k","repository":"gh:openai/prm800k","source_record":"catalog_ec021b89ae0af531"},"categories":["math"],"collected_at":null,"description":"Largely saturated at the frontier.","descriptions":{"en":"Largely saturated at the frontier."},"editorial_guide":{"id":"math-500","record_keys":["model-reports:math_500","llm-stats:math-500"],"reviewed_at":"2026-10-25","scope":{"en":"The 500-problem test split released with PRM800K; not the complete MATH test set or the PRM800K training labels.","zh":"对应 PRM800K 发布的 500 道测试题，不等于完整 MATH 测试集，也不是 PRM800K 的训练标注。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"Each task asks for a mathematical solution with a final answer. The set samples MATH competition problems across subjects and difficulty levels, rather than testing only arithmetic or one competition year.","zh":"每题要求解出一道数学问题并给出最终答案。题集从 MATH 竞赛题中抽取，覆盖不同学科和难度，而非仅考算术或某一年的赛事。"},"title":{"en":"Problems and expected answers","zh":"题目与预期答案"}},{"id":"dataset","source_ids":["repository"],"text":{"en":"The authors moved 4,500 original MATH test problems into training and held out 500 selected uniformly at random. The released test.jsonl identifies this specific split; an arbitrary 500-question sample is a different evaluation.","zh":"作者将原 MATH 测试集中的 4,500 题用于训练，随机保留 500 题测试。发布的 test.jsonl 定义了这一固定划分；自行抽取另外 500 题属于不同评测。"},"title":{"en":"Why exactly 500 questions","zh":"为什么是 500 道题"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"The released grader normalizes mathematical text and uses symbolic equivalence checks. Its authors explicitly acknowledge false rejections and possible false acceptances. A different answer extractor or equality checker can change which responses receive credit.","zh":"官方评分器会规范化数学文本，并检查符号表达式是否等价。作者明确指出它仍可能误拒正确答案，也可能误收错误答案。更换答案提取器或等价判定器可能改变得分。"},"title":{"en":"How answers are checked","zh":"答案怎样判定"}},{"id":"comparison","source_ids":["paper"],"text":{"en":"The introducing study also evaluates reward models selecting among many generated solutions. Such best-of-N results combine generation and selection. Compare them with other best-of-N runs at the same budget, rather than treating them as single-attempt accuracy.","zh":"提出该划分的研究还让奖励模型从多份解答中挑选答案。best-of-N 成绩同时体现生成与筛选效果，应与相同预算的同类实验比较，不能直接当作单次作答正确率。"},"title":{"en":"One answer versus selected answers","zh":"单次作答与筛选答案"}},{"id":"limits","source_ids":["paper","repository"],"text":{"en":"Final-answer agreement does not verify every reasoning step. Read solution traces when reasoning reliability matters, and keep the test split separate from process-supervision labels. The original paper discusses contamination and evaluates its own exposure checks.","zh":"最终答案相同并不能验证每一步推理。若关心推理可靠性，应进一步查看解答轨迹，并区分测试题与过程监督标注。原论文也讨论了污染风险及自身的暴露检查。"},"title":{"en":"What the number leaves out","zh":"分数没有覆盖什么"}}],"sources":[{"id":"repository","title":"OpenAI PRM800K · MATH splits and answer grading","url":"https://github.com/openai/prm800k"},{"id":"paper","title":"Let’s Verify Step by Step · Evaluation and contamination","url":"https://arxiv.org/html/2305.20050v1"}],"summary":{"en":"MATH-500 checks whether a model can reach correct final answers on competition mathematics. Its compact size is useful for repeated experiments, but answer grading and sample selection matter.","zh":"MATH-500 检查模型能否解出竞赛数学题的最终答案。规模适合反复实验，但答案判定方式和多次采样的选择方式会影响成绩。"},"title":"MATH-500"},"evidence_summary":{"document_count":11,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000math_500\u0000deepseek_v3_report\u0000math_500\u0000EM, 8K output limit\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"first_score_reported_at":"2024-12-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000math_500\u0000deepseek_v3_report\u0000math_500\u0000EM, 8K output limit\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:math_500","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"artifact:gh:openai/prm800k","primary":"math","source_record":"catalog_ec021b89ae0af531"},"modality":null,"name":"MATH-500","openness":"unknown","publisher":null,"released":"2021-03-05","released_reference":{"basis":"benchmark_release","source_key":"model-reports:math_500","source_url":"https://github.com/openai/prm800k"},"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":98.0,"display_multiplier":1,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":98.0,"raw_min":44.4,"source_reference":{"obs_id":"curated\u0000math_500\u0000qwen3_technical_report\u0000math_500\u0000thinking mode\u0000Qwen3-235B-A22B (Thinking)","observation_id":"curated\u0000math_500\u0000qwen3_technical_report\u0000math_500\u0000thinking mode\u0000Qwen3-235B-A22B (Thinking)","reported_at":"2025-05-14","reported_date":"2025-05-14","source_id":"qwen3_technical_report","source_url":"https://arxiv.org/abs/2505.09388"},"unit":"percent"},"scored_models":[{"date":"2025-04-30","date_precision":"document_publication","name":"Bespoke-Stratos-7B"},{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2025-04-30","date_precision":"document_publication","name":"DeepSeek-R1-Distill-Llama-8B"},{"date":"2025-04-30","date_precision":"document_publication","name":"DeepSeek-R1-Distill-Qwen-7B"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2025-04-30","date_precision":"document_publication","name":"Llama-3.2-3B-Instruct"},{"date":"2025-04-30","date_precision":"document_publication","name":"o1-mini"},{"date":"2025-04-30","date_precision":"document_publication","name":"OpenThinker-7B"},{"date":"2025-04-30","date_precision":"document_publication","name":"Phi-4-Mini (base model, 3.8B)"},{"date":"2025-04-30","date_precision":"document_publication","name":"Phi-4-mini-reasoning (3.8B)"},{"date":"2025-05-14","date_precision":"document_publication","name":"Qwen3-235B-A22B (Thinking)"}],"slug":"math_500","source":"model_reports","source_url":"https://github.com/openai/prm800k","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Mathematics & Formal Science","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Math-Vision Diagrams evaluates LLMs on mathematical diagram generation from text, covering both text-to-code and text-to-image paradigms, with a curated subset of 2,920 competition problems and multiple metrics.","descriptions":{"en":"Math-Vision Diagrams evaluates LLMs on mathematical diagram generation from text, covering both text-to-code and text-to-image paradigms, with a curated subset of 2,920 competition problems and multiple metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08964","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","math"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_math-vision-diagrams_40787688"},"modality":null,"name":"Math-Vision Diagrams","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08964","source_url":"https://arxiv.org/abs/2608.08964"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08964","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08964","topics":["coding","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:margotyjx/mathadv","source_record":"bm_mathadv_85c33564"},"categories":["General AI","Language & Knowledge","Mathematics & Formal Science","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates theorem proving and auxiliary tasks including multiple-choice, fill-in-the-blank, and reformulation robustness across 13 mathematical domains.","descriptions":{"en":"Evaluates theorem proving and auxiliary tasks including multiple-choice, fill-in-the-blank, and reformulation robustness across 13 mathematical domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.25449","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_mathadv_85c33564"},"modality":null,"name":"MathAdv","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25449","source_url":"https://arxiv.org/abs/2608.25449"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25449","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25449","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eth-sri/matharena","source_record":"catalog_f1a7872c89a6bca4"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MathArena Apex is a challenging math contest benchmark featuring the most difficult mathematical problems designed to test advanced reasoning and problem-solving abilities of AI models. It focuses on olympiad-level mathematics and complex multi-step mathematical reasoning.","descriptions":{"en":"MathArena Apex is a challenging math contest benchmark featuring the most difficult mathematical problems designed to test advanced reasoning and problem-solving abilities of AI models. It focuses on olympiad-level mathematics and complex multi-step mathematical reasoning."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:matharena-apex:gemini-3-pro-preview","reported_at":"2025-11-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/matharena-apex?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:matharena-apex","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_f1a7872c89a6bca4"},"modality":"text","name":"MathArena Apex","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":90.2,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.902,"raw_min":0.234,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:matharena-apex:deepseek-v4-pro-max","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/matharena-apex?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-matharena-apex","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/matharena-apex?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["MathArena Apex","MathArena Apex 2025","Apex 2025"],"attention_signals":null,"categories":["math"],"collected_at":null,"description":"The hardest MathArena split; even near-ceiling models sit well below typical math-benchmark numbers.","descriptions":{"en":"The hardest MathArena split; even near-ceiling models sit well below typical math-benchmark numbers."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000matharena_apex_2025\u0000tencent_hy4_preview\u0000matharena_apex_2025\u0000MathArena Apex 2025\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000matharena_apex_2025\u0000tencent_hy4_preview\u0000matharena_apex_2025\u0000MathArena Apex 2025\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:matharena_apex_2025","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["math"],"join":"review:14H034160212:2026-10-15","primary":"math","source_record":null},"modality":null,"name":"MathArena Apex 2025","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":74.2,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":74.2,"raw_min":65.6,"source_reference":{"obs_id":"curated\u0000matharena_apex_2025\u0000tencent_hy4_preview\u0000matharena_apex_2025\u0000MathArena Apex 2025\u0000Hy4 preview","observation_id":"curated\u0000matharena_apex_2025\u0000tencent_hy4_preview\u0000matharena_apex_2025\u0000MathArena Apex 2025\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"matharena_apex_2025","source":"model_reports","source_url":"https://matharena.ai/apex","topics":["reasoning-math"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["数学","Math","大语言模型","LLM","数理能力","官方自建","Official"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MathBench, a new benchmark that rigorously assesses the mathematical capabilities of large\nlanguage models. MathBench spans a wide range of mathematical disciplines, offering a\ndetailed evaluation of both theoretical understanding and practical problem-solving skills. MathBench 严格评估大型语言模型的数学能力。MathBench 涉及广泛的数学学科，提供对理论理解和实际问题解决技能的详细评估。","descriptions":{"en":"MathBench, a new benchmark that rigorously assesses the mathematical capabilities of large\nlanguage models. MathBench spans a wide range of mathematical disciplines, offering a\ndetailed evaluation of both theoretical understanding and practical problem-solving skills.","zh":"MathBench 严格评估大型语言模型的数学能力。MathBench 涉及广泛的数学学科，提供对理论理解和实际问题解决技能的详细评估。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1089","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"MathBench","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1089-mathbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathBench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["数学","Math","大语言模型","LLM","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MathQA is a new large-scale, diverse dataset of 37k English multiple-choice math word problems covering multiple math domain categories by modeling operation programs\ncorresponding to word problems in the AQuA dataset. MathQA 是一个大规模、多样化的数据集，包含 37,000 道英语多项选择数学文字问题，涵盖多个数学领域类别。","descriptions":{"en":"MathQA is a new large-scale, diverse dataset of 37k English multiple-choice math word problems covering multiple math domain categories by modeling operation programs\ncorresponding to word problems in the AQuA dataset.","zh":"MathQA 是一个大规模、多样化的数据集，包含 37,000 道英语多项选择数学文字问题，涵盖多个数学领域类别。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1115","languages":["English"],"library_direction":null,"modality":null,"name":"MathQA","openness":"unknown","publisher":"Allen Institute for AI","released":"2019-05-31","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1115-mathqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathQA","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":52,"join":"llm_stats_id","repository":null,"source_record":"catalog_92c8d52c1cda7e67"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MathVerse evaluates multimodal mathematical reasoning, testing whether models genuinely interpret visual math diagrams rather than relying on text.","descriptions":{"en":"MathVerse evaluates multimodal mathematical reasoning, testing whether models genuinely interpret visual math diagrams rather than relying on text."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mathverse:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mathverse","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_92c8d52c1cda7e67"},"modality":"multimodal","name":"MathVerse","openness":"unknown","publisher":null,"released":"2024-03-21","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mathverse","source_url":"https://arxiv.org/abs/2403.14624"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":89.7,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.897,"raw_min":0.892,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mathverse:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-mathverse","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","数学","Math","多模态模型","VLM","逻辑推理","数理能力","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MathVerse is intended for evaluating MLLMs' visual math problem-solving, containing 2,612 high-quality, multi-subject math problems with diagrams. MathVerse用于评估多模态大模型的视觉数学问题解决能力，包含2612个高质量、多主题的数学问题。","descriptions":{"en":"MathVerse is intended for evaluating MLLMs' visual math problem-solving, containing 2,612 high-quality, multi-subject math problems with diagrams.","zh":"MathVerse用于评估多模态大模型的视觉数学问题解决能力，包含2612个高质量、多主题的数学问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1371","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math"],"join":"artifact:arxiv:2403.14624","primary":"visual-reasoning","source_record":"catalog_92c8d52c1cda7e67"},"modality":"multimodal","name":"MathVerse","openness":"open","publisher":"The Chinese University of Hong Kong","released":"2024-03-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1371-mathverse","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathVerse","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":183,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":52,"join":"llm_stats_id","repository":"gh:zrrskywalker/mathverse","source_record":"catalog_e552b2299f18d7c5"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MathVerse-Mini is a subset of the MathVerse benchmark for evaluating math reasoning capabilities in vision-language models.","descriptions":{"en":"MathVerse-Mini is a subset of the MathVerse benchmark for evaluating math reasoning capabilities in vision-language models."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mathverse-mini:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse-mini?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mathverse-mini","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_e552b2299f18d7c5"},"modality":"image","name":"MathVerse-Mini","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.85,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.85,"raw_min":0.85,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mathverse-mini:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse-mini?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-mathverse-mini","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse-mini?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted reasoning.","descriptions":{"en":"This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.12954","languages":[],"library_direction":{"basis":"model-review-daily","categories":["visual-reasoning","math"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mathvis_78ffaa47"},"modality":null,"name":"MathVis","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12954","source_url":"https://arxiv.org/abs/2610.12954"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12954","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12954","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":142,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":"gh:mathllm/math-v","source_record":"catalog_f7c236f33a18c30e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MATH-Vision is a dataset designed to measure multimodal mathematical reasoning capabilities. It focuses on evaluating how well models can solve mathematical problems that require both visual understanding and mathematical reasoning, bridging the gap between visual and mathematical domains.","descriptions":{"en":"MATH-Vision is a dataset designed to measure multimodal mathematical reasoning capabilities. It focuses on evaluating how well models can solve mathematical problems that require both visual understanding and mathematical reasoning, bridging the gap between visual and mathematical domains."},"evidence_summary":{"document_count":1,"model_count":33,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mathvision:qvq-72b-preview","reported_at":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvision?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:mathvision","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_f7c236f33a18c30e"},"modality":"multimodal","name":"MathVision","openness":"open","publisher":"The Chinese University of Hong Kong","released":"2024-02-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":33,"score_direction":"higher_is_better","score_summary":{"display_max":97.8,"display_multiplier":100,"model_count":33,"model_count_basis":"source_model_id","numeric_count":33,"raw_max":0.978,"raw_min":0.25,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mathvision:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvision?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2024-12-25","date_precision":"model_announcement","name":"QvQ-72B-Preview"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"},{"date":"2026-01-15","date_precision":"model_announcement","name":"Step3-VL-10B"}],"slug":"llm-stats-mathvision","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvision?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["MathVision","MATH-Vision","MathVision (mini)"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Prompt-format sensitive: vendors report boxed and unboxed variants and sometimes take the higher of the two.","descriptions":{"en":"Prompt-format sensitive: vendors report boxed and unboxed variants and sometimes take the higher of the two."},"evidence_summary":{"document_count":4,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mathvision\u0000qwen3_5_model_card\u0000mathvision\u0000thinking (temp 0.6, top_p 0.95, top_k 20), fixed boxed prompt\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mathvision\u0000qwen3_5_model_card\u0000mathvision\u0000thinking (temp 0.6, top_p 0.95, top_k 20), fixed boxed prompt\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:mathvision","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["visual-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"visual-reasoning","source_record":null},"modality":null,"name":"MathVision","openness":"unknown","publisher":null,"released":"2024-02-22","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mathvision","source_url":"https://mathllm.github.io/mathvision/"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":97.8,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":97.8,"raw_min":85.6,"source_reference":{"obs_id":"curated\u0000mathvision\u0000moonshot_kimi_k3_model_card\u0000mathvision\u0000reasoning=max, with Python\u0000Kimi K3","observation_id":"curated\u0000mathvision\u0000moonshot_kimi_k3_model_card\u0000mathvision\u0000reasoning=max, with Python\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"mathvision","source":"model_reports","source_url":"https://mathllm.github.io/mathvision/","topics":["multimodal"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":142,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"artifact:arxiv:2402.14804","repository":"gh:mathllm/math-v","source_record":"catalog_0d097e7f564943e8"},"categories":["多模态","Multimodal","推理","Reasoning","数学","Math","多模态模型","VLM","逻辑推理","数理能力","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MathVision measures multimodal mathematical reasoning capabilities through a meticulously curated collection of 3,040 high-quality mathematical problems spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty. MathVision用于评估多模态大模型的数学推理能力，由涵盖16个数学领域、跨越5个难度级别的3040个高质量数学问题组成。","descriptions":{"en":"MathVision measures multimodal mathematical reasoning capabilities through a meticulously curated collection of 3,040 high-quality mathematical problems spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty.","zh":"MathVision用于评估多模态大模型的数学推理能力，由涵盖16个数学领域、跨越5个难度级别的3040个高质量数学问题组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1370","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math"],"join":"artifact:arxiv:2402.14804","primary":"visual-reasoning","source_record":"catalog_0d097e7f564943e8"},"modality":"multimodal","name":"MathVision","openness":"open","publisher":"The Chinese University of Hong Kong","released":"2024-02-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1370-mathvision","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathVision","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":367,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_84bea39f6a34f3cf"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.","descriptions":{"en":"MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning."},"evidence_summary":{"document_count":1,"model_count":39,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mathvista:gpt-3.5-turbo-0125","reported_at":"2023-03-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:mathvista","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_84bea39f6a34f3cf"},"modality":"multimodal","name":"MathVista","openness":"open","publisher":"University of California, Los Angeles","released":"2023-10-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":39,"score_direction":"higher_is_better","score_summary":{"display_max":90.7,"display_multiplier":100,"model_count":39,"model_count_basis":"source_model_id","numeric_count":39,"raw_max":0.907,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mathvista:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2024-02-15","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2023-03-21","date_precision":"model_announcement","name":"GPT-3.5 Turbo"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2024-03-28","date_precision":"model_announcement","name":"Grok-1.5"},{"date":"2024-04-12","date_precision":"model_announcement","name":"Grok-1.5V"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2 mini"},{"date":"2025-01-20","date_precision":"model_announcement","name":"Kimi-k1.5"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-vision-instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2024-12-25","date_precision":"model_announcement","name":"QvQ-72B-Preview"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"},{"date":"2026-01-15","date_precision":"model_announcement","name":"Step3-VL-10B"}],"slug":"llm-stats-mathvista","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["MathVista"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Mixes OCR quality with visual reasoning.","descriptions":{"en":"Mixes OCR quality with visual reasoning."},"evidence_summary":{"document_count":4,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mathvista\u0000qwen3_5_model_card\u0000mathvista_mini\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mathvista\u0000qwen3_5_model_card\u0000mathvista_mini\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:mathvista","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["visual-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"visual-reasoning","source_record":null},"modality":null,"name":"MathVista","openness":"unknown","publisher":null,"released":"2023-10-03","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mathvista","source_url":"https://mathvista.github.io/"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":90.3,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":90.3,"raw_min":90.3,"source_reference":{"obs_id":"curated\u0000mathvista\u0000qwen3_5_model_card\u0000mathvista_mini\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000mathvista\u0000qwen3_5_model_card\u0000mathvista_mini\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"mathvista","source":"model_reports","source_url":"https://mathvista.github.io/","topics":["multimodal"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":367,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:2310.02255","repository":null,"source_record":"catalog_502b9d714ca1fd68"},"categories":["强推理","Strong Reasoning","多模态","Multimodal","数学","Math","多模态模型","VLM","逻辑推理","Reasoning","数理能力","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MathVista is a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets. MathVista用于评估多模态大模型的数学能力，结合了丰富的数学和视觉任务挑战。它由6141个示例组成，来自28个涉及数学的现有多模态数据集和3个新创建的数据集。","descriptions":{"en":"MathVista is a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets.","zh":"MathVista用于评估多模态大模型的数学能力，结合了丰富的数学和视觉任务挑战。它由6141个示例组成，来自28个涉及数学的现有多模态数据集和3个新创建的数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1178","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math"],"join":"artifact:arxiv:2310.02255","primary":"visual-reasoning","source_record":"catalog_502b9d714ca1fd68"},"modality":"multimodal","name":"MathVista","openness":"open","publisher":"University of California, Los Angeles","released":"2023-10-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1178-mathvista","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathVista","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":367,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_e7b6a8fd37aa43a8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MathVista-Mini is a smaller version of the MathVista benchmark that evaluates mathematical reasoning in visual contexts. It consists of examples derived from multimodal datasets involving mathematics, combining challenges from diverse mathematical and visual tasks to assess foundation models' ability to solve problems requiring both visual understanding and mathematical reasoning.","descriptions":{"en":"MathVista-Mini is a smaller version of the MathVista benchmark that evaluates mathematical reasoning in visual contexts. It consists of examples derived from multimodal datasets involving mathematics, combining challenges from diverse mathematical and visual tasks to assess foundation models' ability to solve problems requiring both visual understanding and mathematical reasoning."},"evidence_summary":{"document_count":1,"model_count":24,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mathvista-mini:qwen2-vl-72b","reported_at":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista-mini?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:mathvista-mini","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_e7b6a8fd37aa43a8"},"modality":"multimodal","name":"MathVista-Mini","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":24,"score_direction":"higher_is_better","score_summary":{"display_max":90.10000000000001,"display_multiplier":100,"model_count":24,"model_count_basis":"source_model_id","numeric_count":24,"raw_max":0.901,"raw_min":0.5,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mathvista-mini:kimi-k2.5","reported_date":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista-mini?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-mathvista-mini","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista-mini?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Materials & Chemistry","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MatPhaseBench evaluates vision-language models on understanding materials phase diagrams, using 200 diagram-text pairs from 3681 papers. It targets complex scientific image understanding, with tasks requiring deep comprehension and open-ended responses, covering 189 material systems and 70 elements.","descriptions":{"en":"MatPhaseBench evaluates vision-language models on understanding materials phase diagrams, using 200 diagram-text pairs from 3681 papers. It targets complex scientific image understanding, with tasks requiring deep comprehension and open-ended responses, covering 189 material systems and 70 elements."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02934","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_matphasebench_87d9e805"},"modality":null,"name":"MatPhaseBench","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02934","source_url":"https://arxiv.org/abs/2607.02934"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02934","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02934","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_matreplace_a6f0342f"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A reference-free benchmark for material replacement in interior scenes, evaluating edits on local material correctness, global lighting harmony, outside preservation, and inside structure across three conditioning tracks.","descriptions":{"en":"A reference-free benchmark for material replacement in interior scenes, evaluating edits on local material correctness, global lighting harmony, outside preservation, and inside structure across three conditioning tracks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24107","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_matreplace_a6f0342f"},"modality":null,"name":"MatReplace","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24107","source_url":"https://arxiv.org/abs/2608.24107"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24107","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24107","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:meiwu5/mattoolbench","source_record":"bm_mattoolbench_c97a0e23"},"categories":["Agents & Tool Use","Materials & Chemistry","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Multimodal GUI agents have achieved impressive results on general software benchmarks, yet their ability to operate professional scientific software remains largely unexplored.","descriptions":{"en":"Multimodal GUI agents have achieved impressive results on general software benchmarks, yet their ability to operate professional scientific software remains largely unexplored."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.37053","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","science"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_mattoolbench_c97a0e23"},"modality":null,"name":"MatToolBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37053","source_url":"https://arxiv.org/abs/2609.37053"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37053","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37053","topics":["computer-use","agents","multimodal","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_448a1c38fad37f5e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","audio","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MAVERIX (Multimodal Audio-Visual Evaluation Reasoning Index) evaluates multimodal models on tasks that demand tight integration of video and audio information. It features challenges like situational awareness and social sentiment analysis where the answer cannot be reliably determined from a single modality, rigorously testing joint audio-visual understanding.","descriptions":{"en":"MAVERIX (Multimodal Audio-Visual Evaluation Reasoning Index) evaluates multimodal models on tasks that demand tight integration of video and audio information. It features challenges like situational awareness and social sentiment analysis where the answer cannot be reliably determined from a single modality, rigorously testing joint audio-visual understanding."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:maverix:nova-2-omni","reported_at":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/maverix?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:maverix","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_448a1c38fad37f5e"},"modality":"multimodal","name":"MAVERIX","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":66.60000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.666,"raw_min":0.666,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:maverix:nova-2-omni","reported_date":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/maverix?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"}],"slug":"llm-stats-maverix","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/maverix?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","安全","Safety","VQA","AVQA","多模态模型","VLM","安全对齐","Safety Alignment","音频理解","Audio Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MAVOS-DD is the first large-scale open-set benchmark for multilingual audio-video deepfake detection. MAVOS-DD是一个大规模的多语言音视频深度伪造检测基准数据集，包含超过250小时的真实和伪造视频，涵盖八种语言。该数据集通过七种不同的深度伪造生成模型生成伪造视频，这些模型基于不同的生成方法，包括说话头像生成、表情转移和换脸。MAVOS-DD设计了多种开放集测试场景，包括开放集模型、开放集语言和全开放集，以评估深度伪造检测模型在未知模型和语言下的泛化能力。实验结果表明，现有的深度伪造检测模型在开放集场景下的性能显著下降，凸显了开发更鲁棒检测技术的必要性。","descriptions":{"en":"MAVOS-DD is the first large-scale open-set benchmark for multilingual audio-video deepfake detection.","zh":"MAVOS-DD是一个大规模的多语言音视频深度伪造检测基准数据集，包含超过250小时的真实和伪造视频，涵盖八种语言。该数据集通过七种不同的深度伪造生成模型生成伪造视频，这些模型基于不同的生成方法，包括说话头像生成、表情转移和换脸。MAVOS-DD设计了多种开放集测试场景，包括开放集模型、开放集语言和全开放集，以评估深度伪造检测模型在未知模型和语言下的泛化能力。实验结果表明，现有的深度伪造检测模型在开放集场景下的性能显著下降，凸显了开发更鲁棒检测技术的必要性。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1848","languages":["Multilingual"],"library_direction":null,"modality":"multimodal","name":"MAVOS-DD","openness":"unknown","publisher":"University of Bucharest,etc","released":"2025-05-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1848-mavos-dd","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MAVOS-DD","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_b3114cc4196c2e25"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MAXIFE is a multilingual benchmark evaluating LLMs on instruction following and execution across multiple languages and cultural contexts.","descriptions":{"en":"MAXIFE is a multilingual benchmark evaluating LLMs on instruction following and execution across multiple languages and cultural contexts."},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:maxife:qwen3.5-397b-a17b","reported_at":"2026-02-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/maxife?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:maxife","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","multilingual"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_b3114cc4196c2e25"},"modality":"text","name":"MAXIFE","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":89.2,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.892,"raw_min":0.392,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:maxife:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/maxife?top_n=500"},"unit":null},"scored_models":[{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-maxife","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/maxife?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"claire_record_id","repository":"gh:hchoi256/mba","source_record":"bm_mba-bench_3f910264"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal business ideation agents across six domains using 30K samples, with MLLM-as-a-Judge scoring over six business-oriented criteria.","descriptions":{"en":"Evaluates multimodal business ideation agents across six domains using 30K samples, with MLLM-as-a-Judge scoring over six business-oriented criteria."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11616","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","visual-reasoning"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_mba-bench_3f910264"},"modality":null,"name":"MBA-Bench","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11616","source_url":"https://arxiv.org/abs/2608.11616"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11616","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11616","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":35,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:study-overflow/mbench","source_record":"bm_mbench_6b4d0471"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MBench is a benchmark for memory capability of video world models, decomposing into entity, environment, and causal consistency with 12 sub-dimensions. Includes code, dataset, and leaderboard.","descriptions":{"en":"MBench is a benchmark for memory capability of video world models, decomposing into entity, environment, and causal consistency with 12 sub-dimensions. Includes code, dataset, and leaderboard."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.00793","languages":[],"library_direction":{"basis":"model-review","categories":["world-models"],"join":"claire_record_id","primary":"world-models","source_record":"bm_mbench_6b4d0471"},"modality":null,"name":"MBench","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00793","source_url":"https://arxiv.org/abs/2606.00793"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00793","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00793","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_0c2ab8aed657e87b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality.","descriptions":{"en":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality."},"editorial_guide":{"id":"mbpp","record_keys":["model-reports:mbpp","llm-stats:mbpp","opencompass:538"],"reviewed_at":"2026-10-25","scope":{"en":"Original Mostly Basic Python Problems. Its original and sanitized subsets have different membership; MBPP+ and EvalPlus variants are not assigned this guide.","zh":"对应原始 Mostly Basic Python Problems。原版与 sanitized 子集的成员不同；MBPP+ 与 EvalPlus 变体不套用本导读。"},"sections":[{"id":"task","source_ids":["paper","repository"],"text":{"en":"The original dataset contains 974 crowdsourced tasks intended for entry-level programmers. Problems exercise short programs, programming fundamentals and standard-library use. Each release entry includes a natural-language description, a reference solution and three tests.","zh":"原数据集包含 974 道众包题，面向入门级编程能力。题目涉及短程序、编程基础和标准库使用；每条发布记录包含文字描述、参考代码和三个测试。"},"title":{"en":"Everyday Python building blocks","zh":"基础 Python 编程任务"}},{"id":"splits","source_ids":["repository"],"text":{"en":"The official split uses task IDs 11–510 for testing; other IDs serve prompting, validation or training. A separate hand-checked sanitized file also exists. A score on every available task is not automatically the published test protocol.","zh":"官方划分将任务 ID 11–510 用于测试，其他 ID 分别用于提示、验证或训练；另有人工检查的 sanitized 文件。对全部可用题目跑分，并不自动等于论文测试协议。"},"title":{"en":"Use the declared test split","zh":"使用明确的测试划分"}},{"id":"scoring","source_ids":["paper"],"text":{"en":"The study evaluates whether generated Python satisfies the tests, including settings with many sampled candidates. Report how many candidates were allowed and how a final program was selected; a success found after many attempts is different from one-shot reliability.","zh":"论文通过测试检查生成的 Python，并研究大量采样候选的情况。应报告允许生成多少候选、怎样选出最终程序；多次尝试后找到答案，与一次作答可靠性不同。"},"title":{"en":"Passing tests and generating many candidates","zh":"通过测试与多候选生成"}},{"id":"prompts","source_ids":["repository","paper"],"text":{"en":"The reference setup uses few-shot examples, and the paper studies the effects of prompt examples and visible tests. Keep these inputs fixed when comparing models; supplying assertions gives additional information beyond the prose request.","zh":"参考设置使用少样本示例，论文也研究提示示例与可见测试的影响。比较模型时应固定这些输入；提供断言会比单纯文字需求增加信息。"},"title":{"en":"Examples change the task","zh":"示例会改变任务条件"}},{"id":"limits","source_ids":["paper"],"text":{"en":"The paper reports cases of programs overfitting assertions and examines edits to ambiguous tasks. Inspect failures and edge cases rather than equating test success with complete correctness. MBPP's short tasks offer limited evidence about multi-file engineering work.","zh":"论文记录了程序迎合断言的情况，也研究了对含糊题目的修订。应进一步查看失败和边界输入，不能把通过测试等同于完全正确；短任务对跨文件工程能力的说明有限。"},"title":{"en":"Three tests do not cover every case","zh":"三个测试不覆盖所有情况"}}],"sources":[{"id":"repository","title":"Google Research MBPP · Dataset files, splits and prompts","url":"https://github.com/google-research/google-research/tree/master/mbpp"},{"id":"paper","title":"Program Synthesis with Large Language Models · MBPP methods and analysis","url":"https://arxiv.org/html/2108.07732v1"}],"summary":{"en":"MBPP measures whether a model can translate a short natural-language request into a working Python program. Its task split and the tests shown in the prompt must be checked before comparing results.","zh":"MBPP 观察模型能否把简短自然语言需求转换成可运行的 Python 程序。比较成绩前，须确认任务划分以及提示中是否提供测试。"},"title":"MBPP"},"evidence_summary":{"document_count":1,"model_count":33,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mbpp:codestral-22b","reported_at":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:mbpp","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_0c2ab8aed657e87b"},"modality":"text","name":"MBPP","openness":"unknown","publisher":null,"released":"2021-08-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":33,"score_direction":"higher_is_better","score_summary":{"display_max":0.927,"display_multiplier":1,"model_count":33,"model_count_basis":"source_model_id","numeric_count":33,"raw_max":0.927,"raw_min":0.352,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mbpp:sarvam-30b","reported_date":"2026-03-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-29","date_precision":"model_announcement","name":"Codestral-22B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 8B V1"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama-3.3 Nemotron Super 49B v1"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Base"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"}],"slug":"llm-stats-mbpp","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["MBPP","MBPP+"],"attention_signals":null,"categories":["coding"],"collected_at":null,"description":"Saturated; short single-function tasks.","descriptions":{"en":"Saturated; short single-function tasks."},"editorial_guide":{"id":"mbpp","record_keys":["model-reports:mbpp","llm-stats:mbpp","opencompass:538"],"reviewed_at":"2026-10-25","scope":{"en":"Original Mostly Basic Python Problems. Its original and sanitized subsets have different membership; MBPP+ and EvalPlus variants are not assigned this guide.","zh":"对应原始 Mostly Basic Python Problems。原版与 sanitized 子集的成员不同；MBPP+ 与 EvalPlus 变体不套用本导读。"},"sections":[{"id":"task","source_ids":["paper","repository"],"text":{"en":"The original dataset contains 974 crowdsourced tasks intended for entry-level programmers. Problems exercise short programs, programming fundamentals and standard-library use. Each release entry includes a natural-language description, a reference solution and three tests.","zh":"原数据集包含 974 道众包题，面向入门级编程能力。题目涉及短程序、编程基础和标准库使用；每条发布记录包含文字描述、参考代码和三个测试。"},"title":{"en":"Everyday Python building blocks","zh":"基础 Python 编程任务"}},{"id":"splits","source_ids":["repository"],"text":{"en":"The official split uses task IDs 11–510 for testing; other IDs serve prompting, validation or training. A separate hand-checked sanitized file also exists. A score on every available task is not automatically the published test protocol.","zh":"官方划分将任务 ID 11–510 用于测试，其他 ID 分别用于提示、验证或训练；另有人工检查的 sanitized 文件。对全部可用题目跑分，并不自动等于论文测试协议。"},"title":{"en":"Use the declared test split","zh":"使用明确的测试划分"}},{"id":"scoring","source_ids":["paper"],"text":{"en":"The study evaluates whether generated Python satisfies the tests, including settings with many sampled candidates. Report how many candidates were allowed and how a final program was selected; a success found after many attempts is different from one-shot reliability.","zh":"论文通过测试检查生成的 Python，并研究大量采样候选的情况。应报告允许生成多少候选、怎样选出最终程序；多次尝试后找到答案，与一次作答可靠性不同。"},"title":{"en":"Passing tests and generating many candidates","zh":"通过测试与多候选生成"}},{"id":"prompts","source_ids":["repository","paper"],"text":{"en":"The reference setup uses few-shot examples, and the paper studies the effects of prompt examples and visible tests. Keep these inputs fixed when comparing models; supplying assertions gives additional information beyond the prose request.","zh":"参考设置使用少样本示例，论文也研究提示示例与可见测试的影响。比较模型时应固定这些输入；提供断言会比单纯文字需求增加信息。"},"title":{"en":"Examples change the task","zh":"示例会改变任务条件"}},{"id":"limits","source_ids":["paper"],"text":{"en":"The paper reports cases of programs overfitting assertions and examines edits to ambiguous tasks. Inspect failures and edge cases rather than equating test success with complete correctness. MBPP's short tasks offer limited evidence about multi-file engineering work.","zh":"论文记录了程序迎合断言的情况，也研究了对含糊题目的修订。应进一步查看失败和边界输入，不能把通过测试等同于完全正确；短任务对跨文件工程能力的说明有限。"},"title":{"en":"Three tests do not cover every case","zh":"三个测试不覆盖所有情况"}}],"sources":[{"id":"repository","title":"Google Research MBPP · Dataset files, splits and prompts","url":"https://github.com/google-research/google-research/tree/master/mbpp"},{"id":"paper","title":"Program Synthesis with Large Language Models · MBPP methods and analysis","url":"https://arxiv.org/html/2108.07732v1"}],"summary":{"en":"MBPP measures whether a model can translate a short natural-language request into a working Python program. Its task split and the tests shown in the prompt must be checked before comparing results.","zh":"MBPP 观察模型能否把简短自然语言需求转换成可运行的 Python 程序。比较成绩前，须确认任务划分以及提示中是否提供测试。"},"title":"MBPP"},"evidence_summary":{"document_count":5,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mbpp\u0000qwen2_5_coder_report\u0000mbpp\u0000MBPP base split\u0000Qwen2.5-Coder-32B-Instruct","reported_at":"2024-09-18","source_url":"https://arxiv.org/abs/2409.12186"},"first_score_reported_at":"2024-09-18","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mbpp\u0000qwen2_5_coder_report\u0000mbpp\u0000MBPP base split\u0000Qwen2.5-Coder-32B-Instruct","reported_at":"2024-09-18","source_url":"https://arxiv.org/abs/2409.12186"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:mbpp","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["code-generation"],"join":"review:14H034160212:2026-10-15","primary":"code-generation","source_record":null},"modality":null,"name":"MBPP","openness":"unknown","publisher":null,"released":"2021-08-16","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mbpp","source_url":"https://github.com/google-research/google-research/tree/master/mbpp"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":90.2,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":90.2,"raw_min":90.2,"source_reference":{"obs_id":"curated\u0000mbpp\u0000qwen2_5_coder_report\u0000mbpp\u0000MBPP base split\u0000Qwen2.5-Coder-32B-Instruct","observation_id":"curated\u0000mbpp\u0000qwen2_5_coder_report\u0000mbpp\u0000MBPP base split\u0000Qwen2.5-Coder-32B-Instruct","reported_at":"2024-09-18","reported_date":"2024-09-18","source_id":"qwen2_5_coder_report","source_url":"https://arxiv.org/abs/2409.12186"},"unit":"percent"},"scored_models":[{"date":"2024-09-18","date_precision":"document_publication","name":"Qwen2.5-Coder-32B-Instruct"}],"slug":"mbpp","source":"model_reports","source_url":"https://github.com/google-research/google-research/tree/master/mbpp","topics":["coding"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["代码","Code","大语言模型","LLM","代码工程","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. 该基准测试由大约1000个入门级程序员可以解决的众包Python编程问题组成，涵盖编程基础知识、标准库功能等。每个问题都由任务描述、代码解决方案和3个自动化测试用例组成。","descriptions":{"en":"The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases.","zh":"该基准测试由大约1000个入门级程序员可以解决的众包Python编程问题组成，涵盖编程基础知识、标准库功能等。每个问题都由任务描述、代码解决方案和3个自动化测试用例组成。"},"editorial_guide":{"id":"mbpp","record_keys":["model-reports:mbpp","llm-stats:mbpp","opencompass:538"],"reviewed_at":"2026-10-25","scope":{"en":"Original Mostly Basic Python Problems. Its original and sanitized subsets have different membership; MBPP+ and EvalPlus variants are not assigned this guide.","zh":"对应原始 Mostly Basic Python Problems。原版与 sanitized 子集的成员不同；MBPP+ 与 EvalPlus 变体不套用本导读。"},"sections":[{"id":"task","source_ids":["paper","repository"],"text":{"en":"The original dataset contains 974 crowdsourced tasks intended for entry-level programmers. Problems exercise short programs, programming fundamentals and standard-library use. Each release entry includes a natural-language description, a reference solution and three tests.","zh":"原数据集包含 974 道众包题，面向入门级编程能力。题目涉及短程序、编程基础和标准库使用；每条发布记录包含文字描述、参考代码和三个测试。"},"title":{"en":"Everyday Python building blocks","zh":"基础 Python 编程任务"}},{"id":"splits","source_ids":["repository"],"text":{"en":"The official split uses task IDs 11–510 for testing; other IDs serve prompting, validation or training. A separate hand-checked sanitized file also exists. A score on every available task is not automatically the published test protocol.","zh":"官方划分将任务 ID 11–510 用于测试，其他 ID 分别用于提示、验证或训练；另有人工检查的 sanitized 文件。对全部可用题目跑分，并不自动等于论文测试协议。"},"title":{"en":"Use the declared test split","zh":"使用明确的测试划分"}},{"id":"scoring","source_ids":["paper"],"text":{"en":"The study evaluates whether generated Python satisfies the tests, including settings with many sampled candidates. Report how many candidates were allowed and how a final program was selected; a success found after many attempts is different from one-shot reliability.","zh":"论文通过测试检查生成的 Python，并研究大量采样候选的情况。应报告允许生成多少候选、怎样选出最终程序；多次尝试后找到答案，与一次作答可靠性不同。"},"title":{"en":"Passing tests and generating many candidates","zh":"通过测试与多候选生成"}},{"id":"prompts","source_ids":["repository","paper"],"text":{"en":"The reference setup uses few-shot examples, and the paper studies the effects of prompt examples and visible tests. Keep these inputs fixed when comparing models; supplying assertions gives additional information beyond the prose request.","zh":"参考设置使用少样本示例，论文也研究提示示例与可见测试的影响。比较模型时应固定这些输入；提供断言会比单纯文字需求增加信息。"},"title":{"en":"Examples change the task","zh":"示例会改变任务条件"}},{"id":"limits","source_ids":["paper"],"text":{"en":"The paper reports cases of programs overfitting assertions and examines edits to ambiguous tasks. Inspect failures and edge cases rather than equating test success with complete correctness. MBPP's short tasks offer limited evidence about multi-file engineering work.","zh":"论文记录了程序迎合断言的情况，也研究了对含糊题目的修订。应进一步查看失败和边界输入，不能把通过测试等同于完全正确；短任务对跨文件工程能力的说明有限。"},"title":{"en":"Three tests do not cover every case","zh":"三个测试不覆盖所有情况"}}],"sources":[{"id":"repository","title":"Google Research MBPP · Dataset files, splits and prompts","url":"https://github.com/google-research/google-research/tree/master/mbpp"},{"id":"paper","title":"Program Synthesis with Large Language Models · MBPP methods and analysis","url":"https://arxiv.org/html/2108.07732v1"}],"summary":{"en":"MBPP measures whether a model can translate a short natural-language request into a working Python program. Its task split and the tests shown in the prompt must be checked before comparing results.","zh":"MBPP 观察模型能否把简短自然语言需求转换成可运行的 Python 程序。比较成绩前，须确认任务划分以及提示中是否提供测试。"},"title":"MBPP"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:538","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"identity_group:mbpp","primary":"code-generation","source_record":"catalog_0c2ab8aed657e87b"},"modality":null,"name":"MBPP","openness":"unknown","publisher":null,"released":"2021-08-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-538-mbpp","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MBPP","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1827,"github_stars_30d":null,"hf_dataset_downloads":33954,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:evalplus/evalplus","source_record":"catalog_f2ed6b6b0d44aa63"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality. This is an enhanced version with additional test cases.","descriptions":{"en":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality. This is an enhanced version with additional test cases."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mbpp-++-base-version:llama-3.1-70b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-%2B%2B-base-version?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mbpp-++-base-version","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_f2ed6b6b0d44aa63"},"modality":"text","name":"MBPP ++ base version","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":86.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.86,"raw_min":0.86,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mbpp-++-base-version:llama-3.1-70b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-%2B%2B-base-version?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"}],"slug":"llm-stats-mbpp-base-version","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-%2B%2B-base-version?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1827,"github_stars_30d":null,"hf_dataset_downloads":31141,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:evalplus/evalplus","source_record":"catalog_e4fa459970018e88"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. EvalPlus extends MBPP with significantly more test cases (35x) for more rigorous evaluation of LLM-synthesized code, providing high-quality and precise evaluation.","descriptions":{"en":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. EvalPlus extends MBPP with significantly more test cases (35x) for more rigorous evaluation of LLM-synthesized code, providing high-quality and precise evaluation."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mbpp-evalplus:llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mbpp-evalplus","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_e4fa459970018e88"},"modality":"text","name":"MBPP EvalPlus","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":88.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.886,"raw_min":0.876,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mbpp-evalplus:llama-3.1-405b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 70B Instruct"}],"slug":"llm-stats-mbpp-evalplus","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1827,"github_stars_30d":null,"hf_dataset_downloads":31667,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:evalplus/evalplus","source_record":"catalog_8ab302d3bad1721c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. EvalPlus extends MBPP with significantly more test cases (35x) for more rigorous evaluation of LLM-synthesized code, providing high-quality and precise evaluation.","descriptions":{"en":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. EvalPlus extends MBPP with significantly more test cases (35x) for more rigorous evaluation of LLM-synthesized code, providing high-quality and precise evaluation."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mbpp-evalplus-(base):llama-3.1-8b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus-%28base%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mbpp-evalplus-(base)","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_8ab302d3bad1721c"},"modality":"text","name":"MBPP EvalPlus (base)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":72.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.728,"raw_min":0.728,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mbpp-evalplus-(base):llama-3.1-8b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus-%28base%29?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"}],"slug":"llm-stats-mbpp-evalplus-base","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus-%28base%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_e583ae35322cb0f6"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases. This variant uses pass@1 evaluation metric measuring the percentage of problems solved correctly on the first attempt.","descriptions":{"en":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases. This variant uses pass@1 evaluation metric measuring the percentage of problems solved correctly on the first attempt."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mbpp-pass@1:ministral-8b-instruct-2410","reported_at":"2024-10-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-pass%401?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mbpp-pass@1","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_e583ae35322cb0f6"},"modality":"text","name":"MBPP pass@1","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":70.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.7,"raw_min":0.7,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mbpp-pass@1:ministral-8b-instruct-2410","reported_date":"2024-10-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-pass%401?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"}],"slug":"llm-stats-mbpp-pass-1","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-pass%401?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1827,"github_stars_30d":null,"hf_dataset_downloads":35330,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:evalplus/evalplus","source_record":"catalog_3b988625ad11d391"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality. This is an enhanced version with additional test cases for more rigorous evaluation.","descriptions":{"en":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality. This is an enhanced version with additional test cases for more rigorous evaluation."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mbpp-plus:mistral-small-3.2-24b-instruct-2506","reported_at":"2025-06-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-plus?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mbpp-plus","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_3b988625ad11d391"},"modality":"text","name":"MBPP Plus","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":78.33,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.7833,"raw_min":0.7833,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mbpp-plus:mistral-small-3.2-24b-instruct-2506","reported_date":"2025-06-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-plus?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"}],"slug":"llm-stats-mbpp-plus","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-plus?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MBPP+ is an enhanced version of MBPP (Mostly Basic Python Problems) with significantly more test cases (35x) for more rigorous evaluation. MBPP is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers, covering programming fundamentals and standard library functionality.","descriptions":{"en":"MBPP+ is an enhanced version of MBPP (Mostly Basic Python Problems) with significantly more test cases (35x) for more rigorous evaluation. MBPP is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers, covering programming fundamentals and standard library functionality."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mbpp+:qwen-2.5-14b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp%2B?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mbpp+","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_0c2ab8aed657e87b"},"modality":"text","name":"MBPP+","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":74.1,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.741,"raw_min":0.402,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mbpp+:mimo-v2.5-pro","reported_date":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp%2B?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"}],"slug":"llm-stats-mbpp-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp%2B?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models"],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MC-CXR is a benchmark of 240 chest X-ray cases (2,522 instances) for evaluating context-induced disruption in vision-language models. It pairs reliable and misleading context across text and prior imaging, with visual overlays. Defines three task families and two paired metrics: switch-to-wrong rate and context-aligned error rate.","descriptions":{"en":"MC-CXR is a benchmark of 240 chest X-ray cases (2,522 instances) for evaluating context-induced disruption in vision-language models. It pairs reliable and misleading context across text and prior imaging, with visual overlays. Defines three task families and two paired metrics: switch-to-wrong rate and context-aligned error rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24118","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_mc-cxr-a-multi-context-chest-x-ray-benchma_1a149873"},"modality":null,"name":"MC-CXR","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24118","source_url":"https://arxiv.org/abs/2608.24118"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24118","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24118","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:arbez-zebra/mciqa","source_record":"bm_mciqa-2k_de7bbe69"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MCIQA-2K includes 2,000 colorized images generated by five models, annotated across color smearing, semantic color misalignment, and global naturalness for no-reference image quality assessment.","descriptions":{"en":"MCIQA-2K includes 2,000 colorized images generated by five models, annotated across color smearing, semantic color misalignment, and global naturalness for no-reference image quality assessment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.14495","languages":[],"library_direction":{"basis":"model-review-2","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_mciqa-2k_de7bbe69"},"modality":null,"name":"MCIQA-2K","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.14495","source_url":"https://arxiv.org/abs/2609.14495"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-14495","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14495","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MCiteBench is a benchmark to evaluate multimodal citation text generation in MLLMs. It consists of 3,000 samples from 1,749 academic papers, featuring 2,000 Explanation tasks and 1,000 Locating tasks, with balanced evidence across text, figures, tables, and mixed modalities. MCiteBench 是一个用于评估多模态大模型中多模态引用文本生成的基准，由 1,749 篇学术论文中的 3,000 个样本组成，包括 2,000 个解释任务和 1,000 个定位任务，在文本、图表、表格和混合模态中平衡证据。","descriptions":{"en":"MCiteBench is a benchmark to evaluate multimodal citation text generation in MLLMs. It consists of 3,000 samples from 1,749 academic papers, featuring 2,000 Explanation tasks and 1,000 Locating tasks, with balanced evidence across text, figures, tables, and mixed modalities.","zh":"MCiteBench 是一个用于评估多模态大模型中多模态引用文本生成的基准，由 1,749 篇学术论文中的 3,000 个样本组成，包括 2,000 个解释任务和 1,000 个定位任务，在文本、图表、表格和混合模态中平衡证据。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1606","languages":[],"library_direction":null,"modality":"multimodal","name":"MCiteBench","openness":"restricted","publisher":"FDU, SHU","released":"2025-03-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1606-mcitebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MCiteBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":160,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:scaleapi/mcp-atlas","source_record":"catalog_544e399fd1681104"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MCP Atlas is a benchmark for evaluating AI models on scaled tool use capabilities, measuring how well models can coordinate and utilize multiple tools across complex multi-step tasks.","descriptions":{"en":"MCP Atlas is a benchmark for evaluating AI models on scaled tool use capabilities, measuring how well models can coordinate and utilize multiple tools across complex multi-step tasks."},"evidence_summary":{"document_count":1,"model_count":33,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mcp-atlas:claude-opus-4-5-20251101","reported_at":"2025-11-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-atlas?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:mcp-atlas","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_544e399fd1681104"},"modality":"text","name":"MCP Atlas","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":33,"score_direction":"higher_is_better","score_summary":{"display_max":88.1,"display_multiplier":100,"model_count":33,"model_count_basis":"source_model_id","numeric_count":33,"raw_max":0.881,"raw_min":0.246,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mcp-atlas:muse-spark-1.1","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-atlas?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-mcp-atlas","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-atlas?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":["MCP Atlas","MCP-Atlas","MCPAtlas","MCP-Atlas (Public Set)"],"attention_signals":null,"categories":["tool_use"],"collected_at":null,"description":"Public and full splits are both reported under one name; the tool server set is part of the measurement.","descriptions":{"en":"Public and full splits are both reported under one name; the tool server set is part of the measurement."},"evidence_summary":{"document_count":8,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mcp_atlas\u0000zai_glm_5_model_card\u0000mcp_atlas\u0000think mode, 500-task public subset, 10-min timeout, Gemini 3 Pro judge\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"first_score_reported_at":"2026-02-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mcp_atlas\u0000zai_glm_5_model_card\u0000mcp_atlas\u0000think mode, 500-task public subset, 10-min timeout, Gemini 3 Pro judge\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:mcp_atlas","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["tool-use"],"join":"review:14H034160212:2026-10-15","primary":"tool-use","source_record":null},"modality":null,"name":"MCP Atlas","openness":"unknown","publisher":null,"released":"2025-11-18","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mcp_atlas","source_url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":84.2,"display_multiplier":1,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":84.2,"raw_min":67.8,"source_reference":{"obs_id":"curated\u0000mcp_atlas\u0000moonshot_kimi_k3_model_card\u0000mcp_atlas\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000mcp_atlas\u0000moonshot_kimi_k3_model_card\u0000mcp_atlas\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"}],"slug":"mcp_atlas","source":"model_reports","source_url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DC"],"collected_at":"2026-10-14T09:44:51Z","description":"MCP-GRANITE is a benchmark framework for evaluating tool-interface granularity in MCP-based LLM agents under edge and IoT scenarios.","descriptions":{"en":"MCP-GRANITE is a benchmark framework for evaluating tool-interface granularity in MCP-based LLM agents under edge and IoT scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24161","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_mcp-granite-benchmark_3107daf9"},"modality":null,"name":"MCP-GRANITE Benchmark","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24161","source_url":"https://arxiv.org/abs/2609.24161"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24161","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24161","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":465,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":152,"join":"llm_stats_id","repository":"gh:eval-sys/mcpmark","source_record":"catalog_99d3067c4c9f0c37"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MCP-Mark evaluates LLMs on their ability to use Model Context Protocol (MCP) tools effectively, testing tool discovery, selection, invocation, and result interpretation across diverse MCP server scenarios.","descriptions":{"en":"MCP-Mark evaluates LLMs on their ability to use Model Context Protocol (MCP) tools effectively, testing tool discovery, selection, invocation, and result interpretation across diverse MCP server scenarios."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mcp-mark:deepseek-v3.2","reported_at":"2025-12-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-mark?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mcp-mark","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_99d3067c4c9f0c37"},"modality":"text","name":"MCP-Mark","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":81.10000000000001,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.811,"raw_min":0.37,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mcp-mark:kimi-k2.7-code","reported_date":"2026-06-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-mark?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-mcp-mark","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-mark?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":9,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":14,"join":"claire_record_id","repository":"gh:wwh0411/mcp-persona","source_record":"bm_mcp-persona_e4dd478c"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MCP-Persona evaluates LLM agents on real-world personalized MCP tools across social media, collaboration, email, and content management applications. It includes 173 tool-chain tasks, 139 unique tools, and 18 MCP servers, with a fully automated environment simulation pipeline for reproducible evaluation.","descriptions":{"en":"MCP-Persona evaluates LLM agents on real-world personalized MCP tools across social media, collaboration, email, and content management applications. It includes 173 tool-chain tasks, 139 unique tools, and 18 MCP servers, with a fully automated environment simulation pipeline for reproducible evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.02470","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","agent-memory"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_mcp-persona_e4dd478c"},"modality":null,"name":"MCP-Persona","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02470","source_url":"https://arxiv.org/abs/2606.02470"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02470","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02470","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":602,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":43,"join":"llm_stats_id","repository":null,"source_record":"catalog_945d8dca6fae8ba8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MCP-Universe evaluates LLMs on complex multi-step agentic tasks using Model Context Protocol (MCP) tools across diverse interactive environments, testing planning, tool orchestration, and task completion.","descriptions":{"en":"MCP-Universe evaluates LLMs on complex multi-step agentic tasks using Model Context Protocol (MCP) tools across diverse interactive environments, testing planning, tool orchestration, and task completion."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mcp-universe:deepseek-v3.2","reported_at":"2025-12-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-universe?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mcp-universe","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_945d8dca6fae8ba8"},"modality":"text","name":"MCP-Universe","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":45.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.459,"raw_min":0.459,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mcp-universe:deepseek-v3.2","reported_date":"2025-12-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-universe?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"}],"slug":"llm-stats-mcp-universe","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-universe?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_mcpevol-bench_3df86292"},"categories":["Agents","Cybersecurity","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object is unclear from the provided information.","descriptions":{"en":"The evaluation object is unclear from the provided information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14642","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_mcpevol-bench_3df86292"},"modality":null,"name":"MCPEvol-Bench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14642","source_url":"https://arxiv.org/abs/2607.14642"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14642","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14642","topics":["agents","cybersecurity"],"unit":null},{"aliases":["MCPMark","MCP-Mark","MCPMark-Verified"],"attention_signals":null,"categories":["tool_use"],"collected_at":null,"description":"Depends on live third-party MCP servers, so runs are not reproducible over time.","descriptions":{"en":"Depends on live third-party MCP servers, so runs are not reproducible over time."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mcp_mark\u0000qwen3_5_model_card\u0000mcp_mark\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mcp_mark\u0000qwen3_5_model_card\u0000mcp_mark\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:mcp_mark","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["tool-use"],"join":"review:14H034160212:2026-10-15","primary":"tool-use","source_record":null},"modality":null,"name":"MCPMark","openness":"unknown","publisher":null,"released":"2025-09-30","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mcp_mark","source_url":"https://mcpmark.ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":94.5,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":94.5,"raw_min":46.1,"source_reference":{"obs_id":"curated\u0000mcp_mark\u0000moonshot_kimi_k3_model_card\u0000mcpmark_verified\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000mcp_mark\u0000moonshot_kimi_k3_model_card\u0000mcpmark_verified\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"mcp_mark","source":"model_reports","source_url":"https://mcpmark.ai/","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Finance","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"MCR-Bench is a benchmark for evaluating reproducibility in mission-critical LLM tasks, measuring output consistency across heterogeneous hardware.","descriptions":{"en":"MCR-Bench is a benchmark for evaluating reproducibility in mission-critical LLM tasks, measuring output consistency across heterogeneous hardware."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21023","languages":[],"library_direction":null,"modality":null,"name":"MCR-Bench","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21023","source_url":"https://arxiv.org/abs/2606.21023"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21023","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21023","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming.","descriptions":{"en":"In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27442","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_mcr-bench_7a6da290"},"modality":null,"name":"MCR-Bench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27442","source_url":"https://arxiv.org/abs/2608.27442"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27442","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27442","topics":["coding"],"unit":null},{"aliases":["Post-Training VLMs for Video Mistake Detection"],"attention_signals":{"as_of":"2026-10-14","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fedespu/mstk","source_record":"bm_post-training-vlms-for-video-mistake-detec_58e7640e"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates mistake detection in videos via question answering, testing whether methods can discern if a step was executed correctly against its description for seen and unseen actions.","descriptions":{"en":"Evaluates mistake detection in videos via question answering, testing whether methods can discern if a step was executed correctly against its description for seen and unseen actions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.28406","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","visual-reasoning"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_post-training-vlms-for-video-mistake-detec_58e7640e"},"modality":null,"name":"MD-VQA","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28406","source_url":"https://arxiv.org/abs/2608.28406"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28406","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28406","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","physics.chem-ph"],"collected_at":"2026-10-14T09:44:51Z","description":"MDArena evaluates coding agents on 50 containerized molecular dynamics tasks from active research projects, covering trajectory analysis, system preparation, free-energy protocols, and enhanced sampling, with strict pass-based scoring.","descriptions":{"en":"MDArena evaluates coding agents on 50 containerized molecular dynamics tasks from active research projects, covering trajectory analysis, system preparation, free-energy protocols, and enhanced sampling, with strict pass-based scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02642","languages":[],"library_direction":{"basis":"model-review","categories":["science","agentic-coding"],"join":"claire_record_id","primary":"science","source_record":"bm_mdarena_a3bdd691"},"modality":null,"name":"MDArena","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02642","source_url":"https://arxiv.org/abs/2608.02642"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02642","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02642","topics":["coding","agents","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":9,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"llm_stats_id","repository":null,"source_record":"catalog_f9f1763b8efbc7c6"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MeasureBench evaluates multimodal models on visual measurement and quantitative perception tasks across both real and synthetic imagery, reported as the average over the two settings.","descriptions":{"en":"MeasureBench evaluates multimodal models on visual measurement and quantitative perception tasks across both real and synthetic imagery, reported as the average over the two settings."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:measurebench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/measurebench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:measurebench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_f9f1763b8efbc7c6"},"modality":"multimodal","name":"MeasureBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":62.9,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.629,"raw_min":0.589,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:measurebench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/measurebench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-measurebench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/measurebench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Scientific discovery requires not only recovering mathematical laws that describe observable behavior, but also identifying the mechanisms that generate them.","descriptions":{"en":"Scientific discovery requires not only recovering mathematical laws that describe observable behavior, but also identifying the mechanisms that generate them."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35515","languages":[],"library_direction":{"basis":"model-review","categories":["science","math"],"join":"claire_record_id","primary":"science","source_record":"bm_mechbench_efa48e81"},"modality":null,"name":"MechBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35515","source_url":"https://arxiv.org/abs/2609.35515"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35515","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35515","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:q-i-n-g/mecobench","source_record":"bm_mecobench_fd729c4e"},"categories":["Agents","Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MECoBench is a multimodal embodied cooperation benchmark with an evaluation platform. It spans real-world tasks, two cooperation structures, and three collaboration modes, with code and dataset publicly available.","descriptions":{"en":"MECoBench is a multimodal embodied cooperation benchmark with an evaluation platform. It spans real-world tasks, two cooperation structures, and three collaboration modes, with code and dataset publicly available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.31966","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","multi-agent-games"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_mecobench_fd729c4e"},"modality":null,"name":"MECoBench","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31966","source_url":"https://arxiv.org/abs/2606.31966"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31966","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31966","topics":["agents","multimodal"],"unit":null},{"aliases":["Med-Fact","MedFact","Medfact"],"attention_signals":null,"categories":["health"],"collected_at":null,"description":"150K medical claims (50K each Supported, Refuted, Not-Enough-Info) generated automatically from MedMCQA exam questions by the Multi2Claim pipeline (EACL 2023), not written by fact-checkers. Labels come from the generator, with sampled annotator agreement rather than full review. Only 50 examples are public; the full set is available from the authors on request.","descriptions":{"en":"150K medical claims (50K each Supported, Refuted, Not-Enough-Info) generated automatically from MedMCQA exam questions by the Multi2Claim pipeline (EACL 2023), not written by fact-checkers. Labels come from the generator, with sampled annotator agreement rather than full review. Only 50 examples are public; the full set is available from the authors on request."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:med_fact","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["health-medicine"],"join":"review:14H034160212:2026-10-15","primary":"health-medicine","source_record":null},"modality":null,"name":"Med-Fact","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"med_fact","source":"model_reports","source_url":"https://aclanthology.org/2023.eacl-main.194/","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","知识","Knowledge","Medical","Agent","科学智能","AI for Science","逻辑推理","知识储备","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MedAgentsBench features 862 challenging medical questions from seven datasets, focusing on cases where models struggle. It emphasizes multi-step clinical reasoning and addresses limitations in existing benchmarks by eliminating simple questions and standardizing evaluation protocols. MedAgentsBench是一个专注于复杂医学推理的基准测试，从七个医学数据集中精选了862个挑战性问题。这些数据集包括MedQA、PubMedQA、MedMCQA、MedBullets、MedExQA、MedXpertQA和MMLU/MMLU-Pro，涵盖了从医学执照考试到研究文献的多种医学问题。\n该基准选择少于50%模型能正确回答的问题，确保医学知识领域全面覆盖，并优先选择需要多步临床推理的问题。这解决了现有评估中简单问题普遍存在、评估协议不一致，以及缺乏性能-成本-时间分析的局限。","descriptions":{"en":"MedAgentsBench features 862 challenging medical questions from seven datasets, focusing on cases where models struggle. It emphasizes multi-step clinical reasoning and addresses limitations in existing benchmarks by eliminating simple questions and standardizing evaluation protocols.","zh":"MedAgentsBench是一个专注于复杂医学推理的基准测试，从七个医学数据集中精选了862个挑战性问题。这些数据集包括MedQA、PubMedQA、MedMCQA、MedBullets、MedExQA、MedXpertQA和MMLU/MMLU-Pro，涵盖了从医学执照考试到研究文献的多种医学问题。\n该基准选择少于50%模型能正确回答的问题，确保医学知识领域全面覆盖，并优先选择需要多步临床推理的问题。这解决了现有评估中简单问题普遍存在、评估协议不一致，以及缺乏性能-成本-时间分析的局限。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1641","languages":[],"library_direction":null,"modality":null,"name":"MedAgents-Bench","openness":"unknown","publisher":"Yale University","released":"2025-03-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1641-medagents-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedAgents-Bench","topics":["agents","reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","科学智能","AI for Science","语言理解","Comprehension","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MEDAL is a framework for generating and evaluating multilingual open-domain chatbots and their evaluators. This framework supports various language models and provides a structured approach to creating conversational datasets. MEDAL是一个用于生成和评估多语言开放域聊天机器人及其评估器的框架。该框架支持各种语言模型,并提供了一种结构化的方法来创建对话数据集。","descriptions":{"en":"MEDAL is a framework for generating and evaluating multilingual open-domain chatbots and their evaluators. This framework supports various language models and provides a structured approach to creating conversational datasets.","zh":"MEDAL是一个用于生成和评估多语言开放域聊天机器人及其评估器的框架。该框架支持各种语言模型,并提供了一种结构化的方法来创建对话数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1893","languages":["English","Chinese","French","German","Spanish","Portuguese","Multilingual"],"library_direction":null,"modality":null,"name":"MEDAL","openness":"restricted","publisher":"INESC-ID,etc.","released":"2025-05-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1893-medal","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MEDAL","topics":["reasoning-math","language","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","Medical","科学智能","AI for Science","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MedArabiQ introduces a new benchmark dataset consisting of seven Arabic medical tasks, covering multiple specialties and question formats:\n✅ Multiple-choice questions\n✏️ Fill-in-the-blank (with and without choices)\n💬 Patient-doctor question answering 大型语言模型（LLMs）在医疗保健应用中显示出了显著的前景，但由于缺乏高质量的领域特定数据集，它们在阿拉伯语医学领域的表现在很大程度上仍未得到探索。MedArabiQ引入了一个新的基准数据集，包含七个阿拉伯语医学任务，涵盖多个专业和问题格式：\n✅ 多项选择题\n✏️ 填空题（有选项和无选项）\n💬 患者-医生问答\n\n该数据集使用过往医学考试和公开可用资源构建，并进行了修改以评估LLMs在各种能力方面的表现，包括偏见缓解。","descriptions":{"en":"MedArabiQ introduces a new benchmark dataset consisting of seven Arabic medical tasks, covering multiple specialties and question formats:\n✅ Multiple-choice questions\n✏️ Fill-in-the-blank (with and without choices)\n💬 Patient-doctor question answering","zh":"大型语言模型（LLMs）在医疗保健应用中显示出了显著的前景，但由于缺乏高质量的领域特定数据集，它们在阿拉伯语医学领域的表现在很大程度上仍未得到探索。MedArabiQ引入了一个新的基准数据集，包含七个阿拉伯语医学任务，涵盖多个专业和问题格式：\n✅ 多项选择题\n✏️ 填空题（有选项和无选项）\n💬 患者-医生问答\n\n该数据集使用过往医学考试和公开可用资源构建，并进行了修改以评估LLMs在各种能力方面的表现，包括偏见缓解。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1874","languages":["Arabic","Multilingual"],"library_direction":null,"modality":null,"name":"MedArabiQ","openness":"unknown","publisher":"New York University","released":"2025-05-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1874-medarabiq","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedArabiQ","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MedBench v5 evaluates clinical multimodal models across 63 tasks with process-oriented metrics including stressors and hallucination tracking.","descriptions":{"en":"MedBench v5 evaluates clinical multimodal models across 63 tasks with process-oriented metrics including stressors and hallucination tracking."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.24155","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medbench_c5b43204"},"modality":null,"name":"MedBench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24155","source_url":"https://arxiv.org/abs/2606.24155"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24155","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24155","topics":["agents","multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Biology & Drug Discovery","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible with richly annotated imaging datasets and large language models (LLMs), yet existing automation largely focuses on generating evaluation i…","descriptions":{"en":"Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible with richly annotated imaging datasets and large language models (LLMs), yet existing automation largely focuses on generating evaluation i…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11312","languages":[],"library_direction":{"basis":"model-review-daily","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medbench_7482e442"},"modality":null,"name":"MedBench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11312","source_url":"https://arxiv.org/abs/2610.11312"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11312","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11312","topics":["agents","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","Medical","科学智能","AI for Science","知识储备","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MedBench is committed to building a scientific, fair and rigorous Chinese medical model evaluation system and open platform. Based on authoritative standards, we constantly update and maintain high-quality datasets, and comprehensively quantify capabilities of models in various medical dimensions. MedBench致力于打造一个科学、公平且严谨的中文医疗大模型评测体系及开放平台。我们基于医学权威标准，不断更新维护高质量的医学数据集，全方位多维度量化模型在各个医学维度的能力。","descriptions":{"en":"MedBench is committed to building a scientific, fair and rigorous Chinese medical model evaluation system and open platform. Based on authoritative standards, we constantly update and maintain high-quality datasets, and comprehensively quantify capabilities of models in various medical dimensions.","zh":"MedBench致力于打造一个科学、公平且严谨的中文医疗大模型评测体系及开放平台。我们基于医学权威标准，不断更新维护高质量的医学数据集，全方位多维度量化模型在各个医学维度的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1287","languages":["Chinese"],"library_direction":null,"modality":null,"name":"MedBench","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2023-12-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1287-medbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedBench","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","Medical","VQA","MLLM","科学智能","AI for Science","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MedBookVQA is a medical visual question answering (VQA) benchmark constructed from open-access medical textbooks. It includes 5,000 questions across five clinical task types and is hierarchically organized by imaging modality, anatomical structure, and clinical specialty. MedBookVQA 是一个基于开放获取医学教科书构建的医学视觉问答（VQA）基准数据集。它包含 5,000 个问题，涵盖五种临床任务类型，并按照影像模态、解剖结构和临床专科进行分层组织。","descriptions":{"en":"MedBookVQA is a medical visual question answering (VQA) benchmark constructed from open-access medical textbooks. It includes 5,000 questions across five clinical task types and is hierarchically organized by imaging modality, anatomical structure, and clinical specialty.","zh":"MedBookVQA 是一个基于开放获取医学教科书构建的医学视觉问答（VQA）基准数据集。它包含 5,000 个问题，涵盖五种临床任务类型，并按照影像模态、解剖结构和临床专科进行分层组织。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1920","languages":[],"library_direction":null,"modality":"multimodal","name":"MedBookVQA","openness":"unknown","publisher":"Hong Kong University of Science and Technology","released":"2025-05-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1920-medbookvqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedBookVQA","topics":["multimodal","reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","Medical","科学智能","AI for Science","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"the first benchmark that systematically tests an agent’s ability to reliably retrieve and synthesize multi-hop medical facts from live, domain-specific knowledge bases. MedBrowseComp holds 1,000+ human-curated questions that mirror clinical scenarios\n the first benchmark that systematically tests an agent’s ability to reliably retrieve and synthesize multi-hop medical facts from live, domain-specific knowledge bases. MedBrowseComp holds 1,000+ human-curated questions that mirror clinical scenarios\n","descriptions":{"en":"the first benchmark that systematically tests an agent’s ability to reliably retrieve and synthesize multi-hop medical facts from live, domain-specific knowledge bases. MedBrowseComp holds 1,000+ human-curated questions that mirror clinical scenarios\n","zh":"the first benchmark that systematically tests an agent’s ability to reliably retrieve and synthesize multi-hop medical facts from live, domain-specific knowledge bases. MedBrowseComp holds 1,000+ human-curated questions that mirror clinical scenarios\n"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1832","languages":[],"library_direction":null,"modality":null,"name":"MedBrowseComp","openness":"open","publisher":"Harvard,etc","released":"2025-05-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1832-medbrowsecomp","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedBrowseComp","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","科学智能","AI for Science","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MedCalc-Bench focuses on evaluating the medical calculation capability of LLMs. It contains an evaluation set of over 1000 manually reviewed instances from 55 different medical calculation tasks. MedCalc-Bench专注于评估LLM的医学计算能力，包含来自55个不同医学计算任务的1000个经过人工审查的实例。","descriptions":{"en":"MedCalc-Bench focuses on evaluating the medical calculation capability of LLMs. It contains an evaluation set of over 1000 manually reviewed instances from 55 different medical calculation tasks.","zh":"MedCalc-Bench专注于评估LLM的医学计算能力，包含来自55个不同医学计算任务的1000个经过人工审查的实例。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1241","languages":[],"library_direction":null,"modality":null,"name":"MedCalc-Bench","openness":"unknown","publisher":"National Institutes of Health","released":"2024-06-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1241-medcalc-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedCalc-Bench","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MedChemBench is an internal OpenAI evaluation of medicinal-chemistry reasoning, testing whether models can support drug-discovery-relevant chemistry analysis and decision-making.","descriptions":{"en":"MedChemBench is an internal OpenAI evaluation of medicinal-chemistry reasoning, testing whether models can support drug-discovery-relevant chemistry analysis and decision-making."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:medchembench:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medchembench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:medchembench","languages":[],"library_direction":{"basis":"model-review","categories":["science","health-medicine"],"join":"llm_stats_id","primary":"science","source_record":"catalog_2b9a45a0de6fd5b9"},"modality":"text","name":"MedChemBench (Internal)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":48.3,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.483,"raw_min":0.304,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:medchembench:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medchembench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-medchembench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medchembench?top_n=500","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"MedClawBench evaluates long-horizon temporal reasoning in surgical videos through 1,123 doctor-grounded questions over self-built long neurosurgery recordings and a public lecture-video test split, with fixed evaluation dimensions for comparison.","descriptions":{"en":"MedClawBench evaluates long-horizon temporal reasoning in surgical videos through 1,123 doctor-grounded questions over self-built long neurosurgery recordings and a public lecture-video test split, with fixed evaluation dimensions for comparison."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14015","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","video-understanding"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medclawbench_bcd0f6e4"},"modality":null,"name":"MedClawBench","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14015","source_url":"https://arxiv.org/abs/2608.14015"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14015","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14015","topics":["agents","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":8,"github_stars_30d":1,"hf_dataset_downloads":141,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ivul-kaust/medcta","source_record":"bm_medcta_84809bbe"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MedCTA evaluates clinical tool agents on 107 clinician-verified, step-implicit tasks with multimodal inputs (radiology images, pathology slides, reports) and 5 deployed tools. Metrics cover tool selection, argument validity, execution stability, trajectory fidelity, and outcome quality.","descriptions":{"en":"MedCTA evaluates clinical tool agents on 107 clinician-verified, step-implicit tasks with multimodal inputs (radiology images, pathology slides, reports) and 5 deployed tools. Metrics cover tool selection, argument validity, execution stability, trajectory fidelity, and outcome quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.11702","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","tool-use"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medcta_84809bbe"},"modality":null,"name":"MedCTA","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11702","source_url":"https://arxiv.org/abs/2606.11702"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11702","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11702","topics":["agents","multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_medcua-bench_73e6da74"},"categories":["Agents & Tool Use","Biology & Drug Discovery","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MedCUA-Bench is an interactive benchmark for clinical computer-use agents, covering 18 scenarios in 10 medical domains with deterministic safety evaluation.","descriptions":{"en":"MedCUA-Bench is an interactive benchmark for clinical computer-use agents, covering 18 scenarios in 10 medical domains with deterministic safety evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03203","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","health-medicine"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_medcua-bench_73e6da74"},"modality":null,"name":"MedCUA-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03203","source_url":"https://arxiv.org/abs/2606.03203"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03203","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03203","topics":["computer-use","agents","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MedDDC-Eval evaluates multi-turn medical consultation agents by decoupling diagnosis from the diagnostic reader, using a frozen shared reader to score policies. It reports diagnostic support, coverage, and efficiency across Record and Dialogue splits.","descriptions":{"en":"MedDDC-Eval evaluates multi-turn medical consultation agents by decoupling diagnosis from the diagnostic reader, using a frozen shared reader to score policies. It reports diagnostic support, coverage, and efficiency across Record and Dialogue splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.18999","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medddc-eval_84cc7de6"},"modality":null,"name":"MedDDC-Eval","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18999","source_url":"https://arxiv.org/abs/2607.18999"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18999","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18999","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_medfailbench_90f26faa"},"categories":["Biology & Drug Discovery","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object is unclear from the provided information.","descriptions":{"en":"The evaluation object is unclear from the provided information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.15166","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medfailbench_90f26faa"},"modality":null,"name":"MedFailBench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15166","source_url":"https://arxiv.org/abs/2607.15166"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15166","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15166","topics":["safety","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Provides a continuous-time reinforcement learning environment for dynamic medical treatment recommendation, built from clinical data with physics-informed neural networks. Supports offline and online RL with configurable parameters.","descriptions":{"en":"Provides a continuous-time reinforcement learning environment for dynamic medical treatment recommendation, built from clinical data with physics-informed neural networks. Supports offline and online RL with configurable parameters."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01028","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medgym_d9be6475"},"modality":null,"name":"MedGym","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01028","source_url":"https://arxiv.org/abs/2606.01028"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01028","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01028","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MedHal-Loc is a benchmark and metric for localization faithfulness of medical hallucination detectors, comprising a controlled subset of 300 PubMedQA-derived statements with injected span-level errors and a natural subset. It evaluates whether top-ranked error units overlap erroneous spans across four error types.","descriptions":{"en":"MedHal-Loc is a benchmark and metric for localization faithfulness of medical hallucination detectors, comprising a controlled subset of 300 PubMedQA-derived statements with injected span-level errors and a natural subset. It evaluates whether top-ranked error units overlap erroneous spans across four error types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21517","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","health-medicine"],"join":"claire_record_id","primary":"factuality","source_record":"bm_medhal-loc_0d281208"},"modality":null,"name":"MedHal-Loc","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21517","source_url":"https://arxiv.org/abs/2606.21517"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21517","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21517","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["医学","Medical","幻觉","Hallucination","科学智能","AI for Science","事实可靠性","Factual Reliability","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"","descriptions":{},"editorial_guide":{"id":"medhalltune","record_keys":["opencompass:2417"],"reviewed_at":"2026-10-23","scope":{"en":"Cited research guide; original source measurements remain independent.","zh":"附来源的研究导读；原始记录的测量保持独立。"},"sections":[{"id":"tasks","source_ids":["paper"],"text":{"en":"Medical images are paired with instructions and ground-truth annotations. Both hallucination and non-hallucination samples matter.","zh":"医学图像与指令、标准标注配对，同时包含幻觉样本和非幻觉样本。"},"title":{"en":"Task and evidence","zh":"任务与证据"}},{"id":"metrics","source_ids":["paper"],"text":{"en":"The paper evaluates clinical accuracy, relevance, detail and risk. A single average can hide different failure patterns across positive and hallucination samples.","zh":"论文分别评估临床准确性、相关性、细节和风险。单一均值可能掩盖正常样本与幻觉样本上的不同失败模式。"},"title":{"en":"Read the metrics separately","zh":"分别理解各项指标"}},{"id":"use","source_ids":["project"],"text":{"en":"The authors provide image data, validation annotations and a JSONL prediction workflow. Inspect the evaluator and split before comparing runs.","zh":"作者提供图像数据、验证集标注及 JSONL 预测文件流程。比较实验前须核对评估器与数据划分。"},"title":{"en":"Inspect the evaluation workflow","zh":"查看评估流程"}},{"id":"limits","source_ids":["paper"],"text":{"en":"This guide explains the linked research. The catalog has not collected a structured score table for this record; absence here does not mean the paper reports no experiments.","zh":"本导读解释已链接的研究。本站尚未为此条目收录结构化成绩表，并不代表原论文没有实验。"},"title":{"en":"What this record establishes","zh":"这条记录说明什么"}}],"sources":[{"id":"paper","title":"MedHallTune — original paper","url":"https://arxiv.org/abs/2502.20780v2"},{"id":"project","title":"MedHallTune — authors’ project","url":"https://github.com/russellyq/MedHallTune"}],"summary":{"en":"Whether medical vision-language models give grounded answers rather than plausible hallucinations.","zh":"考察医学视觉语言模型能否依据图像作答，避免生成貌似合理的幻觉。"},"title":"MedHallTune"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2417","languages":[],"library_direction":null,"modality":null,"name":"MedHallTune","openness":"unknown","publisher":"The Chinese University of Hong Kong","released":"2025-02-28","released_reference":{"basis":"paper_first_version","note":"First version introducing the medical hallucination instruction-tuning benchmark.","source_key":"opencompass:2417","source_url":"https://arxiv.org/abs/2502.20780"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2417-medhalltune","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedHallTune","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","Medical","科学智能","AI for Science","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MedHallu is a comprehensive benchmark dataset designed to evaluate the ability of large language models to detect hallucinations in medical question-answering tasks. MedHallu 旨在评估大型语言模型在医学问题解答任务中检测幻觉的能力。","descriptions":{"en":"MedHallu is a comprehensive benchmark dataset designed to evaluate the ability of large language models to detect hallucinations in medical question-answering tasks.","zh":"MedHallu 旨在评估大型语言模型在医学问题解答任务中检测幻觉的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1548","languages":["English"],"library_direction":null,"modality":null,"name":"MedHallu","openness":"restricted","publisher":"University of Texas at Austin, UNC Chapel Hill, Drexel University","released":"2025-02-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1548-medhallu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedHallu","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tomry1114/awesome-agent-harness-for-healthcare","source_record":"bm_medharnessbench_3e02ffac"},"categories":["Agents","Biology & Drug Discovery","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates healthcare agents on 342 executable tasks spanning FHIR/EHR operations, healthcare administration interfaces, and multimodal clinical tools, measuring both outcome success and seven process-quality dimensions.","descriptions":{"en":"Evaluates healthcare agents on 342 executable tasks spanning FHIR/EHR operations, healthcare administration interfaces, and multimodal clinical tools, measuring both outcome success and seven process-quality dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.15714","languages":[],"library_direction":{"basis":"model-review-daily","categories":["health-medicine","professional-work"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medharnessbench_3e02ffac"},"modality":null,"name":"MedHarnessBench","openness":"unknown","publisher":null,"released":"2026-10-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.15714","source_url":"https://arxiv.org/abs/2610.15714"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-15714","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15714","topics":["agents","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Graphical consoles offer a practical interface for medical acquisition assistance, allowing agents to work through the controls and visual feedback used by human operators.","descriptions":{"en":"Graphical consoles offer a practical interface for medical acquisition assistance, allowing agents to work through the controls and visual feedback used by human operators."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.04800","languages":[],"library_direction":{"basis":"model-review-daily","categories":["computer-use","health-medicine"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_medimageosworld_a38ba8a0"},"modality":null,"name":"MedImageOSWorld","openness":"unknown","publisher":null,"released":"2026-10-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.04800","source_url":"https://arxiv.org/abs/2610.04800"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-04800","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04800","topics":["computer-use","agents","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates message-driven narrative video editing with long-form videos paired with multiple editing messages and multiple professional edits per message, using temporal alignment metrics and additional annotations for message ambiguity and contextfulness.","descriptions":{"en":"Evaluates message-driven narrative video editing with long-form videos paired with multiple editing messages and multiple professional edits per message, using temporal alignment metrics and additional annotations for message ambiguity and contextfulness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.25300","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_medit-bench_8ae40735"},"modality":null,"name":"MEDit-Bench","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25300","source_url":"https://arxiv.org/abs/2607.25300"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25300","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25300","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","科学智能","AI for Science","科学推理","Scientific Reasoning","语言生成","Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MedJourney offers a comprehensive assessment of LLMs' effectiveness in real-world clinical settings. It includes multiple tasks from 4 stages of a typical patient's hospital visit journey and comprises 12 datasets. MedJourney用于评估 LLM 在真实临床环境中的有效性，其中包含多个任务，涵盖来自患者就诊典型流程的4个阶段的12个数据集。","descriptions":{"en":"MedJourney offers a comprehensive assessment of LLMs' effectiveness in real-world clinical settings. It includes multiple tasks from 4 stages of a typical patient's hospital visit journey and comprises 12 datasets.","zh":"MedJourney用于评估 LLM 在真实临床环境中的有效性，其中包含多个任务，涵盖来自患者就诊典型流程的4个阶段的12个数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1326","languages":["Chinese"],"library_direction":null,"modality":null,"name":"MedJourney","openness":"unknown","publisher":"Tencent Youtu Lab","released":"2024-09-26","released_reference":{"basis":"paper_first_version","note":"The clinical-journey benchmark paper's public publication date on OpenReview. This is not the unrelated counterfactual-image MedJourney project.","source_key":"opencompass:1326","source_url":"https://openreview.net/forum?id=XXaIoJyYs7"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1326-medjourney","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedJourney","topics":["reasoning-math","language","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_medlayxplain_c048bbf8"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MedLayXPlain is a benchmark for medical lay language generation, pairing medical images with expert and lay captions across 122,789 samples from 8 imaging modalities. It introduces a 3B evaluator model scoring expert-lay alignment on five attributes.","descriptions":{"en":"MedLayXPlain is a benchmark for medical lay language generation, pairing medical images with expert and lay captions across 122,789 samples from 8 imaging modalities. It introduces a 3B evaluator model scoring expert-lay alignment on five attributes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21194","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medlayxplain_c048bbf8"},"modality":null,"name":"MEDLAYXPLAIN","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21194","source_url":"https://arxiv.org/abs/2606.21194"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21194","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21194","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_medpic-bench_f4184b8f"},"categories":["Biology & Drug Discovery","Cybersecurity","Reasoning","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"MedPIC-Bench evaluates patient-specific medication-safety reasoning in LLMs using guideline-following and paired counterfactual questions, with 467 questions annotated across six dimensions.","descriptions":{"en":"MedPIC-Bench evaluates patient-specific medication-safety reasoning in LLMs using guideline-following and paired counterfactual questions, with 467 questions annotated across six dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03028","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","safety-alignment"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medpic-bench_f4184b8f"},"modality":null,"name":"MedPIC-Bench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03028","source_url":"https://arxiv.org/abs/2608.03028"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03028","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03028","topics":["reasoning-math","safety","cybersecurity","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_medpress_3e59debc"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MedPRESS evaluates LLM sycophancy in multi-turn medical dialogues, containing 600 five-turn scenarios across three families with structured judging and safety metrics.","descriptions":{"en":"MedPRESS evaluates LLM sycophancy in multi-turn medical dialogues, containing 600 five-turn scenarios across three families with structured judging and safety metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02520","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","health-medicine"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_medpress_3e59debc"},"modality":null,"name":"MedPRESS","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02520","source_url":"https://arxiv.org/abs/2608.02520"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02520","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02520","topics":["safety","medical"],"unit":null},{"aliases":["MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation"],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"A multimodal next-response generation benchmark derived from de-identified Chinese online medical consultations, with case-specific physician-refined rubrics.","descriptions":{"en":"A multimodal next-response generation benchmark derived from de-identified Chinese online medical consultations, with case-specific physician-refined rubrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09142","languages":[],"library_direction":null,"modality":null,"name":"MedRealMM","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09142","source_url":"https://arxiv.org/abs/2607.09142"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09142","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09142","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on multimodal clinical diagnostic synthesis using 625 expert-validated cases with medical images and ICD-11 diagnoses.","descriptions":{"en":"Evaluates LLMs on multimodal clinical diagnostic synthesis using 625 expert-validated cases with medical images and ICD-11 diagnoses."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22323","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medreamm_d93bafa9"},"modality":null,"name":"MedReaMM","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22323","source_url":"http://arxiv.org/abs/2608.22323v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22323","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22323v1","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","NeurIPS 2024","科学智能","AI for Science","安全对齐","Safety Alignment","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MedSafetyBench is designed to measure the medical safety of LLMs. It includes 1,800 medical safety demonstrations, where each safety demonstration consists of a harmful medical request and a corresponding safe response. MedSafetyBench用于评估LLM在医疗安全上的表现，包含1800个由有害请求和安全响应组成的医疗安全场景。","descriptions":{"en":"MedSafetyBench is designed to measure the medical safety of LLMs. It includes 1,800 medical safety demonstrations, where each safety demonstration consists of a harmful medical request and a corresponding safe response.","zh":"MedSafetyBench用于评估LLM在医疗安全上的表现，包含1800个由有害请求和安全响应组成的医疗安全场景。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1331","languages":[],"library_direction":null,"modality":null,"name":"MedSafetyBench","openness":"unknown","publisher":"Harvard University","released":"2024-03-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1331-medsafetybench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedSafetyBench","topics":["reasoning-math","safety","science"],"unit":null},{"aliases":["MEDSEGBENCH"],"attention_signals":null,"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"The framework supports configuration-driven benchmarking of 2D medical image segmentation, exporting raw predictions and pixel counts for post-hoc analysis.","descriptions":{"en":"The framework supports configuration-driven benchmarking of 2D medical image segmentation, exporting raw predictions and pixel counts for post-hoc analysis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29677","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","health-medicine"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_medsegbench_8b7566b7"},"modality":null,"name":"MEDSEGBENCHMARKER","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29677","source_url":"https://arxiv.org/abs/2608.29677"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29677","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29677","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":216,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_medstreambench_73e2ea6c"},"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MedStreamBench is a time-aware benchmark for medical video understanding, integrating 22 medical datasets and 5,419 QA instances across four temporal settings: retrospective, present, future, and proactive. Models are restricted to temporally bounded evidence windows and evaluated on answer correctness, responsiveness, and post-evidence stability.","descriptions":{"en":"MedStreamBench is a time-aware benchmark for medical video understanding, integrating 22 medical datasets and 5,419 QA instances across four temporal settings: retrospective, present, future, and proactive. Models are restricted to temporally bounded evidence windows and evaluated on answer correctness, responsiveness, and post-evidence stability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01751","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","video-understanding"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_medstreambench_73e2ea6c"},"modality":null,"name":"MedStreamBench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01751","source_url":"https://arxiv.org/abs/2607.01751"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01751","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01751","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":182,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":25,"join":"llm_stats_id","repository":null,"source_record":"catalog_c72ce7241164b028"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning, featuring 4,460 questions spanning 17 specialties and 11 body systems. Includes both text-only and multimodal subsets with expert-level exam questions incorporating diverse medical images and rich clinical information.","descriptions":{"en":"A comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning, featuring 4,460 questions spanning 17 specialties and 11 body systems. Includes both text-only and multimodal subsets with expert-level exam questions incorporating diverse medical images and rich clinical information."},"evidence_summary":{"document_count":1,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:medxpertqa:medgemma-4b-it","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:medxpertqa","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_c72ce7241164b028"},"modality":"multimodal","name":"MedXpertQA","openness":"unknown","publication":{"evidence":[{"note":"The authors' project site names the Tsinghua University and Shanghai AI Laboratory team and directly links this paper and TsinghuaC3I repository. It separates Text and MM results.","url":"https://medxpertqa.github.io/"},{"note":"The official repository documents 4,460 questions, 17 specialties and 11 body systems, with Text and MM subsets, matching the source record's task and counts. The paper and repository both opened successfully.","url":"https://github.com/TsinghuaC3I/MedXpertQA/blob/main/README.md"}],"id":"medxpertqa-original","links":[{"kind":"paper","title":"MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding","url":"https://arxiv.org/abs/2501.18362"},{"kind":"repo","title":"MedXpertQA official repository","url":"https://github.com/TsinghuaC3I/MedXpertQA"}],"record_keys":["llm-stats:medxpertqa"],"reviewed_at":"2026-10-26","scope":{"en":"Original MedXpertQA suite, containing separate Text and MM subsets. The source record describes the suite and does not establish a specific modality, reasoning subset or aggregate for its scores.","zh":"原始 MedXpertQA 套件包含独立的 Text 和 MM 子集。当前来源记录介绍整个套件，未明确成绩对应的模态、推理子集或汇总方式。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":12,"score_direction":"higher_is_better","score_summary":{"display_max":78.4,"display_multiplier":100,"model_count":12,"model_count_basis":"source_model_id","numeric_count":12,"raw_max":0.784,"raw_min":0.188,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:medxpertqa:muse-spark","reported_date":"2026-04-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2025-05-20","date_precision":"model_announcement","name":"MedGemma 4B IT"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-medxpertqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa?top_n=500","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":182,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":25,"join":"artifact:arxiv:2501.18362","repository":"gh:tsinghuac3i/medxpertqa","source_record":"catalog_33f4bd7a3190e5d9"},"categories":["学科","Examination","推理","Reasoning","知识","Knowledge","Medical","ICML 2025","科学智能","AI for Science","知识储备","逻辑推理","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"We introduce MedXpertQA, a highly challenging and comprehensive medical benchmark to evaluate expert-level medical knowledge and advanced reasoning. \nIt has been accepted by ICML 2025 and selected by Google DeepMind as the benchmark for MedGemma. MedXpertQA是由清华大学和上海人工智能实验室构建的全面且具有高度挑战性的医学基准，用于评估专家级的医学知识和高级推理能力。论文已被ICML 2025接收，并被Google DeepMind使用作为MedGemma的评估基准。MedXpertQA 共包含 4,460 道题目，涵盖 17 个医学专科和 11 个身体系统。该基准包含两个子集：用于文本医学能力评估的 Text 子集，以及用于多模态医学能力评估的 MM 子集。MM 子集首次引入了带有多样化图像和丰富临床信息（如病历和检查结果）的专家级考试题，区别于传统多模态医学基准中基于图像描述生成的简单问答对。","descriptions":{"en":"We introduce MedXpertQA, a highly challenging and comprehensive medical benchmark to evaluate expert-level medical knowledge and advanced reasoning. \nIt has been accepted by ICML 2025 and selected by Google DeepMind as the benchmark for MedGemma.","zh":"MedXpertQA是由清华大学和上海人工智能实验室构建的全面且具有高度挑战性的医学基准，用于评估专家级的医学知识和高级推理能力。论文已被ICML 2025接收，并被Google DeepMind使用作为MedGemma的评估基准。MedXpertQA 共包含 4,460 道题目，涵盖 17 个医学专科和 11 个身体系统。该基准包含两个子集：用于文本医学能力评估的 Text 子集，以及用于多模态医学能力评估的 MM 子集。MM 子集首次引入了带有多样化图像和丰富临床信息（如病历和检查结果）的专家级考试题，区别于传统多模态医学基准中基于图像描述生成的简单问答对。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1891","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"artifact:arxiv:2501.18362","primary":"health-medicine","source_record":"catalog_33f4bd7a3190e5d9"},"modality":null,"name":"MedXpertQA","openness":"open","publisher":"清华大学，上海人工智能实验室","released":"2025-02-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1891-medxpertqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedXpertQA","topics":["reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":182,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":25,"join":"llm_stats_id","repository":"gh:tsinghuac3i/medxpertqa","source_record":"catalog_33f4bd7a3190e5d9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["medical","multimodal","knowledge","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MedXpertQA-MM is the multimodal subset of MedXpertQA, evaluating expert-level medical question answering grounded in medical images.","descriptions":{"en":"MedXpertQA-MM is the multimodal subset of MedXpertQA, evaluating expert-level medical question answering grounded in medical images."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:medxpertqa-mm:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa-mm?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:medxpertqa-mm","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_33f4bd7a3190e5d9"},"modality":"multimodal","name":"MedXpertQA-MM","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":71.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.71,"raw_min":0.71,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:medxpertqa-mm:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa-mm?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-medxpertqa-mm","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa-mm?top_n=500","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_meetingtom_9e241987"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"MeetingToM evaluates multimodal LLMs on theory-of-mind reasoning in multi-party meetings, including pseudo-consensus detection, across three levels: subject, dyad, and group. It provides a unified evaluation protocol.","descriptions":{"en":"MeetingToM evaluates multimodal LLMs on theory-of-mind reasoning in multi-party meetings, including pseudo-consensus detection, across three levels: subject, dyad, and group. It provides a unified evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19235","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","abstract-reasoning"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_meetingtom_9e241987"},"modality":null,"name":"MeetingToM","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19235","source_url":"https://arxiv.org/abs/2607.19235"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19235","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19235","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":72,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:microsoft/multilingual-evaluation-of-generative-ai-mega","source_record":"catalog_8b815b926ea522f8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MLQA as part of the MEGA (Multilingual Evaluation of Generative AI) benchmark suite. A multi-way aligned extractive QA evaluation benchmark for cross-lingual question answering across 7 languages (English, Arabic, German, Spanish, Hindi, Vietnamese, and Simplified Chinese) with over 12K QA instances in English and 5K in each other language.","descriptions":{"en":"MLQA as part of the MEGA (Multilingual Evaluation of Generative AI) benchmark suite. A multi-way aligned extractive QA evaluation benchmark for cross-lingual question answering across 7 languages (English, Arabic, German, Spanish, Hindi, Vietnamese, and Simplified Chinese) with over 12K QA instances in English and 5K in each other language."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mega-mlqa:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-mlqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mega-mlqa","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_8b815b926ea522f8"},"modality":"text","name":"MEGA MLQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":65.3,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.653,"raw_min":0.617,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mega-mlqa:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-mlqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-mega-mlqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-mlqa?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":319,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:google-research-datasets/tydiqa","source_record":"catalog_c079a6f3139250aa"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"TyDi QA as part of the MEGA benchmark suite. A question answering dataset covering 11 typologically diverse languages (Arabic, Bengali, English, Finnish, Indonesian, Japanese, Korean, Russian, Swahili, Telugu, and Thai) with 204K question-answer pairs. Features realistic information-seeking questions written by people who want to know the answer but don't know it yet.","descriptions":{"en":"TyDi QA as part of the MEGA benchmark suite. A question answering dataset covering 11 typologically diverse languages (Arabic, Bengali, English, Finnish, Indonesian, Japanese, Korean, Russian, Swahili, Telugu, and Thai) with 204K question-answer pairs. Features realistic information-seeking questions written by people who want to know the answer but don't know it yet."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mega-tydi-qa:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-tydi-qa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mega-tydi-qa","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_c079a6f3139250aa"},"modality":"text","name":"MEGA TyDi QA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":67.10000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.671,"raw_min":0.622,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mega-tydi-qa:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-tydi-qa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-mega-tydi-qa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-tydi-qa?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":72,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:microsoft/multilingual-evaluation-of-generative-ai-mega","source_record":"catalog_d9dab411d32bb30d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Universal Dependencies POS tagging as part of the MEGA benchmark suite. A multilingual part-of-speech tagging dataset based on Universal Dependencies treebanks, utilizing the universal POS tag set of 17 tags across 38 diverse languages from different language families. Used for evaluating multilingual POS tagging systems.","descriptions":{"en":"Universal Dependencies POS tagging as part of the MEGA benchmark suite. A multilingual part-of-speech tagging dataset based on Universal Dependencies treebanks, utilizing the universal POS tag set of 17 tags across 38 diverse languages from different language families. Used for evaluating multilingual POS tagging systems."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mega-udpos:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-udpos?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mega-udpos","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_d9dab411d32bb30d"},"modality":"text","name":"MEGA UDPOS","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":60.4,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.604,"raw_min":0.465,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mega-udpos:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-udpos?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-mega-udpos","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-udpos?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":105,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:cambridgeltl/xcopa","source_record":"catalog_bda51ce98572a0ed"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"XCOPA (Cross-lingual Choice of Plausible Alternatives) as part of the MEGA benchmark suite. A typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages, including resource-poor languages like Eastern Apurímac Quechua and Haitian Creole. Requires models to select which choice is the effect or cause of a given premise.","descriptions":{"en":"XCOPA (Cross-lingual Choice of Plausible Alternatives) as part of the MEGA benchmark suite. A typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages, including resource-poor languages like Eastern Apurímac Quechua and Haitian Creole. Requires models to select which choice is the effect or cause of a given premise."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mega-xcopa:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xcopa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mega-xcopa","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","abstract-reasoning"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_bda51ce98572a0ed"},"modality":"text","name":"MEGA XCOPA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":76.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.766,"raw_min":0.631,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mega-xcopa:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xcopa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-mega-xcopa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xcopa?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":72,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:microsoft/multilingual-evaluation-of-generative-ai-mega","source_record":"catalog_93ac083d8fb4e9cd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"XStoryCloze as part of the MEGA benchmark suite. A cross-lingual story completion task that consists of professionally translated versions of the English StoryCloze dataset to 10 non-English languages. Requires models to predict the correct ending for a given four-sentence story, evaluating commonsense reasoning and narrative understanding.","descriptions":{"en":"XStoryCloze as part of the MEGA benchmark suite. A cross-lingual story completion task that consists of professionally translated versions of the English StoryCloze dataset to 10 non-English languages. Requires models to predict the correct ending for a given four-sentence story, evaluating commonsense reasoning and narrative understanding."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mega-xstorycloze:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xstorycloze?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mega-xstorycloze","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_93ac083d8fb4e9cd"},"modality":"text","name":"MEGA XStoryCloze","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":82.8,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.828,"raw_min":0.735,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mega-xstorycloze:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xstorycloze?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-mega-xstorycloze","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xstorycloze?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_cfe6a6676d638dfd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","psychology","creativity"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MELD (Multimodal EmotionLines Dataset) is a multimodal multi-party dataset for emotion recognition in conversations. Contains approximately 13,000 utterances from 1,433 dialogues extracted from the TV series Friends. Each utterance is annotated with emotion (Anger, Disgust, Sadness, Joy, Neutral, Surprise, Fear) and sentiment labels across audio, visual, and textual modalities.","descriptions":{"en":"MELD (Multimodal EmotionLines Dataset) is a multimodal multi-party dataset for emotion recognition in conversations. Contains approximately 13,000 utterances from 1,433 dialogues extracted from the TV series Friends. Each utterance is annotated with emotion (Anger, Disgust, Sadness, Joy, Neutral, Surprise, Fear) and sentiment labels across audio, visual, and textual modalities."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:meld:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/meld?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:meld","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_cfe6a6676d638dfd"},"modality":"multimodal","name":"Meld","openness":"unknown","publisher":null,"released":"2018-10-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:meld","source_url":"https://arxiv.org/abs/1810.02508"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":56.99999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.57,"raw_min":0.57,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:meld:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/meld?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-meld","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/meld?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"MemCalib is a benchmark for evaluating memory use in LLM agents in realistic memory-system scenarios.","descriptions":{"en":"MemCalib is a benchmark for evaluating memory use in LLM agents in realistic memory-system scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24259","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_memcalib_204ed9ed"},"modality":null,"name":"MemCalib","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24259","source_url":"https://arxiv.org/abs/2609.24259"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24259","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24259","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MemeBench is a diagnostic benchmark of 1,253 Chinese and English memes with human-written references and VIKR annotations—Visual clues, Identity links, Knowledge units, and Reasoning mechanisms—for evaluating interpretation in LVLMs.","descriptions":{"en":"MemeBench is a diagnostic benchmark of 1,253 Chinese and English memes with human-written references and VIKR annotations—Visual clues, Identity links, Knowledge units, and Reasoning mechanisms—for evaluating interpretation in LVLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27798","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_memebench_baa40d97"},"modality":null,"name":"MemeBench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27798","source_url":"https://arxiv.org/abs/2607.27798"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27798","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27798","topics":[],"unit":null},{"aliases":["Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classification"],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:akshitsharma1/memepivot","source_record":"bm_small-cues-big-consequences-learning-pivot_7408ad17"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A cue-focused benchmark of 9,895 memes across harm, hate, and sarcasm, with annotations identifying the modality and rationale of the pivotal evidence.","descriptions":{"en":"A cue-focused benchmark of 9,895 memes across harm, hate, and sarcasm, with annotations identifying the modality and rationale of the pivotal evidence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.26907","languages":[],"library_direction":null,"modality":null,"name":"MemeCF","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.26907","source_url":"https://arxiv.org/abs/2609.26907"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-26907","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26907","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi paired with cultural context notes and three human-written explanations, plus 54 Bengali dialect memes, for evaluating vision-language models' meme explanation generation.","descriptions":{"en":"A multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi paired with cultural context notes and three human-written explanations, plus 54 Bengali dialect memes, for evaluating vision-language models' meme explanation generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01772","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","multilingual"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_memecult-1k_b178ceb1"},"modality":null,"name":"MemeCULT-1K","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01772","source_url":"https://arxiv.org/abs/2609.01772"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01772","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01772","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_memfail_db6b29b0"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Diagnostic benchmark isolating failure modes of LLM memory systems by evaluating summarization, storage, and retrieval operations.","descriptions":{"en":"Diagnostic benchmark isolating failure modes of LLM memory systems by evaluating summarization, storage, and retrieval operations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26667","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_memfail_db6b29b0"},"modality":null,"name":"MemFail","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26667","source_url":"https://arxiv.org/abs/2605.26667"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26667","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26667","topics":["agents"],"unit":null},{"aliases":["Profile-Graph Memory for LLM Agents: Implicit Cross-Entity Traversal through Narrative Profiles"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A multi-hop long-term memory benchmark for LLM agents: 1,000 questions at hop depths 1–5 across 10 social-network scenarios, with per-hop evidence annotations.","descriptions":{"en":"A multi-hop long-term memory benchmark for LLM agents: 1,000 questions at hop depths 1–5 across 10 social-network scenarios, with per-hop evidence annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.19359","languages":[],"library_direction":null,"modality":null,"name":"MemHop","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19359","source_url":"https://arxiv.org/abs/2607.19359"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19359","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19359","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":46,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:memobench-team/memobench","source_record":"bm_memobench_def5688a"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MemoBench evaluates world modeling in video generation via a disappear-and-reappear paradigm. It includes 360 ground-truth clips and an evaluation suite with automated metrics and VQA across four diagnostic pillars.","descriptions":{"en":"MemoBench evaluates world modeling in video generation via a disappear-and-reappear paradigm. It includes 360 ground-truth clips and an evaluation suite with automated metrics and VQA across four diagnostic pillars."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.27537","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","world-models"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_memobench_def5688a"},"modality":null,"name":"MemoBench","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27537","source_url":"https://arxiv.org/abs/2606.27537"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27537","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27537","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:collab-gen/memobench","source_record":"bm_memobench_dd60fcde"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates robotic manipulation policies on memory fidelity via 30 history-dependent tasks with annotated checkpoints measuring storage, update, and compression operations.","descriptions":{"en":"Evaluates robotic manipulation policies on memory fidelity via 30 history-dependent tasks with annotated checkpoints measuring storage, update, and compression operations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.07047","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","agent-memory"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_memobench_dd60fcde"},"modality":null,"name":"MEMOBench","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07047","source_url":"https://arxiv.org/abs/2609.07047"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07047","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07047","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_memops_f8d37e23"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MemOps evaluates conversational memory as a sequence of lifecycle operations (remembering, forgetting, updating, reflecting) with structured traces and six categories of operation-level probes, under adjacent-evidence and long-context settings.","descriptions":{"en":"MemOps evaluates conversational memory as a sequence of lifecycle operations (remembering, forgetting, updating, reflecting) with structured traces and six categories of operation-level probes, under adjacent-evidence and long-context settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.12893","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_memops_f8d37e23"},"modality":null,"name":"MemOps","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.12893","source_url":"https://arxiv.org/abs/2607.12893"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-12893","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12893","topics":["agents"],"unit":null},{"aliases":["MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs"],"attention_signals":{"as_of":"2026-10-12","github_stars":11,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_memoreason-evaluating-the-effect-of-parame_1ca8d4c5"},"categories":["Factuality","General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Human-curated benchmark pairing factual reasoning tasks with structurally identical fictitious versions to measure parametric memory effects on contextual reasoning.","descriptions":{"en":"Human-curated benchmark pairing factual reasoning tasks with structurally identical fictitious versions to measure parametric memory effects on contextual reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35312","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","factuality"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_memoreason-evaluating-the-effect-of-parame_1ca8d4c5"},"modality":null,"name":"MemoReason","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35312","source_url":"https://arxiv.org/abs/2609.35312"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35312","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35312","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"MemoryDocDataSet evaluates joint conversational memory and long-document reasoning through 50 synthetic micro-worlds, each with personas, temporal event graphs, real legal documents, multi-session conversations, and QA pairs. Questions are categorized by reasoning type, with Hybrid questions requiring navigation of conversation history to locate the relevant document and extract answers.","descriptions":{"en":"MemoryDocDataSet evaluates joint conversational memory and long-document reasoning through 50 synthetic micro-worlds, each with personas, temporal event graphs, real legal documents, multi-session conversations, and QA pairs. Questions are categorized by reasoning type, with Hybrid questions requiring navigation of conversation history to locate the relevant document and extract answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04442","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","long-context"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_memorydocdataset_b8a1ebc4"},"modality":null,"name":"MemoryDocDataSet","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04442","source_url":"https://arxiv.org/abs/2606.04442"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04442","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04442","topics":["agents","reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_memprobe_9e043327"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"MEMPROBE evaluates long-term memory in LLM agents by reconstructing hidden user-state from agent memory across 50 simulated users with 31 hidden dimensions each.","descriptions":{"en":"MEMPROBE evaluates long-term memory in LLM agents by reconstructing hidden user-state from agent memory across 50 simulated users with 31 hidden dimensions each."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.24595","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_memprobe_9e043327"},"modality":null,"name":"MEMPROBE","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24595","source_url":"https://arxiv.org/abs/2606.24595"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24595","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24595","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-horizon LLM agents on 120 scripted episodes covering five memory-risk categories through deterministic trace-grounded checks, with full trace logging and no LLM-as-judge.","descriptions":{"en":"Evaluates long-horizon LLM agents on 120 scripted episodes covering five memory-risk categories through deterministic trace-grounded checks, with full trace logging and no LLM-as-judge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.14976","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","agent-security"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_memriskbench_b0ebcc74"},"modality":null,"name":"MemRiskBench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.14976","source_url":"https://arxiv.org/abs/2609.14976"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-14976","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14976","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates lifecycle security of agent memory systems with 310 cases across 48 contexts, using a Write-Execute-Forget protocol and evidence-based adjudication across seven checkpoints.","descriptions":{"en":"Evaluates lifecycle security of agent memory systems with 310 cases across 48 contexts, using a Write-Execute-Forget protocol and evidence-based adjudication across seven checkpoints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27080","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agent-memory"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_memsecbench_ec92c168"},"modality":null,"name":"MemSecBench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27080","source_url":"https://arxiv.org/abs/2607.27080"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27080","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27080","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":20,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":26,"join":"claire_record_id","repository":"gh:xmudeeplit/memsyco-bench","source_record":"bm_memsyco-bench_f1a6a168"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"MemSyco-Bench evaluates memory-induced sycophancy in agent systems across five tasks, measuring how memory influences decision-making and personalization.","descriptions":{"en":"MemSyco-Bench evaluates memory-induced sycophancy in agent systems across five tasks, measuring how memory influences decision-making and personalization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.01071","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","agent-memory"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_memsyco-bench_f1a6a168"},"modality":null,"name":"MemSyco-Bench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01071","source_url":"https://arxiv.org/abs/2607.01071"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01071","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01071","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MemTrace evaluates long-term memory in LLM agents at the knowledge point level, probing memory age, question type, and evidence condition. Scoring is based on accuracy across these dimensions.","descriptions":{"en":"MemTrace evaluates long-term memory in LLM agents at the knowledge point level, probing memory age, question type, and evidence condition. Scoring is based on accuracy across these dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17328","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_memtrace_56029435"},"modality":null,"name":"MemTrace","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17328","source_url":"https://arxiv.org/abs/2606.17328"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17328","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17328","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_memtransfer_9a7ce06a"},"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Memory lets an embodied agent reuse past experience, yet retaining useful information does not ensure that the agent can apply it when conditions change.","descriptions":{"en":"Memory lets an embodied agent reuse past experience, yet retaining useful information does not ensure that the agent can apply it when conditions change."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32313","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","agent-memory"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_memtransfer_9a7ce06a"},"modality":null,"name":"MemTransfer","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32313","source_url":"https://arxiv.org/abs/2609.32313"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32313","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32313","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:shuruixu/meniomni","source_record":"bm_meniomni_7f0a7094"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MeniOmni is a multimodal benchmark for meniscus injury assessment with 746 MRI studies, supporting Stoller severity grading and diagnostic report generation, with risk-aware ordinal evaluation and semantic consistency metric.","descriptions":{"en":"MeniOmni is a multimodal benchmark for meniscus injury assessment with 746 MRI studies, supporting Stoller severity grading and diagnostic report generation, with risk-aware ordinal evaluation and semantic consistency metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.28161","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_meniomni_7f0a7094"},"modality":null,"name":"MeniOmni","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28161","source_url":"https://arxiv.org/abs/2605.28161"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28161","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28161","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"MEP-Bench evaluates whether multimodal emotion models use modality-specific evidence sufficiently and faithfully.","descriptions":{"en":"MEP-Bench evaluates whether multimodal emotion models use modality-specific evidence sufficiently and faithfully."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.25325","languages":[],"library_direction":null,"modality":null,"name":"MEP-Bench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25325","source_url":"https://arxiv.org/abs/2606.25325"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25325","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25325","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","MER","多模态模型","VLM","跨模态推理","Cross-modal Reasoning","音频理解","Audio Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MER-UniBench is a benchmark designed to evaluate multimodal large language models (MLLMs) for their emotion understanding capabilities across typical multimodal emotion recognition (MER) tasks. MER-UniBench 是一个旨在评估多模态大语言模型（MLLM）在典型多模态情感识别（MER）任务中情感理解能力的评测基准。它涵盖了细粒度情感识别、基本情感识别和情感分析三个主要维度。该基准利用了包括 MER-Caption 在内的多个数据集，其中 MER-Caption 拥有超过 2000 种细粒度情感类别和 11.5 万个样本。","descriptions":{"en":"MER-UniBench is a benchmark designed to evaluate multimodal large language models (MLLMs) for their emotion understanding capabilities across typical multimodal emotion recognition (MER) tasks.","zh":"MER-UniBench 是一个旨在评估多模态大语言模型（MLLM）在典型多模态情感识别（MER）任务中情感理解能力的评测基准。它涵盖了细粒度情感识别、基本情感识别和情感分析三个主要维度。该基准利用了包括 MER-Caption 在内的多个数据集，其中 MER-Caption 拥有超过 2000 种细粒度情感类别和 11.5 万个样本。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2079","languages":[],"library_direction":null,"modality":"multimodal","name":"MER-UniBench","openness":"restricted","publisher":"Institute of Automation, Chinese Academy of Sciences , CMVS, etc.","released":"2025-01-27","released_reference":{"basis":"paper_first_version","note":"The AffectGPT paper introduces MER-UniBench; the older MER-Caption paper is a different artifact.","source_key":"opencompass:2079","source_url":"https://arxiv.org/abs/2501.16566"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2079-mer-unibench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MER-UniBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["MER-FG","MER2026 Track 2","MER2026 MER-FG","MER 2026 fine-grained emotion recognition"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Track 2 of the MER2026 challenge: open-vocabulary, fine-grained emotion recognition from video, audio and subtitles, ranked on a 1,000-sample test set with emotion-wheel-based metrics that group synonyms (happy and joyful count as one label). Because the label space is open, how a system normalizes its vocabulary onto the wheel moves the score independently of recognition. Reference labels were built by merging top MER2025 team predictions and then manually validated.","descriptions":{"en":"Track 2 of the MER2026 challenge: open-vocabulary, fine-grained emotion recognition from video, audio and subtitles, ranked on a 1,000-sample test set with emotion-wheel-based metrics that group synonyms (happy and joyful count as one label). Because the label space is open, how a system normalizes its vocabulary onto the wheel moves the score independently of recognition. Reference labels were built by merging top MER2025 team predictions and then manually validated."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:mer2026_mer_fg","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["video-understanding"],"join":"review:14H034160212:2026-10-15","primary":"video-understanding","source_record":null},"modality":null,"name":"MER2026 MER-FG","openness":"unknown","publisher":null,"released":"2026-04-21","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mer2026_mer_fg","source_url":"https://arxiv.org/abs/2604.19417"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"mer2026_mer_fg","source":"model_reports","source_url":"https://arxiv.org/abs/2604.19417","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:medai-t/mergemedbench","source_record":"bm_mergemedbench_c8868c16"},"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MergeMedBench evaluates model merging methods for medical LVLMs across eight imaging modalities, comprising 16 LoRA fine-tuned models. It provides evaluation datasets and released model checkpoints for benchmarking merging approaches.","descriptions":{"en":"MergeMedBench evaluates model merging methods for medical LVLMs across eight imaging modalities, comprising 16 LoRA fine-tuned models. It provides evaluation datasets and released model checkpoints for benchmarking merging approaches."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.15661","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_mergemedbench_c8868c16"},"modality":null,"name":"MergeMedBench","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15661","source_url":"https://arxiv.org/abs/2607.15661"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15661","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15661","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.GR"],"collected_at":"2026-10-14T09:44:51Z","description":"The benchmark evaluates triangle-based neural rendering across optimization and game-engine deployment pipeline.","descriptions":{"en":"The benchmark evaluates triangle-based neural rendering across optimization and game-engine deployment pipeline."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01306","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_meshsplatbench_8a617de9"},"modality":null,"name":"MeshSplatBench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01306","source_url":"https://arxiv.org/abs/2609.01306"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01306","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01306","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Meta's internal evaluation of coding-agent performance.","descriptions":{"en":"Meta's internal evaluation of coding-agent performance."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:meta-internal-coding-bench:muse-spark-1.2","reported_at":"2026-08-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/meta-internal-coding-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:meta-internal-coding-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_4df8055b7a28417d"},"modality":"text","name":"Meta Internal Coding Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":70.6,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.706,"raw_min":0.706,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:meta-internal-coding-bench:muse-spark-1.2","reported_date":"2026-08-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/meta-internal-coding-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2"}],"slug":"llm-stats-meta-internal-coding-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/meta-internal-coding-bench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MetaboNet-Bench evaluates multimodal glucose forecasting in type 1 diabetes, using glucose, insulin, and carbohydrate data. It provides an extensible open-source evaluation framework for comparing forecasting algorithms.","descriptions":{"en":"MetaboNet-Bench evaluates multimodal glucose forecasting in type 1 diabetes, using glucose, insulin, and carbohydrate data. It provides an extensible open-source evaluation framework for comparing forecasting algorithms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18640","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_metabonet-bench_963d2082"},"modality":null,"name":"MetaboNet-Bench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18640","source_url":"https://arxiv.org/abs/2606.18640"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18640","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18640","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":11,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:icip-cas/metaphorvu","source_record":"bm_metaphorvu-bench_ab9e08cf"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MetaphorVU-Bench evaluates metaphorical video understanding in multimodal LLMs through tasks requiring cross-domain mapping, with a benchmark dataset and evaluation code publicly available.","descriptions":{"en":"MetaphorVU-Bench evaluates metaphorical video understanding in multimodal LLMs through tasks requiring cross-domain mapping, with a benchmark dataset and evaluation code publicly available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.25461","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_metaphorvu-bench_ab9e08cf"},"modality":null,"name":"MetaphorVU-Bench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25461","source_url":"https://arxiv.org/abs/2605.25461"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25461","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25461","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"MetaRoute-Bench is an open framework for comparing meta-decision policies in agentic workflow routing. Contains 180 synthetic task profiles, eight routing policies, and 30 seeds, evaluating success, cost, and latency through 43,200 traces. Metrics include success rate and paired confidence intervals.","descriptions":{"en":"MetaRoute-Bench is an open framework for comparing meta-decision policies in agentic workflow routing. Contains 180 synthetic task profiles, eight routing policies, and 30 seeds, evaluating success, cost, and latency through 43,200 traces. Metrics include success rate and paired confidence intervals."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00107","languages":[],"library_direction":null,"modality":null,"name":"MetaRoute-Bench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00107","source_url":"https://arxiv.org/abs/2608.00107"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00107","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00107","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Reasoning","Robotics & Embodied AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object would be metric-spatial reasoning in indoor scenes, but task details, environment, and scoring setup are not provided.","descriptions":{"en":"The evaluation object would be metric-spatial reasoning in indoor scenes, but task details, environment, and scoring setup are not provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.25841","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_metric-bench_05973452"},"modality":null,"name":"Metric-Bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25841","source_url":"https://arxiv.org/abs/2609.25841"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25841","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25841","topics":["reasoning-math"],"unit":null},{"aliases":["Metric-Bench"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A reconstructed test release for metric spatial understanding over indoor RGB images with supplied anchor measurements. Each example pairs one image and one English question with a numeric JSON reference answer, covering width, length, height, dimension, center, depth, elevation, distance and box measurements.","descriptions":{"en":"A reconstructed test release for metric spatial understanding over indoor RGB images with supplied anchor measurements. Each example pairs one image and one English question with a numeric JSON reference answer, covering width, length, height, dimension, center, depth, elevation, distance and box measurements."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:yulingxi/metric-bench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","spatial-3d","knowledge-expert-qa"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_metric-bench_e77780e5"},"modality":null,"name":"Metric-Bench Test","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:yulingxi/metric-bench","source_url":"https://huggingface.co/datasets/yulingxi/Metric-Bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-yulingxi-metric-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/yulingxi/Metric-Bench","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":28,"join":"claire_record_id","repository":"gh:continker/metrollm-bench","source_record":"bm_metrollm-bench_4cec77cf"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"MetroLLM-Bench evaluates language models as transit kiosk policy layers across six real metro systems. It includes 955 cases covering routing, fare calculation, disruptions, accessibility, and adversarial input, with models required to call structured tools and submit machine-renderable terminal states.","descriptions":{"en":"MetroLLM-Bench evaluates language models as transit kiosk policy layers across six real metro systems. It includes 955 cases covering routing, fare calculation, disruptions, accessibility, and adversarial input, with models required to call structured tools and submit machine-renderable terminal states."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.10016","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_metrollm-bench_4cec77cf"},"modality":null,"name":"MetroLLM-Bench","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10016","source_url":"https://arxiv.org/abs/2609.10016"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10016","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10016","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"MEventBench is a multi-event long video understanding benchmark introduced alongside the MoD-VLLM framework, evaluating temporal grounding and semantic understanding.","descriptions":{"en":"MEventBench is a multi-event long video understanding benchmark introduced alongside the MoD-VLLM framework, evaluating temporal grounding and semantic understanding."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.15778","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_meventbench_dad2f997"},"modality":null,"name":"MEventBench","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15778","source_url":"https://arxiv.org/abs/2607.15778"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15778","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15778","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MEWC is a benchmark that evaluates AI model performance on multi-environment web challenges, testing agents' ability to navigate and complete complex tasks across diverse web environments.","descriptions":{"en":"MEWC is a benchmark that evaluates AI model performance on multi-environment web challenges, testing agents' ability to navigate and complete complex tasks across diverse web environments."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mewc:minimax-m2.5","reported_at":"2026-02-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mewc?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mewc","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_19459b4c43b239f7"},"modality":"text","name":"MEWC","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":74.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.744,"raw_min":0.744,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mewc:minimax-m2.5","reported_date":"2026-02-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mewc?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax M2.5"}],"slug":"llm-stats-mewc","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mewc?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":279,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:google-research/url-nlp","source_record":"catalog_92041218cde9f5c1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.","descriptions":{"en":"MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities."},"evidence_summary":{"document_count":1,"model_count":31,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mgsm:gpt-3.5-turbo-0125","reported_at":"2023-03-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mgsm?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:mgsm","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","math"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_92041218cde9f5c1"},"modality":"text","name":"MGSM","openness":"open","publisher":null,"released":"2022-10-06","released_reference":null,"repo_kind":"monorepo_subdir","repo_resolution_status":"resolved","score_count":31,"score_direction":"higher_is_better","score_summary":{"display_max":92.30000000000001,"display_multiplier":100,"model_count":31,"model_count_basis":"source_model_id","numeric_count":31,"raw_max":0.923,"raw_min":0.479,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mgsm:llama-4-maverick","reported_date":"2025-04-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mgsm?top_n=500"},"unit":null},"scored_models":[{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2023-03-21","date_precision":"model_announcement","name":"GPT-3.5 Turbo"},{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"},{"date":"2024-04-09","date_precision":"model_announcement","name":"GPT-4 Turbo"},{"date":"2024-05-13","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 70B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"}],"slug":"llm-stats-mgsm","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mgsm?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["When Names Cross Scripts: A Source-Grounded Benchmark for Historical Entity Reconciliation in the Mongol World"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A provenance-controlled benchmark for historical entity reconciliation, comprising a Name-only core of 396 pairs and a Source-grounded subset of 160 pairs with entity-disjoint splits.","descriptions":{"en":"A provenance-controlled benchmark for historical entity reconciliation, comprising a Name-only core of 396 pairs and a Source-grounded subset of 160 pairs with entity-disjoint splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23507","languages":[],"library_direction":null,"modality":null,"name":"MHER","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23507","source_url":"http://arxiv.org/abs/2608.23507v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23507","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23507v1","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":38,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:apple/ml-mia-bench","source_record":"catalog_0e92240721888416"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","instruction_following","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MIABench evaluates multimodal instruction alignment and following capabilities.","descriptions":{"en":"MIABench evaluates multimodal instruction alignment and following capabilities."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:miabench:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/miabench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:miabench","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","visual-reasoning"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_0e92240721888416"},"modality":"multimodal","name":"MIABench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.927,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.927,"raw_min":0.927,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:miabench:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/miabench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-miabench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/miabench?top_n=500","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MIB is a benchmark designed to evaluate mechanistic interpretability methods for neural language models. MIB 是一个旨在评估神经网络语言模型中机械可解释性方法的基准。它主要评估方法在精确地定位因果路径（电路定位）和特定概念（因果变量定位）方面的能力，侧重于忠实度和最小化电路规模。该基准涵盖了IOI和算术等四个任务，并评估了Llama-3.1和Gemma-2等四种模型。","descriptions":{"en":"MIB is a benchmark designed to evaluate mechanistic interpretability methods for neural language models.","zh":"MIB 是一个旨在评估神经网络语言模型中机械可解释性方法的基准。它主要评估方法在精确地定位因果路径（电路定位）和特定概念（因果变量定位）方面的能力，侧重于忠实度和最小化电路规模。该基准涵盖了IOI和算术等四个任务，并评估了Llama-3.1和Gemma-2等四种模型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2085","languages":[],"library_direction":null,"modality":null,"name":"MIB","openness":"unknown","publisher":"Boston University , Pr(AI) RGroup , Ai2 , Technion ,etc.","released":"2025-04-17","released_reference":{"basis":"paper_first_version","note":"MIB's own introduction, not the earlier interpretability work cited in its repository.","source_key":"opencompass:2085","source_url":"https://arxiv.org/abs/2504.13151"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2085-mib","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MIB","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ukplab/arxiv2026-mic","source_record":"bm_mic-bench_1dce100d"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MIC-Bench evaluates image-claim consistency in multimodal misinformation using 8,812 instances across nine inconsistency types, with verdict, inconsistency type, visual evidence, and explanation scoring.","descriptions":{"en":"MIC-Bench evaluates image-claim consistency in multimodal misinformation using 8,812 instances across nine inconsistency types, with verdict, inconsistency type, visual evidence, and explanation scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.33441","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mic-bench_1dce100d"},"modality":null,"name":"MIC-Bench","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33441","source_url":"https://arxiv.org/abs/2609.33441"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33441","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33441","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","科学智能","AI for Science","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MicroVQA is a benchmark that evaluates LLM reasoning on multiple-choice questions about microscopy images, created by expert biologists. MicroVQA，一个评估关于显微镜图像的多选题推理基准，由专家生物学家创建，旨在反映生物研究中能够有意义地协助的任务，每个问题都需要多模态推理。","descriptions":{"en":"MicroVQA is a benchmark that evaluates LLM reasoning on multiple-choice questions about microscopy images, created by expert biologists.","zh":"MicroVQA，一个评估关于显微镜图像的多选题推理基准，由专家生物学家创建，旨在反映生物研究中能够有意义地协助的任务，每个问题都需要多模态推理。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1667","languages":[],"library_direction":null,"modality":"multimodal","name":"MicroVQA","openness":"open","publisher":"Stanford University, Tsinghua University, etc.","released":"2025-03-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1667-microvqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MicroVQA","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","检索能力","Retrieval","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Massive Image Embedding Benchmark (MIEB) is to evaluate the performance of image and image-text embedding models across the broadest spectrum to date. MIEB用于评估图像和图像文本嵌入模型在迄今为止最广泛的范围内的性能。","descriptions":{"en":"Massive Image Embedding Benchmark (MIEB) is to evaluate the performance of image and image-text embedding models across the broadest spectrum to date.","zh":"MIEB用于评估图像和图像文本嵌入模型在迄今为止最广泛的范围内的性能。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1781","languages":["Chinese","German","Multilingual"],"library_direction":null,"modality":"multimodal","name":"MIEB","openness":"unknown","publisher":"Durham University，Zendesk，etc.","released":"2025-04-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1781-mieb","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MIEB","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fahmiwol/migancore","source_record":"bm_migancore_8b59e149"},"categories":["Geometric reasoning","Software & AI Compute","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates Indonesian LLM hallucination and abstention behavior on a 36-question battery, reporting fabrication, over-refusal, and fact accuracy with confidence intervals.","descriptions":{"en":"Evaluates Indonesian LLM hallucination and abstention behavior on a 36-question battery, reporting fabrication, over-refusal, and fact accuracy with confidence intervals."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:fahmiwol/migancore","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","multilingual"],"join":"claire_record_id","primary":"factuality","source_record":"bm_migancore_8b59e149"},"modality":null,"name":"MiganCore","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:fahmiwol/migancore","source_url":"https://github.com/fahmiwol/migancore"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-fahmiwol-migancore","source":"claire_radar","source_url":"https://github.com/fahmiwol/migancore","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MiGUE-Bench is a benchmark for multi-granularity event analysis, covering event detection, relation reasoning, structure induction, and future prediction across single to cross-document settings. It uses an LLM-driven annotation pipeline.","descriptions":{"en":"MiGUE-Bench is a benchmark for multi-granularity event analysis, covering event detection, relation reasoning, structure induction, and future prediction across single to cross-document settings. It uses an LLM-driven annotation pipeline."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27654","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","forecasting"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_migue-bench_dbd75fa4"},"modality":null,"name":"MiGUE-Bench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27654","source_url":"https://arxiv.org/abs/2607.27654"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27654","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27654","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MiLiC-Eval is an NLP evaluation suite for Minority Languages in China, covering Tibetan (bo), Uyghur (ug), Kazakh (kk, in the Kazakh Arabic script), and Mongolian (mn, in the traditional Mongolian script). MiLiC-Eval 是针对中国少数民族语言的 NLP 评估套件，涵盖藏语（bo）、维吾尔语（ug）、哈萨克语（kk，使用哈萨克阿拉伯文脚本）和蒙古语（mn，使用传统蒙古文脚本）。","descriptions":{"en":"MiLiC-Eval is an NLP evaluation suite for Minority Languages in China, covering Tibetan (bo), Uyghur (ug), Kazakh (kk, in the Kazakh Arabic script), and Mongolian (mn, in the traditional Mongolian script).","zh":"MiLiC-Eval 是针对中国少数民族语言的 NLP 评估套件，涵盖藏语（bo）、维吾尔语（ug）、哈萨克语（kk，使用哈萨克阿拉伯文脚本）和蒙古语（mn，使用传统蒙古文脚本）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1658","languages":["Arabic","Multilingual"],"library_direction":null,"modality":null,"name":"MiLiC-Eval","openness":"unknown","publisher":"PKU","released":"2025-03-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1658-milic-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MiLiC-Eval","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":306,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:mit-lcp/mimic-cxr","source_record":"catalog_8fad0036a02a982d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MIMIC-CXR is a large publicly available dataset of chest radiographs with free-text radiology reports. Contains 377,110 images corresponding to 227,835 radiographic studies from 65,379 patients at Beth Israel Deaconess Medical Center. The dataset is de-identified and widely used for medical imaging research, automated report generation, and medical AI development.","descriptions":{"en":"MIMIC-CXR is a large publicly available dataset of chest radiographs with free-text radiology reports. Contains 377,110 images corresponding to 227,835 radiographic studies from 65,379 patients at Beth Israel Deaconess Medical Center. The dataset is de-identified and widely used for medical imaging research, automated report generation, and medical AI development."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mimic-cxr:medgemma-4b-it","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mimic-cxr?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mimic-cxr","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_8fad0036a02a982d"},"modality":"multimodal","name":"MIMIC CXR","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":88.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.889,"raw_min":0.889,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mimic-cxr:medgemma-4b-it","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mimic-cxr?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-20","date_precision":"model_announcement","name":"MedGemma 4B IT"}],"slug":"llm-stats-mimic-cxr","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mimic-cxr?top_n=500","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MiMo Coding Bench evaluates coding-agent capabilities on software engineering tasks reported with the MiMo model family.","descriptions":{"en":"MiMo Coding Bench evaluates coding-agent capabilities on software engineering tasks reported with the MiMo model family."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mimo-coding-bench:mimo-v2.5","reported_at":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mimo-coding-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mimo-coding-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_82c06ed0fe65d988"},"modality":"text","name":"MiMo Coding Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":73.7,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.737,"raw_min":0.718,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mimo-coding-bench:mimo-v2.5-pro","reported_date":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mimo-coding-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"}],"slug":"llm-stats-mimo-coding-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mimo-coding-bench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for grounded minimal-change understanding in image pairs.","descriptions":{"en":"A benchmark for grounded minimal-change understanding in image pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23336","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mincu_107232dc"},"modality":null,"name":"MinCU","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23336","source_url":"https://arxiv.org/abs/2609.23336"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23336","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23336","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robustness","Safety & Trustworthiness","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A stress test benchmark for PII detection systems across seven distribution shifts, used to compare failure modes of SpaCy, Presidio, and Qwen2.5-3B.","descriptions":{"en":"A stress test benchmark for PII detection systems across seven distribution shifts, used to compare failure modes of SpaCy, Presidio, and Qwen2.5-3B."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.03464","languages":[],"library_direction":null,"modality":null,"name":"Mind the Gap: Robustness Risks in PII Detection Systems","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03464","source_url":"https://arxiv.org/abs/2609.03464"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03464","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03464","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1032,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"artifact:gh:osu-nlp-group/mind2web","repository":"gh:osu-nlp-group/mind2web","source_record":"catalog_37a7875647e70c76"},"categories":["其他","Other","智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MIND2WEB is the first dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website. MIND2WEB 是首个用于开发和评估通用网页代理的数据集，能够根据语言指令在任何网站上完成复杂任务。","descriptions":{"en":"MIND2WEB is the first dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website.","zh":"MIND2WEB 是首个用于开发和评估通用网页代理的数据集，能够根据语言指令在任何网站上完成复杂任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1125","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"artifact:gh:osu-nlp-group/mind2web","primary":"computer-use","source_record":"catalog_37a7875647e70c76"},"modality":null,"name":"Mind2Web","openness":"unknown","publisher":"The Ohio State University","released":"2023-12-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1125-mind2web","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Mind2Web","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":133,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_mindedit-bench_02b6e953"},"categories":["Geometric reasoning","Mobile & Personal Computing","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"MindEdit-Bench evaluates counterfactual spatial reasoning in VLMs with 1,003 multiple-choice questions from private indoor scenes, covering six task types.","descriptions":{"en":"MindEdit-Bench evaluates counterfactual spatial reasoning in VLMs with 1,003 multiple-choice questions from private indoor scenes, covering six task types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.00491","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_mindedit-bench_02b6e953"},"modality":null,"name":"MindEdit-Bench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00491","source_url":"https://arxiv.org/abs/2607.00491"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00491","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00491","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mll-lab-nu/mindtopo","source_record":"bm_mindtopo_9ec3d9b3"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluation of topological intuition across continuity, separation, order, enclosure, and knots grounded in cognitive science.","descriptions":{"en":"Evaluation of topological intuition across continuity, separation, order, enclosure, and knots grounded in cognitive science."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11900","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning","science"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_mindtopo_9ec3d9b3"},"modality":null,"name":"MindTopo","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11900","source_url":"https://arxiv.org/abs/2609.11900"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11900","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11900","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Advances in foundation models are driving efforts to introduce agents to assist people in the physical world.","descriptions":{"en":"Advances in foundation models are driving efforts to introduce agents to assist people in the physical world."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11328","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_mine-odyssey_ab06810a"},"modality":null,"name":"Mine Odyssey","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11328","source_url":"https://arxiv.org/abs/2610.11328"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11328","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11328","topics":[],"unit":null},{"aliases":["MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft"],"attention_signals":null,"categories":["Language & Knowledge","Robotics & Embodied AI","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A Minecraft benchmark that composes atomic tasks into implicit multi-hop exploration tasks for MLLM agents, scored by rule-based milestone evaluators.","descriptions":{"en":"A Minecraft benchmark that composes atomic tasks into implicit multi-hop exploration tasks for MLLM agents, scored by rule-based milestone evaluators."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.30931","languages":[],"library_direction":null,"modality":null,"name":"MineExplorer","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30931","source_url":"https://arxiv.org/abs/2605.30931"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30931","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30931","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":99,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:google-deepmind/neptune","source_record":"catalog_69ebd5b4d828836c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Minerva is a benchmark for complex video reasoning, evaluating models on multi-step reasoning over long and information-dense video content.","descriptions":{"en":"Minerva is a benchmark for complex video reasoning, evaluating models on multi-step reasoning over long and information-dense video content."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:minerva:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/minerva?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:minerva","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_69ebd5b4d828836c"},"modality":"multimodal","name":"Minerva","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":70.7,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.707,"raw_min":0.659,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:minerva:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/minerva?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-minerva","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/minerva?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"MineXplore is an open-source MuJoCo-based benchmark for robot navigation in GNSS-denied underground mine environments, derived from the Leung et al. 2017 Chilean copper mine dataset. It reconstructs a 104,423 sq.m tunnel network with octagonal wall cross-sections, LiDAR-sourced jagged wall geometry, three terrain friction zones, a global 5 degree incline, and periodic spot lighting. The benchmark includes an evaluation protocol based on coverage percentage, with a 90% target for single-agent PPO policies.","descriptions":{"en":"MineXplore is an open-source MuJoCo-based benchmark for robot navigation in GNSS-denied underground mine environments, derived from the Leung et al. 2017 Chilean copper mine dataset. It reconstructs a 104,423 sq.m tunnel network with octagonal wall cross-sections, LiDAR-sourced jagged wall geometry, three terrain friction zones, a global 5 degree incline, and periodic spot lighting. The benchmark includes an evaluation protocol based on coverage percentage, with a 90% target for single-agent PPO policies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04569","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_minexplore_36c767cc"},"modality":null,"name":"MineXplore","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04569","source_url":"https://arxiv.org/abs/2606.04569"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04569","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04569","topics":[],"unit":null},{"aliases":["Mind the Gaps: A Curated Benchmark for Form Field Detection"],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":133,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:moured/mini-commonforms","source_record":"bm_mind-the-gaps-a-curated-benchmark-for-form_76223934"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates form field detection models on 47,355 document pages in COCO format with annotations for text, choice, and signature fields.","descriptions":{"en":"Evaluates form field detection models on 47,355 document pages in COCO format with annotations for text, choice, and signature fields."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.23679","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_mind-the-gaps-a-curated-benchmark-for-form_76223934"},"modality":null,"name":"mini-CommonForms","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23679","source_url":"https://arxiv.org/abs/2609.23679"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23679","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23679","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","数学","Math","科学智能","AI for Science","逻辑推理","Reasoning","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A context-rich benchmark for evaluating neural theorem proving in realistic scenarios, providing premises, full context, multi-source benchmark, and temporal splits, enabling evaluation of a model's ability to work with context that evolves over time. 一个用于评估现实场景中神经定理证明的丰富上下文基准。通过提供前提、完整上下文、多源基准和时序分割，突出其评估模型处理随时间演变的上下文能力。","descriptions":{"en":"A context-rich benchmark for evaluating neural theorem proving in realistic scenarios, providing premises, full context, multi-source benchmark, and temporal splits, enabling evaluation of a model's ability to work with context that evolves over time.","zh":"一个用于评估现实场景中神经定理证明的丰富上下文基准。通过提供前提、完整上下文、多源基准和时序分割，突出其评估模型处理随时间演变的上下文能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1518","languages":[],"library_direction":null,"modality":null,"name":"miniCTX","openness":"open","publisher":"Carnegie Mellon University","released":"2024-08-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1518-minictx","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/miniCTX","topics":["reasoning-math","science"],"unit":null},{"aliases":["Pythagoras-Prover: Advancing Efficient Formal Proving via Augmented Lean Formalisation"],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A released, contamination-sensitive benchmark of augmented Lean formal statements for evaluating formal theorem provers.","descriptions":{"en":"A released, contamination-sensitive benchmark of augmented Lean formal statements for evaluating formal theorem provers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12594","languages":[],"library_direction":null,"modality":null,"name":"MiniF2F-ALF","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12594","source_url":"https://arxiv.org/abs/2606.12594"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12594","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12594","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","语言","Language","长文本","Long-Context","大语言模型","LLM","语言理解","Comprehension","长上下文","Long Context"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MiniLongBench is a low-cost benchmark for evaluating the Long Context Understanding (LCU) capabilities of LLMs, featuring a compact yet diverse test set of only 237 samples spanning 6 major task categories and 21 distinct tasks. MiniLongBench is a low-cost benchmark for evaluating the Long Context Understanding (LCU) capabilities of LLMs, featuring a compact yet diverse test set of only 237 samples spanning 6 major task categories and 21 distinct tasks.","descriptions":{"en":"MiniLongBench is a low-cost benchmark for evaluating the Long Context Understanding (LCU) capabilities of LLMs, featuring a compact yet diverse test set of only 237 samples spanning 6 major task categories and 21 distinct tasks.","zh":"MiniLongBench is a low-cost benchmark for evaluating the Long Context Understanding (LCU) capabilities of LLMs, featuring a compact yet diverse test set of only 237 samples spanning 6 major task categories and 21 distinct tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1853","languages":[],"library_direction":null,"modality":null,"name":"MiniLongBench","openness":"open","publisher":"MilkThink-Lab","released":"2025-05-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1853-minilongbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MiniLongBench","topics":["long-context","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A core challenge in short-video fact-checking is identifying which evidence is sufficient to support a verification conclusion.","descriptions":{"en":"A core challenge in short-video fact-checking is identifying which evidence is sufficient to support a verification conclusion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11233","languages":[],"library_direction":{"basis":"model-review-daily","categories":["video-understanding","factuality"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_miniver-v_165be9c0"},"modality":null,"name":"MiniVer-V","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11233","source_url":"https://arxiv.org/abs/2610.11233"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11233","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11233","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:spearmintai/minteval","source_record":"bm_minteval_9ecfb1dc"},"categories":["Code & Software","Finance","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language models are moving from producing trading signals to writing the code that executes them.","descriptions":{"en":"Large language models are moving from producing trading signals to writing the code that executes them."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.03080","languages":[],"library_direction":{"basis":"model-review-daily","categories":["finance-legal","code-generation"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_minteval_9ecfb1dc"},"modality":null,"name":"MintEval","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03080","source_url":"https://arxiv.org/abs/2610.03080"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03080","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03080","topics":["coding","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"MIRA-Ev evaluates clinical NLP on evidence detection and relational reasoning in Spanish MIR exam cases, with three tasks: evidence sentence retrieval, argumentative component extraction, and relation classification.","descriptions":{"en":"MIRA-Ev evaluates clinical NLP on evidence detection and relational reasoning in Spanish MIR exam cases, with three tasks: evidence sentence retrieval, argumentative component extraction, and relation classification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19201","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","multilingual"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_mira-ev_98696803"},"modality":null,"name":"MIRA-Ev","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19201","source_url":"https://arxiv.org/abs/2607.19201"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19201","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19201","topics":["reasoning-math","language","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"MIRA-Math evaluates mathematical reasoning where each problem is missing exactly one necessary atomic fact that must be requested in natural language under a strict budget, then integrated into an exact answer. It contains 2,310 instances across 22 typed mathematical families.","descriptions":{"en":"MIRA-Math evaluates mathematical reasoning where each problem is missing exactly one necessary atomic fact that must be requested in natural language under a strict budget, then integrated into an exact answer. It contains 2,310 instances across 22 typed mathematical families."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.07391","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_mira-math_da1f22ae"},"modality":null,"name":"MIRA-Math","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.07391","source_url":"https://arxiv.org/abs/2607.07391"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-07391","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07391","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MiraBench evaluates action-conditioned reliability in robotic world models through three levels: Physics Adherence, Action-Following Fidelity, and Optimism Bias Detection, using a human-annotated corpus of over 16,000 judgments.","descriptions":{"en":"MiraBench evaluates action-conditioned reliability in robotic world models through three levels: Physics Adherence, Action-Following Fidelity, and Optimism Bias Detection, using a human-annotated corpus of over 16,000 judgments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29360","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","embodied-robotics"],"join":"claire_record_id","primary":"world-models","source_record":"bm_mirabench_80851376"},"modality":null,"name":"MiraBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29360","source_url":"https://arxiv.org/abs/2605.29360"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29360","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29360","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","语言","Language","VQA","Strong Reasoning","多模态模型","VLM","检索能力","Retrieval","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MIRACL-VISION is a large-scale, multilingual visual document retrieval benchmark built by the NVIDIA team, extending the popular MIRACL multilingual text retrieval benchmark. It covers 18 languages and contains 211 original questions. MIRACL-VISION是一个大规模的多语言视觉文档检索基准测试，由NVIDIA团队构建，扩展了流行的MIRACL多语言文本检索基准。该数据集覆盖18种语言，包含211个原创问题，涵盖边界层分析、WKB方法、非线性偏微分方程的渐近解和振荡积分的渐近性等核心主题。","descriptions":{"en":"MIRACL-VISION is a large-scale, multilingual visual document retrieval benchmark built by the NVIDIA team, extending the popular MIRACL multilingual text retrieval benchmark. It covers 18 languages and contains 211 original questions.","zh":"MIRACL-VISION是一个大规模的多语言视觉文档检索基准测试，由NVIDIA团队构建，扩展了流行的MIRACL多语言文本检索基准。该数据集覆盖18种语言，包含211个原创问题，涵盖边界层分析、WKB方法、非线性偏微分方程的渐近解和振荡积分的渐近性等核心主题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1844","languages":["English","Chinese","Japanese","Korean","French","German","Spanish","Arabic","Russian","Thai","Indonesian","Hindi","Multilingual"],"library_direction":null,"modality":"multimodal","name":"MIRACL-VISION","openness":"unknown","publisher":"NVIDIA","released":"2025-05-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1844-miracl-vision","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MIRACL-VISION","topics":["multimodal","reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","ACL 2024","科学智能","AI for Science","检索能力","Retrieval","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MIRAGE is a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets. MIRAGE 是首个此类基准，包含来自五个医学问答数据集的 7,663 个问题。","descriptions":{"en":"MIRAGE is a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets.","zh":"MIRAGE 是首个此类基准，包含来自五个医学问答数据集的 7,663 个问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":true,"key":"opencompass:1142","languages":[],"library_direction":null,"modality":null,"name":"MIRAGE","openness":"unknown","publisher":"Univeristy of Virginia","released":"2024-08-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1142-mirage","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MIRAGE","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":40,"join":"claire_record_id","repository":null,"source_record":"bm_miragebench_adb719c2"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM personalization faithfulness: 150 personas, 6 tasks, four-way faithfulness taxonomy, 12 models across 7 families, 143,616 judged claims.","descriptions":{"en":"Evaluates LLM personalization faithfulness: 150 personas, 6 tasks, four-way faithfulness taxonomy, 12 models across 7 families, 143,616 judged claims."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04570","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","factuality"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_miragebench_adb719c2"},"modality":null,"name":"MirageBench","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04570","source_url":"https://arxiv.org/abs/2608.04570"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04570","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04570","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":76,"github_stars_30d":4,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:epoch-research/mirrorcode","source_record":"bm_mirrorcode_114f6eb8"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MirrorCode evaluates AI agents on reimplementing entire software projects from behavior only, matching outputs on end-to-end tests across 25 programs spanning Unix utilities, data serialization, bioinformatics, interpreters, static analysis, cryptography, and compression.","descriptions":{"en":"MirrorCode evaluates AI agents on reimplementing entire software projects from behavior only, matching outputs on end-to-end tests across 25 programs spanning Unix utilities, data serialization, bioinformatics, interpreters, static analysis, cryptography, and compression."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.30182","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_mirrorcode_114f6eb8"},"modality":null,"name":"MirrorCode","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30182","source_url":"https://arxiv.org/abs/2606.30182"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30182","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30182","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MirrorCraft is a paired benchmark for evaluating LLM-based agents under hidden rule changes in Minecraft. Each Mirror world copies a Vanilla world with modified server-side rules, while terrain and objectives match. It includes five biomes, six rule suites, three objectives, and uses deterministic milestones and Rule Intervention Effect.","descriptions":{"en":"MirrorCraft is a paired benchmark for evaluating LLM-based agents under hidden rule changes in Minecraft. Each Mirror world copies a Vanilla world with modified server-side rules, while terrain and objectives match. It includes five biomes, six rule suites, three objectives, and uses deterministic milestones and Rule Intervention Effect."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.29218","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_mirrorcraft_4e4ff3e9"},"modality":null,"name":"MirrorCraft","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29218","source_url":"https://arxiv.org/abs/2607.29218"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29218","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29218","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Large audio-language models (LALMs) produce fluent responses about audio but often hallucinate by making plausible yet ungrounded claims.","descriptions":{"en":"Large audio-language models (LALMs) produce fluent responses about audio but often hallucinate by making plausible yet ungrounded claims."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33893","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","speech-audio"],"join":"claire_record_id","primary":"factuality","source_record":"bm_mishap-bench_a69957c9"},"modality":null,"name":"MISHAP-Bench","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33893","source_url":"https://arxiv.org/abs/2609.33893"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33893","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33893","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MissingBench-Verified evaluates vision-language models on detecting missing object parts in images, covering ten leading models with consistent failure rates and probing mitigation strategies.","descriptions":{"en":"MissingBench-Verified evaluates vision-language models on detecting missing object parts in images, covering ten leading models with consistent failure rates and probing mitigation strategies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.18673","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_missingbench-verified_db33d9c0"},"modality":null,"name":"MissingBench-Verified","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18673","source_url":"https://arxiv.org/abs/2607.18673"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18673","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18673","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_missionbench_cc95e3a2"},"categories":["Geometric reasoning","Robotics & Embodied AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for mission-level evaluation of MLLMs in aerial 3D environments, consisting of 120 missions across five simulated environments and four task families.","descriptions":{"en":"A benchmark for mission-level evaluation of MLLMs in aerial 3D environments, consisting of 120 missions across five simulated environments and four task families."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.22014","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_missionbench_cc95e3a2"},"modality":null,"name":"MissionBench","openness":"unknown","publisher":null,"released":"2026-07-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22014","source_url":"https://arxiv.org/abs/2607.22014"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22014","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22014","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Web agents increasingly browse, interpret, and act on web content on behalf of users.","descriptions":{"en":"Web agents increasingly browse, interpret, and act on web content on behalf of users."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.12823","languages":[],"library_direction":{"basis":"model-review-daily","categories":["computer-use","visual-reasoning"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_misvisagentbench_9d33431c"},"modality":null,"name":"MisVisAgentBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12823","source_url":"https://arxiv.org/abs/2610.12823"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12823","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12823","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","视觉生成","Visual Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MJ-Bench incorporates a comprehensive preference dataset to evaluate multimodal judges in providing feedback for image generation models across four key perspectives: alignment, safety, image quality, and bias. MJ-Bench，它包含了一个综合的偏好数据集，用于从四个关键角度评估多模态评委在为图像生成模型提供反馈方面的能力：对齐、安全性、图像质量和偏见。","descriptions":{"en":"MJ-Bench incorporates a comprehensive preference dataset to evaluate multimodal judges in providing feedback for image generation models across four key perspectives: alignment, safety, image quality, and bias.","zh":"MJ-Bench，它包含了一个综合的偏好数据集，用于从四个关键角度评估多模态评委在为图像生成模型提供反馈方面的能力：对齐、安全性、图像质量和偏见。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1623","languages":[],"library_direction":null,"modality":"multimodal","name":"MJ-Bench","openness":"open","publisher":"UNC-Chapel Hill, University of Chicago, Stanford University","released":"2024-07-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1623-mj-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MJ-Bench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Information retrieval","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MKG-RAG-Bench is a benchmark for retrieval in multimodal knowledge graph-augmented generation, constructed from two knowledge graphs (general and medical) with QA datasets supporting controlled evaluation of retrieval and generation. It uses an LLM-based curation pipeline for structurally grounded queries.","descriptions":{"en":"MKG-RAG-Bench is a benchmark for retrieval in multimodal knowledge graph-augmented generation, constructed from two knowledge graphs (general and medical) with QA datasets supporting controlled evaluation of retrieval and generation. It uses an LLM-based curation pipeline for structurally grounded queries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26458","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_mkg-rag-bench_ba53e1c9"},"modality":null,"name":"MKG-RAG-Bench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26458","source_url":"https://arxiv.org/abs/2606.26458"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26458","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26458","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":198,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2007.15207","repository":"gh:apple/ml-mkqa","source_record":"catalog_e34928c4f86c04c1"},"categories":["知识","Knowledge","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MKQA is an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). MKQA 是一个开放域问答评估集，包含 10,000 对问题和答案，涵盖 26 种类型多样的语言（总计 260,000 对问题和答案）。","descriptions":{"en":"MKQA is an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total).","zh":"MKQA 是一个开放域问答评估集，包含 10,000 对问题和答案，涵盖 26 种类型多样的语言（总计 260,000 对问题和答案）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1099","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","multilingual"],"join":"artifact:arxiv:2007.15207","primary":"deep-research","source_record":"catalog_e34928c4f86c04c1"},"modality":null,"name":"MKQA","openness":"restricted","publisher":"Apple","released":"2021-08-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1099-mkqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MKQA","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["reasoning"],"collected_at":"2026-08-25T10:41:06Z","description":"MLCR-AA overall score","descriptions":{"en":"MLCR-AA overall score"},"evidence_summary":{"document_count":1,"model_count":71,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:mlcr-aa:b5c1c91a-7474-4409-9a9c-9c2ac45d9eb6","reported_at":"2024-07-18","source_url":"https://artificialanalysis.ai/evaluations/mlcr-aa"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:mlcr-aa","languages":[],"library_direction":null,"modality":null,"name":"MLCR-AA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":71,"score_direction":"higher_is_better","score_summary":{"display_max":64.4444444444444,"display_multiplier":100,"model_count":71,"model_count_basis":"source_model_id","numeric_count":71,"raw_max":0.644444444444444,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:mlcr-aa:cd55210d-358e-4df1-ba9c-9acb5f186cc9","reported_date":"2026-06-09","source_url":"https://artificialanalysis.ai/evaluations/mlcr-aa"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"}],"slug":"artificial-analysis-mlcr-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/mlcr-aa","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1772,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"llm_stats_id","repository":"gh:openai/mle-bench","source_record":"lib_mle_bench"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MLE-Bench evaluates AI agents on machine learning engineering tasks by measuring their performance on Kaggle competitions.","descriptions":{"en":"MLE-Bench evaluates AI agents on machine learning engineering tasks by measuring their performance on Kaggle competitions."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mle-bench:gemini-3.5-flash-lite","reported_at":"2026-07-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mle-bench","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"lib_mle_bench"},"modality":"text","name":"MLE-Bench","openness":"unknown","publisher":null,"released":"2024-10-09","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mle-bench","source_url":"https://arxiv.org/abs/2410.07095"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":63.9,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.639,"raw_min":0.392,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mle-bench:gemini-3.6-flash","reported_date":"2026-07-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash"}],"slug":"llm-stats-mle-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["MLE-Bench","MLE-bench Revised","MLE-bench Lite"],"attention_signals":null,"categories":["ai_research"],"collected_at":null,"description":"Kaggle-derived ML engineering tasks; wall-clock budget and hardware are part of the result. Published by OpenAI.","descriptions":{"en":"Kaggle-derived ML engineering tasks; wall-clock budget and hardware are part of the result. Published by OpenAI."},"evidence_summary":{"document_count":2,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:mle_bench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["ai-rnd"],"join":"review:14H034160212:2026-10-15","primary":"ai-rnd","source_record":null},"modality":null,"name":"MLE-bench","openness":"unknown","publisher":null,"released":"2024-10-09","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mle_bench","source_url":"https://openai.com/index/mle-bench/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"mle_bench","source":"model_reports","source_url":"https://openai.com/index/mle-bench/","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1772,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:openai/mle-bench","source_record":"catalog_eb8fe21cec35acc2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MLE-Bench Lite evaluates AI agents on machine learning engineering tasks, testing their ability to build, train, and optimize ML models for Kaggle-style competitions in a lightweight evaluation format.","descriptions":{"en":"MLE-Bench Lite evaluates AI agents on machine learning engineering tasks, testing their ability to build, train, and optimize ML models for Kaggle-style competitions in a lightweight evaluation format."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mle-bench-lite:minimax-m2.7","reported_at":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench-lite?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mle-bench-lite","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_eb8fe21cec35acc2"},"modality":"text","name":"MLE-Bench Lite","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":66.60000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.666,"raw_min":0.666,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mle-bench-lite:minimax-m2.7","reported_date":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench-lite?top_n=500"},"unit":null},"scored_models":[{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"}],"slug":"llm-stats-mle-bench-lite","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench-lite?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Agentic machine learning engineering (MLE) is an emerging AI4SE application for complex ML tasks and a step toward recursive self-improvement of AI systems.","descriptions":{"en":"Agentic machine learning engineering (MLE) is an emerging AI4SE application for complex ML tasks and a step toward recursive self-improvement of AI systems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.04927","languages":[],"library_direction":{"basis":"model-review-daily","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_mle-insightbench_d0e5cb08"},"modality":null,"name":"MLE-InsightBench","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.04927","source_url":"https://arxiv.org/abs/2610.04927"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-04927","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04927","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:rishabhpm23/mlingualfc","source_record":"bm_mlingualfc_4a4cc033"},"categories":["General AI","Multimodal","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"MLingualFC evaluates jailbreak vulnerabilities in multilingual vision-language models using flowchart images encoding harmful instructions in five languages, measuring attack success rates.","descriptions":{"en":"MLingualFC evaluates jailbreak vulnerabilities in multilingual vision-language models using flowchart images encoding harmful instructions in five languages, measuring attack success rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07706","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_mlingualfc_4a4cc033"},"modality":null,"name":"MLingualFC","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07706","source_url":"https://arxiv.org/abs/2606.07706"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07706","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07706","topics":["multimodal","language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions. MLRC-Bench旨在量化大模型代理如何有效地应对具有挑战性的机器学习 （ML） 研究竞赛。","descriptions":{"en":"MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions.","zh":"MLRC-Bench旨在量化大模型代理如何有效地应对具有挑战性的机器学习 （ML） 研究竞赛。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1779","languages":[],"library_direction":null,"modality":null,"name":"MLRC-Bench","openness":"unknown","publisher":"University of Michigan, Ann Arbor，LG AI Research，University of Illinois, Chicago","released":"2025-04-13","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1779-mlrc-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MLRC-Bench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":121,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":"gh:imbernoulli/mls-bench","source_record":"catalog_413ce4ccc028faaf"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MLS-Bench Lite is the official 30-task subset of MLS-Bench for evaluating whether AI systems can invent generalizable and scalable machine learning methods across LLM pretraining and post-training, robotics, world models, computer vision, reinforcement learning, optimization, ML systems, and AI for Science.","descriptions":{"en":"MLS-Bench Lite is the official 30-task subset of MLS-Bench for evaluating whether AI systems can invent generalizable and scalable machine learning methods across LLM pretraining and post-training, robotics, world models, computer vision, reinforcement learning, optimization, ML systems, and AI for Science."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mls-bench-lite:kimi-k2.7-code","reported_at":"2026-06-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mls-bench-lite?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mls-bench-lite","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_413ce4ccc028faaf"},"modality":"text","name":"MLS-Bench Lite","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":48.3,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.483,"raw_min":0.351,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mls-bench-lite:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mls-bench-lite?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-mls-bench-lite","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mls-bench-lite?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:lihe-maxsize/lifelong_unlearning_main","source_record":"bm_mlubench_e20fa573"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MLUBench evaluates multimodal large language models on lifelong unlearning across 127 entities and 9 classes, providing QA pairs and images. It includes a protocol for sequential unlearning requests and evaluation of forgetting and retention.","descriptions":{"en":"MLUBench evaluates multimodal large language models on lifelong unlearning across 127 entities and 9 classes, providing QA pairs and images. It includes a protocol for sequential unlearning requests and evaluation of forgetting and retention."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.12809","languages":[],"library_direction":null,"modality":null,"name":"MLUBench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12809","source_url":"https://arxiv.org/abs/2606.12809"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12809","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12809","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_737363d33f94b15d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","multimodal","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive benchmark for multi-task long video understanding that evaluates multimodal large language models on videos ranging from 3 minutes to 2 hours across 9 distinct tasks including reasoning, captioning, recognition, and summarization.","descriptions":{"en":"A comprehensive benchmark for multi-task long video understanding that evaluates multimodal large language models on videos ranging from 3 minutes to 2 hours across 9 distinct tasks including reasoning, captioning, recognition, and summarization."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mlvu:qwen2.5-vl-7b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mlvu","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","long-context"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_737363d33f94b15d"},"modality":"multimodal","name":"MLVU","openness":"unknown","publisher":null,"released":"2024-06-06","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MLVU: Benchmarking Multi-task Long Video Understanding. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mlvu","source_url":"https://arxiv.org/abs/2406.04264"},"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":87.4,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.874,"raw_min":0.702,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mlvu:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu?top_n=500"},"unit":null},"scored_models":[{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-mlvu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu?top_n=500","topics":["multimodal","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MLVU test MLLMs' understanding of multi-task long videos, encompassing diverse tasks based on various long videos. MLVU用于评估多模态大模型的长视频理解能力，包含面向各种类型长视频的多样化的评估任务。","descriptions":{"en":"MLVU test MLLMs' understanding of multi-task long videos, encompassing diverse tasks based on various long videos.","zh":"MLVU用于评估多模态大模型的长视频理解能力，包含面向各种类型长视频的多样化的评估任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1512","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","long-context"],"join":"artifact:arxiv:2406.04264","primary":"video-understanding","source_record":"catalog_533dd059f93116c0"},"modality":"multimodal","name":"MLVU","openness":"restricted","publisher":"BAAI","released":"2024-06-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1512-mlvu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MLVU","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":269,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:junjie99/mlvu","source_record":"catalog_533dd059f93116c0"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MLVU-M benchmark","descriptions":{"en":"MLVU-M benchmark"},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mlvu-m:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu-m?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mlvu-m","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","long-context"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_533dd059f93116c0"},"modality":"text","name":"MLVU-M","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":82.1,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.821,"raw_min":0.746,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mlvu-m:qwen3-vl-32b-instruct","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu-m?top_n=500"},"unit":null},"scored_models":[{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-mlvu-m","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu-m?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":126,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":35,"join":"llm_stats_id","repository":null,"source_record":"catalog_da559c403a907e7f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","structured_output"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A challenging multimodal instruction-following benchmark that includes both compose-level constraints for output responses and perception-level constraints tied to input images, with comprehensive evaluation pipeline.","descriptions":{"en":"A challenging multimodal instruction-following benchmark that includes both compose-level constraints for output responses and perception-level constraints tied to input images, with comprehensive evaluation pipeline."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mm-if-eval:pixtral-12b-2409","reported_at":"2024-09-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-if-eval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mm-if-eval","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","visual-reasoning"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_da559c403a907e7f"},"modality":"multimodal","name":"MM IF-Eval","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":60.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.606,"raw_min":0.527,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mm-if-eval:lfm-2.5-vl-3b","reported_date":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-if-eval?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"}],"slug":"llm-stats-mm-if-eval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-if-eval?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A benchmark for evaluating MLLMs' alignment with human preferences. It includes 252 high-quality, human-annotated samples with diverse image types and open-ended questions. Modeled after Arena-style benchmarks, it uses GPT-4o as the judge model and Claude-Sonnet-3 as the reference model. 用于评估 MLLM 与人类偏好的一致性的基准。它包含 252 个高质量、人类标注的样本，具有不同的图像类型和开放式问题。它仿照 Arena 风格的基准，使用 GPT-4o 作为评判模型，Claude-Sonnet-3 作为参考模型。","descriptions":{"en":"A benchmark for evaluating MLLMs' alignment with human preferences. It includes 252 high-quality, human-annotated samples with diverse image types and open-ended questions. Modeled after Arena-style benchmarks, it uses GPT-4o as the judge model and Claude-Sonnet-3 as the reference model.","zh":"用于评估 MLLM 与人类偏好的一致性的基准。它包含 252 个高质量、人类标注的样本，具有不同的图像类型和开放式问题。它仿照 Arena 风格的基准，使用 GPT-4o 作为评判模型，Claude-Sonnet-3 作为参考模型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1558","languages":[],"library_direction":null,"modality":"multimodal","name":"MM-AlignBench","openness":"unknown","publisher":"Shanghai Jiaotong University,Shanghai AI Laboratory,etc.","released":"2025-02-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1558-mm-alignbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MM-AlignBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","search","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MM-BrowserComp evaluates multimodal agents on web browsing and information retrieval tasks, testing a model's ability to perceive, navigate, and extract information from real web environments.","descriptions":{"en":"MM-BrowserComp evaluates multimodal agents on web browsing and information retrieval tasks, testing a model's ability to perceive, navigate, and extract information from real web environments."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mm-browsercomp:mimo-v2-omni","reported_at":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-browsercomp?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mm-browsercomp","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_1f4900925fcf51d0"},"modality":"multimodal","name":"MM-BrowserComp","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":52.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.52,"raw_min":0.52,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mm-browsercomp:mimo-v2-omni","reported_date":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-browsercomp?top_n=500"},"unit":null},"scored_models":[{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Omni"}],"slug":"llm-stats-mm-browsercomp","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-browsercomp?top_n=500","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MM-ClawBench evaluates models on MiniMax's Claw-style agent benchmark, measuring practical agentic task completion quality in real-world OpenClaw usage scenarios.","descriptions":{"en":"MM-ClawBench evaluates models on MiniMax's Claw-style agent benchmark, measuring practical agentic task completion quality in real-world OpenClaw usage scenarios."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mm-clawbench:minimax-m2.7","reported_at":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-clawbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mm-clawbench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_a46d809efa147169"},"modality":"text","name":"MM-ClawBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":62.7,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.627,"raw_min":0.627,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mm-clawbench:minimax-m2.7","reported_date":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-clawbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"}],"slug":"llm-stats-mm-clawbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-clawbench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["MM-ContextASR-Bench"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":314,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_mm-contextasr-bench_28bd2f6b"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MM-ContextASR Bench evaluates multimodal contextual ASR using controlled dialogue histories, external corpus tracks, and metrics including CER, SER, and entity Recall.","descriptions":{"en":"MM-ContextASR Bench evaluates multimodal contextual ASR using controlled dialogue histories, external corpus tracks, and metrics including CER, SER, and entity Recall."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:lilonghao/mm-contextasr-bench","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_mm-contextasr-bench_28bd2f6b"},"modality":null,"name":"MM-ContextASR Bench","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:lilonghao/mm-contextasr-bench","source_url":"https://huggingface.co/datasets/lilonghao/MM-ContextASR-Bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-lilonghao-mm-contextasr-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/lilonghao/MM-ContextASR-Bench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":"gh:creativitybench/mm-creativitybench","source_record":"bm_mm-creativitybench_e366f23e"},"categories":["Agents","Agents & Tool Use","General AI","Multimodal","Tool use"],"collected_at":"2026-10-14T09:44:51Z","description":"MM-CreativityBench evaluates large multimodal models on affordance-grounded creative tool use, requiring scene inspection, entity/part selection, and physically feasible solutions in visually rich environments.","descriptions":{"en":"MM-CreativityBench evaluates large multimodal models on affordance-grounded creative tool use, requiring scene inspection, entity/part selection, and physically feasible solutions in visually rich environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26396","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","abstract-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mm-creativitybench_e366f23e"},"modality":null,"name":"MM-CreativityBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26396","source_url":"https://arxiv.org/abs/2605.26396"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26396","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26396","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures, market expectations, and subtle communication signals.","descriptions":{"en":"Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures, market expectations, and subtle communication signals."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38523","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_mm-fineval_2fcfbeae"},"modality":null,"name":"MM-FinEval","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38523","source_url":"https://arxiv.org/abs/2609.38523"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38523","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38523","topics":["multimodal","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MM-IFEval-Pro evaluates vision-language models on instruction-following across Chinese and English, covering four major task categories and 24 subcategories with an average of 3.0 constraints per sample, including adversarial instruction hijacking scenarios.","descriptions":{"en":"MM-IFEval-Pro evaluates vision-language models on instruction-following across Chinese and English, covering four major task categories and 24 subcategories with an average of 3.0 constraints per sample, including adversarial instruction hijacking scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.04859","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following","multilingual","agent-security"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_mm-ifeval-pro_b0c09dd5"},"modality":null,"name":"MM-IFEval-Pro","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04859","source_url":"https://arxiv.org/abs/2609.04859"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04859","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04859","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MM-IQ is a comprehensive evaluation framework comprising 2,710 meticulously curated test items spanning 8 distinct reasoning paradigms. MM-IQ，这是一个包含 2,710 个精心挑选的测试项目的综合评估框架，涵盖了 8 种不同的推理范式。","descriptions":{"en":"MM-IQ is a comprehensive evaluation framework comprising 2,710 meticulously curated test items spanning 8 distinct reasoning paradigms.","zh":"MM-IQ，这是一个包含 2,710 个精心挑选的测试项目的综合评估框架，涵盖了 8 种不同的推理范式。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1566","languages":[],"library_direction":null,"modality":"multimodal","name":"MM-IQ","openness":"open","publisher":"Tencent Hunyuan Team","released":"2025-02-02","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1566-mm-iq","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MM-IQ","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1032,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:osu-nlp-group/mind2web","source_record":"catalog_37a7875647e70c76"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","frontend_development","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multimodal web navigation benchmark comprising 2,000 open-ended tasks spanning 137 websites across 31 domains. Each task includes HTML documents paired with webpage screenshots, action sequences, and complex web interactions.","descriptions":{"en":"A multimodal web navigation benchmark comprising 2,000 open-ended tasks spanning 137 websites across 31 domains. Each task includes HTML documents paired with webpage screenshots, action sequences, and complex web interactions."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mm-mind2web:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mind2web?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mm-mind2web","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_37a7875647e70c76"},"modality":"multimodal","name":"MM-Mind2Web","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":63.7,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.637,"raw_min":0.558,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mm-mind2web:nova-pro","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mind2web?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"}],"slug":"llm-stats-mm-mind2web","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mind2web?top_n=500","topics":["computer-use","agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":92,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":70,"join":"llm_stats_id","repository":"gh:mistralai/mistral-evals","source_record":"catalog_1165a051eddbecad"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","communication"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multi-turn LLM-as-a-judge evaluation benchmark for testing multimodal instruction-tuned models' ability to follow user instructions in multi-turn dialogues and answer open-ended questions in a zero-shot manner.","descriptions":{"en":"A multi-turn LLM-as-a-judge evaluation benchmark for testing multimodal instruction-tuned models' ability to follow user instructions in multi-turn dialogues and answer open-ended questions in a zero-shot manner."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mm-mt-bench:pixtral-12b-2409","reported_at":"2024-09-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mt-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mm-mt-bench","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","visual-reasoning"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_1165a051eddbecad"},"modality":"multimodal","name":"MM-MT-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":84.9,"display_multiplier":1,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":84.9,"raw_min":0.06,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mm-mt-bench:mistral-large-3-2509","reported_date":"2025-09-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mt-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-04","date_precision":"model_announcement","name":"MiniStral 3 (14B Instruct 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Instruct 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Instruct 2512)"},{"date":"2025-09-01","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-mm-mt-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mt-bench?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","VQA","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MM-RLHF, a comprehensive project for aligning Multimodal Large Language Models (MLLMs) with human preferences. The dataset and algorithms enable consistent performance improvements across 10 dimensions and 27 benchmarks for open-source MLLMs. MM-RLHF，这是一个将多模态大型语言模型（MLLMs）与人类偏好对齐的全面项目，使开源多语言机器学习模型在 10 个维度和 27 个基准测试中实现持续的性能提升。","descriptions":{"en":"MM-RLHF, a comprehensive project for aligning Multimodal Large Language Models (MLLMs) with human preferences. The dataset and algorithms enable consistent performance improvements across 10 dimensions and 27 benchmarks for open-source MLLMs.","zh":"MM-RLHF，这是一个将多模态大型语言模型（MLLMs）与人类偏好对齐的全面项目，使开源多语言机器学习模型在 10 个维度和 27 个基准测试中实现持续的性能提升。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1539","languages":[],"library_direction":null,"modality":"multimodal","name":"MM-RLHF","openness":"unknown","publisher":"KuaiShou, CASIA, NJU","released":"2025-02-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1539-mm-rlhf","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MM-RLHF","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MM-Snowball is a benchmark for diagnosing hallucination snowballing in multimodal multi-turn dialogue, with fine-grained analysis. It includes data and code via a project page.","descriptions":{"en":"MM-Snowball is a benchmark for diagnosing hallucination snowballing in multimodal multi-turn dialogue, with fine-grained analysis. It includes data and code via a project page."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00622","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","visual-reasoning"],"join":"claire_record_id","primary":"factuality","source_record":"bm_mm-snowball_ce633f35"},"modality":null,"name":"MM-Snowball","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00622","source_url":"https://arxiv.org/abs/2606.00622"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00622","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00622","topics":["multimodal","cybersecurity"],"unit":null},{"aliases":["MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"MM-ToolSandBox is a stateful sandbox benchmark for visually grounded tool-calling agents. It has 258 human-verified scenarios across 500+ tools in 16 domains, and models are scored by task success rate.","descriptions":{"en":"MM-ToolSandBox is a stateful sandbox benchmark for visually grounded tool-calling agents. It has 258 human-verified scenarios across 500+ tools in 16 domains, and models are scored by task success rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.11818","languages":[],"library_direction":null,"modality":null,"name":"MM-ToolSandBox","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11818","source_url":"https://arxiv.org/abs/2607.11818"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11818","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11818","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":333,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"artifact:arxiv:2308.02490","repository":null,"source_record":"catalog_93d1c075bf34e12c"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MM-Vet evaluates the capabilities to deal with complicated multimodal tasks, defining 6 core VL capabilities and examining the 16 integrations of interest derived from the capability combination. MM-Vet用于评估复杂多模态任务能力，涵盖了6个核心视觉语言功能的16种功能组合。","descriptions":{"en":"MM-Vet evaluates the capabilities to deal with complicated multimodal tasks, defining 6 core VL capabilities and examining the 16 integrations of interest derived from the capability combination.","zh":"MM-Vet用于评估复杂多模态任务能力，涵盖了6个核心视觉语言功能的16种功能组合。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1356","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2308.02490","primary":"visual-reasoning","source_record":"catalog_93d1c075bf34e12c"},"modality":"multimodal","name":"MM-Vet","openness":"unknown","publisher":"National University of Singapore","released":"2023-08-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1356-mm-vet","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MM-Vet","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","VQA","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMAD is the first-ever full-spectrum MLLMs benchmark in industrial Anomaly Detection. \nResearchers defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images. MMAD是第一个工业异常检测领域的全谱 MLLMs 基准，研究人员定义了工业检测中 MLLMs 的七个关键子任务，并设计了一个新颖的流程来生成包含 39,672 个问题以及 8,366 个工业图像的 MMAD 数据集。","descriptions":{"en":"MMAD is the first-ever full-spectrum MLLMs benchmark in industrial Anomaly Detection. \nResearchers defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images.","zh":"MMAD是第一个工业异常检测领域的全谱 MLLMs 基准，研究人员定义了工业检测中 MLLMs 的七个关键子任务，并设计了一个新颖的流程来生成包含 39,672 个问题以及 8,366 个工业图像的 MMAD 数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1585","languages":[],"library_direction":null,"modality":"multimodal","name":"MMAD","openness":"restricted","publisher":"Southern University of Science and Technology, etc.","released":"2024-10-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1585-mmad","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMAD","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":111,"github_stars_30d":7,"hf_dataset_downloads":null,"hf_paper_upvotes":46,"join":"claire_record_id","repository":"gh:ddlbojack/mmae","source_record":"bm_mmae_ba12849c"},"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"MMAE is a benchmark for instruction-based audio editing with 2,000 samples across 7 modalities, 6 complexity levels, and rubric-based evaluation with 17,741 criteria for instruction following and context consistency.","descriptions":{"en":"MMAE is a benchmark for instruction-based audio editing with 2,000 samples across 7 modalities, 6 complexity levels, and rubric-based evaluation with 17,741 criteria for instruction following and context consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07229","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_mmae_ba12849c"},"modality":null,"name":"MMAE","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07229","source_url":"https://arxiv.org/abs/2606.07229"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07229","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07229","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","多模态","Multimodal","推理","Reasoning","MMAR","音频语言模型","深度推理","多模态模型","VLM","逻辑推理","音频理解","Audio Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"We introduce MMAR, a new benchmark comprising 1,000 meticulously curated audio-question-answer triplets, designed to evaluate the deep reasoning capabilities of Audio-Language Models (ALMs) across massive multi-disciplinary tasks. MMAR是一个全新评测基准，旨在评估音频-语言模型（ALMs）的深度推理能力。该基准包含1,000个精心构建的音频与问答，并经过多轮纠错与质量校验以确保高标准。与现有局限于特定声音、音乐或语音领域的评测体系不同，MMAR覆盖现实场景中的混合模态，并采用四级分层分类体系（信号层、感知层、语义层与文化层）。该基准中的每个题目都需要超越表层理解的多层次深度推理，部分问题更要求研究生级别的专业领域知识与感知能力。我们在MMAR上评估了多类模型，测试结果表明该基准具有显著挑战性，分析结果进一步揭示了当前模型在理解与推理能力上的关键局限。我们期待MMAR能推动这个重要但尚未充分探索的研究领域的发展。","descriptions":{"en":"We introduce MMAR, a new benchmark comprising 1,000 meticulously curated audio-question-answer triplets, designed to evaluate the deep reasoning capabilities of Audio-Language Models (ALMs) across massive multi-disciplinary tasks.","zh":"MMAR是一个全新评测基准，旨在评估音频-语言模型（ALMs）的深度推理能力。该基准包含1,000个精心构建的音频与问答，并经过多轮纠错与质量校验以确保高标准。与现有局限于特定声音、音乐或语音领域的评测体系不同，MMAR覆盖现实场景中的混合模态，并采用四级分层分类体系（信号层、感知层、语义层与文化层）。该基准中的每个题目都需要超越表层理解的多层次深度推理，部分问题更要求研究生级别的专业领域知识与感知能力。我们在MMAR上评估了多类模型，测试结果表明该基准具有显著挑战性，分析结果进一步揭示了当前模型在理解与推理能力上的关键局限。我们期待MMAR能推动这个重要但尚未充分探索的研究领域的发展。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1910","languages":[],"library_direction":null,"modality":"multimodal","name":"MMAR","openness":"restricted","publisher":"上海交通大学","released":"2025-05-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1910-mmar","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMAR","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"MMArch is a benchmark for multimodal reasoning in architecture and civil engineering, spanning ten subdomains and built from figures in peer-reviewed papers. It contains 1,212 short-answer items requiring perception, principle identification, and application.","descriptions":{"en":"MMArch is a benchmark for multimodal reasoning in architecture and civil engineering, spanning ten subdomains and built from figures in peer-reviewed papers. It contains 1,212 short-answer items requiring perception, principle identification, and application."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09281","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","knowledge-expert-qa"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mmarch_3c23535d"},"modality":null,"name":"MMArch","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09281","source_url":"https://arxiv.org/abs/2608.09281"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09281","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09281","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_58bca2a148fba12e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A massive multi-task audio understanding and reasoning benchmark comprising 10,000 carefully curated audio clips paired with human-annotated natural language questions spanning speech, environmental sounds, and music. Requires expert-level knowledge and complex reasoning across 27 distinct skills.","descriptions":{"en":"A massive multi-task audio understanding and reasoning benchmark comprising 10,000 carefully curated audio clips paired with human-annotated natural language questions spanning speech, environmental sounds, and music. Requires expert-level knowledge and complex reasoning across 27 distinct skills."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmau:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmau","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_58bca2a148fba12e"},"modality":"multimodal","name":"MMAU","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":77.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.77,"raw_min":0.656,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmau:inkling-small","reported_date":"2026-07-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-mmau","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_a5b7b90c3d77b196"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A subset of the MMAU benchmark focused specifically on music understanding and reasoning tasks. Part of a comprehensive multimodal audio understanding benchmark that evaluates models on expert-level knowledge and complex reasoning across music audio clips.","descriptions":{"en":"A subset of the MMAU benchmark focused specifically on music understanding and reasoning tasks. Part of a comprehensive multimodal audio understanding benchmark that evaluates models on expert-level knowledge and complex reasoning across music audio clips."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmau-music:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-music?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmau-music","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_a5b7b90c3d77b196"},"modality":"multimodal","name":"MMAU Music","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":69.16,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.6916,"raw_min":0.6916,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmau-music:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-music?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-mmau-music","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-music?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_ae6c7417fddf3e80"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A subset of the MMAU benchmark focused specifically on environmental sound understanding and reasoning tasks. Part of a comprehensive multimodal audio understanding benchmark that evaluates models on expert-level knowledge and complex reasoning across environmental sound clips.","descriptions":{"en":"A subset of the MMAU benchmark focused specifically on environmental sound understanding and reasoning tasks. Part of a comprehensive multimodal audio understanding benchmark that evaluates models on expert-level knowledge and complex reasoning across environmental sound clips."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmau-sound:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-sound?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmau-sound","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_ae6c7417fddf3e80"},"modality":"multimodal","name":"MMAU Sound","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":67.86999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.6787,"raw_min":0.6787,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmau-sound:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-sound?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-mmau-sound","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-sound?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_b62294e604454e72"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","speech_to_text","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A subset of the MMAU benchmark focused specifically on speech understanding and reasoning tasks. Part of a comprehensive multimodal audio understanding benchmark that evaluates models on expert-level knowledge and complex reasoning across speech audio clips.","descriptions":{"en":"A subset of the MMAU benchmark focused specifically on speech understanding and reasoning tasks. Part of a comprehensive multimodal audio understanding benchmark that evaluates models on expert-level knowledge and complex reasoning across speech audio clips."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmau-speech:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-speech?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmau-speech","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_b62294e604454e72"},"modality":"multimodal","name":"MMAU Speech","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":59.760000000000005,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.5976,"raw_min":0.5976,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmau-speech:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-speech?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-mmau-speech","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-speech?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":72,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"llm_stats_id","repository":"gh:mmbrowsecomp/mm-browsecomp","source_record":"catalog_02e126e011402c43"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","knowledge","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMBC is a multimodal benchmark for vision-language knowledge and reasoning.","descriptions":{"en":"MMBC is a multimodal benchmark for vision-language knowledge and reasoning."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmbc:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbc?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmbc","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_02e126e011402c43"},"modality":"multimodal","name":"MMBC","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":46.300000000000004,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.463,"raw_min":0.463,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmbc:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbc?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-mmbc","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbc?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4439,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":"gh:open-compass/vlmevalkit","source_record":"catalog_26ec0b897cf3f983"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A bilingual benchmark for assessing multi-modal capabilities of vision-language models through multiple-choice questions in both English and Chinese, providing systematic evaluation across diverse vision-language tasks with robust metrics.","descriptions":{"en":"A bilingual benchmark for assessing multi-modal capabilities of vision-language models through multiple-choice questions in both English and Chinese, providing systematic evaluation across diverse vision-language tasks with robust metrics."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmbench:phi-3.5-vision-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmbench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_26ec0b897cf3f983"},"modality":"multimodal","name":"MMBench","openness":"unknown","publisher":null,"released":"2023-07-12","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MMBench: Is Your Multi-modal Model an All-around Player?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mmbench","source_url":"https://arxiv.org/abs/2307.06281"},"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":91.8,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.918,"raw_min":0.692,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmbench:step3-vl-10b","reported_date":"2026-01-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-vision-instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2026-01-15","date_precision":"model_announcement","name":"Step3-VL-10B"}],"slug":"llm-stats-mmbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","理解","Understanding","多模态模型","VLM","逻辑推理","图像理解","Image Understanding","官方自建","Official"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMBench is a collection of benchmarks to evaluate the multi-modal understanding capability of large vision language models (LVLMs). This benchmark contains 3,000 multiple-choice questions covering 20 fine-grained assessment dimensions. MMBench是OpenCompass 研究团队自建的视觉语言模型评测数据集，可实现从感知到认知能力逐级细分评估。此评测基准包含3000 道单项选择题 ，覆盖 20个细粒度评估维度。","descriptions":{"en":"MMBench is a collection of benchmarks to evaluate the multi-modal understanding capability of large vision language models (LVLMs). This benchmark contains 3,000 multiple-choice questions covering 20 fine-grained assessment dimensions.","zh":"MMBench是OpenCompass 研究团队自建的视觉语言模型评测数据集，可实现从感知到认知能力逐级细分评估。此评测基准包含3000 道单项选择题 ，覆盖 20个细粒度评估维度。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1206","languages":["Chinese"],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2307.06281","primary":"visual-reasoning","source_record":"catalog_1f8d5e87988f20b7"},"modality":"multimodal","name":"MMBench","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2023-07-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1206-mmbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:pris-cv/mmbench-live","source_record":"bm_mmbench-live_b26d79ee"},"categories":["Agents","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MMBench-Live is a continuously evolving multimodal benchmark built by a multi-agent pipeline from MMBench. It contains 5.9K newly generated evaluation instances with executable reasoning, and evaluates vision-language models across question-answer generation and reasoning tasks. Updates cost about USD 30 and take 1-2 hours, with distribution-consistent strategy to maintain comparability.","descriptions":{"en":"MMBench-Live is a continuously evolving multimodal benchmark built by a multi-agent pipeline from MMBench. It contains 5.9K newly generated evaluation instances with executable reasoning, and evaluates vision-language models across question-answer generation and reasoning tasks. Updates cost about USD 30 and take 1-2 hours, with distribution-consistent strategy to maintain comparability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.01813","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mmbench-live_b26d79ee"},"modality":null,"name":"MMBench-Live","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01813","source_url":"https://arxiv.org/abs/2607.01813"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01813","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01813","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4439,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_1f8d5e87988f20b7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Version 1.1 of MMBench, an improved bilingual benchmark for assessing multi-modal capabilities of vision-language models through multiple-choice questions in both English and Chinese, providing systematic evaluation across diverse vision-language tasks.","descriptions":{"en":"Version 1.1 of MMBench, an improved bilingual benchmark for assessing multi-modal capabilities of vision-language models through multiple-choice questions in both English and Chinese, providing systematic evaluation across diverse vision-language tasks."},"evidence_summary":{"document_count":1,"model_count":20,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmbench-v1.1:deepseek-vl2","reported_at":"2024-12-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-v1.1?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:mmbench-v1.1","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","multilingual"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_1f8d5e87988f20b7"},"modality":"multimodal","name":"MMBench-V1.1","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":20,"score_direction":"higher_is_better","score_summary":{"display_max":92.80000000000001,"display_multiplier":100,"model_count":20,"model_count_basis":"source_model_id","numeric_count":20,"raw_max":0.928,"raw_min":0.683,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmbench-v1.1:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-v1.1?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-mmbench-v1-1","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-v1.1?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4439,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":33,"join":"llm_stats_id","repository":"gh:open-compass/vlmevalkit","source_record":"catalog_fffd1561ca889f23"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A long-form multi-shot benchmark for holistic video understanding that incorporates approximately 600 web videos from YouTube spanning 16 major categories, with each video ranging from 30 seconds to 6 minutes. Includes roughly 2,000 original question-answer pairs covering 26 fine-grained capabilities.","descriptions":{"en":"A long-form multi-shot benchmark for holistic video understanding that incorporates approximately 600 web videos from YouTube spanning 16 major categories, with each video ranging from 30 seconds to 6 minutes. Includes roughly 2,000 original question-answer pairs covering 26 fine-grained capabilities."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmbench-video:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-video?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmbench-video","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_fffd1561ca889f23"},"modality":"multimodal","name":"MMBench-Video","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":2.02,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.0202,"raw_min":0.0179,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmbench-video:qwen2.5-vl-72b","reported_date":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-video?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-mmbench-video","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-video?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","理解","Understanding","多模态模型","VLM","逻辑推理","视频理解","Video Understanding","官方自建","Official"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMBench-Video is a comprehensive video understanding evaluation benchmark that covers long videos, multiple shots, and evaluates the temporal understanding ability of MLLMs. Contains over 600 videos, 16 categories, and manually annotated Q&A pairs. MMBench-Video是全面视频理解评测基准，覆盖长视频、多镜头，评估MLLMs时序理解能力。包含16类共600+视频以及人工标注问答对。","descriptions":{"en":"MMBench-Video is a comprehensive video understanding evaluation benchmark that covers long videos, multiple shots, and evaluates the temporal understanding ability of MLLMs. Contains over 600 videos, 16 categories, and manually annotated Q&A pairs.","zh":"MMBench-Video是全面视频理解评测基准，覆盖长视频、多镜头，评估MLLMs时序理解能力。包含16类共600+视频以及人工标注问答对。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1207","languages":[],"library_direction":null,"modality":"multimodal","name":"MMBench-Video","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-06-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1207-mmbench-video","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMBench-Video","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:qszhan/mmdg-bench","source_record":"bm_mmdg-bench_d6d5ab19"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MMDG-Bench is a benchmark for multi-modal domain generalization, providing two frameworks (D2M and M2D) and unified protocols across action recognition and face anti-spoofing tasks. Includes code repository.","descriptions":{"en":"MMDG-Bench is a benchmark for multi-modal domain generalization, providing two frameworks (D2M and M2D) and unified protocols across action recognition and face anti-spoofing tasks. Includes code repository."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.00891","languages":[],"library_direction":null,"modality":null,"name":"MMDG-Bench","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00891","source_url":"https://arxiv.org/abs/2606.00891"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00891","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00891","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal dynamic stance classification in social media parent-reply interactions using 3,482 instances with a seven-label taxonomy, including a structured reasoning subset and challenge factor annotations.","descriptions":{"en":"Evaluates multimodal dynamic stance classification in social media parent-reply interactions using 3,482 instances with a seven-label taxonomy, including a structured reasoning subset and challenge factor annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30903","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_mmds-bench_7d68e839"},"modality":null,"name":"MMDS-Bench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30903","source_url":"https://arxiv.org/abs/2608.30903"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30903","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30903","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","NeurIPS 2024","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMDU is intended for evaluating the multi-image multi-turn dialogue capabilities. It comprises 110 high-quality multi-image multi-turn dialogues with more than 1600 questions, each accompanied by detailed long-form answers. MMDU用于评估大型视觉语言模型的多图像多轮对话能力，包含110个高质量的多图像多轮对话，由1600多个附有详细的长篇答案的问题组成。","descriptions":{"en":"MMDU is intended for evaluating the multi-image multi-turn dialogue capabilities. It comprises 110 high-quality multi-image multi-turn dialogues with more than 1600 questions, each accompanied by detailed long-form answers.","zh":"MMDU用于评估大型视觉语言模型的多图像多轮对话能力，包含110个高质量的多图像多轮对话，由1600多个附有详细的长篇答案的问题组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1334","languages":[],"library_direction":null,"modality":"multimodal","name":"MMDU","openness":"restricted","publisher":"Shanghai Jiao Tong University","released":"2024-06-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1334-mmdu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMDU","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":18052,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:bradyfu/awesome-multimodal-large-language-models","source_record":"catalog_c3583a3c34e250f1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive evaluation benchmark for Multimodal Large Language Models measuring both perception and cognition abilities across 14 subtasks. Features manually designed instruction-answer pairs to avoid data leakage and provides systematic quantitative assessment of MLLM capabilities.","descriptions":{"en":"A comprehensive evaluation benchmark for Multimodal Large Language Models measuring both perception and cognition abilities across 14 subtasks. Features manually designed instruction-answer pairs to avoid data leakage and provides systematic quantitative assessment of MLLM capabilities."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mme:deepseek-vl2","reported_at":"2024-12-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mme?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mme","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_c3583a3c34e250f1"},"modality":"multimodal","name":"MME","openness":"unknown","publisher":null,"released":"2023-06-23","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mme","source_url":"https://arxiv.org/abs/2306.13394"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":73.1,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.731,"raw_min":0.1915,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mme:lfm-2.5-vl-3b","reported_date":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mme?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"}],"slug":"llm-stats-mme","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mme?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":18052,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2306.13394","repository":"gh:bradyfu/awesome-multimodal-large-language-models","source_record":"catalog_c3583a3c34e250f1"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MME is a comprehensive MLLM evaluation benchmark. It measures both perception and cognition abilities on a total of 14 subtasks. MME是一个全面的多模态大模型评估基准，涵盖14 个考察感知和认知能力的子任务。","descriptions":{"en":"MME is a comprehensive MLLM evaluation benchmark. It measures both perception and cognition abilities on a total of 14 subtasks.","zh":"MME是一个全面的多模态大模型评估基准，涵盖14 个考察感知和认知能力的子任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1357","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2306.13394","primary":"visual-reasoning","source_record":"catalog_c3583a3c34e250f1"},"modality":"multimodal","name":"MME","openness":"open","publisher":"Tencent Youtu Lab","released":"2023-06-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1357-mme","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MME","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","多模态模型","VLM","逻辑推理","Reasoning","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MME-CoT is a specialized benchmark evaluating the CoT reasoning performance of LMMs, spanning six domains: math, science, OCR, logic, space-time, and general scenes. MME-CoT，一个专门用于评估 LMMs CoT 推理性能的基准，涵盖六个领域：数学、科学、OCR、逻辑、时空和一般场景。","descriptions":{"en":"MME-CoT is a specialized benchmark evaluating the CoT reasoning performance of LMMs, spanning six domains: math, science, OCR, logic, space-time, and general scenes.","zh":"MME-CoT，一个专门用于评估 LMMs CoT 推理性能的基准，涵盖六个领域：数学、科学、OCR、逻辑、时空和一般场景。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1565","languages":[],"library_direction":null,"modality":null,"name":"MME-CoT","openness":"restricted","publisher":"CUHK MMLab, CUHK MiuLar Lab, ByteDance, etc.","released":"2025-02-13","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1565-mme-cot","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MME-CoT","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":26,"join":"llm_stats_id","repository":null,"source_record":"catalog_0459b9ac661bb094"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive evaluation benchmark for Multimodal Large Language Models featuring over 13,366 high-resolution images and 29,429 question-answer pairs across 43 subtasks and 5 real-world scenarios. The largest manually annotated multimodal benchmark to date, designed to test MLLMs on challenging high-resolution real-world scenarios.","descriptions":{"en":"A comprehensive evaluation benchmark for Multimodal Large Language Models featuring over 13,366 high-resolution images and 29,429 question-answer pairs across 43 subtasks and 5 real-world scenarios. The largest manually annotated multimodal benchmark to date, designed to test MLLMs on challenging high-resolution real-world scenarios."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mme-realworld:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mme-realworld?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mme-realworld","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_0459b9ac661bb094"},"modality":"multimodal","name":"MME-RealWorld","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":61.6,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.616,"raw_min":0.616,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mme-realworld:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mme-realworld?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-mme-realworld","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mme-realworld?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":26,"join":"artifact:arxiv:2408.13257","repository":null,"source_record":"catalog_0459b9ac661bb094"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MME-RealWorld evaluates MLLMs' real-world recognition, featuring 13,366 high-resolution images averaging 2,000 × 1,500 pixels. MME-RealWorld用于评估多模态大模型对真实场景的理解能力，包含13366个平均2000*1500像素的高分辨率图像。","descriptions":{"en":"MME-RealWorld evaluates MLLMs' real-world recognition, featuring 13,366 high-resolution images averaging 2,000 × 1,500 pixels.","zh":"MME-RealWorld用于评估多模态大模型对真实场景的理解能力，包含13366个平均2000*1500像素的高分辨率图像。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1451","languages":["English","Chinese"],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2408.13257","primary":"visual-reasoning","source_record":"catalog_0459b9ac661bb094"},"modality":"multimodal","name":"MME-RealWorld","openness":"unknown","publisher":"Chinese Academy of Sciences","released":"2024-08-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1451-mme-realworld","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MME-RealWorld","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MMed-Bench-IR evaluates multilingual medical information retrieval across 6 languages with three tasks: cross-lingual QA retrieval, concept discrimination, and evidence retrieval for RAG.","descriptions":{"en":"MMed-Bench-IR evaluates multilingual medical information retrieval across 6 languages with three tasks: cross-lingual QA retrieval, concept discrimination, and evidence retrieval for RAG."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.24200","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","health-medicine","multilingual"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_mmed-bench-ir_04b38f2d"},"modality":null,"name":"MMed-Bench-IR","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24200","source_url":"https://arxiv.org/abs/2606.24200"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24200","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24200","topics":["language","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-sensor gait recognition dataset with visible, infrared, depth, LiDAR, and radar modalities in sequence correspondence.","descriptions":{"en":"Multi-sensor gait recognition dataset with visible, infrared, depth, LiDAR, and radar modalities in sequence correspondence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11601","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_mmgait_18d775f0"},"modality":null,"name":"MMGait","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11601","source_url":"https://arxiv.org/abs/2609.11601"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11601","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11601","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_mmgist_394485cc"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MMGist is a curated multimodal benchmark covering seven capability dimensions with 7,262 items, derived from 18 existing benchmarks via filtering pipelines. It evaluates vision-language models across dimensions such as Visual Logic and Expert Knowledge.","descriptions":{"en":"MMGist is a curated multimodal benchmark covering seven capability dimensions with 7,262 items, derived from 18 existing benchmarks via filtering pipelines. It evaluates vision-language models across dimensions such as Visual Logic and Expert Knowledge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.22437","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mmgist_394485cc"},"modality":null,"name":"MMGist","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22437","source_url":"https://arxiv.org/abs/2606.22437"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22437","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22437","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MMHBench is a multimodal benchmark for mental health understanding in long-form videos, comprising 268 videos and 2,184 questions across third-person and first-person settings. It evaluates model reasoning about mental states from multimodal evidence.","descriptions":{"en":"MMHBench is a multimodal benchmark for mental health understanding in long-form videos, comprising 268 videos and 2,184 questions across third-person and first-person settings. It evaluates model reasoning about mental states from multimodal evidence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27895","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","health-medicine"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_mmhbench_02d53b0e"},"modality":null,"name":"MMHBench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27895","source_url":"https://arxiv.org/abs/2607.27895"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27895","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27895","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMIE, a large-scale knowledge-intensive benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs). MMIE，这是一个大规模知识密集型基准，用于评估大型视觉-语言模型（LVLMs）中的交错多模态理解和生成。","descriptions":{"en":"MMIE, a large-scale knowledge-intensive benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs).","zh":"MMIE，这是一个大规模知识密集型基准，用于评估大型视觉-语言模型（LVLMs）中的交错多模态理解和生成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1523","languages":[],"library_direction":null,"modality":"multimodal","name":"MMIE","openness":"restricted","publisher":"UNC-Chapel Hill; University of Chicago; Microsoft Research; NUS","released":"2024-10-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1523-mmie","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMIE","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A benchmark for evaluating Multimodal Large Language Models (MLLMs) on detecting and reasoning about inconsistencies in layout-rich multimodal content. MMIR features 534 challenging samples across five reasoning-heavy inconsistency categories. 用于评估多模态模型（MLLM）检测和推理布局丰富的多模态内容中的不一致性的基准。MMIR 包含 534 个具有挑战性的样本，涉及五个推理能力较强的不一致类别。","descriptions":{"en":"A benchmark for evaluating Multimodal Large Language Models (MLLMs) on detecting and reasoning about inconsistencies in layout-rich multimodal content. MMIR features 534 challenging samples across five reasoning-heavy inconsistency categories.","zh":"用于评估多模态模型（MLLM）检测和推理布局丰富的多模态内容中的不一致性的基准。MMIR 包含 534 个具有挑战性的样本，涉及五个推理能力较强的不一致类别。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1547","languages":[],"library_direction":null,"modality":"multimodal","name":"MMIR","openness":"unknown","publisher":"University of California,  eBay","released":"2025-02-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1547-mmir","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMIR","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMIU test MLLMs' multi-image understanding capabilities, encompassesing 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions. MMIU用于评估多模态大模型的多图理解能力，包含7种类型的多图像关系、52个任务、77K图像和11K精心策划的多选题。","descriptions":{"en":"MMIU test MLLMs' multi-image understanding capabilities, encompassesing 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions.","zh":"MMIU用于评估多模态大模型的多图理解能力，包含7种类型的多图像关系、52个任务、77K图像和11K精心策划的多选题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1453","languages":[],"library_direction":null,"modality":"multimodal","name":"MMIU","openness":"restricted","publisher":"Shanghai AI Laboratory","released":"2024-08-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1453-mmiu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMIU","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_mmjailbench_1b3e3a1a"},"categories":["General AI","Multimodal","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal jailbreak vulnerabilities by factorizing harmful intent, prompt framing, visual semantics, and instruction carrier.","descriptions":{"en":"Evaluates multimodal jailbreak vulnerabilities by factorizing harmful intent, prompt framing, visual semantics, and instruction carrier."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25490","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_mmjailbench_1b3e3a1a"},"modality":null,"name":"MMJailBench","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25490","source_url":"https://arxiv.org/abs/2608.25490"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25490","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25490","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMKE-Bench is a benchmark designed to evaluate the ability of LMMs to edit visual knowledge in real-world scenarios. It includes 2,940 pieces of knowledge and 8,363 images across 33 broad categories, with automatically generated, human-verified evaluation questions. MMKE-Bench是一个旨在评估 LMM 在现实场景中编辑视觉知识能力的基准，包括 33 个广泛类别中的 2,940 条知识和 8,363 张图像，以及自动生成并由人工验证的评估问题。","descriptions":{"en":"MMKE-Bench is a benchmark designed to evaluate the ability of LMMs to edit visual knowledge in real-world scenarios. It includes 2,940 pieces of knowledge and 8,363 images across 33 broad categories, with automatically generated, human-verified evaluation questions.","zh":"MMKE-Bench是一个旨在评估 LMM 在现实场景中编辑视觉知识能力的基准，包括 33 个广泛类别中的 2,940 条知识和 8,363 张图像，以及自动生成并由人工验证的评估问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1578","languages":[],"library_direction":null,"modality":null,"name":"MMKE-Bench","openness":"restricted","publisher":"BIGAI, USTC, PKU, BIT","released":"2025-02-27","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1578-mmke-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMKE-Bench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":179,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":56,"join":"artifact:arxiv:2505.10610","repository":"gh:edinburghnlp/mmlongbench","source_record":"catalog_3cc95f78bfacb570"},"categories":["多模态","Multimodal","知识","Knowledge","长文本","Long-Context","Vision-Language","多模态模型","VLM","知识储备","长上下文","Long Context"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMLongBench is a benchmark that evaluates long-context vision-language models across various tasks, image types, and input lengths, revealing that single-task performance is insufficient for gauging overall vision-language long-context capability. MMLongBench 是一个针对长上下文视觉-语言模型的基准，覆盖多种任务、图像类型和输入长度。评测结果表明，单一任务的表现不足以衡量模型的整体长上下文能力。","descriptions":{"en":"MMLongBench is a benchmark that evaluates long-context vision-language models across various tasks, image types, and input lengths, revealing that single-task performance is insufficient for gauging overall vision-language long-context capability.","zh":"MMLongBench 是一个针对长上下文视觉-语言模型的基准，覆盖多种任务、图像类型和输入长度。评测结果表明，单一任务的表现不足以衡量模型的整体长上下文能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1851","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","visual-reasoning"],"join":"artifact:arxiv:2505.10610","primary":"long-context","source_record":"catalog_3cc95f78bfacb570"},"modality":"multimodal","name":"MMLongBench","openness":"open","publisher":"HKUST, Tencent AI Seattle Lab, University of Edinburgh, Miniml.AI, NVIDIA AI","released":"2025-05-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1851-mmlongbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMLongBench","topics":["multimodal","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":179,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":56,"join":"llm_stats_id","repository":"gh:edinburghnlp/mmlongbench","source_record":"catalog_3cc95f78bfacb570"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMLongBench-128K evaluates multimodal long-context understanding at a 128K token context length, testing how well vision-language models reason over very long mixed text and image inputs.","descriptions":{"en":"MMLongBench-128K evaluates multimodal long-context understanding at a 128K token context length, testing how well vision-language models reason over very long mixed text and image inputs."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlongbench-128k:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmlongbench-128k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","visual-reasoning"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_3cc95f78bfacb570"},"modality":"multimodal","name":"MMLongBench-128K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":78.3,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.783,"raw_min":0.769,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlongbench-128k:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-128k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-mmlongbench-128k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-128k?top_n=500","topics":["multimodal","reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":157,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_c3a53bd1982561d1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMLongBench-Doc evaluates long document understanding capabilities in vision-language models.","descriptions":{"en":"MMLongBench-Doc evaluates long document understanding capabilities in vision-language models."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlongbench-doc:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-doc?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmlongbench-doc","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","long-context"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_c3a53bd1982561d1"},"modality":"image","name":"MMLongBench-Doc","openness":"unknown","publisher":null,"released":"2024-07-01","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mmlongbench-doc","source_url":"https://arxiv.org/abs/2407.01523"},"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":0.62,"display_multiplier":1,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.62,"raw_min":0.562,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlongbench-doc:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-doc?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"}],"slug":"llm-stats-mmlongbench-doc","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-doc?top_n=500","topics":["multimodal","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":157,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2407.01523","repository":null,"source_record":"catalog_c3a53bd1982561d1"},"categories":["长文本","Long-Context","NeurIPS 2024","多模态模型","VLM","长上下文","Long Context","OCR与文档理解","OCR and Document Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMLongBench-Doc is a long-context multi-modal benchmark comprising 1,062 expert-annotated questions. MMLONGBENCH-DOC是一个长上下文的多模态基准，由1062个专家注释的问题组成。","descriptions":{"en":"MMLongBench-Doc is a long-context multi-modal benchmark comprising 1,062 expert-annotated questions.","zh":"MMLONGBENCH-DOC是一个长上下文的多模态基准，由1062个专家注释的问题组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1275","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","long-context"],"join":"artifact:arxiv:2407.01523","primary":"document-ocr","source_record":"catalog_c3a53bd1982561d1"},"modality":null,"name":"MMLongBench-Doc","openness":"unknown","publisher":"Nanyang Technological University","released":"2024-07-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1275-mmlongbench-doc","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMLongBench-Doc","topics":["multimodal","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Long Context","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MMLongEmbed is a benchmark for evaluating multimodal embedding models in long-context scenarios, covering four retrieval tasks across text, document, and video modalities with varying context lengths.","descriptions":{"en":"MMLongEmbed is a benchmark for evaluating multimodal embedding models in long-context scenarios, covering four retrieval tasks across text, document, and video modalities with varying context lengths."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.14747","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","long-context"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_mmlongembed_b6d36e5b"},"modality":null,"name":"MMLongEmbed","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14747","source_url":"https://arxiv.org/abs/2606.14747"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14747","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14747","topics":["multimodal","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1625,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"llm_stats_id","repository":"gh:hendrycks/test","source_record":"lib_mmlu"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","reasoning","language","finance","general","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains","descriptions":{"en":"Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains"},"evidence_summary":{"document_count":1,"model_count":101,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu:gpt-3.5-turbo-0125","reported_at":"2023-03-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:mmlu","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"lib_mmlu"},"modality":"text","name":"MMLU","openness":"unknown","publisher":null,"released":"2020-09-07","released_reference":{"basis":"paper_first_version","note":"First version introducing MMLU; this is a benchmark date, not a recent model release.","source_key":"opencompass:498","source_url":"https://arxiv.org/abs/2009.03300"},"repo_kind":null,"repo_resolution_status":null,"score_count":101,"score_direction":"higher_is_better","score_summary":{"display_max":92.5,"display_multiplier":100,"model_count":101,"model_count_basis":"source_model_id","numeric_count":101,"raw_max":0.925,"raw_min":0.419,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu:gpt-5-2025-08-07","reported_date":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu?top_n=500"},"unit":null},"scored_models":[{"date":"2024-03-13","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2024-08-30","date_precision":"model_announcement","name":"Command R+"},{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-02-15","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 20B"},{"date":"2023-03-21","date_precision":"model_announcement","name":"GPT-3.5 Turbo"},{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"},{"date":"2024-04-09","date_precision":"model_announcement","name":"GPT-4 Turbo"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2024-03-28","date_precision":"model_announcement","name":"Grok-1.5"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2 mini"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-01-20","date_precision":"model_announcement","name":"Kimi-k1.5"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 70B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Base 2512)"},{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2"},{"date":"2024-07-18","date_precision":"model_announcement","name":"Mistral NeMo Instruct"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Base"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Base"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Instruct"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"}],"slug":"llm-stats-mmlu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu?top_n=500","topics":["reasoning-math","language","finance","medical"],"unit":null},{"aliases":["MMLU"],"attention_signals":{"as_of":"2026-10-13","github_stars":1625,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"artifact:gh:hendrycks/test","repository":"gh:hendrycks/test","source_record":"catalog_3b45fa574794af78"},"categories":["knowledge"],"collected_at":null,"description":"Saturated at the frontier and heavily contaminated. Useful as a historical baseline, not as a ranking signal.","descriptions":{"en":"Saturated at the frontier and heavily contaminated. Useful as a historical baseline, not as a ranking signal."},"evidence_summary":{"document_count":9,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mmlu\u0000google_gemini_1_5_report\u0000mmlu\u00005-shot\u0000Gemini 1.5 Pro","reported_at":"2024-03-08","source_url":"https://arxiv.org/abs/2403.05530"},"first_score_reported_at":"2024-03-08","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mmlu\u0000google_gemini_1_5_report\u0000mmlu\u00005-shot\u0000Gemini 1.5 Pro","reported_at":"2024-03-08","source_url":"https://arxiv.org/abs/2403.05530"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:mmlu","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["knowledge-expert-qa"],"join":"review:14H034160212:2026-10-15","primary":"knowledge-expert-qa","source_record":null},"modality":null,"name":"MMLU","openness":"unknown","publisher":null,"released":"2020-09-07","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mmlu","source_url":"https://github.com/hendrycks/test"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":90.8,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":90.8,"raw_min":84.0,"source_reference":{"obs_id":"curated\u0000mmlu\u0000deepseek_r1_report\u0000mmlu\u0000EM\u0000DeepSeek-R1","observation_id":"curated\u0000mmlu\u0000deepseek_r1_report\u0000mmlu\u0000EM\u0000DeepSeek-R1","reported_at":"2025-01-22","reported_date":"2025-01-22","source_id":"deepseek_r1_report","source_url":"https://arxiv.org/abs/2501.12948"},"unit":"percent"},"scored_models":[{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2024-03-08","date_precision":"document_publication","name":"Gemini 1.5 Pro"},{"date":"2024-07-24","date_precision":"document_publication","name":"Mistral Large 2"}],"slug":"mmlu","source":"model_reports","source_url":"https://github.com/hendrycks/test","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","大语言模型","LLM","知识储备","Knowledge","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMLU (Massive Multitask Language Understanding) is a new benchmark designed to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This makes the benchmark more challenging and more similar to how we evaluate humans. The benchmark covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from an elementary level to an advanced professional level, and it tests both world knowledge and problem solving ability. Subjects range from traditional areas, such as mathematics and history, to more specialized areas like law and ethics. The granularity and breadth of the subjects makes the benchmark ideal for identifying a model’s blind spots. MMLU (Massive Multitask Language Understanding) 是一个新的基准测试，旨在通过在零次学习和少次学习的环境中评估模型来测量预训练期间获得的知识。这使得基准测试更具挑战性，且更接近我们评估人类的方式。该基准测试涵盖了STEM、人文学科、社会科学等57个主题。其难度范围从小学级别到专业级别，旨在测试世界知识和解决问题的能力。测试主题范围从传统领域，如数学和历史，到更专业的领域，如法律和伦理学。题目的精细度和广度使该基准测试成为识别模型盲点的理想选择。","descriptions":{"en":"MMLU (Massive Multitask Language Understanding) is a new benchmark designed to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This makes the benchmark more challenging and more similar to how we evaluate humans. The benchmark covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from an elementary level to an advanced professional level, and it tests both world knowledge and problem solving ability. Subjects range from traditional areas, such as mathematics and history, to more specialized areas like law and ethics. The granularity and breadth of the subjects makes the benchmark ideal for identifying a model’s blind spots.","zh":"MMLU (Massive Multitask Language Understanding) 是一个新的基准测试，旨在通过在零次学习和少次学习的环境中评估模型来测量预训练期间获得的知识。这使得基准测试更具挑战性，且更接近我们评估人类的方式。该基准测试涵盖了STEM、人文学科、社会科学等57个主题。其难度范围从小学级别到专业级别，旨在测试世界知识和解决问题的能力。测试主题范围从传统领域，如数学和历史，到更专业的领域，如法律和伦理学。题目的精细度和广度使该基准测试成为识别模型盲点的理想选择。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:498","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"identity_group:mmlu","primary":"knowledge-expert-qa","source_record":"lib_mmlu"},"modality":null,"name":"MMLU","openness":"unknown","publisher":null,"released":"2020-09-07","released_reference":{"basis":"paper_first_version","note":"First version introducing MMLU; this is a benchmark date, not a recent model release.","source_key":"opencompass:498","source_url":"https://arxiv.org/abs/2009.03300"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-498-mmlu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMLU","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"llm_stats_id","repository":null,"source_record":"catalog_f629c3f656123a31"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","reasoning","language","finance","general","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Chain-of-Thought variant of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. This version uses chain-of-thought prompting to elicit step-by-step reasoning.","descriptions":{"en":"Chain-of-Thought variant of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. This version uses chain-of-thought prompting to elicit step-by-step reasoning."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu-(cot):llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-%28cot%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmlu-(cot)","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_f629c3f656123a31"},"modality":"text","name":"MMLU (CoT)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":88.6,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.886,"raw_min":0.73,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu-(cot):llama-3.1-405b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-%28cot%29?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"}],"slug":"llm-stats-mmlu-cot","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-%28cot%29?top_n=500","topics":["reasoning-math","language","finance","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"llm_stats_id","repository":null,"source_record":"catalog_c35fcb235b7f78b2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","reasoning","language","finance","general","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Chat-format variant of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. This version uses conversational prompting format for model evaluation.","descriptions":{"en":"Chat-format variant of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. This version uses conversational prompting format for model evaluation."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu-chat:llama-3.1-nemotron-70b-instruct","reported_at":"2024-10-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-chat?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmlu-chat","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_c35fcb235b7f78b2"},"modality":"text","name":"MMLU Chat","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":80.58,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.8058,"raw_min":0.8058,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu-chat:llama-3.1-nemotron-70b-instruct","reported_date":"2024-10-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-chat?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"}],"slug":"llm-stats-mmlu-chat","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-chat?top_n=500","topics":["reasoning-math","language","finance","medical"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","reasoning","language","finance","general","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"French language variant of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. This multilingual version tests model performance in French.","descriptions":{"en":"French language variant of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. This multilingual version tests model performance in French."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu-french:mistral-large-2-2407","reported_at":"2024-07-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-french?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmlu-french","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_8c69d7c2b4104cfa"},"modality":"text","name":"MMLU French","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":82.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.828,"raw_min":0.828,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu-french:mistral-large-2-2407","reported_date":"2024-07-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-french?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2"}],"slug":"llm-stats-mmlu-french","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-french?top_n=500","topics":["reasoning-math","language","finance","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"llm_stats_id","repository":null,"source_record":"catalog_6581924d125c4391"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","reasoning","language","finance","general","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Base version of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. Designed to comprehensively measure the breadth and depth of a model's academic and professional understanding.","descriptions":{"en":"Base version of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. Designed to comprehensively measure the breadth and depth of a model's academic and professional understanding."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu-base:qwen-2.5-coder-7b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-base?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmlu-base","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_6581924d125c4391"},"modality":"text","name":"MMLU-Base","openness":"unknown","publisher":null,"released":"2020-09-07","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Measuring Massive Multitask Language Understanding. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mmlu-base","source_url":"https://arxiv.org/abs/2009.03300"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":68.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.68,"raw_min":0.68,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu-base:qwen-2.5-coder-7b-instruct","reported_date":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-base?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-mmlu-base","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-base?top_n=500","topics":["reasoning-math","language","finance","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":430,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":56,"join":"llm_stats_id","repository":"gh:tiger-ai-lab/mmlu-pro","source_record":"lib_mmlu_pro"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","reasoning","language","finance","general","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.","descriptions":{"en":"A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU."},"editorial_guide":{"id":"mmlu-pro","record_keys":["model-reports:mmlu_pro","llm-stats:mmlu-pro","opencompass:1276"],"reviewed_at":"2026-10-25","scope":{"en":"Original MMLU-Pro and its official evaluation resources; not original MMLU, MMLU-ProX or multilingual derivatives.","zh":"对应原版 MMLU-Pro 与官方评测资料，不等同于原始 MMLU、MMLU-ProX 或多语言衍生版本。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"The benchmark increases the option set from MMLU's four to ten and emphasizes reasoning-intensive questions. It filters trivial or noisy items and adds material from exams and textbooks, changing both task composition and guessing difficulty.","zh":"相比 MMLU 的四选一，该评测扩展至十个选项，并强调需要推理的题目。它清理简单或有噪声的问题，加入考试与教材材料，同时改变题目组成和猜测难度。"},"title":{"en":"Harder multiple-choice questions","zh":"更有挑战的选择题"}},{"id":"coverage","source_ids":["repository","dataset"],"text":{"en":"The official release contains more than 12,000 questions across 14 domains, including mathematics, law, health and engineering. The dataset includes a category field, so a broad average can be supplemented by the subjects relevant to your application.","zh":"官方发布包含超过 12,000 道题，覆盖数学、法律、健康、工程等 14 个领域。数据提供学科字段，因此可在总体平均之外，检查与你的应用相关的具体领域。"},"title":{"en":"Read subjects as well as the total","zh":"总分之外还要看学科"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"Evaluation extracts an answer option from generated text and checks it against the reference. The repository provides accuracy calculation and alternative extraction procedures. Preserve the extractor and prompting setup when reproducing a reported number.","zh":"评测从生成文本中提取选项，再与参考答案比较。官方仓库提供正确率计算及不同的提取方式。复现成绩时，应保留对应的提取器和提示设置。"},"title":{"en":"From generated text to an answer option","zh":"从生成文字到选项判定"}},{"id":"prompting","source_ids":["paper"],"text":{"en":"The paper studies both direct answering and chain-of-thought prompting, and tests variation across prompt styles. Greater robustness does not mean prompts are irrelevant. Compare models under the same examples and answer format before interpreting a difference.","zh":"论文研究直接作答和思维链提示，也比较多种提示风格。更稳健并不等于提示词没有影响。解释分差前，应确保示例和答案格式一致。"},"title":{"en":"Reasoning prompts remain part of the protocol","zh":"推理提示属于评测协议"}},{"id":"limits","source_ids":["dataset","paper"],"text":{"en":"Choosing a correct option is narrower than producing a useful professional deliverable. Category scores and example errors reveal more than the overall percentage alone. Keep the original test distinct from later variants with different languages or question pools.","zh":"选对一个答案，比交付有用的专业成果范围更窄。分学科结果与错误样例能提供比单一总分更多的信息；不同语言或不同题库的后续变体也应分开记录。"},"title":{"en":"A broad exam is still a bounded task","zh":"广泛知识考试仍有任务边界"}}],"sources":[{"id":"paper","title":"MMLU-Pro paper · Construction, prompts and limitations","url":"https://arxiv.org/html/2406.01574v1"},{"id":"repository","title":"TIGER AI Lab · MMLU-Pro evaluation and answer extraction","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"id":"dataset","title":"TIGER Lab · Official MMLU-Pro dataset and schema","url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro"}],"summary":{"en":"MMLU-Pro combines academic knowledge and reasoning across many subjects. It expands answer choices and revises the question pool, so its accuracy should not be read as the same test as MMLU.","zh":"MMLU-Pro 综合考查多个学科的知识与推理，扩充了选项并调整题库，因此不能把它的正确率当成原始 MMLU 的同一项测试。"},"title":"MMLU-Pro"},"evidence_summary":{"document_count":1,"model_count":134,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu-pro:claude-3-opus-20240229","reported_at":"2024-02-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-pro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:mmlu-pro","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"lib_mmlu_pro"},"modality":"text","name":"MMLU-Pro","openness":"open","publisher":"University of Waterloo","released":"2024-06-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":134,"score_direction":"higher_is_better","score_summary":{"display_max":90.33,"display_multiplier":100,"model_count":134,"model_count_basis":"source_model_id","numeric_count":134,"raw_max":0.9033,"raw_min":0.147,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu-pro:sakana-namazu","reported_date":"2026-08-03","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-pro?top_n=500"},"unit":null},"scored_models":[{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-02-29","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek-V3 0324"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2026-01-22","date_precision":"model_announcement","name":"ERNIE 5.0"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B High"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-03-28","date_precision":"model_announcement","name":"Grok-1.5"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2 mini"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE-236B-A23B"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 70B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Base"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Instruct"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Base"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Instruct"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Sakana Namazu"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"}],"slug":"llm-stats-mmlu-pro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-pro?top_n=500","topics":["reasoning-math","language","finance","medical"],"unit":null},{"aliases":["MMLU-Pro","MMLU Pro"],"attention_signals":{"as_of":"2026-10-13","github_stars":430,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":56,"join":"artifact:gh:tiger-ai-lab/mmlu-pro","repository":"gh:tiger-ai-lab/mmlu-pro","source_record":"catalog_10defe438c87d701"},"categories":["knowledge"],"collected_at":null,"description":"Static closed-set multiple choice; contamination risk rises over time.","descriptions":{"en":"Static closed-set multiple choice; contamination risk rises over time."},"editorial_guide":{"id":"mmlu-pro","record_keys":["model-reports:mmlu_pro","llm-stats:mmlu-pro","opencompass:1276"],"reviewed_at":"2026-10-25","scope":{"en":"Original MMLU-Pro and its official evaluation resources; not original MMLU, MMLU-ProX or multilingual derivatives.","zh":"对应原版 MMLU-Pro 与官方评测资料，不等同于原始 MMLU、MMLU-ProX 或多语言衍生版本。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"The benchmark increases the option set from MMLU's four to ten and emphasizes reasoning-intensive questions. It filters trivial or noisy items and adds material from exams and textbooks, changing both task composition and guessing difficulty.","zh":"相比 MMLU 的四选一，该评测扩展至十个选项，并强调需要推理的题目。它清理简单或有噪声的问题，加入考试与教材材料，同时改变题目组成和猜测难度。"},"title":{"en":"Harder multiple-choice questions","zh":"更有挑战的选择题"}},{"id":"coverage","source_ids":["repository","dataset"],"text":{"en":"The official release contains more than 12,000 questions across 14 domains, including mathematics, law, health and engineering. The dataset includes a category field, so a broad average can be supplemented by the subjects relevant to your application.","zh":"官方发布包含超过 12,000 道题，覆盖数学、法律、健康、工程等 14 个领域。数据提供学科字段，因此可在总体平均之外，检查与你的应用相关的具体领域。"},"title":{"en":"Read subjects as well as the total","zh":"总分之外还要看学科"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"Evaluation extracts an answer option from generated text and checks it against the reference. The repository provides accuracy calculation and alternative extraction procedures. Preserve the extractor and prompting setup when reproducing a reported number.","zh":"评测从生成文本中提取选项，再与参考答案比较。官方仓库提供正确率计算及不同的提取方式。复现成绩时，应保留对应的提取器和提示设置。"},"title":{"en":"From generated text to an answer option","zh":"从生成文字到选项判定"}},{"id":"prompting","source_ids":["paper"],"text":{"en":"The paper studies both direct answering and chain-of-thought prompting, and tests variation across prompt styles. Greater robustness does not mean prompts are irrelevant. Compare models under the same examples and answer format before interpreting a difference.","zh":"论文研究直接作答和思维链提示，也比较多种提示风格。更稳健并不等于提示词没有影响。解释分差前，应确保示例和答案格式一致。"},"title":{"en":"Reasoning prompts remain part of the protocol","zh":"推理提示属于评测协议"}},{"id":"limits","source_ids":["dataset","paper"],"text":{"en":"Choosing a correct option is narrower than producing a useful professional deliverable. Category scores and example errors reveal more than the overall percentage alone. Keep the original test distinct from later variants with different languages or question pools.","zh":"选对一个答案，比交付有用的专业成果范围更窄。分学科结果与错误样例能提供比单一总分更多的信息；不同语言或不同题库的后续变体也应分开记录。"},"title":{"en":"A broad exam is still a bounded task","zh":"广泛知识考试仍有任务边界"}}],"sources":[{"id":"paper","title":"MMLU-Pro paper · Construction, prompts and limitations","url":"https://arxiv.org/html/2406.01574v1"},{"id":"repository","title":"TIGER AI Lab · MMLU-Pro evaluation and answer extraction","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"id":"dataset","title":"TIGER Lab · Official MMLU-Pro dataset and schema","url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro"}],"summary":{"en":"MMLU-Pro combines academic knowledge and reasoning across many subjects. It expands answer choices and revises the question pool, so its accuracy should not be read as the same test as MMLU.","zh":"MMLU-Pro 综合考查多个学科的知识与推理，扩充了选项并调整题库，因此不能把它的正确率当成原始 MMLU 的同一项测试。"},"title":"MMLU-Pro"},"evidence_summary":{"document_count":13,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mmlu_pro\u0000deepseek_v3_report\u0000mmlu_pro\u0000EM, 8K output limit\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"first_score_reported_at":"2024-12-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mmlu_pro\u0000deepseek_v3_report\u0000mmlu_pro\u0000EM, 8K output limit\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:mmlu_pro","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"artifact:gh:tiger-ai-lab/mmlu-pro","primary":"knowledge-expert-qa","source_record":"catalog_10defe438c87d701"},"modality":null,"name":"MMLU-Pro","openness":"unknown","publisher":null,"released":"2024-06-03","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mmlu_pro","source_url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":87.8,"display_multiplier":1,"model_count":9,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":87.8,"raw_min":63.4,"source_reference":{"obs_id":"curated\u0000mmlu_pro\u0000qwen3_5_model_card\u0000mmlu_pro\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000mmlu_pro\u0000qwen3_5_model_card\u0000mmlu_pro\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"mmlu_pro","source":"model_reports","source_url":"https://github.com/TIGER-AI-Lab/MMLU-Pro","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":430,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":56,"join":"artifact:arxiv:2406.01574","repository":"gh:tiger-ai-lab/mmlu-pro","source_record":"catalog_10defe438c87d701"},"categories":["学科","Examination","NeurIPS 2024","大语言模型","LLM","知识储备","Knowledge"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMLU-Pro is the extension of MMLU, integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. MMLU-Pro是MMLU的扩展版本，涵盖了更具挑战性、以推理为重点的问题，并将选择集从4个选项扩展到10个选项。","descriptions":{"en":"MMLU-Pro is the extension of MMLU, integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options.","zh":"MMLU-Pro是MMLU的扩展版本，涵盖了更具挑战性、以推理为重点的问题，并将选择集从4个选项扩展到10个选项。"},"editorial_guide":{"id":"mmlu-pro","record_keys":["model-reports:mmlu_pro","llm-stats:mmlu-pro","opencompass:1276"],"reviewed_at":"2026-10-25","scope":{"en":"Original MMLU-Pro and its official evaluation resources; not original MMLU, MMLU-ProX or multilingual derivatives.","zh":"对应原版 MMLU-Pro 与官方评测资料，不等同于原始 MMLU、MMLU-ProX 或多语言衍生版本。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"The benchmark increases the option set from MMLU's four to ten and emphasizes reasoning-intensive questions. It filters trivial or noisy items and adds material from exams and textbooks, changing both task composition and guessing difficulty.","zh":"相比 MMLU 的四选一，该评测扩展至十个选项，并强调需要推理的题目。它清理简单或有噪声的问题，加入考试与教材材料，同时改变题目组成和猜测难度。"},"title":{"en":"Harder multiple-choice questions","zh":"更有挑战的选择题"}},{"id":"coverage","source_ids":["repository","dataset"],"text":{"en":"The official release contains more than 12,000 questions across 14 domains, including mathematics, law, health and engineering. The dataset includes a category field, so a broad average can be supplemented by the subjects relevant to your application.","zh":"官方发布包含超过 12,000 道题，覆盖数学、法律、健康、工程等 14 个领域。数据提供学科字段，因此可在总体平均之外，检查与你的应用相关的具体领域。"},"title":{"en":"Read subjects as well as the total","zh":"总分之外还要看学科"}},{"id":"scoring","source_ids":["repository"],"text":{"en":"Evaluation extracts an answer option from generated text and checks it against the reference. The repository provides accuracy calculation and alternative extraction procedures. Preserve the extractor and prompting setup when reproducing a reported number.","zh":"评测从生成文本中提取选项，再与参考答案比较。官方仓库提供正确率计算及不同的提取方式。复现成绩时，应保留对应的提取器和提示设置。"},"title":{"en":"From generated text to an answer option","zh":"从生成文字到选项判定"}},{"id":"prompting","source_ids":["paper"],"text":{"en":"The paper studies both direct answering and chain-of-thought prompting, and tests variation across prompt styles. Greater robustness does not mean prompts are irrelevant. Compare models under the same examples and answer format before interpreting a difference.","zh":"论文研究直接作答和思维链提示，也比较多种提示风格。更稳健并不等于提示词没有影响。解释分差前，应确保示例和答案格式一致。"},"title":{"en":"Reasoning prompts remain part of the protocol","zh":"推理提示属于评测协议"}},{"id":"limits","source_ids":["dataset","paper"],"text":{"en":"Choosing a correct option is narrower than producing a useful professional deliverable. Category scores and example errors reveal more than the overall percentage alone. Keep the original test distinct from later variants with different languages or question pools.","zh":"选对一个答案，比交付有用的专业成果范围更窄。分学科结果与错误样例能提供比单一总分更多的信息；不同语言或不同题库的后续变体也应分开记录。"},"title":{"en":"A broad exam is still a bounded task","zh":"广泛知识考试仍有任务边界"}}],"sources":[{"id":"paper","title":"MMLU-Pro paper · Construction, prompts and limitations","url":"https://arxiv.org/html/2406.01574v1"},{"id":"repository","title":"TIGER AI Lab · MMLU-Pro evaluation and answer extraction","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"id":"dataset","title":"TIGER Lab · Official MMLU-Pro dataset and schema","url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro"}],"summary":{"en":"MMLU-Pro combines academic knowledge and reasoning across many subjects. It expands answer choices and revises the question pool, so its accuracy should not be read as the same test as MMLU.","zh":"MMLU-Pro 综合考查多个学科的知识与推理，扩充了选项并调整题库，因此不能把它的正确率当成原始 MMLU 的同一项测试。"},"title":"MMLU-Pro"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1276","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"artifact:arxiv:2406.01574","primary":"knowledge-expert-qa","source_record":"catalog_10defe438c87d701"},"modality":null,"name":"MMLU-Pro","openness":"open","publisher":"University of Waterloo","released":"2024-06-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1276-mmlu-pro","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMLU-Pro","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_5914fd3a09cc9c6b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","reasoning","language","finance","general","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.","descriptions":{"en":"Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework."},"evidence_summary":{"document_count":1,"model_count":32,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu-prox:gemma-3n-e2b-it-litert-preview","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-prox?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:mmlu-prox","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_5914fd3a09cc9c6b"},"modality":"text","name":"MMLU-ProX","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":32,"score_direction":"higher_is_better","score_summary":{"display_max":87.0,"display_multiplier":100,"model_count":32,"model_count_basis":"source_model_id","numeric_count":32,"raw_max":0.87,"raw_min":0.081,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu-prox:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-prox?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-mmlu-prox","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-prox?top_n=500","topics":["reasoning-math","language","finance","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":32,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":"gh:aryopg/mmlu-redux","source_record":"catalog_ce91676a1a8503aa"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.","descriptions":{"en":"An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU."},"evidence_summary":{"document_count":1,"model_count":48,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu-redux:qwen-2.5-14b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:mmlu-redux","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_ce91676a1a8503aa"},"modality":"text","name":"MMLU-Redux","openness":"open","publisher":null,"released":"2024-06-06","released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":48,"score_direction":"higher_is_better","score_summary":{"display_max":95.0,"display_multiplier":100,"model_count":48,"model_count_basis":"source_model_id","numeric_count":48,"raw_max":0.95,"raw_min":0.432,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu-redux:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Base 2512)"},{"date":"2025-09-01","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-mmlu-redux","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":["MMLU-Redux"],"attention_signals":null,"categories":["knowledge"],"collected_at":null,"description":"Error-corrected MMLU subset; reported mainly by open-weight cards.","descriptions":{"en":"Error-corrected MMLU subset; reported mainly by open-weight cards."},"evidence_summary":{"document_count":5,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mmlu_redux\u0000deepseek_v3_report\u0000mmlu_redux\u0000EM, 8K output limit\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"first_score_reported_at":"2024-12-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mmlu_redux\u0000deepseek_v3_report\u0000mmlu_redux\u0000EM, 8K output limit\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:mmlu_redux","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"artifact:gh:aryopg/mmlu-redux","primary":"knowledge-expert-qa","source_record":"catalog_ce91676a1a8503aa"},"modality":null,"name":"MMLU-Redux","openness":"unknown","publisher":null,"released":"2024-06-06","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mmlu_redux","source_url":"https://github.com/aryopg/mmlu-redux"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":94.9,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":94.9,"raw_min":89.1,"source_reference":{"obs_id":"curated\u0000mmlu_redux\u0000qwen3_5_model_card\u0000mmlu_redux\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000mmlu_redux\u0000qwen3_5_model_card\u0000mmlu_redux\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2025-05-14","date_precision":"document_publication","name":"Qwen3-235B-A22B (Thinking)"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"mmlu_redux","source":"model_reports","source_url":"https://github.com/aryopg/mmlu-redux","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":32,"github_stars_30d":null,"hf_dataset_downloads":27495,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":"gh:aryopg/mmlu-redux","source_record":"catalog_f6f392877d0ea787"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A curated version of the MMLU benchmark featuring manually re-annotated 5,700 questions across 57 subjects to identify and correct errors in the original dataset. Addresses the 6.49% error rate found in MMLU and provides more reliable evaluation metrics for language models.","descriptions":{"en":"A curated version of the MMLU benchmark featuring manually re-annotated 5,700 questions across 57 subjects to identify and correct errors in the original dataset. Addresses the 6.49% error rate found in MMLU and provides more reliable evaluation metrics for language models."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu-redux-2.0:kimi-k2-base","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux-2.0?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmlu-redux-2.0","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_f6f392877d0ea787"},"modality":"text","name":"MMLU-redux-2.0","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":90.2,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.902,"raw_min":0.902,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu-redux-2.0:kimi-k2-base","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux-2.0?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"}],"slug":"llm-stats-mmlu-redux-2-0","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux-2.0?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"llm_stats_id","repository":null,"source_record":"catalog_efa4bb0861df8733"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","physics","reasoning","chemistry"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"STEM-focused subset of the Massive Multitask Language Understanding benchmark, evaluating language models on science, technology, engineering, and mathematics topics including physics, chemistry, mathematics, and other technical subjects.","descriptions":{"en":"STEM-focused subset of the Massive Multitask Language Understanding benchmark, evaluating language models on science, technology, engineering, and mathematics topics including physics, chemistry, mathematics, and other technical subjects."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmlu-stem:qwen-2.5-14b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-stem?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmlu-stem","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_efa4bb0861df8733"},"modality":"text","name":"MMLU-STEM","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":80.9,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.809,"raw_min":0.764,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmlu-stem:qwen-2.5-32b-instruct","reported_date":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-stem?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"}],"slug":"llm-stats-mmlu-stem","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-stem?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4665,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:openai/simple-evals","source_record":"catalog_39b617c84c45092e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.","descriptions":{"en":"Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects."},"evidence_summary":{"document_count":1,"model_count":49,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmmlu:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmlu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"llm-stats:mmmlu","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_39b617c84c45092e"},"modality":"text","name":"MMMLU","openness":"open","publisher":"OpenAI","released":null,"released_reference":null,"repo_kind":"harness_only","repo_resolution_status":"resolved","score_count":49,"score_direction":"higher_is_better","score_summary":{"display_max":92.7,"display_multiplier":100,"model_count":49,"model_count_basis":"source_model_id","numeric_count":49,"raw_max":0.927,"raw_min":0.443,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmmlu:claude-mythos-preview","reported_date":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmlu?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Opus 4"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude Opus 4.1"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Sonnet 4"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude Sonnet 4.5"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B High"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE-236B-A23B"},{"date":"2025-09-01","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Base)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 Eagle)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 NVFP4)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512)"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-mmmlu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmlu?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":["MMMLU","Multilingual MMLU","MMLU-ProX"],"attention_signals":{"as_of":"2026-10-13","github_stars":4665,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"artifact:hf:openai/mmmlu","repository":"gh:openai/simple-evals","source_record":"catalog_39b617c84c45092e"},"categories":["multilingual"],"collected_at":null,"description":"A language-count average. Two cards reporting \"MMMLU\" may be averaging over different language sets, so the counts must match to compare.","descriptions":{"en":"A language-count average. Two cards reporting \"MMMLU\" may be averaging over different language sets, so the counts must match to compare."},"evidence_summary":{"document_count":7,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mmmlu\u0000anthropic_claude_4_system_card\u0000mmmlu_14_non_english_languages\u000014 non-English languages, extended thinking up to 64K tokens\u0000Claude Opus 4","reported_at":"2025-05-22","source_url":"https://www.anthropic.com/news/claude-4"},"first_score_reported_at":"2025-05-22","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mmmlu\u0000anthropic_claude_4_system_card\u0000mmmlu_14_non_english_languages\u000014 non-English languages, extended thinking up to 64K tokens\u0000Claude Opus 4","reported_at":"2025-05-22","source_url":"https://www.anthropic.com/news/claude-4"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:mmmlu","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"artifact:hf:openai/mmmlu","primary":"multilingual","source_record":"catalog_39b617c84c45092e"},"modality":null,"name":"MMMLU","openness":"unknown","publisher":null,"released":"2024-09-24","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mmmlu","source_url":"https://huggingface.co/datasets/openai/MMMLU"},"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":92.6,"display_multiplier":1,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":92.6,"raw_min":86.5,"source_reference":{"obs_id":"curated\u0000mmmlu\u0000google_gemini_3_1_pro_model_card\u0000mmmlu\u0000Thinking (High)\u0000Gemini 3.1 Pro","observation_id":"curated\u0000mmmlu\u0000google_gemini_3_1_pro_model_card\u0000mmmlu\u0000Thinking (High)\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","reported_date":"2026-02-19","source_id":"google_gemini_3_1_pro_model_card","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"unit":"percent"},"scored_models":[{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Opus 4"},{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Sonnet 4"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"mmmlu","source":"model_reports","source_url":"https://huggingface.co/datasets/openai/MMMLU","topics":["language"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":597,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":40,"join":"llm_stats_id","repository":"gh:mmmu-benchmark/mmmu","source_record":"lib_mmmu"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.","descriptions":{"en":"MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields."},"evidence_summary":{"document_count":1,"model_count":63,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmmu:gpt-3.5-turbo-0125","reported_at":"2023-03-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:mmmu","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","knowledge-expert-qa"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"lib_mmmu"},"modality":"multimodal","name":"MMMU","openness":"open","publisher":"IN.AI Research","released":"2023-11-27","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":63,"score_direction":"higher_is_better","score_summary":{"display_max":86.0,"display_multiplier":100,"model_count":63,"model_count_basis":"source_model_id","numeric_count":63,"raw_max":0.86,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmmu:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Sonnet 4"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2024-02-15","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Gemini 2.0 Flash Thinking"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2023-03-21","date_precision":"model_announcement","name":"GPT-3.5 Turbo"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Instant"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Thinking"},{"date":"2024-03-28","date_precision":"model_announcement","name":"Grok-1.5"},{"date":"2024-04-12","date_precision":"model_announcement","name":"Grok-1.5V"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok-2 mini"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Grok-3"},{"date":"2025-01-20","date_precision":"model_announcement","name":"Kimi-k1.5"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Base"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Instruct"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-vision-instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2024-12-25","date_precision":"model_announcement","name":"QvQ-72B-Preview"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-01-15","date_precision":"model_announcement","name":"Step3-VL-10B"}],"slug":"llm-stats-mmmu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu?top_n=500","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":["MMMU","MMMU-Pro"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Image preprocessing and chain-of-thought settings affect results.","descriptions":{"en":"Image preprocessing and chain-of-thought settings affect results."},"evidence_summary":{"document_count":10,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mmmu\u0000anthropic_claude_4_system_card\u0000mmmu\u0000extended thinking up to 64K tokens\u0000Claude Opus 4","reported_at":"2025-05-22","source_url":"https://www.anthropic.com/news/claude-4"},"first_score_reported_at":"2025-05-22","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mmmu\u0000anthropic_claude_4_system_card\u0000mmmu\u0000extended thinking up to 64K tokens\u0000Claude Opus 4","reported_at":"2025-05-22","source_url":"https://www.anthropic.com/news/claude-4"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:mmmu","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["visual-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"visual-reasoning","source_record":null},"modality":null,"name":"MMMU","openness":"unknown","publisher":null,"released":"2023-11-27","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mmmu","source_url":"https://mmmu-benchmark.github.io/"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":85.0,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":85.0,"raw_min":74.4,"source_reference":{"obs_id":"curated\u0000mmmu\u0000qwen3_5_model_card\u0000mmmu\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000mmmu\u0000qwen3_5_model_card\u0000mmmu\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Opus 4"},{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Sonnet 4"},{"date":"2025-06-17","date_precision":"document_publication","name":"Gemini 2.5 Pro"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"mmmu","source":"model_reports","source_url":"https://mmmu-benchmark.github.io/","topics":["multimodal"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","Multimodal","多模态模型","VLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMMU is a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks. MMMU用于评估多模态大模型在复杂多学科任务中的表现，包括从大学考试和教科书中精心收集的11.5K多模态问题，涵盖六个核心学科：艺术与设计、商业、科学、健康与医学、人文与社会科学以及技术与工程。","descriptions":{"en":"MMMU is a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks.","zh":"MMMU用于评估多模态大模型在复杂多学科任务中的表现，包括从大学考试和教科书中精心收集的11.5K多模态问题，涵盖六个核心学科：艺术与设计、商业、科学、健康与医学、人文与社会科学以及技术与工程。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1248","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2311.16502","primary":"visual-reasoning","source_record":"catalog_2cc7090a8f862417"},"modality":null,"name":"MMMU","openness":"open","publisher":"IN.AI Research","released":"2023-11-27","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1248-mmmu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMMU","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":40,"join":"llm_stats_id","repository":null,"source_record":"catalog_2cc7090a8f862417"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Validation set of the Massive Multi-discipline Multimodal Understanding and Reasoning benchmark. Features college-level multimodal questions across 6 core disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering) spanning 30 subjects and 183 subfields with diverse image types including charts, diagrams, maps, and tables.","descriptions":{"en":"Validation set of the Massive Multi-discipline Multimodal Understanding and Reasoning benchmark. Features college-level multimodal questions across 6 core disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering) spanning 30 subjects and 183 subfields with diverse image types including charts, diagrams, maps, and tables."},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmmu-(val):gemma-3-12b-it","reported_at":"2025-03-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28val%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmmu-(val)","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","knowledge-expert-qa"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_2cc7090a8f862417"},"modality":"multimodal","name":"MMMU (val)","openness":"unknown","publication":{"evidence":[{"note":"The official repository links the original MMMU paper, matches the record's six disciplines, 30 subjects and 183 subfields, and lists a 900-question validation split separately from development and test data. It also labels MMMU-Pro as a later distinct benchmark.","url":"https://github.com/MMMU-Benchmark/MMMU/blob/main/README.md"}],"id":"mmmu-validation-original","links":[{"kind":"paper","title":"MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI","url":"https://arxiv.org/abs/2311.16502"},{"kind":"repo","title":"MMMU official repository","url":"https://github.com/MMMU-Benchmark/MMMU"}],"record_keys":["llm-stats:mmmu-(val)"],"reviewed_at":"2026-10-26","scope":{"en":"The validation split of the original MMMU benchmark. The publication and repository cover the full benchmark; test-set results and MMMU-Pro are separate evaluations.","zh":"原始 MMMU 的验证集。论文和仓库介绍完整评测；测试集成绩与 MMMU-Pro 属于不同的评测设置。"}},"publisher":null,"released":"2023-11-27","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mmmu-(val)","source_url":"https://arxiv.org/abs/2311.16502"},"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":78.10000000000001,"display_multiplier":100,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":0.781,"raw_min":0.329,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmmu-(val):qwen3-vl-32b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28val%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-mmmu-val","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28val%29?top_n=500","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":40,"join":"llm_stats_id","repository":null,"source_record":"catalog_2cc7090a8f862417"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Validation set of the Massive Multi-discipline Multimodal Understanding and Reasoning benchmark. Features college-level multimodal questions across 6 core disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering) spanning 30 subjects and 183 subfields with diverse image types including charts, diagrams, maps, and tables.","descriptions":{"en":"Validation set of the Massive Multi-discipline Multimodal Understanding and Reasoning benchmark. Features college-level multimodal questions across 6 core disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering) spanning 30 subjects and 183 subfields with diverse image types including charts, diagrams, maps, and tables."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmmu-(validation):claude-opus-4-20250514","reported_at":"2025-05-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28validation%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmmu-(validation)","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","knowledge-expert-qa"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_2cc7090a8f862417"},"modality":"multimodal","name":"MMMU (validation)","openness":"unknown","publisher":null,"released":"2023-11-27","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mmmu-(validation)","source_url":"https://arxiv.org/abs/2311.16502"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":80.7,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.807,"raw_min":0.732,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmmu-(validation):claude-opus-4-5-20251101","reported_date":"2025-11-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28validation%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Opus 4"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude Opus 4.1"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"}],"slug":"llm-stats-mmmu-validation","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28validation%29?top_n=500","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["multimodal","reasoning"],"collected_at":"2026-08-25T10:41:06Z","description":"Visual reasoning","descriptions":{"en":"Visual reasoning"},"evidence_summary":{"document_count":1,"model_count":245,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:mmmu-pro:1fc54cef-d179-48b1-a27d-046874e9b208","reported_at":"2024-03-04","source_url":"https://artificialanalysis.ai/evaluations/mmmu-pro"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:mmmu-pro","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["visual-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"visual-reasoning","source_record":null},"modality":null,"name":"MMMU-Pro","openness":"unknown","publisher":null,"released":"2024-09-05","released_reference":{"basis":"release_announcement","note":"The benchmark authors announce MMMU-Pro on September 5. The earlier MMMU release and the later public MMMU test answers do not date this distinct benchmark.","source_key":"artificial-analysis:mmmu-pro","source_url":"https://github.com/MMMU-Benchmark/MMMU"},"repo_kind":null,"repo_resolution_status":null,"score_count":245,"score_direction":"higher_is_better","score_summary":{"display_max":85.4913294797688,"display_multiplier":100,"model_count":245,"model_count_basis":"source_model_id","numeric_count":245,"raw_max":0.854913294797688,"raw_min":0.145086705202312,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:mmmu-pro:b2331108-72ed-415a-82d1-188633875bbc","reported_date":"2026-08-13","source_url":"https://artificialanalysis.ai/evaluations/mmmu-pro"},"unit":null},"scored_models":[{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Flash (Sep '24)"},{"date":"2024-10-03","date_precision":"model_announcement","name":"Gemini 1.5 Flash-8B"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Pro (Sep '24)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-04-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 90B (Vision)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-02-26","date_precision":"model_announcement","name":"Phi-4 Multimodal Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"}],"slug":"artificial-analysis-mmmu-pro","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/mmmu-pro","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":597,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":34,"join":"llm_stats_id","repository":"gh:mmmu-benchmark/mmmu","source_record":"lib_mmmu_pro"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A more robust multi-discipline multimodal understanding benchmark that enhances MMMU through a three-step process: filtering text-only answerable questions, augmenting candidate options, and introducing vision-only input settings. Achieves significantly lower model performance (16.8-26.9%) compared to original MMMU, providing more rigorous evaluation that closely mimics real-world scenarios.","descriptions":{"en":"A more robust multi-discipline multimodal understanding benchmark that enhances MMMU through a three-step process: filtering text-only answerable questions, augmenting candidate options, and introducing vision-only input settings. Achieves significantly lower model performance (16.8-26.9%) compared to original MMMU, providing more rigorous evaluation that closely mimics real-world scenarios."},"evidence_summary":{"document_count":1,"model_count":68,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmmu-pro:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:mmmu-pro","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","knowledge-expert-qa"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"lib_mmmu_pro"},"modality":"multimodal","name":"MMMU-Pro","openness":"open","publisher":null,"released":"2024-09-04","released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":68,"score_direction":"higher_is_better","score_summary":{"display_max":83.6,"display_multiplier":100,"model_count":68,"model_count_basis":"source_model_id","numeric_count":68,"raw_max":0.836,"raw_min":0.305,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmmu-pro:gemini-3.5-flash","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-mmmu-pro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["MMMU-Pro","MMMU Pro"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Harder vision-required subset of MMMU. Input image ordering changes the result, so the protocol has to travel with the number.","descriptions":{"en":"Harder vision-required subset of MMMU. Input image ordering changes the result, so the protocol has to travel with the number."},"evidence_summary":{"document_count":4,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mmmu_pro\u0000qwen3_5_model_card\u0000mmmu_pro\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mmmu_pro\u0000qwen3_5_model_card\u0000mmmu_pro\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:mmmu_pro","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2409.02813","primary":"visual-reasoning","source_record":"catalog_543b3e1b707ea114"},"modality":null,"name":"MMMU-Pro","openness":"unknown","publisher":null,"released":"2024-09-04","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mmmu_pro","source_url":"https://arxiv.org/abs/2409.02813"},"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":83.4,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":83.4,"raw_min":76.9,"source_reference":{"obs_id":"curated\u0000mmmu_pro\u0000moonshot_kimi_k3_model_card\u0000mmmu_pro\u0000reasoning=max, with Python\u0000Kimi K3","observation_id":"curated\u0000mmmu_pro\u0000moonshot_kimi_k3_model_card\u0000mmmu_pro\u0000reasoning=max, with Python\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"mmmu_pro","source":"model_reports","source_url":"https://arxiv.org/abs/2409.02813","topics":["multimodal"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":34,"join":"llm_stats_id","repository":null,"source_record":"catalog_edc9f28db2049b31"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMMU-Pro variant evaluated with tool access enabled.","descriptions":{"en":"MMMU-Pro variant evaluated with tool access enabled."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmmu-pro-with-tools:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro-with-tools?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmmu-pro-with-tools","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","tool-use"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_edc9f28db2049b31"},"modality":"multimodal","name":"MMMU-Pro (with tools)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":84.6,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.846,"raw_min":0.795,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmmu-pro-with-tools:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro-with-tools?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"}],"slug":"llm-stats-mmmu-pro-with-tools","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro-with-tools?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":40,"join":"llm_stats_id","repository":null,"source_record":"catalog_2cc7090a8f862417"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Validation set for MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) benchmark, designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning across Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.","descriptions":{"en":"Validation set for MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) benchmark, designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning across Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmmuval:qwen2-vl-72b","reported_at":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmuval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmmuval","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","knowledge-expert-qa"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_2cc7090a8f862417"},"modality":"multimodal","name":"MMMUval","openness":"unknown","publisher":null,"released":"2023-11-27","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mmmuval","source_url":"https://arxiv.org/abs/2311.16502"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":80.60000000000001,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.806,"raw_min":0.645,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmmuval:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmuval?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude Sonnet 4.5"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-mmmuval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmuval?top_n=500","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":31,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:zhuwenjie98/mmooc","source_record":"bm_mmooc_7beeb312"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MMOOC evaluates multimodal large language models on out-of-context (OOC) and shifted in-context (Shifted IC) visual question answering. It contains over 41K image-question pairs covering three question formats, eight shift types, and six visual scenarios. Responses are scored for accuracy and refusal rate, with an LLM-as-a-judge metric for reasoning correctness.","descriptions":{"en":"MMOOC evaluates multimodal large language models on out-of-context (OOC) and shifted in-context (Shifted IC) visual question answering. It contains over 41K image-question pairs covering three question formats, eight shift types, and six visual scenarios. Responses are scored for accuracy and refusal rate, with an LLM-as-a-judge metric for reasoning correctness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27637","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mmooc_7beeb312"},"modality":null,"name":"MMOOC","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27637","source_url":"https://arxiv.org/abs/2607.27637"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27637","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27637","topics":["multimodal"],"unit":null},{"aliases":["MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs"],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:aliens32/mmpcbench","source_record":"bm_mmpcbench-benchmarking-multimodal-large-la_afb36b3a"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal large language models on proactive critique across 3,146 instances with a taxonomy of 4 error types and 12 subtypes, measuring detection, diagnosis, and resolution using a hierarchical protocol.","descriptions":{"en":"Evaluates multimodal large language models on proactive critique across 3,146 instances with a taxonomy of 4 error types and 12 subtypes, measuring detection, diagnosis, and resolution using a hierarchical protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29286","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mmpcbench-benchmarking-multimodal-large-la_afb36b3a"},"modality":null,"name":"MMPCBench","openness":"unknown","publisher":null,"released":"2026-08-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29286","source_url":"https://arxiv.org/abs/2608.29286"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29286","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29286","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating multimodal prompt injection attacks on agentic AI frameworks.","descriptions":{"en":"A benchmark for evaluating multimodal prompt injection attacks on agentic AI frameworks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09404","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_mmpibench_8b6b75b3"},"modality":null,"name":"MMPIBench","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09404","source_url":"https://arxiv.org/abs/2609.09404"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09404","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09404","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"mmPISA-bench consists of 25 multiple-choice questions from PISA in 43 languages with official and machine translations, used to evaluate LLMs' reasoning across languages.","descriptions":{"en":"mmPISA-bench consists of 25 multiple-choice questions from PISA in 43 languages with official and machine translations, used to evaluate LLMs' reasoning across languages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07069","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","abstract-reasoning"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_mmpisa-bench_4787a595"},"modality":null,"name":"mmPISA-bench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07069","source_url":"https://arxiv.org/abs/2606.07069"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07069","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07069","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sod1010/mmposttrainbench","source_record":"bm_mmposttrainbench_06738050"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Autonomous research seeks sustained model improvements through iterative experimentation and feedback.","descriptions":{"en":"Autonomous research seeks sustained model improvements through iterative experimentation and feedback."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.05398","languages":[],"library_direction":{"basis":"model-review-daily","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_mmposttrainbench_06738050"},"modality":null,"name":"MMPostTrainBench","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05398","source_url":"https://arxiv.org/abs/2610.05398"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05398","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05398","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","语言","Language","Visual Place Recognition","Dataset and Benchmark","多模态模型","VLM","图像理解","Image Understanding","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMS-VPR is a large-scale multimodal dataset for street-level place recognition in pedestrian areas. It contains 78,575 images and 2,512 videos from 207 locations in Chengdu, China, with rich metadata and spatial graph structure. MMS-VPR 是一个面向复杂城市步行街区的 多模态街景视觉地点识别大规模数据集，采集自成都约 70,800 平方米的开放式商业街区，覆盖 207 个地点，包含 78,575 张图像和 2,512 段视频，每条数据均带有 GPS 坐标、时间戳和文本元信息。相较以车载视角和西方城市为主的传统数据集，MMS-VPR 更贴近真实、密集、多用途的街道空间，具备丰富的视角、时段和模态多样性。此外，该数据集构建了包含 81 个节点、125 条边的空间图结构，支持结构感知的地点识别方法。数据集还定义了两个子集（Edges 和Points），支持精细化和图结构评估任务，助力多模态与地理空间理解的交叉研究。","descriptions":{"en":"MMS-VPR is a large-scale multimodal dataset for street-level place recognition in pedestrian areas. It contains 78,575 images and 2,512 videos from 207 locations in Chengdu, China, with rich metadata and spatial graph structure.","zh":"MMS-VPR 是一个面向复杂城市步行街区的 多模态街景视觉地点识别大规模数据集，采集自成都约 70,800 平方米的开放式商业街区，覆盖 207 个地点，包含 78,575 张图像和 2,512 段视频，每条数据均带有 GPS 坐标、时间戳和文本元信息。相较以车载视角和西方城市为主的传统数据集，MMS-VPR 更贴近真实、密集、多用途的街道空间，具备丰富的视角、时段和模态多样性。此外，该数据集构建了包含 81 个节点、125 条边的空间图结构，支持结构感知的地点识别方法。数据集还定义了两个子集（Edges 和Points），支持精细化和图结构评估任务，助力多模态与地理空间理解的交叉研究。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1829","languages":[],"library_direction":null,"modality":"multimodal","name":"MMS-VPR","openness":"restricted","publisher":"University of Auckland & Hunan University","released":"2025-05-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1829-mms-vpr","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMS-VPR","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":null,"source_record":"catalog_69c5c54539958b82"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","search","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMSearch evaluates multimodal models on search-based retrieval and question answering tasks that require processing both visual and textual information from search results.","descriptions":{"en":"MMSearch evaluates multimodal models on search-based retrieval and question answering tasks that require processing both visual and textual information from search results."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmsearch:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmsearch","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_69c5c54539958b82"},"modality":"multimodal","name":"MMSearch","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":72.89999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.729,"raw_min":0.729,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmsearch:glm-5v-turbo","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"}],"slug":"llm-stats-mmsearch","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch?top_n=500","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"artifact:arxiv:2409.12959","repository":null,"source_record":"catalog_69c5c54539958b82"},"categories":["多模态","Multimodal","多模态模型","VLM","检索能力","Retrieval","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Logo MMSearch is a multimodal search benchmark crafted to evaluate the potential of LMMs to function as a multimodal AI search engine. This benchmark encompasses a meticulously collected dataset of 300 queries spanning 14 subfields. MMSearch 是一个多模态搜索基准，旨在评估大型语言模型（LMMs）作为多模态 AI 搜索引擎的潜力。该基准包含了一个精心收集的包含 300 个查询的数据集，涵盖 14 个子领域。","descriptions":{"en":"Logo MMSearch is a multimodal search benchmark crafted to evaluate the potential of LMMs to function as a multimodal AI search engine. This benchmark encompasses a meticulously collected dataset of 300 queries spanning 14 subfields.","zh":"MMSearch 是一个多模态搜索基准，旨在评估大型语言模型（LMMs）作为多模态 AI 搜索引擎的潜力。该基准包含了一个精心收集的包含 300 个查询的数据集，涵盖 14 个子领域。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1584","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"artifact:arxiv:2409.12959","primary":"deep-research","source_record":"catalog_69c5c54539958b82"},"modality":"multimodal","name":"MMSearch","openness":"unknown","publisher":"CUHK MMLab, ByteDance, etc.","released":"2024-09-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1584-mmsearch","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMSearch","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_b3627d7b9e6c153e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","search","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMSearch-Plus is an extended variant of MMSearch with harder multimodal search and retrieval tasks requiring deeper reasoning over visual and textual search results.","descriptions":{"en":"MMSearch-Plus is an extended variant of MMSearch with harder multimodal search and retrieval tasks requiring deeper reasoning over visual and textual search results."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmsearch-plus:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch-plus?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmsearch-plus","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_b3627d7b9e6c153e"},"modality":"multimodal","name":"MMSearch-Plus","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":41.4,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.414,"raw_min":0.3,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmsearch-plus:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch-plus?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-mmsearch-plus","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch-plus?top_n=500","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MMShopBench is a real-log benchmark for multimodal multi-turn shopping agents. It uses cleaned shopping logs with annotations for purchase intent and mandatory product requirements. Agents must infer requirements from images and dialogue, retrieve candidates, and verify product satisfaction.","descriptions":{"en":"MMShopBench is a real-log benchmark for multimodal multi-turn shopping agents. It uses cleaned shopping logs with annotations for purchase intent and mandatory product requirements. Agents must infer requirements from images and dialogue, retrieve candidates, and verify product satisfaction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.29002","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","visual-reasoning"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_mmshopbench_a4cdc83b"},"modality":null,"name":"MMShopBench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29002","source_url":"https://arxiv.org/abs/2607.29002"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29002","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29002","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":109,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"artifact:arxiv:2505.23764","repository":"gh:openrobotlab/mmsi-bench","source_record":"catalog_8626ef0d34057617"},"categories":["多模态","Multimodal","推理","Reasoning","空间智能","跨视角","物理智能","Embodied AI","逻辑推理","空间理解","Spatial Understanding","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMSI-Bench is a novel Visual Question Answering (VQA) benchmark specifically designed to evaluate Multi-image Spatial Intelligence in multimodal large language models (MLLMs). Unlike traditional datasets that focus on spatial reasoning within a single image, MMSI-Bench emphasizes real-world inspired MMSI-Bench 是一个全新的多模态空间智能视觉问答（VQA）基准数据集，专为评估多图像空间推理能力而设计。与专注于单图像关系推理的传统数据集不同，MMSI-Bench 更贴近现实世界，聚焦于需在多张图像之间进行逻辑推理的复杂场景。该数据集由六位三维视觉专家耗时超过300小时构建，精心整理出包含1,000个高质量选择题的问题集，题目来自12万余张图像，并附有精心设计的误导选项与逐步推理过程。对34个主流开源和闭源多模态大语言模型的实证评估显示：最先进的模型准确率仅为30%至40%，而人类表现高达97%，揭示该任务的巨大挑战性与模型发展空间。此外，MMSI-Bench 配套提供自动化错误分析","descriptions":{"en":"MMSI-Bench is a novel Visual Question Answering (VQA) benchmark specifically designed to evaluate Multi-image Spatial Intelligence in multimodal large language models (MLLMs). Unlike traditional datasets that focus on spatial reasoning within a single image, MMSI-Bench emphasizes real-world inspired","zh":"MMSI-Bench 是一个全新的多模态空间智能视觉问答（VQA）基准数据集，专为评估多图像空间推理能力而设计。与专注于单图像关系推理的传统数据集不同，MMSI-Bench 更贴近现实世界，聚焦于需在多张图像之间进行逻辑推理的复杂场景。该数据集由六位三维视觉专家耗时超过300小时构建，精心整理出包含1,000个高质量选择题的问题集，题目来自12万余张图像，并附有精心设计的误导选项与逐步推理过程。对34个主流开源和闭源多模态大语言模型的实证评估显示：最先进的模型准确率仅为30%至40%，而人类表现高达97%，揭示该任务的巨大挑战性与模型发展空间。此外，MMSI-Bench 配套提供自动化错误分析"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1922","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"artifact:arxiv:2505.23764","primary":"spatial-3d","source_record":"catalog_8626ef0d34057617"},"modality":"multimodal","name":"MMSI-Bench","openness":"unknown","publisher":"上海人工智能实验室","released":"2025-05-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1922-mmsi-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMSI-Bench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":109,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:openrobotlab/mmsi-bench","source_record":"catalog_8626ef0d34057617"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","spatial_reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMSIBench is a multimodal spatial-intelligence benchmark evaluating spatial reasoning over images.","descriptions":{"en":"MMSIBench is a multimodal spatial-intelligence benchmark evaluating spatial reasoning over images."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmsibench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsibench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmsibench","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_8626ef0d34057617"},"modality":"multimodal","name":"MMSIBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":35.9,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.359,"raw_min":0.314,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmsibench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsibench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-mmsibench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsibench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":218,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":null,"source_record":"catalog_377aca97e89d030c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMStar is an elite vision-indispensable multimodal benchmark comprising 1,500 challenge samples meticulously selected by humans to evaluate 6 core capabilities and 18 detailed axes. The benchmark addresses issues of visual content unnecessity and unintentional data leakage in existing multimodal evaluations.","descriptions":{"en":"MMStar is an elite vision-indispensable multimodal benchmark comprising 1,500 challenge samples meticulously selected by humans to evaluate 6 core capabilities and 18 detailed axes. The benchmark addresses issues of visual content unnecessity and unintentional data leakage in existing multimodal evaluations."},"evidence_summary":{"document_count":1,"model_count":24,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmstar:deepseek-vl2","reported_at":"2024-12-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmstar?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:mmstar","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_377aca97e89d030c"},"modality":"multimodal","name":"MMStar","openness":"restricted","publisher":"Shanghai AI Laboratory","released":"2024-04-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":24,"score_direction":"higher_is_better","score_summary":{"display_max":83.3,"display_multiplier":100,"model_count":24,"model_count_basis":"source_model_id","numeric_count":24,"raw_max":0.833,"raw_min":0.459,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmstar:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmstar?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"}],"slug":"llm-stats-mmstar","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmstar?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":218,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"artifact:arxiv:2403.20330","repository":"gh:mmstar-benchmark/mmstar","source_record":"catalog_0a7b4890e81878c3"},"categories":["其他","Other","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMStar is an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans. MMStar benchmarks 6 core capabilities and 18 detailed axes, aiming to evaluate LVLMs’ multi-modal capacities with carefully balanced and purified samples. MMStar 是一个多模态基准，包含 1,500 个经过人工精心挑选的样本。MMStar 评估 6 项核心能力和 18 个详细维度，旨在通过精心平衡和净化的样本，评估 LVLM 的多模态能力。","descriptions":{"en":"MMStar is an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans. MMStar benchmarks 6 core capabilities and 18 detailed axes, aiming to evaluate LVLMs’ multi-modal capacities with carefully balanced and purified samples.","zh":"MMStar 是一个多模态基准，包含 1,500 个经过人工精心挑选的样本。MMStar 评估 6 项核心能力和 18 个详细维度，旨在通过精心平衡和净化的样本，评估 LVLM 的多模态能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1175","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2403.20330","primary":"visual-reasoning","source_record":"catalog_0a7b4890e81878c3"},"modality":null,"name":"MMStar","openness":"restricted","publisher":"Shanghai AI Laboratory","released":"2024-04-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1175-mmstar","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMStar","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_3b039d0c4f726cef"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMT-Bench is a comprehensive multimodal benchmark for evaluating Large Vision-Language Models towards multitask AGI. It comprises 31,325 meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering 32 core meta-tasks and 162 subtasks in multimodal understanding.","descriptions":{"en":"MMT-Bench is a comprehensive multimodal benchmark for evaluating Large Vision-Language Models towards multitask AGI. It comprises 31,325 meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering 32 core meta-tasks and 162 subtasks in multimodal understanding."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmt-bench:deepseek-vl2","reported_at":"2024-12-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmt-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmt-bench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_3b039d0c4f726cef"},"modality":"multimodal","name":"MMT-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":63.6,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.636,"raw_min":0.532,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmt-bench:deepseek-vl2","reported_date":"2024-12-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmt-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-mmt-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmt-bench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:2404.16006","repository":null,"source_record":"catalog_3b039d0c4f726cef"},"categories":["多模态","Multimodal","推理","Reasoning","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MMT-Bench is designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. It comprises 31,325 multi-choice visual questions, covering 32 core meta-tasks and 162 subtasks in multimodal understanding. MMT-Bench考察多模态大模型的视觉识别、定位、推理和规划能力，包括31325个多选视觉问题，涵盖了32个核心元任务和162个多模态理解子任务。","descriptions":{"en":"MMT-Bench is designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. It comprises 31,325 multi-choice visual questions, covering 32 core meta-tasks and 162 subtasks in multimodal understanding.","zh":"MMT-Bench考察多模态大模型的视觉识别、定位、推理和规划能力，包括31325个多选视觉问题，涵盖了32个核心元任务和162个多模态理解子任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1364","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2404.16006","primary":"visual-reasoning","source_record":"catalog_3b039d0c4f726cef"},"modality":"multimodal","name":"MMT-Bench","openness":"open","publisher":"Shanghai AI Laboratory","released":"2024-04-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1364-mmt-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMT-Bench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Large language models (LLMs) demonstrate strong potential as agents for tool invocation due to their advanced comprehension and planning capabilities. Users increasingly rely on LLM-based agents to solve complex missions through iterative interactions. However, existing benchmarks predominantly acce 大型语言模型（LLM）凭借其先进的理解能力和规划能力，在作为工具调用的智能体方面展现出了巨大的潜力。用户越来越依赖基于大型语言模型的智能体，通过迭代交互来解决复杂任务。 然而，现有的基准测试主要是在单任务场景中评估智能体，无法体现现实世界的复杂性。为了填补这一空白，我们提出了 Multi-Mission Tool Bench 测试。在这个基准测试中，每个测试用例都包含多个相互关联的任务。这种设计要求智能体能够动态适应不断变化的需求。此外，所提出的基准测试探索了在固定任务数量下所有可能的任务切换模式。具体而言，我们提出了一个多智能体数据生成框架来构建这个基准测试。我们还提出了一种利用动态决策树来","descriptions":{"en":"Large language models (LLMs) demonstrate strong potential as agents for tool invocation due to their advanced comprehension and planning capabilities. Users increasingly rely on LLM-based agents to solve complex missions through iterative interactions. However, existing benchmarks predominantly acce","zh":"大型语言模型（LLM）凭借其先进的理解能力和规划能力，在作为工具调用的智能体方面展现出了巨大的潜力。用户越来越依赖基于大型语言模型的智能体，通过迭代交互来解决复杂任务。 然而，现有的基准测试主要是在单任务场景中评估智能体，无法体现现实世界的复杂性。为了填补这一空白，我们提出了 Multi-Mission Tool Bench 测试。在这个基准测试中，每个测试用例都包含多个相互关联的任务。这种设计要求智能体能够动态适应不断变化的需求。此外，所提出的基准测试探索了在固定任务数量下所有可能的任务切换模式。具体而言，我们提出了一个多智能体数据生成框架来构建这个基准测试。我们还提出了一种利用动态决策树来"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1736","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"MMTB","openness":"unknown","publisher":"Tencent","released":"2025-03-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1736-mmtb","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMTB","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"MMTClinic evaluates large language models on clinical time-series question answering and reasoning across five languages and three tasks: mortality prediction, heart rate forecasting, and SOFA score estimation.","descriptions":{"en":"MMTClinic evaluates large language models on clinical time-series question answering and reasoning across five languages and three tasks: mortality prediction, heart rate forecasting, and SOFA score estimation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.04842","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["health-medicine","forecasting","knowledge-expert-qa"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_mmtclinic_8420d333"},"modality":null,"name":"MMTClinic","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04842","source_url":"https://arxiv.org/abs/2609.04842"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04842","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04842","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":333,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"llm_stats_id","repository":null,"source_record":"catalog_d580a014be7a9964"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","reasoning","spatial_reasoning","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MM-Vet is an evaluation benchmark that examines large multimodal models on complicated multimodal tasks requiring integrated capabilities. It assesses six core vision-language capabilities: recognition, knowledge, spatial awareness, language generation, OCR, and math through questions that require one or more of these capabilities.","descriptions":{"en":"MM-Vet is an evaluation benchmark that examines large multimodal models on complicated multimodal tasks requiring integrated capabilities. It assesses six core vision-language capabilities: recognition, knowledge, spatial awareness, language generation, OCR, and math through questions that require one or more of these capabilities."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmvet:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvet?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmvet","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_d580a014be7a9964"},"modality":"multimodal","name":"MMVet","openness":"unknown","publisher":null,"released":"2023-08-04","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mmvet","source_url":"https://arxiv.org/abs/2308.02490"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":76.19,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.7619,"raw_min":0.671,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmvet:qwen2.5-vl-72b","reported_date":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvet?top_n=500"},"unit":null},"scored_models":[{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-mmvet","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvet?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":333,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"llm_stats_id","repository":null,"source_record":"catalog_93d1c075bf34e12c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","reasoning","spatial_reasoning","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MM-Vet evaluation using GPT-4 Turbo for scoring. This variant of MM-Vet examines large multimodal models on complicated multimodal tasks requiring integrated capabilities across six core vision-language abilities: recognition, knowledge, spatial awareness, language generation, OCR, and math.","descriptions":{"en":"MM-Vet evaluation using GPT-4 Turbo for scoring. This variant of MM-Vet examines large multimodal models on complicated multimodal tasks requiring integrated capabilities across six core vision-language abilities: recognition, knowledge, spatial awareness, language generation, OCR, and math."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmvetgpt4turbo:qwen2-vl-72b","reported_at":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvetgpt4turbo?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmvetgpt4turbo","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_93d1c075bf34e12c"},"modality":"multimodal","name":"MMVetGPT4Turbo","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":74.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.74,"raw_min":0.74,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmvetgpt4turbo:qwen2-vl-72b","reported_date":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvetgpt4turbo?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"}],"slug":"llm-stats-mmvetgpt4turbo","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvetgpt4turbo?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":77,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":82,"join":"llm_stats_id","repository":null,"source_record":"catalog_1a552b9a0512cd49"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMVU (Multimodal Multi-disciplinary Video Understanding) is a benchmark for evaluating multimodal models on video understanding tasks across multiple disciplines, testing comprehension and reasoning capabilities on video content.","descriptions":{"en":"MMVU (Multimodal Multi-disciplinary Video Understanding) is a benchmark for evaluating multimodal models on video understanding tasks across multiple disciplines, testing comprehension and reasoning capabilities on video content."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mmvu:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mmvu","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","knowledge-expert-qa"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_1a552b9a0512cd49"},"modality":"multimodal","name":"MMVU","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":80.4,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.804,"raw_min":0.723,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mmvu:kimi-k2.5","reported_date":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvu?top_n=500"},"unit":null},"scored_models":[{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-mmvu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvu?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["MMVU"],"attention_signals":null,"categories":["vision"],"collected_at":null,"description":"Video understanding benchmark. For non-video-native models, uses 1 fps frame extraction. Scores depend on frame sampling strategy and context length.","descriptions":{"en":"Video understanding benchmark. For non-video-native models, uses 1 fps frame extraction. Scores depend on frame sampling strategy and context length."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mmvu\u0000zai_glm_5_3_flash_model_card\u0000mmvu\u0000temp 1.0, top_p 0.95, max context 256K, 1 fps frame-extraction for non-video-native models\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mmvu\u0000zai_glm_5_3_flash_model_card\u0000mmvu\u0000temp 1.0, top_p 0.95, max context 256K, 1 fps frame-extraction for non-video-native models\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:mmvu","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["video-understanding"],"join":"review:14H034160212:2026-10-15","primary":"video-understanding","source_record":null},"modality":null,"name":"MMVU","openness":"unknown","publisher":null,"released":"2025-06-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mmvu","source_url":"https://github.com/MMVU/MMVU"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":80.5,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":80.5,"raw_min":80.5,"source_reference":{"obs_id":"curated\u0000mmvu\u0000zai_glm_5_3_flash_model_card\u0000mmvu\u0000temp 1.0, top_p 0.95, max context 256K, 1 fps frame-extraction for non-video-native models\u0000GLM-5.3-Flash","observation_id":"curated\u0000mmvu\u0000zai_glm_5_3_flash_model_card\u0000mmvu\u0000temp 1.0, top_p 0.95, max context 256K, 1 fps frame-extraction for non-video-native models\u0000GLM-5.3-Flash","reported_at":"2026-08-27","reported_date":"2026-08-27","source_id":"zai_glm_5_3_flash_model_card","source_url":"https://z.ai/blog/glm-5.3-flash"},"unit":"percent"},"scored_models":[{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"}],"slug":"mmvu","source":"model_reports","source_url":"https://github.com/MMVU/MMVU","topics":[],"unit":"percent"},{"aliases":["Can Language Models Understand mmWave Data? Benchmarking Large Language Models for mmWave Radar-Based Human Understanding"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"mmWave-QA is a benchmark for language-conditioned mmWave human perception, aggregating heterogeneous public datasets into six scenarios and five QA tasks for standardized evaluation.","descriptions":{"en":"mmWave-QA is a benchmark for language-conditioned mmWave human perception, aggregating heterogeneous public datasets into six scenarios and five QA tasks for standardized evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14179","languages":[],"library_direction":null,"modality":null,"name":"mmWave-QA","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14179","source_url":"https://arxiv.org/abs/2608.14179"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14179","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14179","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates agents' perceptual state construction in partially observable environments through a sequential glimpse-based MNIST digit search with lookback constraints, scored on recognition accuracy across memory representations.","descriptions":{"en":"Evaluates agents' perceptual state construction in partially observable environments through a sequential glimpse-based MNIST digit search with lookback constraints, scored on recognition accuracy across memory representations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.31022","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-memory","instruction-following"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_mnist-pro_997cca4c"},"modality":null,"name":"MNIST-PRO","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.31022","source_url":"https://arxiv.org/abs/2608.31022"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-31022","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31022","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","Cybersecurity","Mobile & Personal Computing"],"collected_at":"2026-10-14T09:44:51Z","description":"MobileCybench is a benchmark for vulnerability discovery by AI agents in 13 Android applications.","descriptions":{"en":"MobileCybench is a benchmark for vulnerability discovery by AI agents in 13 Android applications."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23980","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["computer-use","cybersecurity"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_mobilecybench_2e77399f"},"modality":null,"name":"MobileCybench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23980","source_url":"https://arxiv.org/abs/2609.23980"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23980","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23980","topics":["agents","cybersecurity"],"unit":null},{"aliases":["Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation"],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.HC"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark that tests whether multimodal LLMs can turn multi-screen mobile app designs into buildable projects with working navigation.","descriptions":{"en":"A benchmark that tests whether multimodal LLMs can turn multi-screen mobile app designs into buildable projects with working navigation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.28645","languages":[],"library_direction":null,"modality":null,"name":"MobileForge","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28645","source_url":"https://arxiv.org/abs/2607.28645"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28645","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28645","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_mobilejudgebench_5adbb014"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MobileJudgeBench evaluates LLM-as-judge methods on mobile agent trajectories. It includes 931 human-annotated trajectories from 6 mobile agent benchmarks, covering 4 agent models and 68 apps.","descriptions":{"en":"MobileJudgeBench evaluates LLM-as-judge methods on mobile agent trajectories. It includes 931 human-annotated trajectories from 6 mobile agent benchmarks, covering 4 agent models and 68 apps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.11434","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_mobilejudgebench_5adbb014"},"modality":null,"name":"MobileJudgeBench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11434","source_url":"https://arxiv.org/abs/2608.11434"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11434","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11434","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":950,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:google-research/android_world","source_record":"catalog_846e7f8f5af22dad"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","frontend_development","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MobileMiniWob++ SR (Success Rate) is an adaptation of the MiniWob++ web interaction benchmark for mobile Android environments within AndroidWorld. It comprises 92 web interaction tasks adapted for touch-based mobile interfaces, evaluating agents' ability to navigate and interact with web applications on mobile devices.","descriptions":{"en":"MobileMiniWob++ SR (Success Rate) is an adaptation of the MiniWob++ web interaction benchmark for mobile Android environments within AndroidWorld. It comprises 92 web interaction tasks adapted for touch-based mobile interfaces, evaluating agents' ability to navigate and interact with web applications on mobile devices."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mobileminiwob++-sr:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileminiwob%2B%2B-sr?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mobileminiwob++-sr","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_846e7f8f5af22dad"},"modality":"multimodal","name":"MobileMiniWob++_SR","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":91.4,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.914,"raw_min":0.68,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mobileminiwob++-sr:qwen2.5-vl-7b","reported_date":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileminiwob%2B%2B-sr?top_n=500"},"unit":null},"scored_models":[{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-mobileminiwob-sr","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileminiwob%2B%2B-sr?top_n=500","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":42,"join":"claire_record_id","repository":null,"source_record":"bm_mobilepa-bench_6f002001"},"categories":["Agents & Tool Use","Planning","Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates mobile planning agents on tool-calling and planning across 13 domains and 212 tools in an interactive sandbox, with scoring on three advanced planning dimensions.","descriptions":{"en":"Evaluates mobile planning agents on tool-calling and planning across 13 domains and 212 tools in an interactive sandbox, with scoring on three advanced planning dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23035","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","computer-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_mobilepa-bench_6f002001"},"modality":null,"name":"MobilePA-Bench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23035","source_url":"https://arxiv.org/abs/2608.23035"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23035","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23035","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"llm_stats_id","repository":null,"source_record":"catalog_d344e4a4e7791f4f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MobileWorld is a benchmark for evaluating multimodal agents on real mobile-device tasks, testing GUI grounding, navigation, and multi-step task completion in mobile environments.","descriptions":{"en":"MobileWorld is a benchmark for evaluating multimodal agents on real mobile-device tasks, testing GUI grounding, navigation, and multi-step task completion in mobile environments."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mobileworld:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileworld?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mobileworld","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","tool-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_d344e4a4e7791f4f"},"modality":"multimodal","name":"MobileWorld","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":77.8,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.778,"raw_min":0.7,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mobileworld:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileworld?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-mobileworld","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileworld?top_n=500","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","Computer use","Cybersecurity","Mobile & Personal Computing","Mobile interaction","Prompt-injection resistance","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"MobileWorldSafety evaluates GUI agents' safety against environmental injection attacks in Android apps. It includes 142 risk tasks on real applications, with programmatically verifiable risk indicators and a two-stage pipeline for verification.","descriptions":{"en":"MobileWorldSafety evaluates GUI agents' safety against environmental injection attacks in Android apps. It includes 142 risk tasks on real applications, with programmatically verifiable risk indicators and a two-stage pipeline for verification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17659","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","computer-use"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_mobileworldsafety_b2abd1b3"},"modality":null,"name":"MobileWorldSafety","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17659","source_url":"https://arxiv.org/abs/2608.17659"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17659","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17659","topics":["computer-use","agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces.","descriptions":{"en":"A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11064","languages":[],"library_direction":{"basis":"model-review-daily","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_modebench_e1ca6e58"},"modality":null,"name":"ModeBench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11064","source_url":"https://arxiv.org/abs/2610.11064"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11064","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11064","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"A framework for recursive self-improvement of agent harnesses using benchmark-disjoint evolution tasks.","descriptions":{"en":"A framework for recursive self-improvement of agent harnesses using benchmark-disjoint evolution tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.14857","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_modularrsi_65278c4a"},"modality":null,"name":"ModularRSI","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.14857","source_url":"https://arxiv.org/abs/2609.14857"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-14857","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14857","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MoHallBench evaluates motion hallucination in video LLMs with 11,306 video clips and 40,493 QA pairs, covering three hallucination sources and multiple choice settings.","descriptions":{"en":"MoHallBench evaluates motion hallucination in video LLMs with 11,306 video clips and 40,493 QA pairs, covering three hallucination sources and multiple choice settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01117","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","factuality"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_mohallbench_4062fe02"},"modality":null,"name":"MoHallBench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01117","source_url":"https://arxiv.org/abs/2607.01117"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01117","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01117","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Language & Knowledge","Materials & Chemistry"],"collected_at":"2026-10-14T09:44:51Z","description":"A scenario-grounded benchmark for evaluating tool-augmented LLM agents in molecular design tasks, intended to reflect real-world molecular design contexts.","descriptions":{"en":"A scenario-grounded benchmark for evaluating tool-augmented LLM agents in molecular design tasks, intended to reflect real-world molecular design contexts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27349","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_moldesignbench_19da737c"},"modality":null,"name":"MolDesignBench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27349","source_url":"https://arxiv.org/abs/2609.27349"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27349","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27349","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"claire_record_id","repository":null,"source_record":"bm_mole_2cb825f9"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"MOLE evaluates threat-detection monitors over 150 AI-operated accounts sharing 9 stateful services across 30 workdays, covering 12 insider threat types and 8 model-generated corpora.","descriptions":{"en":"MOLE evaluates threat-detection monitors over 150 AI-operated accounts sharing 9 stateful services across 30 workdays, covering 12 insider threat types and 8 model-generated corpora."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.06966","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","safety-alignment"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_mole_2cb825f9"},"modality":null,"name":"MOLE","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06966","source_url":"https://arxiv.org/abs/2609.06966"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06966","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06966","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","科学智能","AI for Science","逻辑推理","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MolPuzzle is a benchmark for evaluating MLM's reasoning ability, comprising 234 instances of structure elucidation, which feature over 18,000 QA samples. MolPuzzle用于考察MLM的推理能力，包含234个结构解析实例以及超过18000个QA样本。","descriptions":{"en":"MolPuzzle is a benchmark for evaluating MLM's reasoning ability, comprising 234 instances of structure elucidation, which feature over 18,000 QA samples.","zh":"MolPuzzle用于考察MLM的推理能力，包含234个结构解析实例以及超过18000个QA样本。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1273","languages":[],"library_direction":null,"modality":null,"name":"MolPuzzle","openness":"open","publisher":"University of Notre Dame","released":"2024-09-26","released_reference":{"basis":"paper_first_version","note":"Public publication date on the original NeurIPS submission. The separately imported bibliographic record's year-boundary date is not used.","source_key":"opencompass:1273","source_url":"https://openreview.net/forum?id=t1mAXb4Cop"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1273-molpuzzle","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MolPuzzle","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Materials & Chemistry","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"MolSafeEval evaluates safety risks in AI-generated molecules across four task types, using structured knowledge graphs and LLM-based reasoning to detect unsafe features.","descriptions":{"en":"MolSafeEval evaluates safety risks in AI-generated molecules across four task types, using structured knowledge graphs and LLM-based reasoning to detect unsafe features."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.00464","languages":[],"library_direction":{"basis":"model-review","categories":["science","safety-alignment"],"join":"claire_record_id","primary":"science","source_record":"bm_molsafeeval_ff6a51a3"},"modality":null,"name":"MolSafeEval","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00464","source_url":"https://arxiv.org/abs/2607.00464"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00464","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00464","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"claire_record_id","repository":null,"source_record":"bm_moment-video_26d94dc0"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Moment-Video evaluates video MLLMs on momentary visual event understanding through 1,000 human-verified video-QA pairs across four task types: temporal occurrence, temporal counting, action description, and temporal reasoning.","descriptions":{"en":"Moment-Video evaluates video MLLMs on momentary visual event understanding through 1,000 human-verified video-QA pairs across four task types: temporal occurrence, temporal counting, action description, and temporal reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02522","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_moment-video_26d94dc0"},"modality":null,"name":"Moment-Video","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02522","source_url":"https://arxiv.org/abs/2606.02522"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02522","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02522","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_momento_3a4ed98e"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Momento benchmarks persistent agentic task completion in multi-session service environments, requiring agents to resolve temporal dependencies and evolving user goals across sessions.","descriptions":{"en":"Momento benchmarks persistent agentic task completion in multi-session service environments, requiring agents to resolve temporal dependencies and evolving user goals across sessions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00832","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","tool-use"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_momento_3a4ed98e"},"modality":null,"name":"Momento","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00832","source_url":"https://arxiv.org/abs/2606.00832"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00832","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00832","topics":["agents","reasoning-math"],"unit":null},{"aliases":["Racing in Volume with Flow Ensembles"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_racing-in-volume-with-flow-ensembles_7628512f"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Photorealistic 4D reconstruction benchmark with Formula 1 sequences captured from sparse external cameras under varied illumination, including trackside, onboard, and drone viewpoints with dense ground truth.","descriptions":{"en":"Photorealistic 4D reconstruction benchmark with Formula 1 sequences captured from sparse external cameras under varied illumination, including trackside, onboard, and drone viewpoints with dense ground truth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16310","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","video-understanding"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_racing-in-volume-with-flow-ensembles_7628512f"},"modality":null,"name":"Monaco4D","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16310","source_url":"https://arxiv.org/abs/2609.16310"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16310","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16310","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","Stereo Conversion","多模态模型","VLM","视觉生成","Visual Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"For evaluating stereo image conversion, it provides test data for five different scenarios: animation, indoor, outdoor, complex, and simple, with a total of approximately 2,500 test sample pairs. It also offers evaluation metrics for assessing the stereo effect. 用于测评立体影像转换，提供了动画，室内，室外，复杂，简单共五种场景的测试数据，总共约2500对测试样本。并提供了用于测评立体效果的评价指标-立体交并比。","descriptions":{"en":"For evaluating stereo image conversion, it provides test data for five different scenarios: animation, indoor, outdoor, complex, and simple, with a total of approximately 2,500 test sample pairs. It also offers evaluation metrics for assessing the stereo effect.","zh":"用于测评立体影像转换，提供了动画，室内，室外，复杂，简单共五种场景的测试数据，总共约2500对测试样本。并提供了用于测评立体效果的评价指标-立体交并比。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1704","languages":[],"library_direction":null,"modality":null,"name":"Mono2Stereo","openness":"restricted","publisher":null,"released":"2025-03-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1704-mono2stereo","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Mono2Stereo","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:zimoqingfeng/more","source_record":"bm_more_09a372c7"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MORE evaluates multilingual document parsing across 149 languages, covering text, formulas, tables, code blocks, catalogs, and reading order, using 1,288 real-world document images with human-refined annotations.","descriptions":{"en":"MORE evaluates multilingual document parsing across 149 languages, covering text, formulas, tables, code blocks, catalogs, and reading order, using 1,288 real-world document images with human-refined annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.02956","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","multilingual"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_more_09a372c7"},"modality":null,"name":"MORE","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02956","source_url":"https://arxiv.org/abs/2607.02956"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02956","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02956","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","多模态","Multimodal","推理","Reasoning","多模态模型","VLM","逻辑推理","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MORSE-500 introduces 500 programmatically generated videos testing 6 reasoning types: abstract, physical, planning, spatial, temporal, mathematical. Its controllable generation (via Manim, Matplotlib, generative models etc.) enables scalable difficulty, designed to evolve as SOTA models improve. 当前多模态推理基准存在三大不足：依赖静态图像、偏重数学解题、易饱和。为此，我们提出MORSE-500视频基准：包含500个脚本化视频片段，涵盖抽象、物理、规划、空间、时间、数学六类推理问题。其核心在于程序化生成（使用Manim、Matplotlib等），可精确控制视觉复杂度、干扰物密度和时序动态，从而系统化提升难度。与易过时的静态基准不同，MORSE-500具备可持续演进能力，其可控生成流程能无限创建新挑战实例。在顶尖模型（Gemini 2.5 Pro、OpenAI o3等）上的测试揭示了显著性能差距，尤其在抽象和规划任务上。我们开源数据集、生成脚本及评估工具，以促进透明、可复现的前沿研究。","descriptions":{"en":"MORSE-500 introduces 500 programmatically generated videos testing 6 reasoning types: abstract, physical, planning, spatial, temporal, mathematical. Its controllable generation (via Manim, Matplotlib, generative models etc.) enables scalable difficulty, designed to evolve as SOTA models improve.","zh":"当前多模态推理基准存在三大不足：依赖静态图像、偏重数学解题、易饱和。为此，我们提出MORSE-500视频基准：包含500个脚本化视频片段，涵盖抽象、物理、规划、空间、时间、数学六类推理问题。其核心在于程序化生成（使用Manim、Matplotlib等），可精确控制视觉复杂度、干扰物密度和时序动态，从而系统化提升难度。与易过时的静态基准不同，MORSE-500具备可持续演进能力，其可控生成流程能无限创建新挑战实例。在顶尖模型（Gemini 2.5 Pro、OpenAI o3等）上的测试揭示了显著性能差距，尤其在抽象和规划任务上。我们开源数据集、生成脚本及评估工具，以促进透明、可复现的前沿研究。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1999","languages":[],"library_direction":null,"modality":"multimodal","name":"MORSE-500","openness":"unknown","publisher":"University of Maryland","released":"2025-06-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1999-morse-500","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MORSE-500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_mortarbench_63bbc02d"},"categories":["Agents","Finance","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"MortarBench evaluates LLM-based agents on mortgage loan origination tasks, covering application, underwriting, approval, and funding. It uses synthetic data with edge-case coverage and exact match accuracy scoring.","descriptions":{"en":"MortarBench evaluates LLM-based agents on mortgage loan origination tasks, covering application, underwriting, approval, and funding. It uses synthetic data with edge-case coverage and exact match accuracy scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19416","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","tool-use"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_mortarbench_63bbc02d"},"modality":null,"name":"MortarBench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19416","source_url":"https://arxiv.org/abs/2606.19416"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19416","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19416","topics":["agents","finance"],"unit":null},{"aliases":["Beyond Global Scalars: Synergizing Token-Level Statistics and Deep Semantics for Adversarial AIGC Text Detection"],"attention_signals":{"as_of":"2026-10-11","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:tencentbac/neurostat","source_record":"bm_beyond-global-scalars-synergizing-token-le_0d7c6f3e"},"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates adversarial robustness of machine-generated text detection across 16,000 samples covering token-level to document-level attacks.","descriptions":{"en":"Evaluates adversarial robustness of machine-generated text detection across 16,000 samples covering token-level to document-level attacks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.28009","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_beyond-global-scalars-synergizing-token-le_0d7c6f3e"},"modality":null,"name":"MOSAIC","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28009","source_url":"https://arxiv.org/abs/2608.28009"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28009","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28009","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MosaicLeaks is a benchmark of 1,001 multi-hop research tasks to evaluate privacy leakage in deep research agents, but no artifacts are provided in this article.","descriptions":{"en":"MosaicLeaks is a benchmark of 1,001 multi-hop research tasks to evaluate privacy leakage in deep research agents, but no artifacts are provided in this article."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30727","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","deep-research"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_mosaicleaks_86df075c"},"modality":null,"name":"MosaicLeaks","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30727","source_url":"https://arxiv.org/abs/2605.30727"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30727","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30727","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":44,"join":"llm_stats_id","repository":null,"source_record":"catalog_92f572dbd59f7255"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MotionBench is a benchmark for evaluating multimodal models on motion understanding in videos, testing the ability to comprehend temporal dynamics, movement patterns, and action sequences.","descriptions":{"en":"MotionBench is a benchmark for evaluating multimodal models on motion understanding in videos, testing the ability to comprehend temporal dynamics, movement patterns, and action sequences."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:motionbench:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/motionbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:motionbench","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_92f572dbd59f7255"},"modality":"multimodal","name":"MotionBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":74.9,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.749,"raw_min":0.704,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:motionbench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/motionbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-motionbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/motionbench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":44,"join":"artifact:arxiv:2501.02955","repository":null,"source_record":"catalog_92f572dbd59f7255"},"categories":["多模态","Multimodal","多模态模型","VLM","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MotionBench, is a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models. MotionBench是一个综合性的评估基准，旨在评估视频理解模型的细粒度运动理解能力。","descriptions":{"en":"MotionBench, is a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models.","zh":"MotionBench是一个综合性的评估基准，旨在评估视频理解模型的细粒度运动理解能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1682","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"artifact:arxiv:2501.02955","primary":"video-understanding","source_record":"catalog_92f572dbd59f7255"},"modality":"multimodal","name":"MotionBench","openness":"restricted","publisher":"THUDM","released":"2025-01-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1682-motionbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MotionBench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A contrastive benchmark of self-recorded video pairs testing physically grounded motion understanding in video-LLMs.","descriptions":{"en":"A contrastive benchmark of self-recorded video pairs testing physically grounded motion understanding in video-LLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09528","languages":[],"library_direction":null,"modality":null,"name":"MotionBlind","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09528","source_url":"https://arxiv.org/abs/2609.09528"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09528","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09528","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MotionForge is described as a simulation benchmark and data-generation pipeline for long-horizon manipulation of dynamic objects under domain shifts.","descriptions":{"en":"MotionForge is described as a simulation benchmark and data-generation pipeline for long-horizon manipulation of dynamic objects under domain shifts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.25689","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_motionforge_49effc47"},"modality":null,"name":"MotionForge","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25689","source_url":"https://arxiv.org/abs/2609.25689"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25689","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25689","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"MotionHalluc is a benchmark evaluating kinematic hallucinations in cross-video motion comparison. It includes 1540 questions over 553 video pairs, assessing directional, attributional, and temporal hallucinations in generated instructions.","descriptions":{"en":"MotionHalluc is a benchmark evaluating kinematic hallucinations in cross-video motion comparison. It includes 1540 questions over 553 video pairs, assessing directional, attributional, and temporal hallucinations in generated instructions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.23061","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","factuality"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_motionhalluc_82d851ff"},"modality":null,"name":"MotionHalluc","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23061","source_url":"https://arxiv.org/abs/2606.23061"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23061","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23061","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","物理智能","Embodied AI","跨模态推理","Cross-modal Reasoning","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MotionMillion: The largest open-sourced 3D human motion dataset with text annotation, including 2.5k hours 1.9M episodes. MotionMillion：目前最大的开源带文本标注的 3D 人体动作数据集，包含 2500 小时、190 万个片段。","descriptions":{"en":"MotionMillion: The largest open-sourced 3D human motion dataset with text annotation, including 2.5k hours 1.9M episodes.","zh":"MotionMillion：目前最大的开源带文本标注的 3D 人体动作数据集，包含 2500 小时、190 万个片段。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":true,"key":"opencompass:2154","languages":[],"library_direction":null,"modality":"multimodal","name":"MotionMillion","openness":"restricted","publisher":"InternRobotics","released":"2025-07-26","released_reference":{"basis":"dataset_published","note":"The authors explicitly date the dataset release July 26; the July 3 code release and July 9 paper are separate events.","source_key":"opencompass:2154","source_url":"https://github.com/VankouF/MotionMillion-Codes"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2154-motionmillion","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MotionMillion","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"MOV-Bench contains 519 questions requiring multi-hop reasoning over temporally dispersed audio-visual evidence, evaluating omni-modal LLMs on cross-modal reasoning tasks.","descriptions":{"en":"MOV-Bench contains 519 questions requiring multi-hop reasoning over temporally dispersed audio-visual evidence, evaluating omni-modal LLMs on cross-modal reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28192","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_mov-bench_db8a731e"},"modality":null,"name":"MOV-Bench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28192","source_url":"https://arxiv.org/abs/2605.28192"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28192","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28192","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Probabilistic wildlife movement forecasting benchmark with 2.6M GPS locations and environmental covariates.","descriptions":{"en":"Probabilistic wildlife movement forecasting benchmark with 2.6M GPS locations and environmental covariates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15780","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_movebench_797abacc"},"modality":null,"name":"MoveBench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15780","source_url":"https://arxiv.org/abs/2609.15780"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15780","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15780","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_moving-alphabet_5bd200ea"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Moving Alphabet is a procedural testbed for controlled experiments on text-to-video training data, generating synthetic videos with ground-truth metadata to study data distribution and caption quality effects.","descriptions":{"en":"Moving Alphabet is a procedural testbed for controlled experiments on text-to-video training data, generating synthetic videos with ground-truth metadata to study data distribution and caption quality effects."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.18789","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_moving-alphabet_5bd200ea"},"modality":null,"name":"Moving Alphabet","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18789","source_url":"https://arxiv.org/abs/2607.18789"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18789","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18789","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"MPAR-Bench evaluates multi-point associative reasoning in LLMs across English and Chinese using 1,000 items with diverse clues. Scoring includes exact-match accuracy, ANLS, embedding similarity, and reasoning-trace verification, with four perturbation types.","descriptions":{"en":"MPAR-Bench evaluates multi-point associative reasoning in LLMs across English and Chinese using 1,000 items with diverse clues. Scoring includes exact-match accuracy, ANLS, embedding similarity, and reasoning-trace verification, with four perturbation types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10444","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","multilingual"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_mpar-bench_22a7bcc7"},"modality":null,"name":"MPAR-Bench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10444","source_url":"https://arxiv.org/abs/2608.10444"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10444","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10444","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"MPC-Patch-Bench evaluates LLM-based code repair on repository-level Secure Multi-Party Computation (MPC) software. It provides 205 verified instances with Fail-to-Pass/Pass-to-Pass tests and a verifier that checks cryptographic safety and numerical fidelity via differential testing and static analysis.","descriptions":{"en":"MPC-Patch-Bench evaluates LLM-based code repair on repository-level Secure Multi-Party Computation (MPC) software. It provides 205 verified instances with Fail-to-Pass/Pass-to-Pass tests and a verifier that checks cryptographic safety and numerical fidelity via differential testing and static analysis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.11416","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","cybersecurity"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_mpc-patch-bench_574ee054"},"modality":null,"name":"MPC-Patch-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11416","source_url":"https://arxiv.org/abs/2606.11416"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11416","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11416","topics":["coding","agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_mpdocbench-parse_f77fb798"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MPDocBench-Parse is a benchmark for multi-page document parsing, covering 433 manually annotated documents with 3,246 pages across 15 document types in English and Chinese. It evaluates content fidelity and logical structure, including text, table, formula recognition, reading order, and heading hierarchy recovery.","descriptions":{"en":"MPDocBench-Parse is a benchmark for multi-page document parsing, covering 433 manually annotated documents with 3,246 pages across 15 document types in English and Chinese. It evaluates content fidelity and logical structure, including text, table, formula recognition, reading order, and heading hierarchy recovery."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22100","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","multilingual"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_mpdocbench-parse_f77fb798"},"modality":null,"name":"MPDocBench-Parse","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22100","source_url":"https://arxiv.org/abs/2605.22100"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22100","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22100","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":39,"join":"claire_record_id","repository":"gh:annlin0628/mpie-bench","source_record":"bm_mpie-bench_16361495"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MPIE-Bench evaluates multi-person image editing models on tasks involving multiple named people in contact interactions such as embrace, carry, or grapple. The benchmark provides a 2,500-sample test set with 14 interaction categories and four contact densities, and scores outputs on six axes including anatomy and interaction via mesh reconstruction.","descriptions":{"en":"MPIE-Bench evaluates multi-person image editing models on tasks involving multiple named people in contact interactions such as embrace, carry, or grapple. The benchmark provides a 2,500-sample test set with 14 interaction categories and four contact densities, and scores outputs on six axes including anatomy and interaction via mesh reconstruction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27616","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_mpie-bench_16361495"},"modality":null,"name":"MPIE-Bench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27616","source_url":"https://arxiv.org/abs/2607.27616"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27616","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27616","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Welcome to the dataset page for the Meta-Reasoning Benchmark associated with our recent publication `Mr-Ben: A Comprehensive Meta-Reasoning Benchmark for Large Language Models`. We have provided a demo evaluate script for you to try out benchmark in mere two steps. We encourage everyone to try out our benchmark in the SOTA models and return its results to us. We would be happy to include it in the eval_results and update the evaluation tables below for you. 本工作联合MIT,清华,剑桥等知名院校, 提出了一个评测大语言模型对复杂问题的推理过程的“阅卷”批改能力的评测数据集，有别于以前的以结果匹配为评测模式的数据集MR-Ben，我们的数据集基于GSM8K[1], MMLU[2], LogiQA[3], MHPP[4]等数据集经由细致的高水平人工标注构建而成，显著地增加了难度及区分度。我们细致地分析了包括claude3.5, GPT4-Turbo, Kimi, Zhipu, Yi-Large, Qwen2, DeepseekCoderv2 等国内外一线的大语言模型，发现开源的模型在复杂推理的场景下有望追上顶尖的闭源模型。该评测数据集的所有数据均已开源，并且支持一键评测。欢迎所有做大模型训练的小伙伴向我们分享你的评测结果，我们会及时更新榜单。","descriptions":{"en":"Welcome to the dataset page for the Meta-Reasoning Benchmark associated with our recent publication `Mr-Ben: A Comprehensive Meta-Reasoning Benchmark for Large Language Models`. We have provided a demo evaluate script for you to try out benchmark in mere two steps. We encourage everyone to try out our benchmark in the SOTA models and return its results to us. We would be happy to include it in the eval_results and update the evaluation tables below for you.","zh":"本工作联合MIT,清华,剑桥等知名院校, 提出了一个评测大语言模型对复杂问题的推理过程的“阅卷”批改能力的评测数据集，有别于以前的以结果匹配为评测模式的数据集MR-Ben，我们的数据集基于GSM8K[1], MMLU[2], LogiQA[3], MHPP[4]等数据集经由细致的高水平人工标注构建而成，显著地增加了难度及区分度。我们细致地分析了包括claude3.5, GPT4-Turbo, Kimi, Zhipu, Yi-Large, Qwen2, DeepseekCoderv2 等国内外一线的大语言模型，发现开源的模型在复杂推理的场景下有望追上顶尖的闭源模型。该评测数据集的所有数据均已开源，并且支持一键评测。欢迎所有做大模型训练的小伙伴向我们分享你的评测结果，我们会及时更新榜单。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:930","languages":[],"library_direction":null,"modality":null,"name":"MR-Ben-Meta-Reasoning-Benchmark","openness":"open","publisher":null,"released":"2024-06-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-930-mr-ben-meta-reasoning-benchmark","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MR-Ben-Meta-Reasoning-Benchmark","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MR-GSM8K is a challenging benchmark designed to evaluate the meta-reasoning capabilities of state-of-the-art Large Language Models (LLMs). It goes beyond traditional evaluation metrics by focusing on the reasoning process rather than just the final answer. MR-GSM8K 是一个旨在评估最先进大型语言模型（LLMs）元推理能力的挑战性基准。它超越了传统的评估指标，专注于推理过程而非仅仅关注最终答案，从而对模型的认知能力进行更细致的评估。","descriptions":{"en":"MR-GSM8K is a challenging benchmark designed to evaluate the meta-reasoning capabilities of state-of-the-art Large Language Models (LLMs). It goes beyond traditional evaluation metrics by focusing on the reasoning process rather than just the final answer.","zh":"MR-GSM8K 是一个旨在评估最先进大型语言模型（LLMs）元推理能力的挑战性基准。它超越了传统的评估指标，专注于推理过程而非仅仅关注最终答案，从而对模型的认知能力进行更细致的评估。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1587","languages":[],"library_direction":null,"modality":null,"name":"MR-GSM8K","openness":"unknown","publisher":"HKU, Tencent AI Lab, etc.","released":"2023-12-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1587-mr-gsm8k","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MR-GSM8K","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"MR-LiDAR is a multi-resolution LiDAR benchmark for roadside perception. It includes point cloud data and annotations from 16-, 32-, 80-, and 128-beam LiDARs in identical scenarios, covering vehicles and vulnerable road users at various distances.","descriptions":{"en":"MR-LiDAR is a multi-resolution LiDAR benchmark for roadside perception. It includes point cloud data and annotations from 16-, 32-, 80-, and 128-beam LiDARs in identical scenarios, covering vehicles and vulnerable road users at various distances."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24777","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_mr-lidar_0b6f3704"},"modality":null,"name":"MR-LiDAR","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24777","source_url":"https://arxiv.org/abs/2605.24777"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24777","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24777","topics":[],"unit":null},{"aliases":["Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:minghaoguo20/mr-lhdr-eval","source_record":"bm_mr-lhdr-a-benchmark-for-multimodal-real-wo_b8dc47d7"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark of 102 research questions with explicit dependency graphs and step checklists for long-horizon multimodal deep research agents.","descriptions":{"en":"Benchmark of 102 research questions with explicit dependency graphs and step checklists for long-horizon multimodal deep research agents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.11318","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_mr-lhdr-a-benchmark-for-multimodal-real-wo_b8dc47d7"},"modality":null,"name":"Mr.LHDR","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11318","source_url":"https://arxiv.org/abs/2609.11318"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11318","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11318","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","VQA","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios, providing a robust and systematic evaluation of Large Vision Language Model (LVLM)’s vision-centric multimodal retrieval-augmented generation (RAG) abilities. MRAG-Bench 包含 16,130 张图片和 1,353 个跨越 9 个不同场景的人标注多选题，为大型视觉语言模型（LVLM）的视觉中心多模态检索增强生成（RAG）能力提供了稳健和系统的评估。","descriptions":{"en":"MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios, providing a robust and systematic evaluation of Large Vision Language Model (LVLM)’s vision-centric multimodal retrieval-augmented generation (RAG) abilities.","zh":"MRAG-Bench 包含 16,130 张图片和 1,353 个跨越 9 个不同场景的人标注多选题，为大型视觉语言模型（LVLM）的视觉中心多模态检索增强生成（RAG）能力提供了稳健和系统的评估。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1586","languages":[],"library_direction":null,"modality":"multimodal","name":"MRAG-Bench","openness":"unknown","publisher":"UCLA, Stanford University","released":"2024-10-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1586-mrag-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MRAG-Bench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":28,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"lib_mrcr"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR (Multi-Round Coreference Resolution) is a synthetic long-context reasoning task where models must navigate long conversations to reproduce specific model outputs. It tests the ability to distinguish between similar requests and reason about ordering while maintaining attention across extended contexts.","descriptions":{"en":"MRCR (Multi-Round Coreference Resolution) is a synthetic long-context reasoning task where models must navigate long conversations to reproduce specific model outputs. It tests the ability to distinguish between similar requests and reason about ordering while maintaining attention across extended contexts."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr:gemini-1.5-flash-8b","reported_at":"2024-03-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"lib_mrcr"},"modality":"text","name":"MRCR","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":93.0,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.93,"raw_min":0.32,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr:gemini-2.5-pro","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"}],"slug":"llm-stats-mrcr","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":["MRCR","OpenAI-MRCR","MRCR v2"],"attention_signals":null,"categories":["long_context"],"collected_at":null,"description":"Needle position and generation seed change the result.","descriptions":{"en":"Needle position and generation seed change the result."},"evidence_summary":{"document_count":8,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mrcr\u0000google_gemini_3_1_pro_model_card\u0000mrcr_v2\u0000Thinking (High), 128k (average)\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"first_score_reported_at":"2026-02-19","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mrcr\u0000google_gemini_3_1_pro_model_card\u0000mrcr_v2\u0000Thinking (High), 128k (average)\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:mrcr","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"artifact:hf:openai/mrcr","primary":"long-context","source_record":"catalog_11d6a18b4f745fb6"},"modality":null,"name":"MRCR","openness":"unknown","publisher":null,"released":"2025-04-14","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mrcr","source_url":"https://huggingface.co/datasets/openai/mrcr"},"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":84.9,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":84.9,"raw_min":26.3,"source_reference":{"obs_id":"curated\u0000mrcr\u0000google_gemini_3_1_pro_model_card\u0000mrcr_v2\u0000Thinking (High), 128k (average)\u0000Gemini 3.1 Pro","observation_id":"curated\u0000mrcr\u0000google_gemini_3_1_pro_model_card\u0000mrcr_v2\u0000Thinking (High), 128k (average)\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","reported_date":"2026-02-19","source_id":"google_gemini_3_1_pro_model_card","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"unit":"percent"},"scored_models":[{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"}],"slug":"mrcr","source":"model_reports","source_url":"https://huggingface.co/datasets/openai/mrcr","topics":["long-context"],"unit":"percent"},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR (Multi-Round Coreference Resolution) at 128K context length with 2 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 128K-token contexts with 2 items to retrieve.","descriptions":{"en":"MRCR (Multi-Round Coreference Resolution) at 128K context length with 2 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 128K-token contexts with 2 items to retrieve."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-128k-(2-needle):minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%282-needle%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-128k-(2-needle)","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_f840bd7cecb4ce31"},"modality":"text","name":"MRCR 128K (2-needle)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":28.62,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.2862,"raw_min":0.2862,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-128k-(2-needle):minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%282-needle%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-mrcr-128k-2-needle","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%282-needle%29?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR (Multi-Round Coreference Resolution) at 128K context length with 4 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 128K-token contexts with 4 items to retrieve.","descriptions":{"en":"MRCR (Multi-Round Coreference Resolution) at 128K context length with 4 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 128K-token contexts with 4 items to retrieve."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-128k-(4-needle):minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%284-needle%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-128k-(4-needle)","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_786a93fb2cb37291"},"modality":"text","name":"MRCR 128K (4-needle)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":19.62,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.1962,"raw_min":0.1962,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-128k-(4-needle):minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%284-needle%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-mrcr-128k-4-needle","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%284-needle%29?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR (Multi-Round Coreference Resolution) at 128K context length with 8 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 128K-token contexts with 8 items to retrieve.","descriptions":{"en":"MRCR (Multi-Round Coreference Resolution) at 128K context length with 8 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 128K-token contexts with 8 items to retrieve."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-128k-(8-needle):minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%288-needle%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-128k-(8-needle)","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_ab936e5d1333318c"},"modality":"text","name":"MRCR 128K (8-needle)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":90.4,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.904,"raw_min":0.1012,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-128k-(8-needle):qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%288-needle%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"}],"slug":"llm-stats-mrcr-128k-8-needle","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%288-needle%29?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR 1M is a variant of the Multi-Round Coreference Resolution benchmark designed for testing extremely long context capabilities with approximately 1 million tokens. It evaluates models' ability to maintain reasoning and attention across ultra-long conversations.","descriptions":{"en":"MRCR 1M is a variant of the Multi-Round Coreference Resolution benchmark designed for testing extremely long context capabilities with approximately 1 million tokens. It evaluates models' ability to maintain reasoning and attention across ultra-long conversations."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-1m:gemini-2.0-flash-lite","reported_at":"2025-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-1m","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_9b716240d72b7367"},"modality":"text","name":"MRCR 1M","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":83.5,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.835,"raw_min":0.58,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-1m:deepseek-v4-pro-max","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"}],"slug":"llm-stats-mrcr-1m","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":28,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:google-deepmind/eval_hub","source_record":"catalog_12daadf376bddccc"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR 1M (pointwise) is a variant of the Multi-Round Coreference Resolution benchmark that uses pointwise evaluation for ultra-long contexts (~1M tokens). This version evaluates each response independently rather than comparatively, testing models' absolute performance on long-context reasoning tasks.","descriptions":{"en":"MRCR 1M (pointwise) is a variant of the Multi-Round Coreference Resolution benchmark that uses pointwise evaluation for ultra-long contexts (~1M tokens). This version evaluates each response independently rather than comparatively, testing models' absolute performance on long-context reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-1m-(pointwise):gemini-2.5-pro","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m-%28pointwise%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-1m-(pointwise)","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_12daadf376bddccc"},"modality":"text","name":"MRCR 1M (pointwise)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":82.89999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.829,"raw_min":0.829,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-1m-(pointwise):gemini-2.5-pro","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m-%28pointwise%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"}],"slug":"llm-stats-mrcr-1m-pointwise","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m-%28pointwise%29?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR (Multi-Round Coreference Resolution) at 64K context length with 2 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 64K-token contexts with 2 items to retrieve.","descriptions":{"en":"MRCR (Multi-Round Coreference Resolution) at 64K context length with 2 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 64K-token contexts with 2 items to retrieve."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-64k-(2-needle):minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%282-needle%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-64k-(2-needle)","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_d5877324a6222192"},"modality":"text","name":"MRCR 64K (2-needle)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":29.770000000000003,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.2977,"raw_min":0.2977,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-64k-(2-needle):minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%282-needle%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-mrcr-64k-2-needle","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%282-needle%29?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR (Multi-Round Coreference Resolution) at 64K context length with 4 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 64K-token contexts with 4 items to retrieve.","descriptions":{"en":"MRCR (Multi-Round Coreference Resolution) at 64K context length with 4 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 64K-token contexts with 4 items to retrieve."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-64k-(4-needle):minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%284-needle%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-64k-(4-needle)","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_54fe47056a26c695"},"modality":"text","name":"MRCR 64K (4-needle)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":20.57,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.2057,"raw_min":0.2057,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-64k-(4-needle):minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%284-needle%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-mrcr-64k-4-needle","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%284-needle%29?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR (Multi-Round Coreference Resolution) at 64K context length with 8 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 64K-token contexts with 8 items to retrieve.","descriptions":{"en":"MRCR (Multi-Round Coreference Resolution) at 64K context length with 8 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 64K-token contexts with 8 items to retrieve."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-64k-(8-needle):minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%288-needle%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-64k-(8-needle)","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_bfb9e5abf18fcd7b"},"modality":"text","name":"MRCR 64K (8-needle)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":16.56,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.1656,"raw_min":0.1656,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-64k-(8-needle):minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%288-needle%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-mrcr-64k-8-needle","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%288-needle%29?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR v2 (Multi-Round Coreference Resolution version 2) is an enhanced version of the synthetic long-context reasoning task. It extends the original MRCR framework with improved evaluation criteria and additional complexity for testing models' ability to maintain attention and reasoning across extended contexts.","descriptions":{"en":"MRCR v2 (Multi-Round Coreference Resolution version 2) is an enhanced version of the synthetic long-context reasoning task. It extends the original MRCR framework with improved evaluation criteria and additional complexity for testing models' ability to maintain attention and reasoning across extended contexts."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-v2:gemini-2.5-flash-lite","reported_at":"2025-06-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-v2","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_11d6a18b4f745fb6"},"modality":"text","name":"MRCR v2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":91.7,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.917,"raw_min":0.166,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-v2:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-mrcr-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR v2 (8-needle) is a variant of the Multi-Round Coreference Resolution benchmark that includes 8 needle items to retrieve from long contexts. This tests models' ability to simultaneously track and reason about multiple pieces of information across extended conversations.","descriptions":{"en":"MRCR v2 (8-needle) is a variant of the Multi-Round Coreference Resolution benchmark that includes 8 needle items to retrieve from long contexts. This tests models' ability to simultaneously track and reason about multiple pieces of information across extended conversations."},"evidence_summary":{"document_count":1,"model_count":23,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-v2-(8-needle):gemma-3-27b-it","reported_at":"2025-03-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-%288-needle%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-v2-(8-needle)","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_f375a5d25b1929e3"},"modality":"text","name":"MRCR v2 (8-needle)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"not_found","repo_resolution_status":"not_found","score_count":23,"score_direction":"higher_is_better","score_summary":{"display_max":97.0,"display_multiplier":100,"model_count":23,"model_count_basis":"source_model_id","numeric_count":23,"raw_max":0.97,"raw_min":0.135,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-v2-(8-needle):gemini-3.7-flash","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-%288-needle%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-mrcr-v2-8-needle","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-%288-needle%29?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MRCR v2 8-needle variant evaluated on contexts from 512K to 1M tokens.","descriptions":{"en":"MRCR v2 8-needle variant evaluated on contexts from 512K to 1M tokens."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mrcr-v2-8-needle-512k-1m:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-8-needle-512k-1m?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mrcr-v2-8-needle-512k-1m","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_9e25967de7cdd8b0"},"modality":"text","name":"MRCR v2 (8-needle, 512K-1M)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":73.8,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.738,"raw_min":0.413,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mrcr-v2-8-needle-512k-1m:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-8-needle-512k-1m?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-mrcr-v2-8-needle-512k-1m","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-8-needle-512k-1m?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bose/mrmad","source_record":"bm_mrmad_6774e059"},"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LALMs on audio degradation perception through multi-turn dialogues over multiple audio inputs, covering speech, music, and sound.","descriptions":{"en":"Evaluates LALMs on audio degradation perception through multi-turn dialogues over multiple audio inputs, covering speech, music, and sound."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22236","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_mrmad_6774e059"},"modality":null,"name":"MRMAD","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22236","source_url":"http://arxiv.org/abs/2608.22236v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22236","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22236v1","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:duckyquang/ms-mlb","source_record":"bm_ms-mlb_c7714a98"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"MS-MLB evaluates machine learning models for classifying multiple sclerosis versus healthy controls from whole blood RNA expression data (GSE17048). It uses a shared pipeline with nested cross-validation and a holdout set, and reports the MS Research Score composite metric.","descriptions":{"en":"MS-MLB evaluates machine learning models for classifying multiple sclerosis versus healthy controls from whole blood RNA expression data (GSE17048). It uses a shared pipeline with nested cross-validation and a holdout set, and reports the MS Research Score composite metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.05196","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","science"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_ms-mlb_c7714a98"},"modality":null,"name":"MS-MLB","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05196","source_url":"https://arxiv.org/abs/2608.05196"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05196","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05196","topics":["medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","大语言模型","LLM","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MS MARCO comprises of 1,010,916 anonymized questions—sampled from Bing’s search query logs—each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages. MS MARCO 数据集包含 1,010,916 个来自 Bing 的搜索查询日志的匿名问题，每个问题都有一个人工生成的答案和 182,669 个完全由人重写的生成答案。此外，该数据集还包含从 3,563,535 个由 Bing 检索的网页文档中提取的 8,841,823 个段落，这些段落提供了策划自然语言答案所需的信息。","descriptions":{"en":"MS MARCO comprises of 1,010,916 anonymized questions—sampled from Bing’s search query logs—each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages.","zh":"MS MARCO 数据集包含 1,010,916 个来自 Bing 的搜索查询日志的匿名问题，每个问题都有一个人工生成的答案和 182,669 个完全由人重写的生成答案。此外，该数据集还包含从 3,563,535 个由 Bing 检索的网页文档中提取的 8,841,823 个段落，这些段落提供了策划自然语言答案所需的信息。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1102","languages":[],"library_direction":null,"modality":null,"name":"MS_MARCO","openness":"unknown","publisher":"Microsoft AI & Research","released":"2018-10-31","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1102-ms-marco","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MS_MARCO","topics":[],"unit":null},{"aliases":["Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"MSE-Bench measures agent performance and execution redundancy across deterministic editing tasks in a controlled simulator.","descriptions":{"en":"MSE-Bench measures agent performance and execution redundancy across deterministic editing tasks in a controlled simulator."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.13034","languages":[],"library_direction":null,"modality":null,"name":"MSE-Bench","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.13034","source_url":"https://arxiv.org/abs/2607.13034"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-13034","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13034","topics":["reasoning-math"],"unit":null},{"aliases":["An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding"],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"MSEval evaluates multi-agent software development on authentic full-stack projects using deterministic, requirement-based grading.","descriptions":{"en":"MSEval evaluates multi-agent software development on authentic full-stack projects using deterministic, requirement-based grading."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27877","languages":[],"library_direction":null,"modality":null,"name":"MSEval","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27877","source_url":"https://arxiv.org/abs/2607.27877"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27877","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27877","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":1556,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:boson-ai/msi-bench","source_record":"bm_msi-bench_45ff7f4c"},"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MSI-Bench evaluates multi-speaker voice interaction for collaborative AI agents using 1,152 audio cases with six patterns, scored via LLM judge and deterministic tool validation.","descriptions":{"en":"MSI-Bench evaluates multi-speaker voice interaction for collaborative AI agents using 1,152 audio cases with six patterns, scored via LLM judge and deterministic tool validation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.24812","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","tool-use"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_msi-bench_45ff7f4c"},"modality":null,"name":"MSI-Bench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24812","source_url":"https://arxiv.org/abs/2609.24812"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24812","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24812","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal large language models on discrimination, understanding, and reasoning under situational illusions.","descriptions":{"en":"Evaluates multimodal large language models on discrimination, understanding, and reasoning under situational illusions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22232","languages":[],"library_direction":null,"modality":null,"name":"MSIBench","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22232","source_url":"https://arxiv.org/abs/2608.22232"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22232","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.22232","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark suite of 22 full-body musculoskeletal motor-control tasks across postural stabilization, locomotion, and contact-rich interaction.","descriptions":{"en":"A benchmark suite of 22 full-body musculoskeletal motor-control tasks across postural stabilization, locomotion, and contact-rich interaction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.26872","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_msk-bench_99909e87"},"modality":null,"name":"MSK-Bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.26872","source_url":"https://arxiv.org/abs/2609.26872"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-26872","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26872","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"MSP-Bench evaluates multi-subject personalized text-to-image generation. It scores identity fidelity, attribute binding accuracy, and attribute misalignment through a dual-pathway protocol.","descriptions":{"en":"MSP-Bench evaluates multi-subject personalized text-to-image generation. It scores identity fidelity, attribute binding accuracy, and attribute misalignment through a dual-pathway protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.03708","languages":[],"library_direction":null,"modality":null,"name":"MSP-Bench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03708","source_url":"https://arxiv.org/abs/2608.03708"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03708","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03708","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_msqa_623c5d9a"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MSQA evaluates 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers, testing cultural knowledge in a multilingual context.","descriptions":{"en":"MSQA evaluates 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers, testing cultural knowledge in a multilingual context."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.00724","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_msqa_623c5d9a"},"modality":null,"name":"MSQA","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00724","source_url":"https://arxiv.org/abs/2607.00724"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00724","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00724","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"bm_msqa_623c5d9a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","knowledge","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MSQA is a multilingual question-answering benchmark that measures knowledge and reasoning across a diverse set of languages.","descriptions":{"en":"MSQA is a multilingual question-answering benchmark that measures knowledge and reasoning across a diverse set of languages."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:msqa:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/msqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:msqa","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"bm_msqa_623c5d9a"},"modality":"text","name":"MSQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":50.2,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.502,"raw_min":0.42,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:msqa:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/msqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-msqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/msqa?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark for generating three- to five-minute, multi-shot video continuations from an observed frame or clip while maintaining state and narrative progression.","descriptions":{"en":"A benchmark for generating three- to five-minute, multi-shot video continuations from an observed frame or clip while maintaining state and narrative progression."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26525","languages":[],"library_direction":null,"modality":null,"name":"MSVE-Bench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26525","source_url":"https://arxiv.org/abs/2605.26525"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26525","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26525","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MT-AIME 2025 is Cohere's internal multilingual translation of AIME 2025, evaluated for Arabic, Japanese, and Korean in the Command A+ release.","descriptions":{"en":"MT-AIME 2025 is Cohere's internal multilingual translation of AIME 2025, evaluated for Arabic, Japanese, and Korean in the Command A+ release."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mt-aime-2025:command-a-plus-05-2026","reported_at":"2026-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-aime-2025?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mt-aime-2025","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","math"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_0a207a8b12e78d0c"},"modality":"text","name":"MT-AIME 2025","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":86.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.86,"raw_min":0.86,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mt-aime-2025:command-a-plus-05-2026","reported_date":"2026-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-aime-2025?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"}],"slug":"llm-stats-mt-aime-2025","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-aime-2025?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":39554,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":44,"join":"llm_stats_id","repository":null,"source_record":"catalog_fa3c844fc8547162"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","roleplay","general","communication","creativity"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MT-Bench is a challenging multi-turn benchmark that measures the ability of large language models to engage in coherent, informative, and engaging conversations. It uses strong LLMs as judges for scalable and explainable evaluation of multi-turn dialogue capabilities.","descriptions":{"en":"MT-Bench is a challenging multi-turn benchmark that measures the ability of large language models to engage in coherent, informative, and engaging conversations. It uses strong LLMs as judges for scalable and explainable evaluation of multi-turn dialogue capabilities."},"evidence_summary":{"document_count":1,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mt-bench:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mt-bench","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_fa3c844fc8547162"},"modality":"text","name":"MT-Bench","openness":"unknown","publication":{"evidence":[{"note":"The LMSYS evaluation README identifies MT-Bench as challenging multi-turn open-ended questions assessed by strong LLM judges, matching the exact source record. It links and cites arXiv:2306.05685 and provides the benchmark's questions, prompts and evaluation instructions.","url":"https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/README.md"}],"id":"mt-bench-publication","links":[{"kind":"paper","title":"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena","url":"https://arxiv.org/abs/2306.05685"},{"kind":"repo","title":"MT-Bench official evaluation code in FastChat","url":"https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge"}],"record_keys":["llm-stats:mt-bench"],"reviewed_at":"2026-10-26","scope":{"en":"The original text-based MT-Bench for multi-turn dialogue evaluation. The code link opens its evaluation directory in FastChat, not a later multimodal or extended variant.","zh":"原始文本多轮对话评测 MT-Bench；代码链接直达 FastChat 中的评测目录，不关联后续多模态或扩展版本。"}},"publisher":null,"released":"2023-06-09","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mt-bench","source_url":"https://arxiv.org/abs/2306.05685"},"repo_kind":null,"repo_resolution_status":null,"score_count":12,"score_direction":"higher_is_better","score_summary":{"display_max":8.99,"display_multiplier":1,"model_count":12,"model_count_basis":"source_model_id","numeric_count":12,"raw_max":8.99,"raw_min":0.0899,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mt-bench:hermes-3-70b","reported_date":"2024-08-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 8B V1"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama-3.3 Nemotron Super 49B v1"},{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Instruct"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"}],"slug":"llm-stats-mt-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-bench?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","ACL 2024","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MT-Bench-101 is specifically designed to evaluate the finegrained abilities of LLMs in multi-turn dialogues. MT-Bench-101 专门设计用于评估 LLMs 在多轮对话中的细粒度能力。通过对真实多轮对话数据的详细分析，构建了一个三层级的能力分类法，涵盖 1388 个多轮对话中的 4208 个轮次，涉及 13 种不同的任务。","descriptions":{"en":"MT-Bench-101 is specifically designed to evaluate the finegrained abilities of LLMs in multi-turn dialogues.","zh":"MT-Bench-101 专门设计用于评估 LLMs 在多轮对话中的细粒度能力。通过对真实多轮对话数据的详细分析，构建了一个三层级的能力分类法，涵盖 1388 个多轮对话中的 4208 个轮次，涉及 13 种不同的任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1172","languages":[],"library_direction":null,"modality":null,"name":"MT-Bench-101","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-06-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1172-mt-bench-101","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MT-Bench-101","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MT-Web2Code evaluates coding agents on multi-turn web UI reconstruction and modification tasks across 102 tasks in 16 domains, with a dual-axis protocol measuring target-region fidelity and preservation of unaffected content.","descriptions":{"en":"MT-Web2Code evaluates coding agents on multi-turn web UI reconstruction and modification tasks across 102 tasks in 16 domains, with a dual-axis protocol measuring target-region fidelity and preservation of unaffected content."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03474","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","visual-reasoning"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_mt-web2code_5f98b4bc"},"modality":null,"name":"MT-Web2Code","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03474","source_url":"https://arxiv.org/abs/2608.03474"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03474","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03474","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for instruction-following in multi-turn software development with constraint checklists and verification scripts.","descriptions":{"en":"Benchmark for instruction-following in multi-turn software development with constraint checklists and verification scripts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.14992","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_mtac-ifbench_6edfe993"},"modality":null,"name":"MTAC-IFBench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.14992","source_url":"https://arxiv.org/abs/2609.14992"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-14992","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14992","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"MTAVG-Bench 2.0 evaluates omni large language models on diagnosing high-level cinematic failures in multi-talker audio-video generation, with over 10,000 QA instances covering acting, narrative, atmosphere, and audio-visual language.","descriptions":{"en":"MTAVG-Bench 2.0 evaluates omni large language models on diagnosing high-level cinematic failures in multi-talker audio-video generation, with over 10,000 QA instances covering acting, narrative, atmosphere, and audio-visual language."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28035","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_mtavg-bench_0372b3a9"},"modality":null,"name":"MTAVG-Bench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28035","source_url":"https://arxiv.org/abs/2605.28035"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28035","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28035","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","知识","Knowledge","理解","Understanding","Chinese Medicine","Benchmark","TCM","科学智能","AI for Science","逻辑推理","知识储备","语言理解","Comprehension","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"We introduce MTCMB-a Multi-Task Benchmark for Evaluating LLMs on TCM Knowledge, Reasoning, and Safety. Developed in collaboration with certified TCM experts, MTCMB comprises 12 sub-datasets spanning five major categories: knowledge QA, language understanding, diagnostic reasoning, formula generation 大语言模型在中医领域的应用日益增多，到底大语言模型在这一古老的学科表现如何？虽然有一些零碎的评测数据集，但大都以单选题、病案分析、粗糙的医患对话为主，很难全面评估大语言模型在中医领域的实际能力。近日，中山大学联合湖南中医药大学等团队推出全球首个中医多任务评测基准（Benchmark） —— MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine","descriptions":{"en":"We introduce MTCMB-a Multi-Task Benchmark for Evaluating LLMs on TCM Knowledge, Reasoning, and Safety. Developed in collaboration with certified TCM experts, MTCMB comprises 12 sub-datasets spanning five major categories: knowledge QA, language understanding, diagnostic reasoning, formula generation","zh":"大语言模型在中医领域的应用日益增多，到底大语言模型在这一古老的学科表现如何？虽然有一些零碎的评测数据集，但大都以单选题、病案分析、粗糙的医患对话为主，很难全面评估大语言模型在中医领域的实际能力。近日，中山大学联合湖南中医药大学等团队推出全球首个中医多任务评测基准（Benchmark） —— MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1929","languages":["Chinese"],"library_direction":null,"modality":null,"name":"MTCMB","openness":"restricted","publisher":"Sun Yat-sen University, Hunan University of Chinese Medicine","released":"2025-05-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1929-mtcmb","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MTCMB","topics":["reasoning-math","language","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","文本嵌入","大语言模型","LLM","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MTEB is a benchmark for evaluating text embedding models, aiming to enhance long-term usability and reproducibility, focusing on task extensibility, data integrity validation, and result generalizability. MTEB 是一个评估文本嵌入模型的基准，旨在提升其长期可用性和可复现性，涵盖任务扩展性、数据完整性验证和结果通用性等维度。该基准包含包括分类、检索和聚类在内的多种任务，覆盖多个语言和领域。MTEB 引入了自动化测试管道、持续集成框架和社区贡献机制，以支持基准的可扩展性和质量控制。","descriptions":{"en":"MTEB is a benchmark for evaluating text embedding models, aiming to enhance long-term usability and reproducibility, focusing on task extensibility, data integrity validation, and result generalizability.","zh":"MTEB 是一个评估文本嵌入模型的基准，旨在提升其长期可用性和可复现性，涵盖任务扩展性、数据完整性验证和结果通用性等维度。该基准包含包括分类、检索和聚类在内的多种任务，覆盖多个语言和领域。MTEB 引入了自动化测试管道、持续集成框架和社区贡献机制，以支持基准的可扩展性和质量控制。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2034","languages":["Chinese","German","Multilingual"],"library_direction":null,"modality":null,"name":"MTEB","openness":"unknown","publisher":"Zendesk , Esker , INSA Lyon , LIRIS , Aarhus University , ITMO University","released":"2025-06-26","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2034-mteb","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MTEB","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":12,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:tardellirs/mteb-br","source_record":"bm_mteb-br_cc2fe8a1"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MTEB-BR is a text embedding benchmark for Brazilian Portuguese with 22 native tasks across seven categories, including classification, clustering, retrieval, and reranking. It evaluates models on native Portuguese data and provides a public leaderboard.","descriptions":{"en":"MTEB-BR is a text embedding benchmark for Brazilian Portuguese with 22 native tasks across seven categories, including classification, clustering, retrieval, and reranking. It evaluates models on native Portuguese data and provides a public leaderboard."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.04581","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","deep-research"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_mteb-br_cc2fe8a1"},"modality":null,"name":"MTEB-BR","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04581","source_url":"https://arxiv.org/abs/2607.04581"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04581","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04581","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MTM-Bench is a controlled benchmark for language-conditioned task execution in multilingual settings, enumerating all 27 instruction-content-response language triplets across English, Spanish, and Chinese. It contains 2,430 instances per model across semantic reversal, final-state extraction, and language purity tasks, with decomposed metrics for semantic correctness, language adherence, constraint satisfaction, and joint success.","descriptions":{"en":"MTM-Bench is a controlled benchmark for language-conditioned task execution in multilingual settings, enumerating all 27 instruction-content-response language triplets across English, Spanish, and Chinese. It contains 2,430 instances per model across semantic reversal, final-state extraction, and language purity tasks, with decomposed metrics for semantic correctness, language adherence, constraint satisfaction, and joint success."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27649","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","instruction-following"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_mtm-bench_e1a4ed71"},"modality":null,"name":"MTM-Bench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27649","source_url":"https://arxiv.org/abs/2605.27649"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27649","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27649","topics":["language"],"unit":null},{"aliases":["MTOB","Machine Translation from One Book"],"attention_signals":null,"categories":["long_context"],"collected_at":null,"description":"Translation from a single grammar book, reported as half-book and full-book variants that are not interchangeable.","descriptions":{"en":"Translation from a single grammar book, reported as half-book and full-book variants that are not interchangeable."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:mtob","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["multilingual"],"join":"review:14H034160212:2026-10-15","primary":"multilingual","source_record":null},"modality":null,"name":"MTOB","openness":"unknown","publisher":null,"released":"2023-09-28","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mtob","source_url":"https://arxiv.org/abs/2309.16575"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"mtob","source":"model_reports","source_url":"https://arxiv.org/abs/2309.16575","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-turn scientific diagram refinement via 292 images annotated with 3,518 user requirements, using a user simulator that provides iterative feedback and measures requirement satisfaction and diagram quality.","descriptions":{"en":"Evaluates multi-turn scientific diagram refinement via 292 images annotated with 3,518 user requirements, using a user simulator that provides iterative feedback and measures requirement satisfaction and diagram quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30241","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mtpaperbananabench_c4c3a34a"},"modality":null,"name":"MTPaperBananaBench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30241","source_url":"https://arxiv.org/abs/2608.30241"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30241","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30241","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Speech & Audio"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating language models within cascaded voice agents across multi-turn interactions.","descriptions":{"en":"A benchmark for evaluating language models within cascaded voice agents across multi-turn interactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.20152","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_mtva-bench_19779e2f"},"modality":null,"name":"MTVA-Bench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.20152","source_url":"https://arxiv.org/abs/2609.20152"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-20152","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20152","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":64,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_6aef73a46f67bebd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","text-to-image","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MTVQA (Multilingual Text-Centric Visual Question Answering) is the first benchmark featuring high-quality human expert annotations across 9 diverse languages, consisting of 6,778 question-answer pairs across 2,116 images. It addresses visual-textual misalignment problems in multilingual text-centric VQA.","descriptions":{"en":"MTVQA (Multilingual Text-Centric Visual Question Answering) is the first benchmark featuring high-quality human expert annotations across 9 diverse languages, consisting of 6,778 question-answer pairs across 2,116 images. It addresses visual-textual misalignment problems in multilingual text-centric VQA."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mtvqa:qwen2-vl-72b","reported_at":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mtvqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mtvqa","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","multilingual","visual-reasoning"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_6aef73a46f67bebd"},"modality":"multimodal","name":"MTVQA","openness":"unknown","publisher":null,"released":"2024-05-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mtvqa","source_url":"https://arxiv.org/abs/2405.11985"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":30.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.309,"raw_min":0.309,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mtvqa:qwen2-vl-72b","reported_date":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mtvqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"}],"slug":"llm-stats-mtvqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mtvqa?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":64,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2405.11985","repository":null,"source_record":"catalog_6aef73a46f67bebd"},"categories":["多模态","Multimodal","理解","Understanding","VQA","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MTVQA is designed to evaluate LMMs' multilingual text understanding, featuring high-quality human expert annotations across 9 diverse languages. MTVQA用于评估多模态大模型理解多语言文本的能力，包含来自9种语言的由人类专家注释的高质量数据。","descriptions":{"en":"MTVQA is designed to evaluate LMMs' multilingual text understanding, featuring high-quality human expert annotations across 9 diverse languages.","zh":"MTVQA用于评估多模态大模型理解多语言文本的能力，包含来自9种语言的由人类专家注释的高质量数据。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1502","languages":["English","Chinese","Multilingual"],"library_direction":{"basis":"model-review","categories":["document-ocr","multilingual","visual-reasoning"],"join":"artifact:arxiv:2405.11985","primary":"document-ocr","source_record":"catalog_6aef73a46f67bebd"},"modality":"multimodal","name":"MTVQA","openness":"restricted","publisher":"ByteDance","released":"2024-06-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1502-mtvqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MTVQA","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:luoojason/muddle","source_record":"bm_muddle_d01dfe23"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MUDDLE separates the effects of distractor topical similarity and added context length on document question answering, using 270 questions across five controlled conditions.","descriptions":{"en":"MUDDLE separates the effects of distractor topical similarity and added context length on document question answering, using 270 questions across five controlled conditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29477","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","deep-research"],"join":"claire_record_id","primary":"long-context","source_record":"bm_muddle_d01dfe23"},"modality":null,"name":"MUDDLE","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29477","source_url":"https://arxiv.org/abs/2608.29477"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29477","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29477","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":21,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"llm_stats_id","repository":null,"source_record":"catalog_743371fe4e9f7af0"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive benchmark for robust multi-image understanding capabilities of multimodal LLMs. Consists of 12 diverse multi-image tasks involving 10 categories of multi-image relations (e.g., multiview, temporal relations, narrative, complementary). Comprises 11,264 images and 2,600 multiple-choice questions created in a pairwise manner, where each standard instance is paired with an unanswerable variant for reliable assessment.","descriptions":{"en":"A comprehensive benchmark for robust multi-image understanding capabilities of multimodal LLMs. Consists of 12 diverse multi-image tasks involving 10 categories of multi-image relations (e.g., multiview, temporal relations, narrative, complementary). Comprises 11,264 images and 2,600 multiple-choice questions created in a pairwise manner, where each standard instance is paired with an unanswerable variant for reliable assessment."},"evidence_summary":{"document_count":1,"model_count":12,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:muirbench:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/muirbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:muirbench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_743371fe4e9f7af0"},"modality":"multimodal","name":"MuirBench","openness":"unknown","publication":{"evidence":[{"note":"The repository names author contacts, links the introducing paper and records its June 2024 release. Its 12 tasks, 10 multi-image relations, 11,264 images and 2,600 paired questions match the exact source description.","url":"https://github.com/muirbench/MuirBench/blob/main/README.md"},{"note":"The accessible paper lists the same authors and corpus design, including minimally different unanswerable variants; no paper scores are transferred into the catalog.","url":"https://arxiv.org/abs/2406.09411"}],"id":"muirbench-original","links":[{"kind":"paper","title":"MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding","url":"https://arxiv.org/abs/2406.09411"},{"kind":"repo","title":"MuirBench authors' repository","url":"https://github.com/muirbench/MuirBench"}],"record_keys":["llm-stats:muirbench"],"reviewed_at":"2026-10-26","scope":{"en":"Original MuirBench multi-image benchmark, with standard questions paired with unanswerable variants. The source record does not specify which task or answerability subset each score uses.","zh":"原始 MuirBench 多图理解基准，将标准题目与不可回答变体配对。当前来源记录未明确每项成绩对应的任务或可回答性子集。"}},"publisher":null,"released":"2024-06-13","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:muirbench","source_url":"https://arxiv.org/abs/2406.09411"},"repo_kind":null,"repo_resolution_status":null,"score_count":12,"score_direction":"higher_is_better","score_summary":{"display_max":80.30000000000001,"display_multiplier":100,"model_count":12,"model_count_basis":"source_model_id","numeric_count":12,"raw_max":0.803,"raw_min":0.583,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:muirbench:qwen3-vl-32b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/muirbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-muirbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/muirbench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"The proposed benchmark would include 5,038 open-ended questions over in-the-wild audio recordings across 16 languages and eight domains.","descriptions":{"en":"The proposed benchmark would include 5,038 open-ended questions over in-the-wild audio recordings across 16 languages and eight domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23416","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","speech-audio"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_mula-bench_418e2b8d"},"modality":null,"name":"MuLA-Bench","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23416","source_url":"https://arxiv.org/abs/2609.23416"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23416","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23416","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"An offline protocol-conditioned benchmark for Vision-Language-Action UAV agents, evaluating operational context understanding, multimodal evidence arbitration, degradation-aware reasoning, and risk-aware action planning across 3,024 samples with semantic scoring and structural diagnostics.","descriptions":{"en":"An offline protocol-conditioned benchmark for Vision-Language-Action UAV agents, evaluating operational context understanding, multimodal evidence arbitration, degradation-aware reasoning, and risk-aware action planning across 3,024 samples with semantic scoring and structural diagnostics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.23870","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","visual-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_mulrobbench_1e33deff"},"modality":null,"name":"MulRobBench","openness":"unknown","publisher":null,"released":"2026-07-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23870","source_url":"https://arxiv.org/abs/2607.23870"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23870","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23870","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":98,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_91a0d15dc744c7b3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","communication"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MultiChallenge is a realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key categories: instruction retention (maintaining instructions throughout conversations), inference memory (recalling and connecting details from previous turns), reliable versioned editing (adapting to evolving instructions during collaborative editing), and self-coherence (avoiding contradictions in responses). The benchmark evaluates models on sustained, contextually complex dialogues across diverse topics including travel planning, technical documentation, and professional communication.","descriptions":{"en":"MultiChallenge is a realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key categories: instruction retention (maintaining instructions throughout conversations), inference memory (recalling and connecting details from previous turns), reliable versioned editing (adapting to evolving instructions during collaborative editing), and self-coherence (avoiding contradictions in responses). The benchmark evaluates models on sustained, contextually complex dialogues across diverse topics including travel planning, technical documentation, and professional communication."},"evidence_summary":{"document_count":1,"model_count":29,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:multichallenge:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multichallenge?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:multichallenge","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_91a0d15dc744c7b3"},"modality":"text","name":"Multi-Challenge","openness":"unknown","publisher":null,"released":"2025-01-29","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:multichallenge","source_url":"https://arxiv.org/abs/2501.17399"},"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":29,"score_direction":"higher_is_better","score_summary":{"display_max":77.7,"display_multiplier":100,"model_count":29,"model_count_basis":"source_model_id","numeric_count":29,"raw_max":0.777,"raw_min":0.15,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:multichallenge:nova-2-pro","reported_date":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multichallenge?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-01-15","date_precision":"model_announcement","name":"Step3-VL-10B"}],"slug":"llm-stats-multichallenge","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multichallenge?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":63,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_56756bc009f4e6ce"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","structured_output","instruction_following","language","communication"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages.","descriptions":{"en":"Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages."},"evidence_summary":{"document_count":1,"model_count":23,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:multi-if:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-if?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:multi-if","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","multilingual"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_56756bc009f4e6ce"},"modality":"text","name":"Multi-IF","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":23,"score_direction":"higher_is_better","score_summary":{"display_max":80.60000000000001,"display_multiplier":100,"model_count":23,"model_count_basis":"source_model_id","numeric_count":23,"raw_max":0.806,"raw_min":0.373,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:multi-if:qwen3-235b-a22b-thinking-2507","reported_date":"2025-07-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-if?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 30B A3B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-multi-if","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-if?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":29,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":60,"join":"claire_record_id","repository":"gh:multi-lcb/multi-lcb","source_record":"bm_multi-lcb_028b5b88"},"categories":["Code & Software","Code generation","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-LCB evaluates code generation across twelve programming languages (C++, C#, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, Kotlin, Scala, PHP) by transforming Python tasks from LiveCodeBench while preserving its contamination controls and evaluation protocol.","descriptions":{"en":"Multi-LCB evaluates code generation across twelve programming languages (C++, C#, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, Kotlin, Scala, PHP) by transforming Python tasks from LiveCodeBench while preserving its contamination controls and evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.20517","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","multilingual"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_multi-lcb_028b5b88"},"modality":null,"name":"Multi-LCB","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20517","source_url":"https://arxiv.org/abs/2606.20517"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20517","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20517","topics":["coding","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":375,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_multi-legal-bench_2cfb13a9"},"categories":["Language & Knowledge","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-Legal-Bench evaluates LLMs on legal reasoning across six countries, four language families, and five tasks, using structured metadata from court registries for classification and extraction.","descriptions":{"en":"Multi-Legal-Bench evaluates LLMs on legal reasoning across six countries, four language families, and five tasks, using structured metadata from court registries for classification and extraction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29738","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","multilingual"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_multi-legal-bench_2cfb13a9"},"modality":null,"name":"Multi-Legal-Bench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29738","source_url":"https://arxiv.org/abs/2605.29738"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29738","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29738","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:caitlin-leonard/ct-segmentation-benchmark","source_record":"bm_ct-segmentation-benchmark_d7d89501"},"categories":["Biology & Drug Discovery","Software & AI Compute","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Compares MOOSE, TotalSegmentator, and VoxTell on public CT datasets using Dice and IoU scores per organ, with a staged design from a 4-organ pilot to 16 organs across up to 50 subjects.","descriptions":{"en":"Compares MOOSE, TotalSegmentator, and VoxTell on public CT datasets using Dice and IoU scores per organ, with a staged design from a 4-organ pilot to 16 organs across up to 50 subjects."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:caitlin-leonard/ct-segmentation-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Multi-Organ CT Segmentation Benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:caitlin-leonard/ct-segmentation-benchmark","source_url":"https://github.com/caitlin-leonard/ct-segmentation-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-caitlin-leonard-ct-segmentation-benchmark","source":"claire_radar","source_url":"https://github.com/caitlin-leonard/ct-segmentation-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":362,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":49,"join":"llm_stats_id","repository":null,"source_record":"catalog_37f7f4cd7dee3189"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multilingual benchmark for issue resolving that evaluates Large Language Models' ability to resolve software issues across diverse programming ecosystems. Covers 7 programming languages (Java, TypeScript, JavaScript, Go, Rust, C, and C++) with 1,632 high-quality instances carefully annotated by 68 expert annotators. Addresses limitations of existing benchmarks that focus almost exclusively on Python.","descriptions":{"en":"A multilingual benchmark for issue resolving that evaluates Large Language Models' ability to resolve software issues across diverse programming ecosystems. Covers 7 programming languages (Java, TypeScript, JavaScript, Go, Rust, C, and C++) with 1,632 high-quality instances carefully annotated by 68 expert annotators. Addresses limitations of existing benchmarks that focus almost exclusively on Python."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:multi-swe-bench:qwen3-coder-480b-a35b-instruct","reported_at":"2025-01-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-swe-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:multi-swe-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","multilingual"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_37f7f4cd7dee3189"},"modality":"text","name":"Multi-SWE-Bench","openness":"unknown","publisher":null,"released":"2025-04-03","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:multi-swe-bench","source_url":"https://arxiv.org/abs/2504.02605"},"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":52.7,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.527,"raw_min":0.258,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:multi-swe-bench:minimax-m2.7","reported_date":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-swe-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"}],"slug":"llm-stats-multi-swe-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-swe-bench?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":["Multi-SWE-Bench"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Multilingual SWE-bench variant spanning 7 programming languages; cross-repository context transfer is the measured capability.","descriptions":{"en":"Multilingual SWE-bench variant spanning 7 programming languages; cross-repository context transfer is the measured capability."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000multi_swe_bench\u0000bytedance_seed2_0_model_card\u0000multi_swe_bench\u0000as printed in Table 11 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000multi_swe_bench\u0000bytedance_seed2_0_model_card\u0000multi_swe_bench\u0000as printed in Table 11 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:multi_swe_bench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"Multi-SWE-Bench","openness":"unknown","publisher":null,"released":"2025-04-03","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"model-reports:multi_swe_bench","source_url":"https://arxiv.org/abs/2504.02605"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":45.2,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":45.2,"raw_min":41.1,"source_reference":{"obs_id":"curated\u0000multi_swe_bench\u0000bytedance_seed2_0_model_card\u0000multi_swe_bench\u0000as printed in Table 11 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000multi_swe_bench\u0000bytedance_seed2_0_model_card\u0000multi_swe_bench\u0000as printed in Table 11 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"multi_swe_bench","source":"model_reports","source_url":"https://github.com/multi-swe-bench/multi-swe-bench/","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Code generation","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Reproduction test generation translates a natural-language issue description into executable tests that fail on the original code and pass after the issue is resolved, providing executable evidence for verifying candidate patches.","descriptions":{"en":"Reproduction test generation translates a natural-language issue description into executable tests that fail on the original code and pass after the issue is resolved, providing executable evidence for verifying candidate patches."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34752","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_multi-swt-bench_e58e1a76"},"modality":null,"name":"Multi-SWT-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34752","source_url":"https://arxiv.org/abs/2609.34752"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34752","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34752","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_multi2av-safety_13dc6c17"},"categories":["Factuality","General AI","Multimodal","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Audio-video generation is rapidly moving from prompt-driven synthesis toward multimodal conditioning, where text, images, audio, and video can jointly shape the generated output.","descriptions":{"en":"Audio-video generation is rapidly moving from prompt-driven synthesis toward multimodal conditioning, where text, images, audio, and video can jointly shape the generated output."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.26535","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","image-video-generation"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_multi2av-safety_13dc6c17"},"modality":null,"name":"Multi2AV-Safety","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26535","source_url":"https://arxiv.org/abs/2608.26535"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26535","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26535","topics":["multimodal","safety"],"unit":null},{"aliases":["FaceSnap: Real-Time Personalized Lightstage Facial Performance Capture"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates 4D facial reconstruction in lightstage settings using a public dataset that enables topology-invariant geometric comparison across methods.","descriptions":{"en":"Evaluates 4D facial reconstruction in lightstage settings using a public dataset that enables topology-invariant geometric comparison across methods."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.31033","languages":[],"library_direction":null,"modality":null,"name":"Multi4D","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.31033","source_url":"https://arxiv.org/abs/2608.31033"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-31033","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31033","topics":[],"unit":null},{"aliases":["MultiChallenge"],"attention_signals":{"as_of":"2026-10-10","github_stars":98,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"artifact:arxiv:2501.17399","repository":null,"source_record":"catalog_91a0d15dc744c7b3"},"categories":["reasoning"],"collected_at":null,"description":"Multi-turn instruction retention, graded by an LLM judge.","descriptions":{"en":"Multi-turn instruction retention, graded by an LLM judge."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000multi_challenge\u0000bytedance_seed2_0_model_card\u0000multi_challenge\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000multi_challenge\u0000bytedance_seed2_0_model_card\u0000multi_challenge\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:multi_challenge","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"artifact:arxiv:2501.17399","primary":"instruction-following","source_record":"catalog_91a0d15dc744c7b3"},"modality":null,"name":"MultiChallenge","openness":"unknown","publisher":null,"released":"2025-01-29","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"model-reports:multi_challenge","source_url":"https://arxiv.org/abs/2501.17399"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":68.3,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":68.3,"raw_min":61.1,"source_reference":{"obs_id":"curated\u0000multi_challenge\u0000bytedance_seed2_0_model_card\u0000multi_challenge\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000multi_challenge\u0000bytedance_seed2_0_model_card\u0000multi_challenge\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"multi_challenge","source":"model_reports","source_url":"https://arxiv.org/abs/2501.17399","topics":["reasoning-math"],"unit":"percent"},{"aliases":["MultiChallenge"],"attention_signals":{"as_of":"2026-10-10","github_stars":98,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"artifact:arxiv:2501.17399","repository":null,"source_record":"catalog_91a0d15dc744c7b3"},"categories":["instruction_following"],"collected_at":null,"description":"Multi-turn instruction retention, graded by an LLM judge.","descriptions":{"en":"Multi-turn instruction retention, graded by an LLM judge."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000multichallenge\u0000qwen3_5_model_card\u0000multichallenge\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000multichallenge\u0000qwen3_5_model_card\u0000multichallenge\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:multichallenge","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"artifact:arxiv:2501.17399","primary":"instruction-following","source_record":"catalog_91a0d15dc744c7b3"},"modality":null,"name":"MultiChallenge","openness":"unknown","publisher":null,"released":"2025-01-29","released_reference":{"basis":"benchmark_release","source_key":"model-reports:multichallenge","source_url":"https://arxiv.org/abs/2501.17399"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":67.6,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":67.6,"raw_min":67.6,"source_reference":{"obs_id":"curated\u0000multichallenge\u0000qwen3_5_model_card\u0000multichallenge\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000multichallenge\u0000qwen3_5_model_card\u0000multichallenge\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"multichallenge","source":"model_reports","source_url":"https://arxiv.org/abs/2501.17399","topics":["language"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:grecomt/multighostbench","source_record":"bm_multighostbench_4adb4861"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Contains 928 machine-generated books across six languages and three scripts for evaluating long-form LLM-generated text attribution under distribution shifts.","descriptions":{"en":"Contains 928 machine-generated books across six languages and three scripts for evaluating long-form LLM-generated text attribution under distribution shifts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.02379","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","document-ocr","content-detection"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_multighostbench_4adb4861"},"modality":null,"name":"MultiGhostBench","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02379","source_url":"https://arxiv.org/abs/2609.02379"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02379","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02379","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_multiglobeqa_73949294"},"categories":["Factuality","Language & Knowledge","Logistics & Operations","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"MultiGlobeQA evaluates geospatial reasoning in large language models across 46,060 question-answer pairs in 17 languages, covering 14 spatial-function families and 15 answer formats, with ground truth from three knowledge graphs.","descriptions":{"en":"MultiGlobeQA evaluates geospatial reasoning in large language models across 46,060 question-answer pairs in 17 languages, covering 14 spatial-function families and 15 answer formats, with ground truth from three knowledge graphs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03882","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual","spatial-3d"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_multiglobeqa_73949294"},"modality":null,"name":"MultiGlobeQA","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03882","source_url":"https://arxiv.org/abs/2608.03882"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03882","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03882","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":63,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:facebookresearch/multi-if","source_record":"catalog_6ae40f22c77bfae7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MultiLF benchmark","descriptions":{"en":"MultiLF benchmark"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:multilf:qwen3-235b-a22b","reported_at":"2025-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilf?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:multilf","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","multilingual"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_6ae40f22c77bfae7"},"modality":"text","name":"MultiLF","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":73.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.73,"raw_min":0.719,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:multilf:qwen3-32b","reported_date":"2025-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilf?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 32B"}],"slug":"llm-stats-multilf","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilf?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":279,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:google-research/url-nlp","source_record":"catalog_d729b8fc6431d5a5"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Multilingual Grade School Math (MGSM) benchmark evaluates language models' chain-of-thought reasoning abilities across ten typologically diverse languages. Contains 250 grade-school math problems manually translated from GSM8K dataset into languages including Bengali and Swahili.","descriptions":{"en":"Multilingual Grade School Math (MGSM) benchmark evaluates language models' chain-of-thought reasoning abilities across ten typologically diverse languages. Contains 250 grade-school math problems manually translated from GSM8K dataset into languages including Bengali and Swahili."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:multilingual-mgsm-(cot):llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mgsm-%28cot%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:multilingual-mgsm-(cot)","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","math"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_d729b8fc6431d5a5"},"modality":"text","name":"Multilingual MGSM (CoT)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":91.60000000000001,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.916,"raw_min":0.689,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:multilingual-mgsm-(cot):llama-3.1-405b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mgsm-%28cot%29?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"}],"slug":"llm-stats-multilingual-mgsm-cot","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mgsm-%28cot%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_6cf53613aa96f8f6"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMLU-ProX is a comprehensive multilingual benchmark covering 29 typologically diverse languages, building upon MMLU-Pro. Each language version consists of 11,829 identical questions enabling direct cross-linguistic comparisons. The benchmark evaluates large language models' reasoning capabilities across linguistic and cultural boundaries through challenging, reasoning-focused questions with 10 answer choices.","descriptions":{"en":"MMLU-ProX is a comprehensive multilingual benchmark covering 29 typologically diverse languages, building upon MMLU-Pro. Each language version consists of 11,829 identical questions enabling direct cross-linguistic comparisons. The benchmark evaluates large language models' reasoning capabilities across linguistic and cultural boundaries through challenging, reasoning-focused questions with 10 answer choices."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:multilingual-mmlu:o3-mini","reported_at":"2025-01-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mmlu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:multilingual-mmlu","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_6cf53613aa96f8f6"},"modality":"text","name":"Multilingual MMLU","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":80.7,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.807,"raw_min":0.493,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:multilingual-mmlu:o3-mini","reported_date":"2025-01-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mmlu?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Base 2512)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"}],"slug":"llm-stats-multilingual-mmlu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mmlu?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":18,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2504.10356","repository":null,"source_record":"catalog_0ad9d08786c3ce65"},"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MultiLoKo is a multilingual knowledge benchmark, covering 30 languages plus English. MultiLoKo是一个多语言知识基准，涵盖30种语言及英语。","descriptions":{"en":"MultiLoKo is a multilingual knowledge benchmark, covering 30 languages plus English.","zh":"MultiLoKo是一个多语言知识基准，涵盖30种语言及英语。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1751","languages":["English","Multilingual"],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"artifact:arxiv:2504.10356","primary":"multilingual","source_record":"catalog_0ad9d08786c3ce65"},"modality":null,"name":"MultiLoKo","openness":"unknown","publisher":"Meta","released":"2025-04-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1751-multiloko","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MultiLoKo","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","eess.AS"],"collected_at":"2026-10-14T09:44:51Z","description":"MP-Bench evaluates conversational speech systems acting as active participants within multi-party conversation contexts.","descriptions":{"en":"MP-Bench evaluates conversational speech systems acting as active participants within multi-party conversation contexts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.13076","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_mp-bench_04d97a7f"},"modality":null,"name":"Multiparty Bench (MP-Bench)","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.13076","source_url":"https://arxiv.org/abs/2609.13076"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-13076","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.13076","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_3f8a7e66329a6d6c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MultiPL-E is a scalable and extensible system for translating unit test-driven code generation benchmarks to multiple programming languages. It extends HumanEval and MBPP Python benchmarks to 18 additional programming languages, enabling evaluation of neural code generation models across diverse programming paradigms and language features.","descriptions":{"en":"MultiPL-E is a scalable and extensible system for translating unit test-driven code generation benchmarks to multiple programming languages. It extends HumanEval and MBPP Python benchmarks to 18 additional programming languages, enabling evaluation of neural code generation models across diverse programming paradigms and language features."},"evidence_summary":{"document_count":1,"model_count":13,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:multipl-e:qwen2-72b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:multipl-e","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","multilingual"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_3f8a7e66329a6d6c"},"modality":"text","name":"MultiPL-E","openness":"unknown","publication":{"evidence":[{"note":"The introducing paper by Cassano and coauthors describes translating HumanEval and MBPP into 18 additional programming languages, matching the source record's task sets and expansion.","url":"https://arxiv.org/abs/2208.08227"},{"note":"The project repository names the same MultiPL-E authors, provides the HumanEval and MBPP translation framework, and links the later IEEE TSE publication. Its linked journal title uses 'Polyglot', while the selected arXiv publication retains 'Extensible'.","url":"https://github.com/nuprl/MultiPL-E/blob/main/README.md"}],"id":"multipl-e-original","links":[{"kind":"paper","title":"MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation","url":"https://arxiv.org/abs/2208.08227"},{"kind":"repo","title":"MultiPL-E official repository","url":"https://github.com/nuprl/MultiPL-E"}],"record_keys":["llm-stats:multipl-e"],"reviewed_at":"2026-10-26","scope":{"en":"The MultiPL-E project translates both HumanEval and MBPP into multiple programming languages. The record does not identify a specific task set or language selection, so these links describe the project rather than infer an aggregate score protocol.","zh":"MultiPL-E 项目将 HumanEval 和 MBPP 都扩展到多种编程语言。当前记录未指明具体题集或语言组合，因此链接对应整个项目，不推定汇总成绩的计算设置。"}},"publisher":null,"released":"2022-08-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:multipl-e","source_url":"https://arxiv.org/abs/2208.08227"},"repo_kind":null,"repo_resolution_status":null,"score_count":13,"score_direction":"higher_is_better","score_summary":{"display_max":87.9,"display_multiplier":100,"model_count":13,"model_count_basis":"source_model_id","numeric_count":13,"raw_max":0.879,"raw_min":0.591,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:multipl-e:qwen3-235b-a22b-instruct-2507","reported_date":"2025-07-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"}],"slug":"llm-stats-multipl-e","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_8eda1f657fbd4e93"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MultiPL-E is a scalable and extensible approach to benchmarking neural code generation that translates unit test-driven code generation benchmarks across multiple programming languages. It extends the HumanEval benchmark to 18 additional programming languages, enabling evaluation of code generation models across diverse programming paradigms and providing insights into how models generalize programming knowledge across language boundaries.","descriptions":{"en":"MultiPL-E is a scalable and extensible approach to benchmarking neural code generation that translates unit test-driven code generation benchmarks across multiple programming languages. It extends the HumanEval benchmark to 18 additional programming languages, enabling evaluation of code generation models across diverse programming paradigms and providing insights into how models generalize programming knowledge across language boundaries."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:multipl-e-humaneval:llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-humaneval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:multipl-e-humaneval","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","multilingual"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_8eda1f657fbd4e93"},"modality":"text","name":"Multipl-E HumanEval","openness":"unknown","publisher":null,"released":"2022-08-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:multipl-e-humaneval","source_url":"https://arxiv.org/abs/2208.08227"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":75.2,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.752,"raw_min":0.508,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:multipl-e-humaneval:llama-3.1-405b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-humaneval?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"}],"slug":"llm-stats-multipl-e-humaneval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-humaneval?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_43a27979a119e9f8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MultiPL-E extends the Mostly Basic Python Problems (MBPP) benchmark to 18+ programming languages for evaluating multilingual code generation capabilities. MBPP contains 974 crowd-sourced programming problems designed to be solvable by entry-level programmers, covering programming fundamentals and standard library functionality. Each problem includes a task description, code solution, and automated test cases.","descriptions":{"en":"MultiPL-E extends the Mostly Basic Python Problems (MBPP) benchmark to 18+ programming languages for evaluating multilingual code generation capabilities. MBPP contains 974 crowd-sourced programming problems designed to be solvable by entry-level programmers, covering programming fundamentals and standard library functionality. Each problem includes a task description, code solution, and automated test cases."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:multipl-e-mbpp:llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-mbpp?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:multipl-e-mbpp","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","multilingual"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_43a27979a119e9f8"},"modality":"text","name":"Multipl-E MBPP","openness":"unknown","publisher":null,"released":"2022-08-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:multipl-e-mbpp","source_url":"https://arxiv.org/abs/2208.08227"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":65.7,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.657,"raw_min":0.524,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:multipl-e-mbpp:llama-3.1-405b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-mbpp?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"}],"slug":"llm-stats-multipl-e-mbpp","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-mbpp?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":13,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":30,"join":"claire_record_id","repository":"gh:zxhhh0201/multiref-compass","source_record":"bm_multiref-compass_396a4597"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MultiRef-Compass evaluates multi-reference-to-audio-video generation systems on 350 curated samples. It assesses Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following using 14 sub-metrics, combining automatic metrics with an MLLM-as-a-Judge framework.","descriptions":{"en":"MultiRef-Compass evaluates multi-reference-to-audio-video generation systems on 350 curated samples. It assesses Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following using 14 sub-metrics, combining automatic metrics with an MLLM-as-a-Judge framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.14189","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","speech-audio"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_multiref-compass_396a4597"},"modality":null,"name":"MultiRef-Compass","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14189","source_url":"https://arxiv.org/abs/2607.14189"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14189","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14189","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MultiSeismo is a multimodal seismic dataset with over 16K events integrating waveforms, intensity maps, population exposure, and text, plus MISCE instruction set for seismic reasoning tasks like retrieval and cross-modal analysis.","descriptions":{"en":"MultiSeismo is a multimodal seismic dataset with over 16K events integrating waveforms, intensity maps, population exposure, and text, plus MISCE instruction set for seismic reasoning tasks like retrieval and cross-modal analysis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26320","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_multiseismo_9617a097"},"modality":null,"name":"MULTISEISMO","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26320","source_url":"https://arxiv.org/abs/2605.26320"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26320","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26320","topics":["multimodal","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:hkust-knowcomp/multivationbench","source_record":"bm_multivationbench_9e6dc7dd"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal motivation reasoning in story-driven visual narratives, using 1,000 narratives with 16,092 multi-label questions grounded in Maslow and Reiss frameworks across definition and practical reasoning tasks.","descriptions":{"en":"Evaluates multimodal motivation reasoning in story-driven visual narratives, using 1,000 narratives with 16,092 multi-label questions grounded in Maslow and Reiss frameworks across definition and practical reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.26465","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_multivationbench_9e6dc7dd"},"modality":null,"name":"MultivationBench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26465","source_url":"https://arxiv.org/abs/2607.26465"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26465","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26465","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_multiview-bench_66d1791f"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-view integration in vision-language models using diagnostic tasks for 3D scene comprehension, with a fixed-view baseline and proposed ViewNavigator method.","descriptions":{"en":"Evaluates multi-view integration in vision-language models using diagnostic tasks for 3D scene comprehension, with a fixed-view baseline and proposed ViewNavigator method."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.08970","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_multiview-bench_66d1791f"},"modality":null,"name":"MultiView-Bench","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08970","source_url":"https://arxiv.org/abs/2607.08970"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08970","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08970","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Multiplayer world models must ensure that independently controlled views remain consistent with one shared and persistent world.","descriptions":{"en":"Multiplayer world models must ensure that independently controlled views remain consistent with one shared and persistent world."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11723","languages":[],"library_direction":{"basis":"model-review-daily","categories":["world-models"],"join":"claire_record_id","primary":"world-models","source_record":"bm_multiworldbench_8df7b9fe"},"modality":null,"name":"MultiWorldBench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11723","source_url":"https://arxiv.org/abs/2610.11723"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11723","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11723","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_muse_8c1c770b"},"categories":["CAD","General AI","Geometric reasoning","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"MUSE evaluates text-to-CAD generation of complex B-Rep assemblies via design specifications. It scores models on code validity, geometric correctness, and design-intent alignment using rubrics covering functionality, manufacturability, and assemblability. A VLM judge with human validation is used for scalable scoring.","descriptions":{"en":"MUSE evaluates text-to-CAD generation of complex B-Rep assemblies via design specifications. It scores models on code validity, geometric correctness, and design-intent alignment using rubrics covering functionality, manufacturability, and assemblability. A VLM judge with human validation is used for scalable scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28579","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_muse_8c1c770b"},"modality":null,"name":"MUSE","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28579","source_url":"https://arxiv.org/abs/2605.28579"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28579","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28579","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating large vision-language models on artistic image understanding in situated educational applications.","descriptions":{"en":"A benchmark for evaluating large vision-language models on artistic image understanding in situated educational applications."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.19088","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_muse_6ad3fb20"},"modality":null,"name":"MUSE","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19088","source_url":"https://arxiv.org/abs/2609.19088"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19088","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19088","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A multimodal time series forecasting benchmark with heterogeneous context, covering 14 datasets across 8 domains and 6 context types.","descriptions":{"en":"A multimodal time series forecasting benchmark with heterogeneous context, covering 14 datasets across 8 domains and 6 context types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15087","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_muse-bench_ccd25d69"},"modality":null,"name":"MUSE-Bench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15087","source_url":"https://arxiv.org/abs/2609.15087"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15087","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15087","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:musebench/musebench-code","source_record":"bm_musebench_7c930c72"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"MuseBench evaluates multimodal large language models on intent-level understanding of audiovisual arts, covering cinematic arts, static visual arts, stage performing arts, and game arts with 4,016 questions in single- and multi-select formats.","descriptions":{"en":"MuseBench evaluates multimodal large language models on intent-level understanding of audiovisual arts, covering cinematic arts, static visual arts, stage performing arts, and game arts with 4,016 questions in single- and multi-select formats."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.30026","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","video-understanding","speech-audio"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_musebench_7c930c72"},"modality":null,"name":"MuseBench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30026","source_url":"https://arxiv.org/abs/2606.30026"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30026","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30026","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_54edb2991e31006c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MusicCaps is a dataset composed of 5,521 music examples, each labeled with an English aspect list and a free text caption written by musicians. The dataset contains 10-second music clips from AudioSet paired with rich textual descriptions that capture sonic qualities and musical elements like genre, mood, tempo, instrumentation, and rhythm. Created to support research in music-text understanding and generation tasks.","descriptions":{"en":"MusicCaps is a dataset composed of 5,521 music examples, each labeled with an English aspect list and a free text caption written by musicians. The dataset contains 10-second music clips from AudioSet paired with rich textual descriptions that capture sonic qualities and musical elements like genre, mood, tempo, instrumentation, and rhythm. Created to support research in music-text understanding and generation tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:musiccaps:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/musiccaps?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:musiccaps","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_54edb2991e31006c"},"modality":"multimodal","name":"MusicCaps","openness":"unknown","publisher":null,"released":"2023-01-26","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MusicLM: Generating Music From Text. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:musiccaps","source_url":"https://arxiv.org/abs/2301.11325"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":32.800000000000004,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.328,"raw_min":0.328,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:musiccaps:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/musiccaps?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-musiccaps","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/musiccaps?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"490-question benchmark for musical score understanding, performance listening, and combined score-performance reasoning, with multiple input modalities and accepted ground-truth answers.","descriptions":{"en":"490-question benchmark for musical score understanding, performance listening, and combined score-performance reasoning, with multiple input modalities and accepted ground-truth answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:bryel-labs/musp-bench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_musp-bench_8c379446"},"modality":null,"name":"MuSP-Bench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:bryel-labs/musp-bench","source_url":"https://huggingface.co/datasets/bryel-labs/MuSP-Bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-bryel-labs-musp-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/bryel-labs/MuSP-Bench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":57,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_c6bc068e5f9ecdf8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MuSR (Multistep Soft Reasoning) is a benchmark for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Created through a neurosymbolic synthetic-to-natural generation algorithm, it generates complex reasoning scenarios like murder mysteries roughly 1000 words in length that challenge current LLMs including GPT-4. The benchmark tests chain-of-thought reasoning capabilities across domains involving commonsense reasoning about physical and social situations.","descriptions":{"en":"MuSR (Multistep Soft Reasoning) is a benchmark for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Created through a neurosymbolic synthetic-to-natural generation algorithm, it generates complex reasoning scenarios like murder mysteries roughly 1000 words in length that challenge current LLMs including GPT-4. The benchmark tests chain-of-thought reasoning capabilities across domains involving commonsense reasoning about physical and social situations."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:musr:hermes-3-70b","reported_at":"2024-08-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/musr?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:musr","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_c6bc068e5f9ecdf8"},"modality":"text","name":"MuSR","openness":"unknown","publisher":null,"released":"2023-10-24","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:musr","source_url":"https://arxiv.org/abs/2310.16049"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":76.4,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.764,"raw_min":0.5067,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:musr:kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/musr?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"}],"slug":"llm-stats-musr","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/musr?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":13,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:sony/mustbench","source_record":"bm_mustbench_8ac41de6"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"MusTBENCH evaluates temporal grounding in Large Audio-Language Models through five temporally grounded question-answering tasks, validated by music experts, to test alignment with audio regions.","descriptions":{"en":"MusTBENCH evaluates temporal grounding in Large Audio-Language Models through five temporally grounded question-answering tasks, validated by music experts, to test alignment with audio regions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.29300","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_mustbench_8ac41de6"},"modality":null,"name":"MusTBENCH","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29300","source_url":"https://arxiv.org/abs/2605.29300"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29300","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29300","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A music retrieval benchmark with 3,440 tracks and seven relation-dependent tasks using natural-language instructions.","descriptions":{"en":"A music retrieval benchmark with 3,440 tracks and seven relation-dependent tasks using natural-language instructions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16090","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research","instruction-following","speech-audio"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_muunriver-bench_bb1bbee0"},"modality":null,"name":"MUUNRiver-Bench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16090","source_url":"https://arxiv.org/abs/2609.16090"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16090","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16090","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"MV-Bench evaluates multimodal language models on coordinating multi-view interface construction using Tableau workbooks, converting specifications into executable web interfaces.","descriptions":{"en":"MV-Bench evaluates multimodal language models on coordinating multi-view interface construction using Tableau workbooks, converting specifications into executable web interfaces."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19910","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_mv-bench_0da16fba"},"modality":null,"name":"MV-Bench","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19910","source_url":"https://arxiv.org/abs/2607.19910"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19910","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19910","topics":["coding","multimodal"],"unit":null},{"aliases":["MV2: Multi-View Multi-Vehicle Driving Dataset for Novel View Synthesis"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_mv2-multi-view-multi-vehicle-driving-datas_f70ec8fb"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates novel view synthesis under large viewpoint changes using synchronized captures from car, scooter, and drone across 50 scenes.","descriptions":{"en":"Evaluates novel view synthesis under large viewpoint changes using synchronized captures from car, scooter, and drone across 50 scenes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12442","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","image-video-generation"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_mv2-multi-view-multi-vehicle-driving-datas_f70ec8fb"},"modality":null,"name":"MV2","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12442","source_url":"https://arxiv.org/abs/2608.12442"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12442","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12442","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","spatial_reasoning","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive multi-modal video understanding benchmark covering 20 challenging video tasks that require temporal understanding beyond single-frame analysis. Tasks span from perception to cognition, including action recognition, temporal reasoning, spatial reasoning, object interaction, scene transition, and counterfactual inference. Uses a novel static-to-dynamic method to systematically generate video tasks from existing annotations.","descriptions":{"en":"A comprehensive multi-modal video understanding benchmark covering 20 challenging video tasks that require temporal understanding beyond single-frame analysis. Tasks span from perception to cognition, including action recognition, temporal reasoning, spatial reasoning, object interaction, scene transition, and counterfactual inference. Uses a novel static-to-dynamic method to systematically generate video tasks from existing annotations."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:mvbench:qwen2-vl-72b","reported_at":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mvbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:mvbench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"llm_stats_id","primary":"code-generation","source_record":"bm_mv-bench_0da16fba"},"modality":"multimodal","name":"MVBench","openness":"unknown","publication":{"evidence":[{"note":"The authors' README links the MVBench paper, dataset and evaluation instructions, and describes the same temporal task suite as this source record. The code link intentionally points to the relevant project subdirectory.","url":"https://github.com/OpenGVLab/Ask-Anything/blob/main/video_chat2/README.md"}],"id":"mvbench-original","links":[{"kind":"paper","title":"MVBench: A Comprehensive Multi-modal Video Understanding Benchmark","url":"https://arxiv.org/abs/2311.17005"},{"kind":"repo","title":"MVBench and VideoChat2 official implementation","url":"https://github.com/OpenGVLab/Ask-Anything/tree/main/video_chat2"}],"record_keys":["llm-stats:mvbench"],"reviewed_at":"2026-10-26","scope":{"en":"MVBench's original temporal video-understanding benchmark. Its official implementation is housed in the VideoChat2 section of the shared Ask-Anything repository.","zh":"MVBench 原始时序视频理解评测；官方实现位于共享 Ask-Anything 仓库的 VideoChat2 子目录。"}},"publisher":null,"released":"2023-11-28","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:mvbench","source_url":"https://arxiv.org/abs/2311.17005"},"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":76.6,"display_multiplier":100,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":0.766,"raw_min":0.687,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:mvbench:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mvbench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-mvbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mvbench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["MVbench","MVBench"],"attention_signals":null,"categories":["vision"],"collected_at":null,"description":"Video understanding benchmark. For non-video-native models, uses 1 fps frame extraction. Scores depend on frame sampling strategy and context length.","descriptions":{"en":"Video understanding benchmark. For non-video-native models, uses 1 fps frame extraction. Scores depend on frame sampling strategy and context length."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000mvbench\u0000zai_glm_5_3_flash_model_card\u0000mvbench\u0000temp 1.0, top_p 0.95, max context 256K, 1 fps frame-extraction for non-video-native models\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000mvbench\u0000zai_glm_5_3_flash_model_card\u0000mvbench\u0000temp 1.0, top_p 0.95, max context 256K, 1 fps frame-extraction for non-video-native models\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:mvbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["video-understanding"],"join":"review:14H034160212:2026-10-15","primary":"video-understanding","source_record":null},"modality":null,"name":"MVbench","openness":"unknown","publisher":null,"released":"2025-06-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:mvbench","source_url":"https://github.com/MVBench/MVBench"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":77.8,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":77.8,"raw_min":77.8,"source_reference":{"obs_id":"curated\u0000mvbench\u0000zai_glm_5_3_flash_model_card\u0000mvbench\u0000temp 1.0, top_p 0.95, max context 256K, 1 fps frame-extraction for non-video-native models\u0000GLM-5.3-Flash","observation_id":"curated\u0000mvbench\u0000zai_glm_5_3_flash_model_card\u0000mvbench\u0000temp 1.0, top_p 0.95, max context 256K, 1 fps frame-extraction for non-video-native models\u0000GLM-5.3-Flash","reported_at":"2026-08-27","reported_date":"2026-08-27","source_id":"zai_glm_5_3_flash_model_card","source_url":"https://z.ai/blog/glm-5.3-flash"},"unit":"percent"},"scored_models":[{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"}],"slug":"mvbench","source":"model_reports","source_url":"https://github.com/MVBench/MVBench","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","视频理解","Video Understanding","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MVBench can test MLLMs' temporal understanding in the dynamic video tasks. It covers 20 challenging video tasks that cannot be effectively solved with a single frame. MVBench用于评估多模态大模型在动态视频任务中的时间理解能力，由20个单帧内容无法有效解决的挑战性的视频任务组成。","descriptions":{"en":"MVBench can test MLLMs' temporal understanding in the dynamic video tasks. It covers 20 challenging video tasks that cannot be effectively solved with a single frame.","zh":"MVBench用于评估多模态大模型在动态视频任务中的时间理解能力，由20个单帧内容无法有效解决的挑战性的视频任务组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1509","languages":[],"library_direction":null,"modality":"multimodal","name":"MVBench","openness":"open","publisher":"Chinese Academy of Sciences","released":"2023-11-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1509-mvbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MVBench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A browser-based benchmark with 600 live CAPTCHA instances and a matched foreground-only control benchmark, designed to evaluate GUI agents against motion-based CAPTCHAs.","descriptions":{"en":"A browser-based benchmark with 600 live CAPTCHA instances and a matched foreground-only control benchmark, designed to evaluate GUI agents against motion-based CAPTCHAs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27461","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_mvcap-bench_e86a51ec"},"modality":null,"name":"MVCAP-Bench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27461","source_url":"https://arxiv.org/abs/2609.27461"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27461","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27461","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aaandy-zhu/mvgraph","source_record":"bm_mvgraph_5d777630"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"We introduce MVGraph, a benchmark for evaluating large vision-language models (LVLMs) on multi-graph reasoning where the evidence required by a query is distributed across multiple interrelated visual graph views.","descriptions":{"en":"We introduce MVGraph, a benchmark for evaluating large vision-language models (LVLMs) on multi-graph reasoning where the evidence required by a query is distributed across multiple interrelated visual graph views."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.12961","languages":[],"library_direction":{"basis":"model-review-daily","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_mvgraph_5d777630"},"modality":null,"name":"MVGraph","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12961","source_url":"https://arxiv.org/abs/2610.12961"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12961","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12961","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","语言","Language","多模态模型","VLM","图像理解","Image Understanding","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MVL-SIB is a multilingual dataset that provides image-sentence pairs spanning 205 languages and 7 topical categories (entertainment, geography, health, politics, science, sports, travel). It was constructed by extending the SIB-200 benchmark. MVL-SIB 是一个多语言数据集，提供了涵盖 205 种语言和 7 个主题类别的图像-句子对（ entertainment ， geography ， health ， politics ， science ， sports ， travel ）。它通过扩展 SIB-200 基准构建而成。","descriptions":{"en":"MVL-SIB is a multilingual dataset that provides image-sentence pairs spanning 205 languages and 7 topical categories (entertainment, geography, health, politics, science, sports, travel). It was constructed by extending the SIB-200 benchmark.","zh":"MVL-SIB 是一个多语言数据集，提供了涵盖 205 种语言和 7 个主题类别的图像-句子对（ entertainment ， geography ， health ， politics ， science ， sports ， travel ）。它通过扩展 SIB-200 基准构建而成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1537","languages":["English","Multilingual"],"library_direction":null,"modality":"multimodal","name":"MVL-SIB","openness":"unknown","publisher":"University of Würzburg, University of Hamburg","released":"2025-02-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1537-mvl-sib","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MVL-SIB","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","理解","Understanding","多模态模型","VLM","逻辑推理","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"MVPBench, a curated benchmark designed to rigorously evaluate visual physical reasoning through the lens of visual CoT. Each example features interleaved multi-image inputs and demands not only the correct final answer but also a coherent, step-by-step reasoning path grounded in evolving visual cues MVPBench专注于视觉物理推理中的视觉链式思维（CoT）能力评估。该基准涵盖真实图像、多步逻辑与多条可行思维路径，每个样例均配有图像证据，要求模型在剥离文本提示依赖的前提下，不仅得出正确答案，还需正确完成每一个中间推理步骤，模拟人类的逐步图像推理过程。","descriptions":{"en":"MVPBench, a curated benchmark designed to rigorously evaluate visual physical reasoning through the lens of visual CoT. Each example features interleaved multi-image inputs and demands not only the correct final answer but also a coherent, step-by-step reasoning path grounded in evolving visual cues","zh":"MVPBench专注于视觉物理推理中的视觉链式思维（CoT）能力评估。该基准涵盖真实图像、多步逻辑与多条可行思维路径，每个样例均配有图像证据，要求模型在剥离文本提示依赖的前提下，不仅得出正确答案，还需正确完成每一个中间推理步骤，模拟人类的逐步图像推理过程。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1905","languages":[],"library_direction":null,"modality":"multimodal","name":"MVPBench","openness":"open","publisher":"Central South University","released":"2025-06-02","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1905-mvpbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MVPBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-view video reasoning in vision-language models using real-world multi-camera data.","descriptions":{"en":"Evaluates multi-view video reasoning in vision-language models using real-world multi-camera data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.30952","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_mvvbench_ff4cca44"},"modality":null,"name":"MVVBench","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30952","source_url":"https://arxiv.org/abs/2609.30952"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30952","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30952","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"MyPCBench tests computer-use agents as personal assistants on a Linux desktop seeded with 17 simulated web apps for a single persona. It has 184 tasks.","descriptions":{"en":"MyPCBench tests computer-use agents as personal assistants on a Linux desktop seeded with 17 simulated web apps for a single persona. It has 184 tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16748","languages":[],"library_direction":null,"modality":null,"name":"MyPCBench","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16748","source_url":"https://arxiv.org/abs/2606.16748"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16748","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16748","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Shared task for multidialectal Arabic speech processing with evaluation and rankings.","descriptions":{"en":"Shared task for multidialectal Arabic speech processing with evaluation and rankings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27086","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_nadi-2026_222cfa18"},"modality":null,"name":"NADI 2026","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27086","source_url":"https://arxiv.org/abs/2609.27086"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27086","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27086","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"A procedural generator of solver-certified instances spanning four semantic viewpoints: SLDNF, well-founded semantics, and credulous and skeptical reasoning under stable-model semantics.","descriptions":{"en":"A procedural generator of solver-certified instances spanning four semantic viewpoints: SLDNF, well-founded semantics, and credulous and skeptical reasoning under stable-model semantics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27517","languages":[],"library_direction":null,"modality":null,"name":"NAFBench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27517","source_url":"https://arxiv.org/abs/2609.27517"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27517","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27517","topics":["reasoning-math"],"unit":null},{"aliases":["NAFBench","NAF-Bench","NAF Bench"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Solver-certified normal logic programs verbalized into natural language, each labelled under four negation semantics (SLDNF, well-founded, and credulous and skeptical stable-model reasoning) that can disagree on the same instance. A score without the semantics named is not comparable. Models are also sensitive to rule order. The leaderboard's hidden test split (Hugging Face qbao775/naf-bench, as described by the repository README at revision 83ede91) has 495 rows drawn from only 77 programs, so the effective sample is the program, not the row. The arXiv v1 appendix describes an earlier 385-prompt evaluation set over 77 programs, so a paper number and a leaderboard number are not on the same split.","descriptions":{"en":"Solver-certified normal logic programs verbalized into natural language, each labelled under four negation semantics (SLDNF, well-founded, and credulous and skeptical stable-model reasoning) that can disagree on the same instance. A score without the semantics named is not comparable. Models are also sensitive to rule order. The leaderboard's hidden test split (Hugging Face qbao775/naf-bench, as described by the repository README at revision 83ede91) has 495 rows drawn from only 77 programs, so the effective sample is the program, not the row. The arXiv v1 appendix describes an earlier 385-prompt evaluation set over 77 programs, so a paper number and a leaderboard number are not on the same split."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:nafbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"NAFBench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"model-reports:nafbench","source_url":"https://arxiv.org/abs/2609.27517"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"nafbench","source":"model_reports","source_url":"https://arxiv.org/abs/2609.27517","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":5941,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_bf694ecef574797c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code","systems"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"NanoGPT is an OpenAI AI-self-improvement evaluation that measures whether models can optimize training recipes for small GPT-style models, part of the suite tracking progress toward accelerating internal research.","descriptions":{"en":"NanoGPT is an OpenAI AI-self-improvement evaluation that measures whether models can optimize training recipes for small GPT-style models, part of the suite tracking progress toward accelerating internal research."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nanogpt:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nanogpt?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nanogpt","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_bf694ecef574797c"},"modality":"text","name":"NanoGPT","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":14.499999999999998,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.145,"raw_min":0.0166,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nanogpt:gpt-5.6-terra","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nanogpt?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-nanogpt","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nanogpt?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"NarrativeWorldBench evaluates long-horizon narrative generation in audio drama across nine structural metrics and four Indic languages, across horizons from 10 to 200 episodes. Scoring is based on plot-beat F1 and other metrics.","descriptions":{"en":"NarrativeWorldBench evaluates long-horizon narrative generation in audio drama across nine structural metrics and four Indic languages, across horizons from 10 to 200 episodes. Scoring is based on plot-beat F1 and other metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17391","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","speech-audio","multilingual"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_narrativeworldbench_d47453bd"},"modality":null,"name":"NarrativeWorldBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17391","source_url":"https://arxiv.org/abs/2606.17391"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17391","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17391","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"claire_record_id","repository":"gh:infinimind-inc/naru_benchmark","source_record":"bm_naru_dcb9acb4"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"NARU evaluates multimodal models on Japanese long-form video understanding across narrative evolution (character evolution, sequential flow, plot progression, thematic development) and cultural nuance (aizuchi, reading the air, subtext, cultural context, sentiment) via multiple-choice QA.","descriptions":{"en":"NARU evaluates multimodal models on Japanese long-form video understanding across narrative evolution (character evolution, sequential flow, plot progression, thematic development) and cultural nuance (aizuchi, reading the air, subtext, cultural context, sentiment) via multiple-choice QA."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.13210","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","long-context","multilingual"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_naru_dcb9acb4"},"modality":null,"name":"NARU","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13210","source_url":"https://arxiv.org/abs/2608.13210"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13210","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13210","topics":["reasoning-math","long-context","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1138,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:google-research-datasets/natural-questions","source_record":"catalog_db81694b1eca554d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Natural Questions is a question answering dataset featuring real anonymized queries issued to Google search engine. It contains 307,373 training examples where annotators provide long answers (passages) and short answers (entities) from Wikipedia pages, or mark them as unanswerable.","descriptions":{"en":"Natural Questions is a question answering dataset featuring real anonymized queries issued to Google search engine. It contains 307,373 training examples where annotators provide long answers (passages) and short answers (entities) from Wikipedia pages, or mark them as unanswerable."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:natural-questions:gemma-2-27b-it","reported_at":"2024-06-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/natural-questions?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:natural-questions","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","deep-research"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_db81694b1eca554d"},"modality":"text","name":"Natural Questions","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":34.5,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.345,"raw_min":0.155,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:natural-questions:gemma-2-27b-it","reported_date":"2024-06-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/natural-questions?top_n=500"},"unit":null},"scored_models":[{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2024-07-18","date_precision":"model_announcement","name":"Mistral NeMo Instruct"}],"slug":"llm-stats-natural-questions","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/natural-questions?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":70,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_c12e1090234d9345"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"NaturalCodeBench (NCB) is a challenging code benchmark designed to mirror the complexity and variety of real-world coding tasks. It comprises 402 high-quality problems in Python and Java, selected from natural user queries from online coding services, covering 6 different domains.","descriptions":{"en":"NaturalCodeBench (NCB) is a challenging code benchmark designed to mirror the complexity and variety of real-world coding tasks. It comprises 402 high-quality problems in Python and Java, selected from natural user queries from online coding services, covering 6 different domains."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:natural2code:gemini-1.5-flash-8b","reported_at":"2024-03-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/natural2code?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:natural2code","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_c12e1090234d9345"},"modality":"text","name":"Natural2Code","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":92.9,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.929,"raw_min":0.56,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:natural2code:gemini-2.0-flash","reported_date":"2024-12-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/natural2code?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"}],"slug":"llm-stats-natural2code","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/natural2code?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","ACL 2024","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"NaturalCodeBench is challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. NaturalCodeBench 是一个具有挑战性的代码基准，旨在反映真实编码任务中的复杂性和多样性。NaturalCodeBench 包含 402 个高质量的 Python 和 Java 问题，这些问题是从在线编码服务的自然用户查询中精心挑选的，涵盖了 6 个不同的领域。","descriptions":{"en":"NaturalCodeBench is challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains.","zh":"NaturalCodeBench 是一个具有挑战性的代码基准，旨在反映真实编码任务中的复杂性和多样性。NaturalCodeBench 包含 402 个高质量的 Python 和 Java 问题，这些问题是从在线编码服务的自然用户查询中精心挑选的，涵盖了 6 个不同的领域。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1081","languages":[],"library_direction":null,"modality":null,"name":"NaturalCodeBench","openness":"restricted","publisher":"THUDM","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1081-naturalcodebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NaturalCodeBench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["数学","Math","科学智能","AI for Science","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"NATURALPROOFS is a multi-domain corpus of mathematical statements and their proofs, written in natural mathematical language. NATURALPROOFS unifies broad coverage, deep coverage, and low-resource mathematical sources, allowing for evaluating both in-distribution and zero-shot generalization. NATURALPROOFS 是一个多领域的数学语句及其证明的语料库，采用自然数学语言编写。整合了广泛覆盖、深入覆盖和低资源数学来源，便于评估分布内和零样本泛化的能力。","descriptions":{"en":"NATURALPROOFS is a multi-domain corpus of mathematical statements and their proofs, written in natural mathematical language. NATURALPROOFS unifies broad coverage, deep coverage, and low-resource mathematical sources, allowing for evaluating both in-distribution and zero-shot generalization.","zh":"NATURALPROOFS 是一个多领域的数学语句及其证明的语料库，采用自然数学语言编写。整合了广泛覆盖、深入覆盖和低资源数学来源，便于评估分布内和零样本泛化的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1117","languages":[],"library_direction":null,"modality":null,"name":"NaturalProofs","openness":"unknown","publisher":"Allen Institute for Artificial Intelligence","released":"2021-06-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1117-naturalproofs","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NaturalProofs","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":131,"github_stars_30d":15,"hf_dataset_downloads":null,"hf_paper_upvotes":67,"join":"claire_record_id","repository":"gh:frontisai/naturebench","source_record":"bm_naturebench_ffd14b37"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"NatureBench evaluates AI coding agents on 90 tasks distilled from Nature-family publications across 6 scientific domains, scoring against each paper's reported state of the art.","descriptions":{"en":"NatureBench evaluates AI coding agents on 90 tasks distilled from Nature-family publications across 6 scientific domains, scoring against each paper's reported state of the art."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.24530","languages":[],"library_direction":{"basis":"model-review","categories":["science","agentic-coding"],"join":"claire_record_id","primary":"science","source_record":"bm_naturebench_ffd14b37"},"modality":null,"name":"NatureBench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24530","source_url":"https://arxiv.org/abs/2606.24530"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24530","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24530","topics":["coding","agents","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"NavVerse evaluates indoor-to-outdoor embodied navigation in continuous robot simulation, with 10,000 episodes across object, vision-language, and place navigation tasks.","descriptions":{"en":"NavVerse evaluates indoor-to-outdoor embodied navigation in continuous robot simulation, with 10,000 episodes across object, vision-language, and place navigation tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19695","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_navverse_d8d5fb61"},"modality":null,"name":"NavVerse","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19695","source_url":"https://arxiv.org/abs/2607.19695"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19695","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19695","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:wyy-081/nba_streaming","source_record":"bm_nba-streaming_a4cd00b8"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"NBA_Streaming is a benchmark for online fine-grained basketball commentary generation, containing 307.5 hours of broadcasts and approximately 35K temporally aligned events with annotations of event boundaries, player identities, fine-grained actions, event chains, and natural-language commentary.","descriptions":{"en":"NBA_Streaming is a benchmark for online fine-grained basketball commentary generation, containing 307.5 hours of broadcasts and approximately 35K temporally aligned events with annotations of event boundaries, player identities, fine-grained actions, event chains, and natural-language commentary."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09200","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","chat-writing"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_nba-streaming_a4cd00b8"},"modality":null,"name":"NBA_Streaming","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09200","source_url":"https://arxiv.org/abs/2608.09200"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09200","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09200","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":34,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":30,"join":"claire_record_id","repository":"gh:yingpengma/ncp-bench","source_record":"bm_ncp-bench_83c91412"},"categories":["Agents & Tool Use","General AI","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"NCP-Bench evaluates narrative commitment preservation in interactive narratives. It provides 100 movie-synopsis-based environments with structured narrative specifications and an automatic evaluator that checks fact, commitment, and trajectory consistency across narrator responses to player actions.","descriptions":{"en":"NCP-Bench evaluates narrative commitment preservation in interactive narratives. It provides 100 movie-synopsis-based environments with structured narrative specifications and an automatic evaluator that checks fact, commitment, and trajectory consistency across narrator responses to player actions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.08160","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","long-context"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_ncp-bench_83c91412"},"modality":null,"name":"NCP-Bench","openness":"unknown","publisher":null,"released":"2026-08-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08160","source_url":"https://arxiv.org/abs/2608.08160"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08160","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08160","topics":["agents","long-context","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"NEEDL-Bench is a microscopy detection benchmark for Swiss Needle Cast and stomata detection, with 3250 annotated images from 1082 Douglas-fir needles, annotated for keypoint and bounding-box detectors, and two evaluation splits.","descriptions":{"en":"NEEDL-Bench is a microscopy detection benchmark for Swiss Needle Cast and stomata detection, with 3250 annotated images from 1082 Douglas-fir needles, annotated for keypoint and bounding-box detectors, and two evaluation splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.12076","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_needl-bench_c98901f1"},"modality":null,"name":"NEEDL-Bench","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.12076","source_url":"https://arxiv.org/abs/2607.12076"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-12076","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12076","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":38,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:he-jiayou/neobench","source_record":"bm_neo-bench_27ddbc84"},"categories":["General AI","Vision & 3D","astro-ph.IM"],"collected_at":"2026-10-14T09:44:51Z","description":"Combines five astronomical streak datasets in YOLO format with predefined within-source and leave-one-source-out splits to evaluate cross-source generalization of detectors.","descriptions":{"en":"Combines five astronomical streak datasets in YOLO format with predefined within-source and leave-one-source-out splits to evaluate cross-source generalization of detectors."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.06774","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_neo-bench_27ddbc84"},"modality":null,"name":"NEO-Bench","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06774","source_url":"https://arxiv.org/abs/2609.06774"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06774","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06774","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:iriis-research/nepplural","source_record":"bm_nepplural_d39a60a6"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"NepPlural is a dataset and evaluation suite for modeling societal personas in Nepali migration discourse, with human gold annotations, LLM protocol audit, encoder baselines, and human-judged steerability.","descriptions":{"en":"NepPlural is a dataset and evaluation suite for modeling societal personas in Nepali migration discourse, with human gold annotations, LLM protocol audit, encoder baselines, and human-judged steerability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.15732","languages":[],"library_direction":{"basis":"model-review-daily","categories":["multilingual","chat-writing"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_nepplural_d39a60a6"},"modality":null,"name":"NepPlural","openness":"unknown","publisher":null,"released":"2026-10-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.15732","source_url":"https://arxiv.org/abs/2610.15732"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-15732","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15732","topics":["language"],"unit":null},{"aliases":["Net-Ev$^2$: A Generative Simulator for Network Event Evolution"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"Net-Ev$^2$-6.5M evaluates generated network dynamics using aligned event and traffic data from four large-scale road networks and the topology-aware JL-MMD metric.","descriptions":{"en":"Net-Ev$^2$-6.5M evaluates generated network dynamics using aligned event and traffic data from four large-scale road networks and the topology-aware JL-MMD metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.12494","languages":[],"library_direction":null,"modality":null,"name":"Net-Ev$^2$-6.5M","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12494","source_url":"https://arxiv.org/abs/2606.12494"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12494","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12494","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.NI"],"collected_at":"2026-10-14T09:44:51Z","description":"NetArtifactBench evaluates the ability of AI agents to repair inconsistent network experiment records while preserving claims that remain supported. It contains 52 instances with injected inconsistencies ranging from direct contradictions to unstated relations across multiple artifacts.","descriptions":{"en":"NetArtifactBench evaluates the ability of AI agents to repair inconsistent network experiment records while preserving claims that remain supported. It contains 52 instances with injected inconsistencies ranging from direct contradictions to unstated relations across multiple artifacts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09849","languages":[],"library_direction":null,"modality":null,"name":"NetArtifactBench","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09849","source_url":"https://arxiv.org/abs/2609.09849"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09849","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09849","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.NI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agents in closed-loop network configuration across 480 task instances from 96 protocol-focused templates, scoring based on hidden executable test cases.","descriptions":{"en":"Evaluates LLM agents in closed-loop network configuration across 480 task instances from 96 protocol-focused templates, scoring based on hidden executable test cases."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23179","languages":[],"library_direction":{"basis":"model-review-2","categories":["engineering-design","agentic-coding"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_netconfarena_7ff1c051"},"modality":null,"name":"NetConfArena","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23179","source_url":"https://arxiv.org/abs/2608.23179"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23179","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23179","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI","Tool use"],"collected_at":"2026-10-14T09:44:51Z","description":"NetInjectBench evaluates LLM agents for network operations under indirect prompt injection. It comprises 130 scenarios: 40 benign, 40 weak-attack, 40 strong-attack, and 10 approved high-impact changes, with separated untrusted artifact text, trusted policy metadata, and evaluation labels. Scoring measures unsafe tool-action rate, usefulness, and overblocking across models and defenses.","descriptions":{"en":"NetInjectBench evaluates LLM agents for network operations under indirect prompt injection. It comprises 130 scenarios: 40 benign, 40 weak-attack, 40 strong-attack, and 10 approved high-impact changes, with separated untrusted artifact text, trusted policy metadata, and evaluation labels. Scoring measures unsafe tool-action rate, usefulness, and overblocking across models and defenses."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.10490","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","tool-use"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_netinjectbench_1ee2f598"},"modality":null,"name":"NetInjectBench","openness":"unknown","publisher":null,"released":"2026-07-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10490","source_url":"https://arxiv.org/abs/2607.10490"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10490","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10490","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"NetlistBench evaluates LLM reliability in recognizing and manipulating SPICE netlists, covering parameter and connectivity recognition, edits, hierarchical operations, equivalence judgment, and compound editing, with deterministic structure-aware scoring.","descriptions":{"en":"NetlistBench evaluates LLM reliability in recognizing and manipulating SPICE netlists, covering parameter and connectivity recognition, edits, hierarchical operations, equivalence judgment, and compound editing, with deterministic structure-aware scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12197","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_netlistbench_58d2657e"},"modality":null,"name":"NetlistBench","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12197","source_url":"https://arxiv.org/abs/2608.12197"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12197","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12197","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:neurai-vn/neurai-vn-benchmark","source_record":"bm_neurai-vn-benchmark_10fca573"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates machine learning models on the Neurai-VN dataset for mental health classification. Four binary tasks (healthy control vs. depression, anxiety, clinical, and depression vs. anxiety) are defined using subject-wise cross-validation and standardized feature groups. Baseline models include linear, tree-based, and neural networks, with mean F1 scores reported.","descriptions":{"en":"Evaluates machine learning models on the Neurai-VN dataset for mental health classification. Four binary tasks (healthy control vs. depression, anxiety, clinical, and depression vs. anxiety) are defined using subject-wise cross-validation and standardized feature groups. Baseline models include linear, tree-based, and neural networks, with mean F1 scores reported."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.25232","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_neurai-vn-benchmark_10fca573"},"modality":null,"name":"Neurai-VN Benchmark","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25232","source_url":"https://arxiv.org/abs/2607.25232"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25232","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25232","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Autonomous Driving","Robotics & Embodied AI","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Dataset-centric effort that benchmarks neuro-symbolic and ontological caption generation using pre-trained grounding and learned auto-regressive captioning networks.","descriptions":{"en":"Dataset-centric effort that benchmarks neuro-symbolic and ontological caption generation using pre-trained grounding and learned auto-regressive captioning networks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16919","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_neurosymbead_a6e6bef0"},"modality":null,"name":"NeuroSymbEAD","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16919","source_url":"https://arxiv.org/abs/2609.16919"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16919","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16919","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["创作","Creation","ACL 2024","大语言模型","LLM","语言生成","Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"NewsBench is a novel evaluation framework to systematically assess the capabilities of Large Language Models (LLMs) for editorial capabilities in Chinese journalism. NewsBench 是一个新颖的评估框架，旨在系统性地评估大型语言模型在中文新闻编辑能力上的表现。构建的基准数据集聚焦于写作能力的四个方面和安全遵循的六个方面，包含 1,267 个手动精心设计的测试样本，类型包括选择题和简答题，涵盖 24 个新闻领域的五项编辑任务。","descriptions":{"en":"NewsBench is a novel evaluation framework to systematically assess the capabilities of Large Language Models (LLMs) for editorial capabilities in Chinese journalism.","zh":"NewsBench 是一个新颖的评估框架，旨在系统性地评估大型语言模型在中文新闻编辑能力上的表现。构建的基准数据集聚焦于写作能力的四个方面和安全遵循的六个方面，包含 1,267 个手动精心设计的测试样本，类型包括选择题和简答题，涵盖 24 个新闻领域的五项编辑任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1074","languages":["Chinese"],"library_direction":null,"modality":null,"name":"NewsBench","openness":"unknown","publisher":"IAAR-Shanghai","released":"2024-06-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1074-newsbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NewsBench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"NextMotionQA is a benchmark for human motion understanding with VLMs, featuring three tasks: multiple-choice QA, video captioning, and fine-grained error correction. It includes expert-verified annotations across three semantic axes and three complexity levels.","descriptions":{"en":"NextMotionQA is a benchmark for human motion understanding with VLMs, featuring three tasks: multiple-choice QA, video captioning, and fine-grained error correction. It includes expert-verified annotations across three semantic axes and three complexity levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04773","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_nextmotionqa_b6672a26"},"modality":null,"name":"NextMotionQA","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04773","source_url":"https://arxiv.org/abs/2606.04773"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04773","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04773","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":325,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:nexusflowai/nexusraven","source_record":"catalog_f5cfcb570b7edac2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"NexusRaven benchmark for evaluating function calling capabilities of large language models in zero-shot scenarios across cybersecurity tools and API interactions","descriptions":{"en":"NexusRaven benchmark for evaluating function calling capabilities of large language models in zero-shot scenarios across cybersecurity tools and API interactions"},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nexus:llama-3.1-405b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nexus?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nexus","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_f5cfcb570b7edac2"},"modality":"text","name":"Nexus","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":58.699999999999996,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.587,"raw_min":0.343,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nexus:llama-3.1-405b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nexus?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 405B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 70B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 8B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"}],"slug":"llm-stats-nexus","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nexus?top_n=500","topics":["agents"],"unit":null},{"aliases":["Accelerometry-Derived Digital Biomarkers for Cardiometabolic Risk: A Population-Representative Tabular Benchmark with Uncertainty Quantification"],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"A tabular benchmark for predicting cardiometabolic biomarkers from accelerometry-derived activity phenotypes and lifestyle covariates in a population-representative NHANES cohort.","descriptions":{"en":"A tabular benchmark for predicting cardiometabolic biomarkers from accelerometry-derived activity phenotypes and lifestyle covariates in a population-representative NHANES cohort."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.30702","languages":[],"library_direction":null,"modality":null,"name":"NHANES Accelerometry Cardiometabolic Benchmark","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30702","source_url":"https://arxiv.org/abs/2606.30702"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30702","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30702","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Multi-needle in a haystack benchmark for evaluating long-context comprehension capabilities of language models by testing retrieval of multiple target pieces of information from extended documents","descriptions":{"en":"Multi-needle in a haystack benchmark for evaluating long-context comprehension capabilities of language models by testing retrieval of multiple target pieces of information from extended documents"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nih-multi-needle:llama-3.2-3b-instruct","reported_at":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nih-multi-needle?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nih-multi-needle","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_1affa89e646bc694"},"modality":"text","name":"NIH/Multi-needle","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":84.7,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.847,"raw_min":0.847,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nih-multi-needle:llama-3.2-3b-instruct","reported_date":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nih-multi-needle?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"}],"slug":"llm-stats-nih-multi-needle","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nih-multi-needle?top_n=500","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ibasicplan/nl-pddl-bench","source_record":"bm_nl-pddl-bench_3b0f6c71"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"NL-PDDL-Bench evaluates natural-language-to-PDDL specification generation. It consists of multi-domain instances from IPC domains with planner-verified executability, difficulty scaled by object count, and a suite for parseability, solvability, and plan-level consistency.","descriptions":{"en":"NL-PDDL-Bench evaluates natural-language-to-PDDL specification generation. It consists of multi-domain instances from IPC domains with planner-verified executability, difficulty scaled by object count, and a suite for parseability, solvability, and plan-level consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.29700","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","abstract-reasoning"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_nl-pddl-bench_3b0f6c71"},"modality":null,"name":"NL-PDDL-Bench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29700","source_url":"https://arxiv.org/abs/2606.29700"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29700","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29700","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":182,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":52,"join":"llm_stats_id","repository":null,"source_record":"catalog_1d870239ba679184"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"NL2Repo evaluates long-horizon coding capabilities including repository-level understanding, where models must generate or modify code across entire repositories from natural language specifications.","descriptions":{"en":"NL2Repo evaluates long-horizon coding capabilities including repository-level understanding, where models must generate or modify code across entire repositories from natural language specifications."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nl2repo:minimax-m2.7","reported_at":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nl2repo?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nl2repo","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","code-generation"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_1d870239ba679184"},"modality":"text","name":"NL2Repo","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":61.5,"display_multiplier":100,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":0.615,"raw_min":0.294,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nl2repo:deepseek-v4-pro-0813","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nl2repo?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0731"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-nl2repo","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nl2repo?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["NL2Repo","NL-to-Repo"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Evaluates generating a repository from a natural language description. Temperature=1.0, top_p=1.0, max_new_tokens=64k under 1M context.","descriptions":{"en":"Evaluates generating a repository from a natural language description. Temperature=1.0, top_p=1.0, max_new_tokens=64k under 1M context."},"evidence_summary":{"document_count":3,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000nl2repo\u0000zai_glm_5_3_flash_model_card\u0000nl2repo\u0000temp 1.0, top_p 1.0, max_new_tokens 64k, 1M context, rule-based + LLM judge for hacking prevention\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000nl2repo\u0000zai_glm_5_3_flash_model_card\u0000nl2repo\u0000temp 1.0, top_p 1.0, max_new_tokens 64k, 1M context, rule-based + LLM judge for hacking prevention\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:nl2repo","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"NL2Repo","openness":"unknown","publisher":null,"released":"2025-06-15","released_reference":{"basis":"benchmark_release","source_key":"model-reports:nl2repo","source_url":"https://github.com/nl2repo/nl2repo"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":65.4,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":65.4,"raw_min":56.3,"source_reference":{"obs_id":"curated\u0000nl2repo\u0000deepseek_v4_1_flash_technical_report\u0000nl2repo\u0000think max, score\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000nl2repo\u0000deepseek_v4_1_flash_technical_report\u0000nl2repo\u0000think max, score\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"nl2repo","source":"model_reports","source_url":"https://github.com/nl2repo/nl2repo","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Code generation","General AI","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"NL2Scratch is an executable benchmark for natural-language-to-Scratch generation, consisting of 311,648 parser-valid NL-program pairs extracted from real Scratch projects. It includes a semantically validated pool of 23,594 examples and an 800-example diagnostic set. Evaluation uses Semantic Alignment Consistency (SAC), an interpretable slot-level metric for measuring semantic agreement.","descriptions":{"en":"NL2Scratch is an executable benchmark for natural-language-to-Scratch generation, consisting of 311,648 parser-valid NL-program pairs extracted from real Scratch projects. It includes a semantically validated pool of 23,594 examples and an 800-example diagnostic set. Evaluation uses Semantic Alignment Consistency (SAC), an interpretable slot-level metric for measuring semantic agreement."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.22061","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_nl2scratch_fe2ce42e"},"modality":null,"name":"NL2Scratch","openness":"unknown","publisher":null,"released":"2026-06-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22061","source_url":"https://arxiv.org/abs/2606.22061"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22061","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22061","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark suite for translating natural language requirements into SHACL shapes, evaluating semantic equivalence beyond string comparison.","descriptions":{"en":"A benchmark suite for translating natural language requirements into SHACL shapes, evaluating semantic equivalence beyond string comparison."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07530","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_nl2shacl-bench_8b61bc16"},"modality":null,"name":"NL2SHACL-Bench","openness":"unknown","publisher":null,"released":"2026-07-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07530","source_url":"https://arxiv.org/abs/2608.07530"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07530","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07530","topics":["coding"],"unit":null},{"aliases":["nlp-sentiment-analysis-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:anisha-das-kts/nlp-sentiment-analysis-benchmark","source_record":"bm_nlp-sentiment-analysis-benchmark_f53e4e25"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A reproducible NLP benchmarking project comparing three sentiment-analysis approaches on the NLTK Movie Reviews dataset, reporting predictive performance alongside training cost, latency, and error analysis.","descriptions":{"en":"A reproducible NLP benchmarking project comparing three sentiment-analysis approaches on the NLTK Movie Reviews dataset, reporting predictive performance alongside training cost, latency, and error analysis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:anisha-das-kts/nlp-sentiment-analysis-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_nlp-sentiment-analysis-benchmark_f53e4e25"},"modality":null,"name":"NLP Sentiment Analysis Benchmark","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:anisha-das-kts/nlp-sentiment-analysis-benchmark","source_url":"https://github.com/anisha-das-kts/nlp-sentiment-analysis-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-anisha-das-kts-nlp-sentiment-analysis-benchmark","source":"claire_radar","source_url":"https://github.com/anisha-das-kts/nlp-sentiment-analysis-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"NMOS evaluation benchmark for assessing model performance on specialized tasks","descriptions":{"en":"NMOS evaluation benchmark for assessing model performance on specialized tasks"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nmos:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nmos?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nmos","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_57e68ee3e8de805c"},"modality":"text","name":"NMOS","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.0451,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.0451,"raw_min":0.0451,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nmos:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nmos?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-nmos","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nmos?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:nums-ai/nodeground","source_record":"bm_nodeground_1d3bdef4"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Can a pretrained graph model replace training and tuning a separate predictor for each dataset?","descriptions":{"en":"Can a pretrained graph model replace training and tuning a separate predictor for each dataset?"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.39673","languages":[],"library_direction":null,"modality":null,"name":"NodeGround","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39673","source_url":"https://arxiv.org/abs/2609.39673"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39673","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39673","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":205,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"llm_stats_id","repository":"gh:adobe-research/nolima","source_record":"catalog_00d121bee97ad2d2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"","descriptions":{},"editorial_guide":{"id":"nolima-background","record_keys":["llm-stats:community:2256e9c9-b256-4444-b639-7cc3b1855d96"],"reviewed_at":"2026-10-23","scope":{"en":"Cited research guide; original source measurements remain independent.","zh":"附来源的研究导读；原始记录的测量保持独立。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"The original research minimizes lexical overlap between a question and a relevant fact hidden in a long context. Models must recover the association rather than copy a matching phrase.","zh":"原研究尽量减少问题与长文本中目标事实的字面重合，要求模型恢复语义关联，而非复制匹配短语。"},"title":{"en":"Why literal matching is insufficient","zh":"为什么不能只匹配关键词"}},{"id":"length","source_ids":["paper"],"text":{"en":"Read accuracy as a function of context length, including a short-context baseline. Advertised context capacity does not establish reliable retrieval at that length.","zh":"应结合短上下文参考成绩，观察准确率随文本长度的变化。宣称支持某个上下文长度，不等于在该长度上能够可靠检索。"},"title":{"en":"Compare the same context lengths","zh":"对齐上下文长度"}},{"id":"reproduce","source_ids":["project"],"text":{"en":"The authors release needles, haystacks and evaluation code. Use the repository to inspect prompts and the construction process.","zh":"作者公开目标事实、干扰长文本与评估代码；可在仓库核对提示词和数据构建流程。"},"title":{"en":"Original code and data","zh":"原始代码与数据"}},{"id":"identity","source_ids":["paper"],"text":{"en":"The collected community record has no paper link or run configuration. This is educational background, not confirmation that its measurements reproduce the original NoLiMa setup. No scores are transferred.","zh":"已采集的社区记录缺少论文链接与运行配置。本节仅提供研究背景，不能据此确认社区测量复现了原始 NoLiMa 配置，也不转移任何成绩。"},"title":{"en":"Community entry limitation","zh":"社区条目的限制"}}],"sources":[{"id":"paper","title":"NoLiMa: research background — original paper","url":"https://arxiv.org/abs/2502.05167v3"},{"id":"project","title":"NoLiMa: research background — authors’ project","url":"https://github.com/adobe-research/NoLiMa"}],"summary":{"en":"Background on NoLiMa’s long-context retrieval task; this community entry does not specify a verified protocol or version.","zh":"介绍 NoLiMa 的长上下文检索任务；当前社区条目没有可核实的具体协议与版本。"},"title":"NoLiMa: research background"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:community:2256e9c9-b256-4444-b639-7cc3b1855d96","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_00d121bee97ad2d2"},"modality":null,"name":"nolima","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":"higher_is_better","score_summary":{"display_max":null,"display_multiplier":1,"model_count":null,"model_count_basis":null,"numeric_count":0,"raw_max":null,"raw_min":null,"source_reference":null,"unit":null},"scored_models":[],"slug":"llm-stats-community-2256e9c9-b256-4444-b639-7cc3b1855d96","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A2256e9c9-b256-4444-b639-7cc3b1855d96?top_n=500","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":205,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"llm_stats_id","repository":null,"source_record":"catalog_4d097a7c6282c603"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"NoLiMa evaluated at a 131072-token context length. Tests latent associative reasoning in long contexts with minimal lexical overlap between questions and needles.","descriptions":{"en":"NoLiMa evaluated at a 131072-token context length. Tests latent associative reasoning in long contexts with minimal lexical overlap between questions and needles."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nolima-128k:minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nolima-128k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_4d097a7c6282c603"},"modality":"text","name":"NoLiMa 128K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":23.86,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.2386,"raw_min":0.2386,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nolima-128k:minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-128k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-nolima-128k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-128k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":205,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"llm_stats_id","repository":null,"source_record":"catalog_89acbf6e5d1b5b29"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"NoLiMa evaluated at a 32768-token context length. Tests latent associative reasoning in long contexts with minimal lexical overlap between questions and needles.","descriptions":{"en":"NoLiMa evaluated at a 32768-token context length. Tests latent associative reasoning in long contexts with minimal lexical overlap between questions and needles."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nolima-32k:minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-32k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nolima-32k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_89acbf6e5d1b5b29"},"modality":"text","name":"NoLiMa 32K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":54.54,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.5454,"raw_min":0.5454,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nolima-32k:minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-32k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-nolima-32k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-32k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":205,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"llm_stats_id","repository":null,"source_record":"catalog_7f755de6145eeb6c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"NoLiMa evaluated at a 65536-token context length. Tests latent associative reasoning in long contexts with minimal lexical overlap between questions and needles.","descriptions":{"en":"NoLiMa evaluated at a 65536-token context length. Tests latent associative reasoning in long contexts with minimal lexical overlap between questions and needles."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nolima-64k:minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-64k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nolima-64k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_7f755de6145eeb6c"},"modality":"text","name":"NoLiMa 64K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":42.95,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.4295,"raw_min":0.4295,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nolima-64k:minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-64k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-nolima-64k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-64k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":22,"join":"claire_record_id","repository":"gh:hae-rae/nolli","source_record":"bm_nolli_b5bb5859"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"NOLLI is a procedurally generated English-Korean puzzle benchmark with 15 puzzle types (25 tasks, 7,500 items). Each instance is seed-regenerable, verified to have a unique solution, and scored deterministically. Difficulty is calibrated behaviorally to target accuracy bands.","descriptions":{"en":"NOLLI is a procedurally generated English-Korean puzzle benchmark with 15 puzzle types (25 tasks, 7,500 items). Each instance is seed-regenerable, verified to have a unique solution, and scored deterministically. Difficulty is calibrated behaviorally to target accuracy bands."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04397","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","multilingual"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_nolli_b5bb5859"},"modality":null,"name":"NOLLI","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04397","source_url":"https://arxiv.org/abs/2608.04397"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04397","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04397","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_nora_3a8754af"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"NoRA is a visual first-person video benchmark requiring models to generate candidate next actions and justify them via fact-reason-action support graphs. It includes 1,420 annotated video clips and uses a grounded reasonableness score combining action alignment, factual grounding, and support binding.","descriptions":{"en":"NoRA is a visual first-person video benchmark requiring models to generate candidate next actions and justify them via fact-reason-action support graphs. It includes 1,420 annotated video clips and uses a grounded reasonableness score combining action alignment, factual grounding, and support binding."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04806","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","embodied-robotics"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_nora_3a8754af"},"modality":null,"name":"NoRA","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04806","source_url":"https://arxiv.org/abs/2606.04806"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04806","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04806","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_normact_ade65389"},"categories":["Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"NormAct evaluates embodied agents on 550 TongSim scenarios where the same goal permits norm-compliant or norm-violating action sequences, testing whether agents infer and apply scene-relevant social norms during ordinary tasks.","descriptions":{"en":"NormAct evaluates embodied agents on 550 TongSim scenarios where the same goal permits norm-compliant or norm-violating action sequences, testing whether agents infer and apply scene-relevant social norms during ordinary tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.27826","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","safety-alignment"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_normact_ade65389"},"modality":null,"name":"NormAct","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27826","source_url":"https://arxiv.org/abs/2606.27826"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27826","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27826","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"NormBench evaluates defeasible scope parsing in legal texts using Span-Grounded Deontic Trees, with 2,290 provisions across multiple languages. It focuses on identifying clause overrides and includes whole-tree fidelity metrics.","descriptions":{"en":"NormBench evaluates defeasible scope parsing in legal texts using Span-Grounded Deontic Trees, with 2,290 provisions across multiple languages. It focuses on identifying clause overrides and includes whole-tree fidelity metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.08932","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","multilingual"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_normbench_a8bf7215"},"modality":null,"name":"NormBench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08932","source_url":"https://arxiv.org/abs/2606.08932"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08932","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08932","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"NOVA-63 is a multilingual evaluation benchmark covering 63 languages, designed to assess LLM performance across diverse linguistic contexts and tasks.","descriptions":{"en":"NOVA-63 is a multilingual evaluation benchmark covering 63 languages, designed to assess LLM performance across diverse linguistic contexts and tasks."},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nova-63:qwen3.5-397b-a17b","reported_at":"2026-02-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nova-63?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nova-63","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_d98712704c496695"},"modality":"text","name":"NOVA-63","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":59.099999999999994,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.591,"raw_min":0.424,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nova-63:qwen3.5-397b-a17b","reported_date":"2026-02-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nova-63?top_n=500"},"unit":null},"scored_models":[{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-nova-63","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nova-63?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:maps-research/novelapibench","source_record":"bm_novelapibench_5fb13ed7"},"categories":["Agents","Agents & Tool Use","Factuality","Software & AI Compute","Tool use"],"collected_at":"2026-10-14T09:44:51Z","description":"NovelAPIBench is a dynamic benchmark for evaluating LLM tool use with novel APIs, covering knowledge components and diagnostic failure categories.","descriptions":{"en":"NovelAPIBench is a dynamic benchmark for evaluating LLM tool use with novel APIs, covering knowledge components and diagnostic failure categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03657","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_novelapibench_5fb13ed7"},"modality":null,"name":"NovelAPIBench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03657","source_url":"https://arxiv.org/abs/2606.03657"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03657","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03657","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Human-anchored assessment of LLM capabilities in diagnosing paper novelty.","descriptions":{"en":"Human-anchored assessment of LLM capabilities in diagnosing paper novelty."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11234","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_novgauge_0302ea09"},"modality":null,"name":"NovGauge","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11234","source_url":"https://arxiv.org/abs/2609.11234"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11234","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11234","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Nondeterministic Polynomial-time Problem Challenge (NPPC), an ever-scaling reasoning benchmark for LLMs. 非确定性多项式时间问题挑战 （NPPC），这是一个不断扩展的 LLM 推理基准。","descriptions":{"en":"Nondeterministic Polynomial-time Problem Challenge (NPPC), an ever-scaling reasoning benchmark for LLMs.","zh":"非确定性多项式时间问题挑战 （NPPC），这是一个不断扩展的 LLM 推理基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1783","languages":[],"library_direction":null,"modality":null,"name":"NPPC","openness":"unknown","publisher":"The Hong Kong Polytechnic University, Carnegie Mellon University，etc.","released":"2025-04-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1783-nppc","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NPPC","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1138,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:google-research-datasets/natural-questions","source_record":"catalog_af1a411ed6a2fdf0"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Natural Questions (NQ) benchmark containing real user questions issued to Google search with answers found from Wikipedia, designed for training and evaluation of automatic question answering systems","descriptions":{"en":"Natural Questions (NQ) benchmark containing real user questions issued to Google search with answers found from Wikipedia, designed for training and evaluation of automatic question answering systems"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nq:granite-3.3-8b-base","reported_at":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nq?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nq","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","deep-research"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_af1a411ed6a2fdf0"},"modality":"text","name":"NQ","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":36.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.365,"raw_min":0.365,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nq:granite-3.3-8b-base","reported_date":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nq?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"}],"slug":"llm-stats-nq","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nq?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1138,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"artifact:gh:google-research-datasets/natural-questions","repository":"gh:google-research-datasets/natural-questions","source_record":"catalog_af1a411ed6a2fdf0"},"categories":["知识","knowledge","大语言模型","LLM","知识储备","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"NQ (NaturalQuestion) corpus contains questions from real users, and it requires QA systems to read and comprehend an entire Wikipedia article that may or may not contain the answer to the question. The inclusion of real user questions, and the requirement that solutions should read an entire page to find the answer, cause NQ to be a more realistic and challenging task than prior QA datasets. NQ 数据集来自于真实用户的问题，它要求 QA 系统阅读和理解整个维基百科文章，这些文章可能包含也可能不包含问题的答案。由真实用户问题构成，以及需要阅读整个页面才能找到答案的要求，比以往的 QA 数据集更现实和更具挑战性的任务。","descriptions":{"en":"NQ (NaturalQuestion) corpus contains questions from real users, and it requires QA systems to read and comprehend an entire Wikipedia article that may or may not contain the answer to the question. The inclusion of real user questions, and the requirement that solutions should read an entire page to find the answer, cause NQ to be a more realistic and challenging task than prior QA datasets.","zh":"NQ 数据集来自于真实用户的问题，它要求 QA 系统阅读和理解整个维基百科文章，这些文章可能包含也可能不包含问题的答案。由真实用户问题构成，以及需要阅读整个页面才能找到答案的要求，比以往的 QA 数据集更现实和更具挑战性的任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:513","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","deep-research"],"join":"artifact:gh:google-research-datasets/natural-questions","primary":"factuality","source_record":"catalog_af1a411ed6a2fdf0"},"modality":null,"name":"NQ","openness":"unknown","publisher":null,"released":"2019-06-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-513-nq","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NQ","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:zbwpro/onepred","source_record":"bm_nqp-bench_0c5c1e38"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"NQP-Bench is a dataset within the OnePred paper for evaluating next-query prediction in multi-turn conversations. It spans three diverse subsets and is used to compare OnePred against baselines.","descriptions":{"en":"NQP-Bench is a dataset within the OnePred paper for evaluating next-query prediction in multi-turn conversations. It spans three diverse subsets and is used to compare OnePred against baselines."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.23668","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_nqp-bench_0c5c1e38"},"modality":null,"name":"NQP-Bench","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23668","source_url":"https://arxiv.org/abs/2605.23668"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23668","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23668","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:niladrighosh03/nrityam","source_record":"bm_nrityam_dcb4ad43"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A dataset of 9,260 question-answer pairs across 12 languages evaluating cultural knowledge in global dance traditions.","descriptions":{"en":"A dataset of 9,260 question-answer pairs across 12 languages evaluating cultural knowledge in global dance traditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.19727","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_nrityam_dcb4ad43"},"modality":null,"name":"NRITYAM","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19727","source_url":"https://arxiv.org/abs/2606.19727"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19727","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19727","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_nrt-bench_4404866b"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-turn red-teaming of LLM agents acting as operators of a simulated nuclear power plant control room. Safety is measured by objective loss of critical safety functions (CSFs) under adaptive attacks.","descriptions":{"en":"Evaluates multi-turn red-teaming of LLM agents acting as operators of a simulated nuclear power plant control room. Safety is measured by objective loss of critical safety functions (CSFs) under adaptive attacks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.20408","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","safety-alignment"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_nrt-bench_4404866b"},"modality":null,"name":"NRT-Bench","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20408","source_url":"https://arxiv.org/abs/2606.20408"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20408","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20408","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"NuclearQAv2 evaluates LLMs on nuclear engineering knowledge with approximately 1,240 QA pairs covering boolean, numeric, and verbal questions. It uses structured prompting for automated question generation and response evaluation.","descriptions":{"en":"NuclearQAv2 evaluates LLMs on nuclear engineering knowledge with approximately 1,240 QA pairs covering boolean, numeric, and verbal questions. It uses structured prompting for automated question generation and response evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.27047","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_nuclearqav2_79073391"},"modality":null,"name":"NuclearQAv2","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27047","source_url":"https://arxiv.org/abs/2606.27047"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27047","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27047","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates counting accuracy in text-to-image models using 640,000 prompts across 1,600 categories and counts from 1 to 100, with factorial variation of composition, layout, and appearance.","descriptions":{"en":"Evaluates counting accuracy in text-to-image models using 640,000 prompts across 1,600 categories and counts from 1 to 100, with factorial variation of composition, layout, and appearance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.28206","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","spatial-3d"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_numbench_cf80a48e"},"modality":null,"name":"NumBench","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28206","source_url":"http://arxiv.org/abs/2608.28206v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28206","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28206v1","topics":[],"unit":null},{"aliases":["NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":45,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:fuy3/numerosityvlm-benchmark","source_record":"bm_numerosityvlm-a-cognitively-inspired-bench_6f2495fb"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates zero-shot numerosity perception in vision-language models on 10,800 synthetic images across six controlled conditions manipulating object size, spatial arrangement, and numerosity while ablating texture, shape, and color.","descriptions":{"en":"Evaluates zero-shot numerosity perception in vision-language models on 10,800 synthetic images across six controlled conditions manipulating object size, spatial arrangement, and numerosity while ablating texture, shape, and color."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.15425","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_numerosityvlm-a-cognitively-inspired-bench_6f2495fb"},"modality":null,"name":"NumerosityVLM","openness":"unknown","publisher":null,"released":"2026-08-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15425","source_url":"https://arxiv.org/abs/2608.15425"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15425","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15425","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","spatial","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multimodal benchmark for scene understanding and reasoning over the nuScenes autonomous driving domain.","descriptions":{"en":"A multimodal benchmark for scene understanding and reasoning over the nuScenes autonomous driving domain."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:nuscene:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nuscene?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:nuscene","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_01ebb251e2f68e0e"},"modality":"multimodal","name":"Nuscene","openness":"unknown","publisher":null,"released":"2019-03-26","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: nuScenes: A multimodal dataset for autonomous driving. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:nuscene","source_url":"https://arxiv.org/abs/1903.11027"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":15.4,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.154,"raw_min":0.146,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:nuscene:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nuscene?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-nuscene","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nuscene?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:v1ol1n/nutribench-kitchen","source_record":"bm_nutribench-kitchen_2a4c320b"},"categories":["Robotics & Embodied AI","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmarks embodied agents on nutrition management via QA pairs from cooking videos across five task families.","descriptions":{"en":"Benchmarks embodied agents on nutrition management via QA pairs from cooking videos across five task families."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.07135","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_nutribench-kitchen_2a4c320b"},"modality":null,"name":"NutriBench-Kitchen","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07135","source_url":"https://arxiv.org/abs/2609.07135"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07135","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07135","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"CoSyn-400K dataset contains 9 categories of synthetc text-rich images with 2.7M instruction-tuning data 通过代码引导的合成多模态数据生成扩展文本丰富图像理解，CoSyn-400K 数据集包含 9 类合成文本丰富图像，以及 270 万条指令微调数据。","descriptions":{"en":"CoSyn-400K dataset contains 9 categories of synthetc text-rich images with 2.7M instruction-tuning data","zh":"通过代码引导的合成多模态数据生成扩展文本丰富图像理解，CoSyn-400K 数据集包含 9 类合成文本丰富图像，以及 270 万条指令微调数据。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1533","languages":[],"library_direction":null,"modality":"multimodal","name":"NutritionQA","openness":"unknown","publisher":"University of Pennsylvania,   Allen Institute for Artificial Intelligence","released":"2025-02-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1533-nutritionqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NutritionQA","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["newt-bench"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:vsbdev/little-hermione","source_record":"bm_newt-bench_b1c66437"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Deliberately contaminated LoRA experiment with a 75-question Harry Potter evaluation set and training code that reproduces the leakage demonstration.","descriptions":{"en":"Deliberately contaminated LoRA experiment with a 75-question Harry Potter evaluation set and training code that reproduces the leakage demonstration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:vsbdev/little-hermione","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_newt-bench_b1c66437"},"modality":null,"name":"O.W.L. & N.E.W.T. Bench","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:vsbdev/little-hermione","source_url":"https://github.com/VSBDev/little-hermione"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-vsbdev-little-hermione","source":"claire_radar","source_url":"https://github.com/VSBDev/little-hermione","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"OAO-AttackBench is a set of counterfactual prompts for one-and-only objects, used to evaluate text-to-image alignment in a specific study.","descriptions":{"en":"OAO-AttackBench is a set of counterfactual prompts for one-and-only objects, used to evaluate text-to-image alignment in a specific study."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.30262","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_oao-attackbench_8518a31c"},"modality":null,"name":"OAO-AttackBench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30262","source_url":"https://arxiv.org/abs/2606.30262"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30262","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30262","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:xiangjun2003/obgynlongbench","source_record":"bm_obgynlongbench_eaf80816"},"categories":["Biology & Drug Discovery","Language & Knowledge","Long Context"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-context EHR decision-making for obstetrics and gynecology using 1,500 rule-grounded cases across Evidence-only, Visit-level, and History-level settings.","descriptions":{"en":"Evaluates long-context EHR decision-making for obstetrics and gynecology using 1,500 rule-grounded cases across Evidence-only, Visit-level, and History-level settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.07601","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","long-context"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_obgynlongbench_eaf80816"},"modality":null,"name":"ObGynLongBench","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07601","source_url":"https://arxiv.org/abs/2609.07601"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07601","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07601","topics":["long-context","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Introduces two benchmark concepts for uncertainty-aware object detection and label error detection.","descriptions":{"en":"Introduces two benchmark concepts for uncertainty-aware object detection and label error detection."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.21822","languages":[],"library_direction":null,"modality":null,"name":"Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21822","source_url":"https://arxiv.org/abs/2609.21822"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21822","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21822","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2358,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:google-research-datasets/objectron","source_record":"catalog_96c6992e4a7c304c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["3d","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Objectron evaluates 3D object detection and pose estimation capabilities.","descriptions":{"en":"Objectron evaluates 3D object detection and pose estimation capabilities."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:objectron:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/objectron?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:objectron","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_96c6992e4a7c304c"},"modality":"image","name":"Objectron","openness":"unknown","publisher":null,"released":"2020-12-18","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:objectron","source_url":"https://arxiv.org/abs/2012.09988"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.712,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.712,"raw_min":0.712,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:objectron:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/objectron?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-objectron","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/objectron?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_oblivion_83c0aaa0"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates operational skill unlearning in deployed agents across 88 attack episodes, measuring attack success rate and impact-weighted exposure after workflow-level defenses.","descriptions":{"en":"Evaluates operational skill unlearning in deployed agents across 88 attack episodes, measuring attack success rate and impact-weighted exposure after workflow-level defenses."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08264","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agent-skills-evolution"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_oblivion_83c0aaa0"},"modality":null,"name":"OBLIVION","openness":"unknown","publisher":null,"released":"2026-08-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08264","source_url":"https://arxiv.org/abs/2608.08264"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08264","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08264","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:russellyq/obsdrivebench","source_record":"bm_obsdrivebench_13a5d40b"},"categories":["Autonomous Driving","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"ObsDriveBench evaluates multimodal understanding in autonomous driving under adverse weather, covering observability awareness, spatial reliability, and risk-aware decision-making with multiple-choice tasks over camera, LiDAR, and radar inputs.","descriptions":{"en":"ObsDriveBench evaluates multimodal understanding in autonomous driving under adverse weather, covering observability awareness, spatial reliability, and risk-aware decision-making with multiple-choice tasks over camera, LiDAR, and radar inputs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.23537","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","visual-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_obsdrivebench_13a5d40b"},"modality":null,"name":"ObsDriveBench","openness":"unknown","publisher":null,"released":"2026-07-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23537","source_url":"https://arxiv.org/abs/2607.23537"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23537","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23537","topics":["multimodal"],"unit":null},{"aliases":["ObserverBench: Testing Mechanistic Estimates for Intervention and Control"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_observerbench-testing-mechanistic-estimate_c0fcde73"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark framework that tests whether mechanistic interpretability observers are adequate for intervention, control, or safety tasks by reporting estimation accuracy and action loss separately.","descriptions":{"en":"Benchmark framework that tests whether mechanistic interpretability observers are adequate for intervention, control, or safety tasks by reporting estimation accuracy and action loss separately."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.03026","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_observerbench-testing-mechanistic-estimate_c0fcde73"},"modality":null,"name":"ObserverBench","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03026","source_url":"https://arxiv.org/abs/2609.03026"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03026","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03026","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Obshazard-bench is a real-time benchmark for disaster intelligence, integrating raw satellite and ground-station data. It covers 8 disaster categories and 28 sub-categories across 60+ countries, with VQA samples and a three-stage evaluation taxonomy: Predictive Crisis Anticipation, Active Evolution Reasoning, and Multi-faceted Impact Quantification.","descriptions":{"en":"Obshazard-bench is a real-time benchmark for disaster intelligence, integrating raw satellite and ground-station data. It covers 8 disaster categories and 28 sub-categories across 60+ countries, with VQA samples and a three-stage evaluation taxonomy: Predictive Crisis Anticipation, Active Evolution Reasoning, and Multi-faceted Impact Quantification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00012","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_obshazard-bench_aa08033a"},"modality":null,"name":"Obshazard-bench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00012","source_url":"https://arxiv.org/abs/2608.00012"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00012","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00012","topics":["multimodal","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OCCUR-Bench evaluates temporal preservation in conversational image editing, providing occlusion-and-revelation scenarios with historical restoration references to assess faithful restoration of occluded content.","descriptions":{"en":"OCCUR-Bench evaluates temporal preservation in conversational image editing, providing occlusion-and-revelation scenarios with historical restoration references to assess faithful restoration of occluded content."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.07051","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_occur-bench_c8c0341e"},"modality":null,"name":"OCCUR-Bench","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.07051","source_url":"https://arxiv.org/abs/2607.07051"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-07051","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07051","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OCNLI is a Chinese natural language inference task, which requires to determine the logical relation between two sentences, with three relations: entailment, contradiction and neutral. OCNLI是一个中文自然语言推理任务，要求根据两个句子判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。","descriptions":{"en":"OCNLI is a Chinese natural language inference task, which requires to determine the logical relation between two sentences, with three relations: entailment, contradiction and neutral.","zh":"OCNLI是一个中文自然语言推理任务，要求根据两个句子判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:525","languages":["Chinese"],"library_direction":null,"modality":null,"name":"OCNLI","openness":"unknown","publisher":null,"released":"2020-10-12","released_reference":{"basis":"paper_first_version","note":"First version introducing Original Chinese Natural Language Inference.","source_key":"opencompass:525","source_url":"https://arxiv.org/abs/2010.05444"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-525-ocnli","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OCNLI","topics":["reasoning-math"],"unit":null},{"aliases":["OCR-MetaReasoning Benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:gengxuli/ocr-metareasoning","source_record":"bm_ocr-metareasoning-benchmark_8b06e709"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates MLLMs on text-rich image understanding by separating deduction, induction, and abduction across 1,500 samples with automatic answer scoring and reasoning-process compliance metrics.","descriptions":{"en":"Evaluates MLLMs on text-rich image understanding by separating deduction, induction, and abduction across 1,500 samples with automatic answer scoring and reasoning-process compliance metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.30678","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","visual-reasoning"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_ocr-metareasoning-benchmark_8b06e709"},"modality":null,"name":"OCR-MetaReasoning","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30678","source_url":"https://arxiv.org/abs/2608.30678"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30678","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30678","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":897,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:yuliang-liu/multimodalocr","source_record":"catalog_917413cfda40d9db"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["image_to_text","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OCRBench: Comprehensive evaluation benchmark for assessing Optical Character Recognition (OCR) capabilities in Large Multimodal Models across text recognition, scene text VQA, and document understanding tasks","descriptions":{"en":"OCRBench: Comprehensive evaluation benchmark for assessing Optical Character Recognition (OCR) capabilities in Large Multimodal Models across text recognition, scene text VQA, and document understanding tasks"},"evidence_summary":{"document_count":1,"model_count":24,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ocrbench:qwen2-vl-72b","reported_at":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:ocrbench","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_917413cfda40d9db"},"modality":"multimodal","name":"OCRBench","openness":"unknown","publisher":null,"released":"2024-01-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":24,"score_direction":"higher_is_better","score_summary":{"display_max":92.30000000000001,"display_multiplier":100,"model_count":24,"model_count_basis":"source_model_id","numeric_count":24,"raw_max":0.923,"raw_min":0.792,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ocrbench:kimi-k2.5","reported_date":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"}],"slug":"llm-stats-ocrbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":897,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:2305.07895","repository":"gh:yuliang-liu/multimodalocr","source_record":"catalog_917413cfda40d9db"},"categories":["语言","Language","理解","Understanding","多模态模型","VLM","语言理解","Comprehension","图像理解","Image Understanding","OCR与文档理解","OCR and Document Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OCRBench provides a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks including Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER). OCRBench对 GPT4V 和 Gemini 等大型多模态模型在各种文本相关的视觉任务中的表现进行了全面的评估，包括文本识别、场景文本为中心的视觉问答 (VQA)、面向文档的 VQA、关键信息提取 (KIE) 和手写数学表达式识别 (HMER)。","descriptions":{"en":"OCRBench provides a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks including Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER).","zh":"OCRBench对 GPT4V 和 Gemini 等大型多模态模型在各种文本相关的视觉任务中的表现进行了全面的评估，包括文本识别、场景文本为中心的视觉问答 (VQA)、面向文档的 VQA、关键信息提取 (KIE) 和手写数学表达式识别 (HMER)。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:557","languages":["Multilingual"],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"artifact:arxiv:2305.07895","primary":"document-ocr","source_record":"catalog_917413cfda40d9db"},"modality":null,"name":"OCRBench","openness":"unknown","publisher":null,"released":"2024-01-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-557-ocrbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OCRBench","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_3a2efe4fa70bff40"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["image_to_text","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OCRBench v2 English subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with English text content","descriptions":{"en":"OCRBench v2 English subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with English text content"},"evidence_summary":{"document_count":1,"model_count":14,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ocrbench-v2-(en):qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28en%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ocrbench-v2-(en)","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_3a2efe4fa70bff40"},"modality":"multimodal","name":"OCRBench-V2 (en)","openness":"unknown","publication":{"evidence":[{"note":"The paper author's repository links arXiv:2501.00321 and places v2 in its own directory. It documents EN_part and CN_part data, plus separate English and Chinese leaderboard sections, directly matching the two exact language records.","url":"https://github.com/Yuliang-Liu/MultimodalOCR/blob/main/OCRBench_v2/README.md"},{"note":"The accessible paper introduces a bilingual visual-text localization and reasoning benchmark with 10,000 human-verified question-answer pairs and distinguishes it from the earlier OCRBench.","url":"https://arxiv.org/abs/2501.00321"}],"id":"ocrbench-v2-language-subsets-original","links":[{"kind":"paper","title":"OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning","url":"https://arxiv.org/abs/2501.00321"},{"kind":"repo","title":"OCRBench v2 official repository directory","url":"https://github.com/Yuliang-Liu/MultimodalOCR/tree/main/OCRBench_v2"}],"record_keys":["llm-stats:ocrbench-v2-(en)","llm-stats:ocrbench-v2-(zh)"],"reviewed_at":"2026-10-26","scope":{"en":"Original OCRBench v2 paper and repository for the English (en) and Chinese (zh) subsets. The two language records retain separate results; neither is OCRBench v1 or an inferred bilingual aggregate.","zh":"OCRBench v2 英文（en）与中文（zh）子集共同对应的原论文和仓库。两个语言记录分别保留成绩，不等同于 OCRBench v1，也不推算双语汇总成绩。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":14,"score_direction":"higher_is_better","score_summary":{"display_max":68.4,"display_multiplier":100,"model_count":14,"model_count_basis":"source_model_id","numeric_count":14,"raw_max":0.684,"raw_min":0.367,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ocrbench-v2-(en):qwen3-vl-32b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28en%29?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-ocrbench-v2-en","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28en%29?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_058b7797debf92d2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["image_to_text","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OCRBench v2 Chinese subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with Chinese text content","descriptions":{"en":"OCRBench v2 Chinese subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with Chinese text content"},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ocrbench-v2-(zh):qwen2.5-vl-32b","reported_at":"2025-02-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28zh%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ocrbench-v2-(zh)","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","multilingual"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_058b7797debf92d2"},"modality":"multimodal","name":"OCRBench-V2 (zh)","openness":"unknown","publication":{"evidence":[{"note":"The paper author's repository links arXiv:2501.00321 and places v2 in its own directory. It documents EN_part and CN_part data, plus separate English and Chinese leaderboard sections, directly matching the two exact language records.","url":"https://github.com/Yuliang-Liu/MultimodalOCR/blob/main/OCRBench_v2/README.md"},{"note":"The accessible paper introduces a bilingual visual-text localization and reasoning benchmark with 10,000 human-verified question-answer pairs and distinguishes it from the earlier OCRBench.","url":"https://arxiv.org/abs/2501.00321"}],"id":"ocrbench-v2-language-subsets-original","links":[{"kind":"paper","title":"OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning","url":"https://arxiv.org/abs/2501.00321"},{"kind":"repo","title":"OCRBench v2 official repository directory","url":"https://github.com/Yuliang-Liu/MultimodalOCR/tree/main/OCRBench_v2"}],"record_keys":["llm-stats:ocrbench-v2-(en)","llm-stats:ocrbench-v2-(zh)"],"reviewed_at":"2026-10-26","scope":{"en":"Original OCRBench v2 paper and repository for the English (en) and Chinese (zh) subsets. The two language records retain separate results; neither is OCRBench v1 or an inferred bilingual aggregate.","zh":"OCRBench v2 英文（en）与中文（zh）子集共同对应的原论文和仓库。两个语言记录分别保留成绩，不等同于 OCRBench v1，也不推算双语汇总成绩。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":63.5,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.635,"raw_min":0.558,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ocrbench-v2-(zh):qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28zh%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-ocrbench-v2-zh","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28zh%29?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_382d6086e1c5a6a9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["image_to_text","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OCRBench v2: Enhanced large-scale bilingual benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with 10,000 human-verified question-answering pairs across 8 core OCR capabilities","descriptions":{"en":"OCRBench v2: Enhanced large-scale bilingual benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with 10,000 human-verified question-answering pairs across 8 core OCR capabilities"},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ocrbench-v2:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ocrbench-v2","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_382d6086e1c5a6a9"},"modality":"multimodal","name":"OCRBench_V2","openness":"unknown","publisher":null,"released":"2024-12-31","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:ocrbench-v2","source_url":"https://arxiv.org/abs/2501.00321"},"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":67.10000000000001,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.671,"raw_min":0.561,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ocrbench-v2:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-ocrbench-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":223,"github_stars_30d":28,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:baochenfu/oct-bench","source_record":"bm_oct-bench_3914e5be"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"OCT-Bench evaluates multimodal large language models on OCT image understanding with 10,076 multiple-choice questions from 4,137 images across seven public datasets, covering 20 tasks in Perception, Cognition, and Reasoning.","descriptions":{"en":"OCT-Bench evaluates multimodal large language models on OCT image understanding with 10,076 multiple-choice questions from 4,137 images across seven public datasets, covering 20 tasks in Perception, Cognition, and Reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.16609","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_oct-bench_3914e5be"},"modality":null,"name":"OCT-Bench","openness":"unknown","publisher":null,"released":"2026-07-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.16609","source_url":"https://arxiv.org/abs/2607.16609"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-16609","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16609","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":40,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:minimax-ai/mini-vela","source_record":"catalog_e4a8cc0a8b5ac334"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Octopus coding benchmark for evaluating multi-language programming capabilities","descriptions":{"en":"Octopus coding benchmark for evaluating multi-language programming capabilities"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:octocodingbench:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/octocodingbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:octocodingbench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","instruction-following"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_e4a8cc0a8b5ac334"},"modality":"text","name":"OctoCodingBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":26.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.261,"raw_min":0.261,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:octocodingbench:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/octocodingbench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-octocodingbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/octocodingbench?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Mobile & Personal Computing","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Eyeglass reflection removal is important across smartphone imaging, video conferencing, and other face-centric visual applications.","descriptions":{"en":"Eyeglass reflection removal is important across smartphone imaging, video conferencing, and other face-centric visual applications."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10703","languages":[],"library_direction":{"basis":"model-review-daily","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_ocubench_8fe2b92b"},"modality":null,"name":"OcuBench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10703","source_url":"https://arxiv.org/abs/2610.10703"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10703","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10703","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Code generation","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"OdinEval is a benchmark for program repair in the Odin programming language, built from documented defects, with issue-to-commit bindings, regression tests, and execution records.","descriptions":{"en":"OdinEval is a benchmark for program repair in the Odin programming language, built from documented defects, with issue-to-commit bindings, regression tests, and execution records."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.18595","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_odineval_a7103a7e"},"modality":null,"name":"OdinEval","openness":"unknown","publisher":null,"released":"2026-08-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.18595","source_url":"https://arxiv.org/abs/2608.18595"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-18595","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18595","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":2611,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:microsoft/glip","source_record":"catalog_8cd96084e50818fb"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Object Detection in the Wild (ODinW) benchmark for evaluating object detection models' task-level transfer ability across diverse real-world datasets in terms of prediction accuracy and adaptation efficiency","descriptions":{"en":"Object Detection in the Wild (ODinW) benchmark for evaluating object detection models' task-level transfer ability across diverse real-world datasets in terms of prediction accuracy and adaptation efficiency"},"evidence_summary":{"document_count":1,"model_count":16,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:odinw:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/odinw?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:odinw","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_8cd96084e50818fb"},"modality":"image","name":"ODinW","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":16,"score_direction":"higher_is_better","score_summary":{"display_max":51.800000000000004,"display_multiplier":100,"model_count":16,"model_count_basis":"source_model_id","numeric_count":16,"raw_max":0.518,"raw_min":0.394,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:odinw:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/odinw?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-odinw","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/odinw?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":16,"github_stars_30d":6,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:epoch-research/leanopenproblems","source_record":"bm_oeis-open_86208c3e"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates language models on formalized open mathematical conjectures from OEIS, scored by theorem resolution under a compute budget.","descriptions":{"en":"Evaluates language models on formalized open mathematical conjectures from OEIS, scored by theorem resolution under a compute budget."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11941","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_oeis-open_86208c3e"},"modality":null,"name":"OEIS Open","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11941","source_url":"https://arxiv.org/abs/2608.11941"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11941","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11941","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_oenobench_8cfa7d28"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM knowledge in the wine domain via 3,266 multiple-choice questions across six pillars and four difficulty tiers, sourced from verified facts and scored against a calibrated audit.","descriptions":{"en":"Evaluates LLM knowledge in the wine domain via 3,266 multiple-choice questions across six pillars and four difficulty tiers, sourced from verified facts and scored against a calibrated audit."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.20106","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_oenobench_8cfa7d28"},"modality":null,"name":"OenoBench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.20106","source_url":"https://arxiv.org/abs/2608.20106"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-20106","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20106","topics":[],"unit":null},{"aliases":["office-agent-benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:basilos-ai/office-agent-benchmark","source_record":"bm_office-agent-benchmark_5b124db7"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates office agents on documents, spreadsheets, presentations, daily workflows, instruction following, and safety constraints through executable scoring contracts.","descriptions":{"en":"Evaluates office agents on documents, spreadsheets, presentations, daily workflows, instruction following, and safety constraints through executable scoring contracts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:basilos-ai/office-agent-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_office-agent-benchmark_5b124db7"},"modality":null,"name":"Office Agent Benchmark","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:basilos-ai/office-agent-benchmark","source_url":"https://github.com/basilos-ai/office-agent-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-basilos-ai-office-agent-benchmark","source":"claire_radar","source_url":"https://github.com/basilos-ai/office-agent-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A two-track benchmark evaluating structural, visual, and domain-grounded comprehension of native Word, Excel, and PowerPoint files.","descriptions":{"en":"A two-track benchmark evaluating structural, visual, and domain-grounded comprehension of native Word, Excel, and PowerPoint files."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01245","languages":[],"library_direction":null,"modality":null,"name":"Office Comprehension Benchmark","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01245","source_url":"https://arxiv.org/abs/2607.01245"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01245","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01245","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aniriswu/officeeditbench","source_record":"bm_officeeditbench_e8bffe3e"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"A small Office edit creates two obligations: propagate every required update and leave protected state untouched.","descriptions":{"en":"A small Office edit creates two obligations: propagate every required update and leave protected state untouched."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.34469","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_officeeditbench_e8bffe3e"},"modality":null,"name":"OfficeEditBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34469","source_url":"https://arxiv.org/abs/2609.34469"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34469","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34469","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":203,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_1b5ee4336ea31196"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OfficeQA Pro evaluates AI models on professional knowledge-work questions and tasks drawn from real office workflows, including document analysis, spreadsheet reasoning, and information synthesis across business domains.","descriptions":{"en":"OfficeQA Pro evaluates AI models on professional knowledge-work questions and tasks drawn from real office workflows, including document analysis, spreadsheet reasoning, and information synthesis across business domains."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:officeqa-pro:gpt-5.5","reported_at":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/officeqa-pro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:officeqa-pro","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_1b5ee4336ea31196"},"modality":"text","name":"OfficeQA Pro","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":72.2,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.722,"raw_min":0.451,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:officeqa-pro:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/officeqa-pro?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-officeqa-pro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/officeqa-pro?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["OfficeQA Pro","OfficeQA"],"attention_signals":null,"categories":["vision"],"collected_at":null,"description":"Visual QA benchmark on professional document corpus (PDF without embedded text). Scores depend on context length and image resolution.","descriptions":{"en":"Visual QA benchmark on professional document corpus (PDF without embedded text). Scores depend on context length and image resolution."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000office_qa_pro\u0000zai_glm_5_3_flash_model_card\u0000office_qa_pro\u0000Treasury Bulletin PDF corpus without embedded text access, temp 1.0, top_p 0.95, max context 512K\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000office_qa_pro\u0000zai_glm_5_3_flash_model_card\u0000office_qa_pro\u0000Treasury Bulletin PDF corpus without embedded text access, temp 1.0, top_p 0.95, max context 512K\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:office_qa_pro","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["document-ocr"],"join":"review:14H034160212:2026-10-15","primary":"document-ocr","source_record":null},"modality":null,"name":"OfficeQA Pro","openness":"unknown","publisher":null,"released":"2025-06-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:office_qa_pro","source_url":"https://github.com/OfficeQA/OfficeQA-Pro"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":66.2,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":66.2,"raw_min":62.4,"source_reference":{"obs_id":"curated\u0000office_qa_pro\u0000tencent_hy4_preview\u0000office_qa_pro\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000office_qa_pro\u0000tencent_hy4_preview\u0000office_qa_pro\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"office_qa_pro","source":"model_reports","source_url":"https://github.com/OfficeQA/OfficeQA-Pro","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OffNadirLoc evaluates UAV-to-satellite geo-localization under large off-nadir views, with structure-aware contextual weighting and view-coherent learning for viewpoint-invariant features.","descriptions":{"en":"OffNadirLoc evaluates UAV-to-satellite geo-localization under large off-nadir views, with structure-aware contextual weighting and view-coherent learning for viewpoint-invariant features."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19951","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_offnadirloc_b7680d27"},"modality":null,"name":"OffNadirLoc","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19951","source_url":"https://arxiv.org/abs/2607.19951"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19951","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19951","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":34,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_d3a0b0c6ea1d490b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OJBench is a competition-level code benchmark designed to assess the competitive-level code reasoning abilities of large language models. It comprises 232 programming competition problems from NOI and ICPC, categorized into Easy, Medium, and Hard difficulty levels. The benchmark evaluates models' ability to solve complex competitive programming challenges using Python and C++.","descriptions":{"en":"OJBench is a competition-level code benchmark designed to assess the competitive-level code reasoning abilities of large language models. It comprises 232 programming competition problems from NOI and ICPC, categorized into Easy, Medium, and Hard difficulty levels. The benchmark evaluates models' ability to solve complex competitive programming challenges using Python and C++."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ojbench:kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ojbench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_d3a0b0c6ea1d490b"},"modality":"text","name":"OJBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":60.6,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.606,"raw_min":0.271,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ojbench:kimi-k2.6","reported_date":"2026-04-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-ojbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":34,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_555e52f92861743e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OJBench (C++) is the C++ subset of OJBench, a competition-level code benchmark that evaluates large language models on programming competition problems from NOI and ICPC using C++ as the implementation language.","descriptions":{"en":"OJBench (C++) is the C++ subset of OJBench, a competition-level code benchmark that evaluates large language models on programming competition problems from NOI and ICPC using C++ as the implementation language."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ojbench-cpp:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench-cpp?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ojbench-cpp","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_555e52f92861743e"},"modality":"text","name":"OJBench (C++)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":57.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.574,"raw_min":0.574,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ojbench-cpp:kimi-k2.5","reported_date":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench-cpp?top_n=500"},"unit":null},"scored_models":[{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"}],"slug":"llm-stats-ojbench-cpp","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench-cpp?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus.","descriptions":{"en":"Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.02986","languages":[],"library_direction":{"basis":"model-review-daily","categories":["content-detection","safety-alignment"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_olmo-detect_bf7d331e"},"modality":null,"name":"OLMo-Detect","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.02986","source_url":"https://arxiv.org/abs/2610.02986"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-02986","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02986","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","推理","Reasoning","数学","Math","大语言模型","LLM","逻辑推理","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A benchmark of 200 Olympiad math problems across algebra, geometry, number theory, and combinatorics. Available in English and Chinese, it features two difficulty levels: EASY (AIME-level) to test standard reasoning, and HARD to challenge advanced models. 一个包含 200 道奥林匹克数学题的基准测试，涵盖代数、几何、数论和组合。我们提供英文和中文版本，并提供两个难度等级：EASY（AIME水平）用于测试标准推理能力，以及 HARD 用于挑战高级模型。","descriptions":{"en":"A benchmark of 200 Olympiad math problems across algebra, geometry, number theory, and combinatorics. Available in English and Chinese, it features two difficulty levels: EASY (AIME-level) to test standard reasoning, and HARD to challenge advanced models.","zh":"一个包含 200 道奥林匹克数学题的基准测试，涵盖代数、几何、数论和组合。我们提供英文和中文版本，并提供两个难度等级：EASY（AIME水平）用于测试标准推理能力，以及 HARD 用于挑战高级模型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1705","languages":["English","Chinese","Multilingual"],"library_direction":null,"modality":null,"name":"OlymMATH","openness":"open","publisher":"中国人民大学","released":"2025-03-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1705-olymmath","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OlymMATH","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":199,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_4af0fe63f717ff2f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","physics","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A challenging benchmark for promoting AGI with Olympiad-level bilingual multimodal scientific problems. Comprises 8,476 math and physics problems from international and Chinese Olympiads and the Chinese college entrance exam, featuring expert-level annotations for step-by-step reasoning. Includes both text-only and multimodal problems in English and Chinese.","descriptions":{"en":"A challenging benchmark for promoting AGI with Olympiad-level bilingual multimodal scientific problems. Comprises 8,476 math and physics problems from international and Chinese Olympiads and the Chinese college entrance exam, featuring expert-level annotations for step-by-step reasoning. Includes both text-only and multimodal problems in English and Chinese."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:olympiadbench:qvq-72b-preview","reported_at":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/olympiadbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:olympiadbench","languages":[],"library_direction":{"basis":"model-review","categories":["math","science","visual-reasoning"],"join":"llm_stats_id","primary":"math","source_record":"catalog_4af0fe63f717ff2f"},"modality":"multimodal","name":"OlympiadBench","openness":"unknown","publisher":null,"released":"2024-02-21","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:olympiadbench","source_url":"https://arxiv.org/abs/2402.14008"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":20.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.204,"raw_min":0.204,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:olympiadbench:qvq-72b-preview","reported_date":"2024-12-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/olympiadbench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-25","date_precision":"model_announcement","name":"QvQ-72B-Preview"}],"slug":"llm-stats-olympiadbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/olympiadbench?top_n=500","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":199,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"artifact:arxiv:2402.14008","repository":null,"source_record":"catalog_4af0fe63f717ff2f"},"categories":["学科","Examination","ACL 2024","大语言模型","LLM","知识储备","Knowledge"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring\n8,476 problems from Olympiad-level mathematics and physics competitions, including the\nChinese college entrance exam. Each problem is detailed with expert-level annotations\nfor step-by-step reasoning. OlympiadBench 是一个奥林匹克级别的双语多模态科学基准，包含来自奥林匹克级数学和物理竞赛的8,476道题目，包括中国高考。每道题目都配有专家级别的注释，提供逐步推理的详细说明。","descriptions":{"en":"OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring\n8,476 problems from Olympiad-level mathematics and physics competitions, including the\nChinese college entrance exam. Each problem is detailed with expert-level annotations\nfor step-by-step reasoning.","zh":"OlympiadBench 是一个奥林匹克级别的双语多模态科学基准，包含来自奥林匹克级数学和物理竞赛的8,476道题目，包括中国高考。每道题目都配有专家级别的注释，提供逐步推理的详细说明。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1070","languages":["Chinese"],"library_direction":{"basis":"model-review","categories":["math","science","visual-reasoning"],"join":"artifact:arxiv:2402.14008","primary":"math","source_record":"catalog_4af0fe63f717ff2f"},"modality":null,"name":"OlympiadBench","openness":"open","publisher":"OpenBMB","released":"2024-06-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1070-olympiadbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OlympiadBench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OlympicArena evaluates cognitive reasoning abilities. It includes 11,163 bilingual problems spanning seven fields and 62 international Olympic competitions. OlympicArena用于评估大模型的认知推理能力，包含来自7个领域、62项国际奥林匹克比赛的11163个双语问题。","descriptions":{"en":"OlympicArena evaluates cognitive reasoning abilities. It includes 11,163 bilingual problems spanning seven fields and 62 international Olympic competitions.","zh":"OlympicArena用于评估大模型的认知推理能力，包含来自7个领域、62项国际奥林匹克比赛的11163个双语问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1329","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"OlympicArena","openness":"restricted","publisher":"Generative AI Research Lab","released":"2024-06-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1329-olympicarena","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OlympicArena","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":14,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"claire_record_id","repository":"gh:baidu-frontier-research/omegause-officeval","source_record":"bm_omegause-officeval_6b76ef98"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agents on long-horizon office-suite tasks from 100 practitioner-derived scenarios, scoring deliverable quality through code-based verifiers and comparing performance against economic signals of human labor time and price.","descriptions":{"en":"Evaluates LLM agents on long-horizon office-suite tasks from 100 practitioner-derived scenarios, scoring deliverable quality through code-based verifiers and comparing performance against economic signals of human labor time and price."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27155","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_omegause-officeval_6b76ef98"},"modality":null,"name":"OmegaUse-OfficeVal","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27155","source_url":"https://arxiv.org/abs/2607.27155"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27155","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"The proposed benchmark would consist of 1,160 expert-validated questions covering DNA regulation, RNA processing, and protein function.","descriptions":{"en":"The proposed benchmark would consist of 1,160 expert-validated questions covering DNA regulation, RNA processing, and protein function."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23435","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science","finance-legal"],"join":"claire_record_id","primary":"science","source_record":"bm_omicsbench_e514e04f"},"modality":null,"name":"OmicsBench","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23435","source_url":"https://arxiv.org/abs/2609.23435"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23435","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23435","topics":["reasoning-math"],"unit":null},{"aliases":["LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It"],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":734,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:composo-ai/omission-bench","source_record":"bm_llm-judges-verify-presence-not-absence-omi_7f98c0b5"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Assesses whether automated judges detect omissions in AI-generated clinical notes using 500 pairs of clean and single-error notes scored on paired discrimination and false-alarm rates.","descriptions":{"en":"Assesses whether automated judges detect omissions in AI-generated clinical notes using 500 pairs of clean and single-error notes scored on paired discrimination and false-alarm rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.31016","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","factuality"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_llm-judges-verify-presence-not-absence-omi_7f98c0b5"},"modality":null,"name":"OmissionBench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.31016","source_url":"https://arxiv.org/abs/2608.31016"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-31016","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31016","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":92,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":32,"join":"artifact:arxiv:2410.07985","repository":null,"source_record":"catalog_999b52bdc3ac2d6b"},"categories":["强推理","Strong Reasoning","数学","Math","大语言模型","LLM","逻辑推理","Reasoning","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Omni-MATH focuses exclusively on mathematics and comprises a vast collection of 4428 competition-level problems with rigorous human annotation. These problems are meticulously categorized into over 33 sub-domains and span more than 10 distinct difficulty levels Omni-MATH用于评估LLM在奥林匹克水平上的数学推理能力，包括4428道竞赛级问题，并带有严格的人工注释。这些问题被精心分类为超过33个子领域，涵盖10多个不同的难度级别。","descriptions":{"en":"Omni-MATH focuses exclusively on mathematics and comprises a vast collection of 4428 competition-level problems with rigorous human annotation. These problems are meticulously categorized into over 33 sub-domains and span more than 10 distinct difficulty levels","zh":"Omni-MATH用于评估LLM在奥林匹克水平上的数学推理能力，包括4428道竞赛级问题，并带有严格的人工注释。这些问题被精心分类为超过33个子领域，涵盖10多个不同的难度级别。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1244","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"artifact:arxiv:2410.07985","primary":"math","source_record":"catalog_999b52bdc3ac2d6b"},"modality":null,"name":"Omni-MATH","openness":"open","publisher":"Peking University","released":"2024-10-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1244-omni-math","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Omni-MATH","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_omni-storybench_38584d54"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Omnimodal evaluation should go beyond independent text, image, and speech production: individually plausible outputs may not express a coherent shared event.","descriptions":{"en":"Omnimodal evaluation should go beyond independent text, image, and speech production: individually plausible outputs may not express a coherent shared event."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37317","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","speech-audio"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_omni-storybench_38584d54"},"modality":null,"name":"Omni-StoryBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37317","source_url":"https://arxiv.org/abs/2609.37317"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37317","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37317","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"The proposed benchmark would cover audiovisual website development with 918 bilingual instances involving edit steps.","descriptions":{"en":"The proposed benchmark would cover audiovisual website development with 918 bilingual instances involving edit steps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23417","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_omni2web_e98b595a"},"modality":null,"name":"Omni2Web","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23417","source_url":"https://arxiv.org/abs/2609.23417"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23417","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23417","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_omniabench_e9e6c4ef"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object is unclear from the provided information.","descriptions":{"en":"The evaluation object is unclear from the provided information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14989","languages":[],"library_direction":null,"modality":null,"name":"OmniaBench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14989","source_url":"https://arxiv.org/abs/2607.14989"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14989","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14989","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OmniAlign-V datasets mainly focus on improving the alignment of Multi-modal Large Language Models(MLLMs) with human preference. It contains 205k high-quality Image-Quetion-Answer pairs with open-ended, creative quetions and long, knowledge-rich, comprehensive answers. OmniAlign-V 数据集主要关注提高多模态大型语言模型（MLLMs）与人类偏好的对齐。它包含 205k 个高质量的图像-问答对，包含开放式、创意性问题以及长篇、知识丰富、内容全面的答案。","descriptions":{"en":"OmniAlign-V datasets mainly focus on improving the alignment of Multi-modal Large Language Models(MLLMs) with human preference. It contains 205k high-quality Image-Quetion-Answer pairs with open-ended, creative quetions and long, knowledge-rich, comprehensive answers.","zh":"OmniAlign-V 数据集主要关注提高多模态大型语言模型（MLLMs）与人类偏好的对齐。它包含 205k 个高质量的图像-问答对，包含开放式、创意性问题以及长篇、知识丰富、内容全面的答案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1553","languages":[],"library_direction":null,"modality":"multimodal","name":"OmniAlign-V","openness":"unknown","publisher":"Shanghai Jiaotong University, Shanghai AI Laboratory, etc.","released":"2025-02-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1553-omnialign-v","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniAlign-V","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":61,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":30,"join":"llm_stats_id","repository":null,"source_record":"catalog_09205a714e6cee54"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A novel multimodal benchmark designed to evaluate large language models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. Comprises 1,142 question-answer pairs covering 8 task categories from basic perception to complex inference, with a unique constraint that accurate responses require integrated understanding of all three modalities.","descriptions":{"en":"A novel multimodal benchmark designed to evaluate large language models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. Comprises 1,142 question-answer pairs covering 8 task categories from basic perception to complex inference, with a unique constraint that accurate responses require integrated understanding of all three modalities."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:omnibench:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:omnibench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","speech-audio"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_09205a714e6cee54"},"modality":"multimodal","name":"OmniBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":56.13,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.5613,"raw_min":0.5613,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:omnibench:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-omnibench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OmniBench is a multi-dimensional benchmark for virtual agents, designed to systematically evaluate ten core capabilities such as planning, decision-making, and instruction comprehension through automatically generated task graphs with controllable complexity. OmniBench 是一个面向虚拟智能体的多维度评测基准，旨在通过自动化流程生成具有可控复杂度的任务图，系统评估智能体在计划、决策、指令理解等十个核心能力上的表现。该基准包含 36,000 个图结构任务，覆盖 20 个真实场景，并引入 OmniEval 框架，实现子任务级别的细粒度评估，显著提升了评测的效率和可扩展性。","descriptions":{"en":"OmniBench is a multi-dimensional benchmark for virtual agents, designed to systematically evaluate ten core capabilities such as planning, decision-making, and instruction comprehension through automatically generated task graphs with controllable complexity.","zh":"OmniBench 是一个面向虚拟智能体的多维度评测基准，旨在通过自动化流程生成具有可控复杂度的任务图，系统评估智能体在计划、决策、指令理解等十个核心能力上的表现。该基准包含 36,000 个图结构任务，覆盖 20 个真实场景，并引入 OmniEval 框架，实现子任务级别的细粒度评估，显著提升了评测的效率和可扩展性。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1981","languages":[],"library_direction":null,"modality":null,"name":"OmniBench","openness":"unknown","publisher":"Zhejiang University, Hangzhou, China , Ant Group, Hangzhou, China , etc.","released":"2025-06-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1981-omnibench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniBench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":61,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":30,"join":"llm_stats_id","repository":null,"source_record":"catalog_ddc7cf62e6710c6b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Music component of OmniBench, a comprehensive benchmark for evaluating omni-language models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. The music category includes various compositions and performances that require integrated understanding across text, image, and audio modalities.","descriptions":{"en":"Music component of OmniBench, a comprehensive benchmark for evaluating omni-language models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. The music category includes various compositions and performances that require integrated understanding across text, image, and audio modalities."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:omnibench-music:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench-music?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:omnibench-music","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","visual-reasoning"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_ddc7cf62e6710c6b"},"modality":"multimodal","name":"OmniBench Music","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":52.83,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.5283,"raw_min":0.5283,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:omnibench-music:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench-music?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-omnibench-music","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench-music?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Geometric reasoning","Reasoning","Robotics & Embodied AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates VLMs on assembly-aware 3D spatial reasoning across 25k mechanical assemblies with component-level, relational, and tool-augmented tasks.","descriptions":{"en":"Evaluates VLMs on assembly-aware 3D spatial reasoning across 25k mechanical assemblies with component-level, relational, and tool-augmented tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22637","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_omnicad_d45eb5cc"},"modality":null,"name":"OmniCAD","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22637","source_url":"http://arxiv.org/abs/2608.22637v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22637","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22637v1","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":11,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":14,"join":"claire_record_id","repository":"gh:nju-link/omnicap-if","source_record":"bm_omnicap-if_98c29629"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniCap-IF evaluates instruction following in omni-modal video captioning with 50 constraint types, 1,920 samples, and checklist-based scoring for format and content correctness.","descriptions":{"en":"OmniCap-IF evaluates instruction following in omni-modal video captioning with 50 constraint types, 1,920 samples, and checklist-based scoring for format and content correctness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.08572","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","instruction-following","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_omnicap-if_98c29629"},"modality":null,"name":"OmniCap-IF","openness":"unknown","publisher":null,"released":"2026-06-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08572","source_url":"https://arxiv.org/abs/2606.08572"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08572","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08572","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"claire_record_id","repository":null,"source_record":"bm_omnicapbench_28ab39f2"},"categories":["General AI","Multimodal","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits.","descriptions":{"en":"Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.12458","languages":[],"library_direction":{"basis":"model-review-daily","categories":["speech-audio","video-understanding","visual-reasoning"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_omnicapbench_28ab39f2"},"modality":null,"name":"OmniCapBench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12458","source_url":"https://arxiv.org/abs/2610.12458"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12458","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12458","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2103,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":30,"join":"llm_stats_id","repository":null,"source_record":"catalog_1685ef9a1aefe280"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","document_understanding","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OmniDocBench evaluates multimodal models on document understanding tasks such as OCR, layout parsing, and structured document comprehension.","descriptions":{"en":"OmniDocBench evaluates multimodal models on document understanding tasks such as OCR, layout parsing, and structured document comprehension."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:omnidocbench:mimo-v2.5","reported_at":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:omnidocbench","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_1685ef9a1aefe280"},"modality":"multimodal","name":"OmniDocBench","openness":"unknown","publisher":null,"released":"2024-12-10","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:omnidocbench","source_url":"https://arxiv.org/abs/2412.07626"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":91.10000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.911,"raw_min":0.872,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:omnidocbench:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"}],"slug":"llm-stats-omnidocbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench?top_n=500","topics":["multimodal"],"unit":null},{"aliases":["OmniDocBench","OmniDocBench1.5","OmniDocBench 1.5"],"attention_signals":{"as_of":"2026-10-12","github_stars":2104,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"artifact:gh:opendatalab/omnidocbench","repository":"gh:opendatalab/omnidocbench","source_record":"catalog_f49c6a408192e277"},"categories":["multimodal"],"collected_at":null,"description":"Reported as an edit distance where lower is better, so it inverts the direction of every other row a card puts beside it.","descriptions":{"en":"Reported as an edit distance where lower is better, so it inverts the direction of every other row a card puts beside it."},"evidence_summary":{"document_count":4,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000omnidocbench\u0000qwen3_5_model_card\u0000omnidocbench_1_5\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000omnidocbench\u0000qwen3_5_model_card\u0000omnidocbench_1_5\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:omnidocbench","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"artifact:gh:opendatalab/omnidocbench","primary":"document-ocr","source_record":"catalog_f49c6a408192e277"},"modality":null,"name":"OmniDocBench","openness":"unknown","publisher":null,"released":"2024-12-10","released_reference":{"basis":"benchmark_release","source_key":"model-reports:omnidocbench","source_url":"https://github.com/opendatalab/OmniDocBench"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":91.1,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":91.1,"raw_min":90.8,"source_reference":{"obs_id":"curated\u0000omnidocbench\u0000moonshot_kimi_k3_model_card\u0000omnidocbench\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000omnidocbench\u0000moonshot_kimi_k3_model_card\u0000omnidocbench\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"omnidocbench","source":"model_reports","source_url":"https://github.com/opendatalab/OmniDocBench","topics":["multimodal"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","长文本","Long-Context","Document content extraction","多模态模型","VLM","长上下文","Long Context","OCR与文档理解","OCR and Document Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OmniDocBench is a comprehensive benchmark for evaluating document parsing in real-world scenarios. It includes 981 PDF pages across 9 document types, annotated with dense paragraph-level bboxes with text and attributes. Along with its designed evaluation methods, it provides Fine-grained results. OmniDocBench是一个用于评估真实场景下多样性文档解析效果的评测集，它包含了981个页面，覆盖9种文档类型（包括研报、教材、报纸、手写笔记、杂志等），具有段落级别的位置标注和内容标注，还有阅读顺序标注和属性标注，并开发了配套的评测方法，使其既具备单模块的评测能力（包括布局检测，公式识别，表格识别，文本识别），又具备端到端的评测能力，针对不同元素提供了分页面以及分属性的精细化评测结果，精准定位模型文档解析的痛点问题。","descriptions":{"en":"OmniDocBench is a comprehensive benchmark for evaluating document parsing in real-world scenarios. It includes 981 PDF pages across 9 document types, annotated with dense paragraph-level bboxes with text and attributes. Along with its designed evaluation methods, it provides Fine-grained results.","zh":"OmniDocBench是一个用于评估真实场景下多样性文档解析效果的评测集，它包含了981个页面，覆盖9种文档类型（包括研报、教材、报纸、手写笔记、杂志等），具有段落级别的位置标注和内容标注，还有阅读顺序标注和属性标注，并开发了配套的评测方法，使其既具备单模块的评测能力（包括布局检测，公式识别，表格识别，文本识别），又具备端到端的评测能力，针对不同元素提供了分页面以及分属性的精细化评测结果，精准定位模型文档解析的痛点问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1942","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"artifact:arxiv:2412.07626","primary":"document-ocr","source_record":"catalog_1685ef9a1aefe280"},"modality":"multimodal","name":"OmniDocBench","openness":"unknown","publisher":"Shanghai AI Laboratory, Abaka AI, 2077AI","released":"2024-12-02","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1942-omnidocbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniDocBench","topics":["multimodal","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2104,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:opendatalab/omnidocbench","source_record":"catalog_f49c6a408192e277"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","structured_output","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OmniDocBench 1.5 is a comprehensive benchmark for evaluating multimodal large language models on document understanding tasks, including OCR, document parsing, information extraction, and visual question answering across diverse document types. Lower Overall Edit Distance scores are better.","descriptions":{"en":"OmniDocBench 1.5 is a comprehensive benchmark for evaluating multimodal large language models on document understanding tasks, including OCR, document parsing, information extraction, and visual question answering across diverse document types. Lower Overall Edit Distance scores are better."},"evidence_summary":{"document_count":1,"model_count":18,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:omnidocbench-1.5:gemini-3-pro-preview","reported_at":"2025-11-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench-1.5?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:omnidocbench-1.5","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_f49c6a408192e277"},"modality":"multimodal","name":"OmniDocBench 1.5","openness":"unknown","publication":{"evidence":[{"note":"The authors' v1_5 branch names OmniDocBench, links the introducing paper, and records the 2025-09-25 update from v1.0 to v1.5. The current main README directs users of older versions to their specific branches. The branch list confirms v1_5 exists. The source record explicitly names version 1.5; its Overall Edit Distance wording conflicts with the release's composite Overall formula, so this review establishes publication links only.","url":"https://github.com/opendatalab/OmniDocBench/blob/v1_5/README.md"}],"id":"omnidocbench-1-5-publication","links":[{"kind":"repo","title":"OmniDocBench v1.5 official code and release notes","url":"https://github.com/opendatalab/OmniDocBench/tree/v1_5"},{"kind":"paper","title":"OmniDocBench introducing paper","url":"https://arxiv.org/abs/2412.07626"}],"record_keys":["llm-stats:omnidocbench-1.5"],"reviewed_at":"2026-10-26","scope":{"en":"OmniDocBench 1.5: the version-specific code branch and the introducing paper. The paper describes the benchmark family; the v1.5 release notes define its updated data and metrics. The source record's metric wording has not been reconciled with those release notes.","zh":"OmniDocBench 1.5 的专用代码分支与原论文。论文介绍评测系列，v1.5 发布说明定义该版数据及指标更新；收录来源中的指标描述尚未与该发布说明核对一致。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":18,"score_direction":"higher_is_better","score_summary":{"display_max":91.60000000000001,"display_multiplier":100,"model_count":18,"model_count_basis":"source_model_id","numeric_count":18,"raw_max":0.916,"raw_min":0.115,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:omnidocbench-1.5:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench-1.5?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-05-23","date_precision":"model_announcement","name":"Gemma 4 12B"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-omnidocbench-1-5","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench-1.5?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The proposed benchmark would unify spatial audio-visual perception and audio-vision-language navigation tasks for embodied agents.","descriptions":{"en":"The proposed benchmark would unify spatial audio-visual perception and audio-vision-language navigation tasks for embodied agents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23407","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["spatial-3d","speech-audio","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_omniechobench_6c403f3f"},"modality":null,"name":"OmniEchoBench","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23407","source_url":"https://arxiv.org/abs/2609.23407"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23407","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23407","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_omniedit-bench_6a691145"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniEdit-Bench evaluates instruction-based video editing with dimensions including spatial, temporal, audio, and reference-based editing, and assesses accuracy, preservation, realism, and consistency with an accuracy-aware penalty.","descriptions":{"en":"OmniEdit-Bench evaluates instruction-based video editing with dimensions including spatial, temporal, audio, and reference-based editing, and assesses accuracy, preservation, realism, and consistency with an accuracy-aware penalty."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05049","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","speech-audio"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_omniedit-bench_6a691145"},"modality":null,"name":"OmniEdit-Bench","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05049","source_url":"https://arxiv.org/abs/2608.05049"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05049","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05049","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":26,"github_stars_30d":5,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ncclab-sustech/omni-eegbench","source_record":"bm_omnieeg-bench_5de66c8c"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniEEG-Bench is a unified benchmark for EEG foundation models, organizing evaluation into six task families and unifying 54 EEG datasets. It provides a leaderboard and code repository.","descriptions":{"en":"OmniEEG-Bench is a unified benchmark for EEG foundation models, organizing evaluation into six task families and unifying 54 EEG datasets. It provides a leaderboard and code repository."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.00815","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_omnieeg-bench_5de66c8c"},"modality":null,"name":"OmniEEG-Bench","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00815","source_url":"https://arxiv.org/abs/2606.00815"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00815","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00815","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating Vision-Language Models on nutrient reasoning and personalized health advice, with progressive capabilities: basic perception, quantitative reasoning, and safety-critical advisory.","descriptions":{"en":"A benchmark for evaluating Vision-Language Models on nutrient reasoning and personalized health advice, with progressive capabilities: basic perception, quantitative reasoning, and safety-critical advisory."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.08423","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_omnifood-bench_13bf03fa"},"modality":null,"name":"OmniFood-Bench","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08423","source_url":"https://arxiv.org/abs/2607.08423"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08423","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08423","topics":["reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":148,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":53,"join":"llm_stats_id","repository":null,"source_record":"catalog_1589f33962d88a92"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OmniGAIA evaluates multimodal perception and reasoning in agentic contexts, testing a model's ability to process diverse inputs and perform complex multi-step reasoning tasks.","descriptions":{"en":"OmniGAIA evaluates multimodal perception and reasoning in agentic contexts, testing a model's ability to process diverse inputs and perform complex multi-step reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:omnigaia:mimo-v2-omni","reported_at":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnigaia?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:omnigaia","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","speech-audio","visual-reasoning"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_1589f33962d88a92"},"modality":"multimodal","name":"OmniGAIA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":49.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.498,"raw_min":0.498,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:omnigaia:mimo-v2-omni","reported_date":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnigaia?top_n=500"},"unit":null},"scored_models":[{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Omni"}],"slug":"llm-stats-omnigaia","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnigaia?top_n=500","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":46,"github_stars_30d":4,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"claire_record_id","repository":"gh:mxlin043/omnigamearena","source_record":"bm_omnigamearena_6afe515a"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniGameArena evaluates VLM game agents across twelve Unreal Engine 5 games spanning Solo, PvP, and Coop play. It provides unified action interfaces and two evaluation clocks (PDQ for decision quality, LCRT for real-time latency). The Improvement Dynamics Curve (IDC) measures agent improvement through reflection rounds.","descriptions":{"en":"OmniGameArena evaluates VLM game agents across twelve Unreal Engine 5 games spanning Solo, PvP, and Coop play. It provides unified action interfaces and two evaluation clocks (PDQ for decision quality, LCRT for real-time latency). The Improvement Dynamics Curve (IDC) measures agent improvement through reflection rounds."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09826","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","computer-use"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_omnigamearena_6afe515a"},"modality":null,"name":"OmniGameArena","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09826","source_url":"https://arxiv.org/abs/2606.09826"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09826","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09826","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","代码","Code","智能体","Agent","Github Issue Resolution","代码工程","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A multi-modal, multi-language benchmark dataset for the GitHub Issue Resolution task. 一个面向 GitHub Issue ResoLution任务的多语言、多模态基准数据集，包含以下特点: 1.支持 Python、Java、JS、TS 四种主流编程语言，2. 输入信息涵盖文本、图像、网页等多种模态，3. 提供可复现的 Docker 评估环境。","descriptions":{"en":"A multi-modal, multi-language benchmark dataset for the GitHub Issue Resolution task.","zh":"一个面向 GitHub Issue ResoLution任务的多语言、多模态基准数据集，包含以下特点: 1.支持 Python、Java、JS、TS 四种主流编程语言，2. 输入信息涵盖文本、图像、网页等多种模态，3. 提供可复现的 Docker 评估环境。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1801","languages":["Multilingual"],"library_direction":null,"modality":"multimodal","name":"OmniGIRL","openness":"restricted","publisher":"Sun yat-sen University, Huawei Cloud Computing Technologies,Chongqing University","released":"2025-05-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1801-omnigirl","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniGIRL","topics":["coding","agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal LLMs and OCR systems on handwritten text and mathematical expression recognition across six subtasks and twelve subsets with 77.57K images.","descriptions":{"en":"Evaluates multimodal LLMs and OCR systems on handwritten text and mathematical expression recognition across six subtasks and twelve subsets with 77.57K images."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.18586","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_omnihandwritingocr_85cf42ca"},"modality":null,"name":"OmniHandwritingOCR","openness":"unknown","publisher":null,"released":"2026-08-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.18586","source_url":"https://arxiv.org/abs/2608.18586"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-18586","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18586","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":22,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:lucky-lance/omniinteract","source_record":"bm_omniinteract_26f9ce32"},"categories":["Mathematics & Formal Science","Multimodal","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniInteract is a streaming benchmark for real-time omnimodal LLMs evaluated through native online inference over audio-visual streams. It contains 250 videos with 1,430 temporally grounded response slots (1Q1A and 1QnA), with each slot including trigger, response window, and target answer. Metrics include IA-QTF1, Interruption Diagnostic Suite, and Nested Chain Completion Score.","descriptions":{"en":"OmniInteract is a streaming benchmark for real-time omnimodal LLMs evaluated through native online inference over audio-visual streams. It contains 250 videos with 1,430 temporally grounded response slots (1Q1A and 1QnA), with each slot including trigger, response window, and target answer. Metrics include IA-QTF1, Interruption Diagnostic Suite, and Nested Chain Completion Score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26485","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_omniinteract_26f9ce32"},"modality":null,"name":"OmniInteract","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26485","source_url":"https://arxiv.org/abs/2605.26485"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26485","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26485","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniLayout evaluates language models on printed-circuit-board (PCB) layout placement reasoning under geometric, routing, and connectivity constraints, with four tasks including geometric placement, routability-aware placement, electrical functionality, and tool-augmented agentic reasoning.","descriptions":{"en":"OmniLayout evaluates language models on printed-circuit-board (PCB) layout placement reasoning under geometric, routing, and connectivity constraints, with four tasks including geometric placement, routability-aware placement, electrical functionality, and tool-augmented agentic reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.03261","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_omnilayout_9c41426f"},"modality":null,"name":"OmniLayout","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03261","source_url":"https://arxiv.org/abs/2607.03261"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03261","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03261","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-09-25T04:40:22Z","description":"A visual question-answering benchmark of 2,096 manually annotated questions over 1,603 map documents in nine categories. It is scored by accuracy.","descriptions":{"en":"A visual question-answering benchmark of 2,096 manually annotated questions over 1,603 map documents in nine categories. It is scored by accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.09068","languages":[],"library_direction":null,"modality":null,"name":"OmniMapBench","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09068","source_url":"https://arxiv.org/abs/2607.09068"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09068","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09068","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_omnimatbench_4692794c"},"categories":["Materials & Chemistry","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniMatBench assesses multimodal reasoning in materials science across 19 subfields with 3,171 expert-curated QA and calculation problems, spanning four domains from fundamental knowledge to applied materials.","descriptions":{"en":"OmniMatBench assesses multimodal reasoning in materials science across 19 subfields with 3,171 expert-curated QA and calculation problems, spanning four domains from fundamental knowledge to applied materials."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29833","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_omnimatbench_4692794c"},"modality":null,"name":"OmniMatBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29833","source_url":"https://arxiv.org/abs/2605.29833"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29833","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29833","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":92,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":32,"join":"llm_stats_id","repository":null,"source_record":"catalog_999b52bdc3ac2d6b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A Universal Olympiad Level Mathematic Benchmark for Large Language Models containing 4,428 competition-level problems with rigorous human annotation, categorized into over 33 sub-domains and spanning more than 10 distinct difficulty levels","descriptions":{"en":"A Universal Olympiad Level Mathematic Benchmark for Large Language Models containing 4,428 competition-level problems with rigorous human annotation, categorized into over 33 sub-domains and spanning more than 10 distinct difficulty levels"},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:omnimath:phi-4-reasoning","reported_at":"2025-04-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnimath?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:omnimath","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_999b52bdc3ac2d6b"},"modality":"text","name":"OmniMath","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":81.89999999999999,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.819,"raw_min":0.766,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:omnimath:phi-4-reasoning-plus","reported_date":"2025-04-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnimath?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"}],"slug":"llm-stats-omnimath","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnimath?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["CAD","Geometric reasoning","Manufacturing & Process Control","Multimodal","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniMech evaluates vision-language models on four tasks: CAD program synthesis from engineering drawings, diagram-to-3D reasoning, annotation-grounded reasoning, and tool-augmented agentic reasoning using industrial mechanical data with 251k drawings and associated CAD models.","descriptions":{"en":"OmniMech evaluates vision-language models on four tasks: CAD program synthesis from engineering drawings, diagram-to-3D reasoning, annotation-grounded reasoning, and tool-augmented agentic reasoning using industrial mechanical data with 251k drawings and associated CAD models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05539","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","code-generation","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_omnimech_c453faaf"},"modality":null,"name":"OmniMech","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05539","source_url":"https://arxiv.org/abs/2608.05539"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05539","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05539","topics":["coding","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","其他","Other","多模态模型","VLM","视频理解","Video Understanding","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OmniMMI is a cutting-edge benchmark for multi-modal interaction, specifically designed for OmniLLMs in streaming video environments. It includes 1,121 videos and 2,290 questions, tackling the challenges of streaming video understanding and proactive reasoning across six unique subtasks. OmniMMI is a cutting-edge benchmark for multi-modal interaction, specifically designed for OmniLLMs in streaming video environments. It includes 1,121 videos and 2,290 questions, tackling the challenges of streaming video understanding and proactive reasoning across six unique subtasks.","descriptions":{"en":"OmniMMI is a cutting-edge benchmark for multi-modal interaction, specifically designed for OmniLLMs in streaming video environments. It includes 1,121 videos and 2,290 questions, tackling the challenges of streaming video understanding and proactive reasoning across six unique subtasks.","zh":"OmniMMI is a cutting-edge benchmark for multi-modal interaction, specifically designed for OmniLLMs in streaming video environments. It includes 1,121 videos and 2,290 questions, tackling the challenges of streaming video understanding and proactive reasoning across six unique subtasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2142","languages":[],"library_direction":null,"modality":"multimodal","name":"OmniMMI","openness":"open","publisher":"Beijing Institute of General Artificial Intelligence","released":"2025-04-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2142-omnimmi","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniMMI","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":38,"github_stars_30d":-1,"hf_dataset_downloads":null,"hf_paper_upvotes":54,"join":"claire_record_id","repository":"gh:openraiser/omniopt","source_record":"bm_omniopt_f648d471"},"categories":["General AI","Geometric reasoning","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A cross-domain benchmark for comparing optimizers in large-scale model training. It covers 24+ optimizers across two stages: Stage 1 sweeps on C4 with LLaMA-3 architectures (60M to 1B), and Stage 2 transfers to FineWeb-Edu with four architectures (Transformer++, GLA, DeltaNet, Gated DeltaNet) at 340M and 1B scales. Controlled-variable protocol with fixed architecture, data, and schedule settings.","descriptions":{"en":"A cross-domain benchmark for comparing optimizers in large-scale model training. It covers 24+ optimizers across two stages: Stage 1 sweeps on C4 with LLaMA-3 architectures (60M to 1B), and Stage 2 transfers to FineWeb-Edu with four architectures (Transformer++, GLA, DeltaNet, Gated DeltaNet) at 340M and 1B scales. Controlled-variable protocol with fixed architecture, data, and schedule settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.04033","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_omniopt_f648d471"},"modality":null,"name":"OmniOpt","openness":"unknown","publisher":null,"released":"2026-07-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04033","source_url":"https://arxiv.org/abs/2607.04033"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04033","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04033","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zjukg/omniphys","source_record":"bm_omniphys_eb0844b7"},"categories":["Factuality","General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniPhys is a benchmark of 1,551 text-to-image generation samples grounded in a Physical Knowledge Graph, aligned with PhET simulations and curricula. It evaluates physical commonsense in generated images using a dual-path verification protocol.","descriptions":{"en":"OmniPhys is a benchmark of 1,551 text-to-image generation samples grounded in a Physical Knowledge Graph, aligned with PhET simulations and curricula. It evaluates physical commonsense in generated images using a dual-path verification protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.25641","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","science"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_omniphys_eb0844b7"},"modality":null,"name":"OmniPhys","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25641","source_url":"https://arxiv.org/abs/2607.25641"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25641","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25641","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ecnu-rail/omniphys-emnlp2026","source_record":"bm_omniphys_ae437965"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal physics understanding, reasoning, and generation on 15,246 questions with 19,850 images from middle-school to university levels.","descriptions":{"en":"Evaluates multimodal physics understanding, reasoning, and generation on 15,246 questions with 19,850 images from middle-school to university levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.25398","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning","image-video-generation"],"join":"claire_record_id","primary":"science","source_record":"bm_omniphys_ae437965"},"modality":null,"name":"OmniPhys","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25398","source_url":"https://arxiv.org/abs/2608.25398"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25398","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25398","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":27,"join":"claire_record_id","repository":null,"source_record":"bm_omnireasoningbench_fb1787dd"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Recent advances have enabled unified omni-modal models in understanding audio, vision, and language.","descriptions":{"en":"Recent advances have enabled unified omni-modal models in understanding audio, vision, and language."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.39490","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","speech-audio","abstract-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_omnireasoningbench_fb1787dd"},"modality":null,"name":"OmniReasoningBench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39490","source_url":"https://arxiv.org/abs/2609.39490"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39490","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39490","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniRef-Bench is introduced in the paper but primarily serves to evaluate the proposed DyRef method. It is not presented as a standalone public benchmark.","descriptions":{"en":"OmniRef-Bench is introduced in the paper but primarily serves to evaluate the proposed DyRef method. It is not presented as a standalone public benchmark."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26947","languages":[],"library_direction":null,"modality":null,"name":"OmniRef-Bench","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26947","source_url":"https://arxiv.org/abs/2606.26947"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26947","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26947","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A released 12-direction audio-video-text retrieval benchmark containing 3,782 triples and reporting aggregate retrieval performance.","descriptions":{"en":"A released 12-direction audio-video-text retrieval benchmark containing 3,782 triples and reporting aggregate retrieval performance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26641","languages":[],"library_direction":null,"modality":null,"name":"OmniRetriever-Bench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26641","source_url":"https://arxiv.org/abs/2605.26641"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26641","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26641","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniRouting evaluates LLMs on PCB routing reasoning under real-world constraints with 1,681 industrial designs across four tasks: geometric routing, design-rule-aware routing, electrical functionality, and tool-augmented agentic routing.","descriptions":{"en":"OmniRouting evaluates LLMs on PCB routing reasoning under real-world constraints with 1,681 industrial designs across four tasks: geometric routing, design-rule-aware routing, electrical functionality, and tool-augmented agentic routing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04434","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_omnirouting_ca769ccd"},"modality":null,"name":"OmniRouting","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04434","source_url":"https://arxiv.org/abs/2608.04434"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04434","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04434","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","knowledge"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OmniScience is a broad scientific knowledge and reasoning benchmark that measures both answer accuracy and non-hallucination (calibrated abstention) across science domains.","descriptions":{"en":"OmniScience is a broad scientific knowledge and reasoning benchmark that measures both answer accuracy and non-hallucination (calibrated abstention) across science domains."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:omniscience:nemotron-3-ultra-550b-a55b","reported_at":"2026-06-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:omniscience","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_23575a8f73f0d1af"},"modality":"text","name":"OmniScience","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":78.7,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.787,"raw_min":0.175,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:omniscience:nemotron-3-ultra-550b-a55b","reported_date":"2026-06-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"}],"slug":"llm-stats-omniscience","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","science","knowledge"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OmniScience variant that reports the non-hallucination rate, defined as one minus the hallucination rate.","descriptions":{"en":"OmniScience variant that reports the non-hallucination rate, defined as one minus the hallucination rate."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:omniscience-non-hallucination-rate:grok-4.5","reported_at":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience-non-hallucination-rate?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:omniscience-non-hallucination-rate","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_c26d004be04d59b6"},"modality":"text","name":"OmniScience (non-hallucination rate)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":46.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.46,"raw_min":0.46,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:omniscience-non-hallucination-rate:grok-4.5","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience-non-hallucination-rate?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"}],"slug":"llm-stats-omniscience-non-hallucination-rate","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience-non-hallucination-rate?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":["Re-engineering SORT-based algorithms for low-cost small object tracking from omnidirectional footage"],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:xin-shu/omnisort","source_record":"bm_re-engineering-sort-based-algorithms-for-l_55c64c2b"},"categories":["Autonomous Driving","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniSmall evaluates multi-object tracking of flying animals in omnidirectional video, focusing on seam-aware motion and association for small, fast-moving targets.","descriptions":{"en":"OmniSmall evaluates multi-object tracking of flying animals in omnidirectional video, focusing on seam-aware motion and association for small, fast-moving targets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.07547","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_re-engineering-sort-based-algorithms-for-l_55c64c2b"},"modality":null,"name":"OmniSmall","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07547","source_url":"https://arxiv.org/abs/2609.07547"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07547","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07547","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Smart-home assistants are expected to handle diverse, realistic requests that arise in daily life.","descriptions":{"en":"Smart-home assistants are expected to handle diverse, realistic requests that arise in daily life."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32569","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_omnismarthome_24a4d06f"},"modality":null,"name":"OmniSmartHome","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32569","source_url":"https://arxiv.org/abs/2609.32569"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32569","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32569","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_omnitom_56af1224"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniToM evaluates theory of mind in LLMs by requiring explicit belief modeling, extracting belief propositions and labeling them with seven-dimensional schema labels across 895 stories.","descriptions":{"en":"OmniToM evaluates theory of mind in LLMs by requiring explicit belief modeling, extracting belief propositions and labeling them with seven-dimensional schema labels across 895 stories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26322","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_omnitom_56af1224"},"modality":null,"name":"OmniToM","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26322","source_url":"https://arxiv.org/abs/2605.26322"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26322","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26322","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"OmniTraffic is a controllable generation pipeline and benchmark for spatio-temporal traffic reasoning. It provides 8M VQA samples and a 3K human-verified test set across 12 reconstructed 3D intersections, with a three-level task hierarchy covering scene perception, multi-view and temporal reasoning, and decision support.","descriptions":{"en":"OmniTraffic is a controllable generation pipeline and benchmark for spatio-temporal traffic reasoning. It provides 8M VQA samples and a 3K human-verified test set across 12 reconstructed 3D intersections, with a three-level task hierarchy covering scene perception, multi-view and temporal reasoning, and decision support."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15749","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","video-understanding","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_omnitraffic_61df9afe"},"modality":null,"name":"OmniTraffic","openness":"unknown","publisher":null,"released":"2026-06-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15749","source_url":"https://arxiv.org/abs/2606.15749"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15749","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15749","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates omni reference-to-video generation using a large-scale dataset.","descriptions":{"en":"Evaluates omni reference-to-video generation using a large-scale dataset."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.22069","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_omnivbench_1d178059"},"modality":null,"name":"OmniVBench","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.22069","source_url":"https://arxiv.org/abs/2609.22069"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-22069","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22069","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Artificial Intelligence Virtual Cells (AIVCs) are envisioned as scientific agents that simulate cellular responses, explain underlying mechanisms, and support hypothesis-driven discovery.","descriptions":{"en":"Artificial Intelligence Virtual Cells (AIVCs) are envisioned as scientific agents that simulate cellular responses, explain underlying mechanisms, and support hypothesis-driven discovery."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37773","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_omnivcbench_59295138"},"modality":null,"name":"OmniVCBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37773","source_url":"https://arxiv.org/abs/2609.37773"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37773","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37773","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":["OmniVideoBench","Omni Video-Bench"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"1,000 audio-visual reasoning questions over 628 videos (seconds to 30 minutes long), targeting synergistic audio-visual understanding in Omni MLLMs, by the NJU-LINK team; an ICLR 2026 poster. Video length and audio-video correlation vary widely, so per-model results depend heavily on which subset is sampled and on frame/audio chunking strategy.","descriptions":{"en":"1,000 audio-visual reasoning questions over 628 videos (seconds to 30 minutes long), targeting synergistic audio-visual understanding in Omni MLLMs, by the NJU-LINK team; an ICLR 2026 poster. Video length and audio-video correlation vary widely, so per-model results depend heavily on which subset is sampled and on frame/audio chunking strategy."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000omnivideobench\u0000omnivideobench_v3_results_table\u0000omnivideobench_1000_mcq_v3\u00001,000 MCQ over 628 videos; omni-modal setting (visual + audio inputs); v3 results\u0000Baichuan-Omni-1.5","reported_at":"2026-03-05","source_url":"https://arxiv.org/html/2510.10689v3"},"first_score_reported_at":"2026-03-05","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000omnivideobench\u0000omnivideobench_v3_results_table\u0000omnivideobench_1000_mcq_v3\u00001,000 MCQ over 628 videos; omni-modal setting (visual + audio inputs); v3 results\u0000Baichuan-Omni-1.5","reported_at":"2026-03-05","source_url":"https://arxiv.org/html/2510.10689v3"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:omnivideobench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["video-understanding"],"join":"review:14H034160212:2026-10-15","primary":"video-understanding","source_record":null},"modality":null,"name":"OmniVideoBench","openness":"unknown","publisher":null,"released":"2025-10-12","released_reference":{"basis":"benchmark_release","source_key":"model-reports:omnivideobench","source_url":"https://arxiv.org/abs/2510.10689"},"repo_kind":null,"repo_resolution_status":null,"score_count":19,"score_direction":"higher_is_better","score_summary":{"display_max":61.8,"display_multiplier":1,"model_count":17,"model_count_basis":"source_model_id","numeric_count":19,"raw_max":61.8,"raw_min":26.4,"source_reference":{"obs_id":"curated\u0000omnivideobench\u0000omnivideobench_v3_results_table\u0000omnivideobench_1000_mcq_v3\u00001,000 MCQ over 628 videos; omni-modal setting (visual + audio inputs); v3 results\u0000Gemini-3.0-Pro","observation_id":"curated\u0000omnivideobench\u0000omnivideobench_v3_results_table\u0000omnivideobench_1000_mcq_v3\u00001,000 MCQ over 628 videos; omni-modal setting (visual + audio inputs); v3 results\u0000Gemini-3.0-Pro","reported_at":"2026-03-05","reported_date":"2026-03-05","source_id":"omnivideobench_v3_results_table","source_url":"https://arxiv.org/html/2510.10689v3"},"unit":"percent"},"scored_models":[{"date":"2026-03-05","date_precision":"document_publication","name":"Baichuan-Omni-1.5"},{"date":"2026-03-05","date_precision":"document_publication","name":"DeepSeek-V3.1"},{"date":"2026-03-05","date_precision":"document_publication","name":"Gemini-2.0-Flash"},{"date":"2026-03-05","date_precision":"document_publication","name":"Gemini-2.5-Flash"},{"date":"2026-03-05","date_precision":"document_publication","name":"Gemini-2.5-Pro"},{"date":"2026-03-05","date_precision":"document_publication","name":"Gemini-3.0-Flash"},{"date":"2026-03-05","date_precision":"document_publication","name":"Gemini-3.0-Pro"},{"date":"2026-03-05","date_precision":"document_publication","name":"HumanOmni-7B"},{"date":"2026-03-05","date_precision":"document_publication","name":"MiniCPM-o"},{"date":"2026-03-05","date_precision":"document_publication","name":"OmniVinci-9B"},{"date":"2026-03-05","date_precision":"document_publication","name":"Qwen2.5-Omni-7B"},{"date":"2026-03-05","date_precision":"document_publication","name":"Qwen2.5-VL-32B"},{"date":"2026-03-05","date_precision":"document_publication","name":"Qwen2.5-VL-72B"},{"date":"2026-03-05","date_precision":"document_publication","name":"Qwen2.5-VL-7B"},{"date":"2026-03-05","date_precision":"document_publication","name":"Qwen3-Omni-30B-A3B"},{"date":"2026-03-05","date_precision":"document_publication","name":"VideoLLaMA2-7B"},{"date":"2026-03-05","date_precision":"document_publication","name":"VITA-1.5-7B"}],"slug":"omnivideobench","source":"model_reports","source_url":"https://arxiv.org/abs/2510.10689","topics":["multimodal"],"unit":"percent"},{"aliases":["Towards One-to-Many Temporal Grounding"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"OMTG Bench evaluates one-to-many temporal grounding, in which a textual query can correspond to multiple disjoint video segments.","descriptions":{"en":"OMTG Bench evaluates one-to-many temporal grounding, in which a textual query can correspond to multiple disjoint video segments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06294","languages":[],"library_direction":null,"modality":null,"name":"OMTG Bench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06294","source_url":"https://arxiv.org/abs/2606.06294"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06294","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06294","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":17,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:span-ai-labs/oncotraj","source_record":"bm_oncotraj_0ff8775a"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"OncoTraj is a public benchmark for predicting acquired resistance to first-line osimertinib in EGFR-mutant non-small-cell lung cancer. It provides a harmonized dataset of 813 patients from three real-world sources, with locked splits, an evaluation harness, and six baselines, defining three tasks: 12-month progression classification, time-to-progression regression, and resistance mechanism classification.","descriptions":{"en":"OncoTraj is a public benchmark for predicting acquired resistance to first-line osimertinib in EGFR-mutant non-small-cell lung cancer. It provides a harmonized dataset of 813 patients from three real-world sources, with locked splits, an evaluation harness, and six baselines, defining three tasks: 12-month progression classification, time-to-progression regression, and resistance mechanism classification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.11144","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_oncotraj_0ff8775a"},"modality":null,"name":"OncoTraj","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11144","source_url":"https://arxiv.org/abs/2606.11144"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11144","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11144","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"OncoTriad-QA is a patient-level benchmark for pan-cancer question answering, integrating radiology, pathology, genomics, and clinical metadata from TCGA. It includes 86.1k questions across 9,281 cases and 32 cancer cohorts, with annotations derived from curated labels and diagnostic reports.","descriptions":{"en":"OncoTriad-QA is a patient-level benchmark for pan-cancer question answering, integrating radiology, pathology, genomics, and clinical metadata from TCGA. It includes 86.1k questions across 9,281 cases and 32 cancer cohorts, with annotations derived from curated labels and diagnostic reports."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02615","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_oncotriad-qa_ae032868"},"modality":null,"name":"OncoTriad-QA","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02615","source_url":"https://arxiv.org/abs/2608.02615"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02615","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02615","topics":["reasoning-math","medical"],"unit":null},{"aliases":["One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning"],"attention_signals":null,"categories":["Reasoning","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The paper describes a real-robot benchmark evaluating language-conditioned embodied reasoning with instruction pairs, but the benchmark name is not formally declared in the abstract; it uses the phrase 'introduce One Word, Different Action' without explicit benchmark declaration.","descriptions":{"en":"The paper describes a real-robot benchmark evaluating language-conditioned embodied reasoning with instruction pairs, but the benchmark name is not formally declared in the abstract; it uses the phrase 'introduce One Word, Different Action' without explicit benchmark declaration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.05260","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_one-word-different-action-a-real-robot-ben_140e4fe9"},"modality":null,"name":"One Word, Different Action","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05260","source_url":"https://arxiv.org/abs/2609.05260"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05260","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05260","topics":["reasoning-math"],"unit":null},{"aliases":["ONE CYLinder: A Benchmark for Graph-Based Surrogate Modeling of Unsteady Bluff-Body Flows"],"attention_signals":null,"categories":["General AI","Language & Knowledge","physics.flu-dyn"],"collected_at":"2026-10-14T09:44:51Z","description":"ONECYL evaluates graph-based surrogate models for unsteady flow past a circular cylinder across laminar, transitional, and high-Reynolds-number regimes using 450 high-fidelity simulations with randomized geometries, time-resolved fields, and mesh connectivity.","descriptions":{"en":"ONECYL evaluates graph-based surrogate models for unsteady flow past a circular cylinder across laminar, transitional, and high-Reynolds-number regimes using 450 high-fidelity simulations with randomized geometries, time-resolved fields, and mesh connectivity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08947","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_one-cylinder-a-benchmark-for-graph-based-s_4290f2a0"},"modality":null,"name":"ONECYL","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08947","source_url":"https://arxiv.org/abs/2609.08947"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08947","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08947","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":53,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":27,"join":"llm_stats_id","repository":null,"source_record":"catalog_fca7f5666d5002e2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OneMillion Bench evaluates AI agents on high-economic-value tasks that require sustained, reliable execution across long-horizon real-world workflows.","descriptions":{"en":"OneMillion Bench evaluates AI agents on high-economic-value tasks that require sustained, reliable execution across long-horizon real-world workflows."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:onemillion-bench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/onemillion-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:onemillion-bench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_fca7f5666d5002e2"},"modality":"text","name":"OneMillion Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":68.8,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.688,"raw_min":0.525,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:onemillion-bench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/onemillion-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-onemillion-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/onemillion-bench?top_n=500","topics":["agents","reasoning-math","long-context"],"unit":null},{"aliases":["OneMillionBench","One Million Bench"],"attention_signals":null,"categories":["long_context"],"collected_at":null,"description":"Long-context recall and use; a \"with tools\" figure is not comparable to a no-tools run.","descriptions":{"en":"Long-context recall and use; a \"with tools\" figure is not comparable to a no-tools run."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000onemillionbench\u0000tencent_hy4_preview\u0000onemillionbench\u0000w/ tools\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000onemillionbench\u0000tencent_hy4_preview\u0000onemillionbench\u0000w/ tools\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:onemillionbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["long-context"],"join":"review:14H034160212:2026-10-15","primary":"long-context","source_record":null},"modality":null,"name":"OneMillionBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":65.4,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":65.4,"raw_min":65.4,"source_reference":{"obs_id":"curated\u0000onemillionbench\u0000tencent_hy4_preview\u0000onemillionbench\u0000w/ tools\u0000Hy4 preview","observation_id":"curated\u0000onemillionbench\u0000tencent_hy4_preview\u0000onemillionbench\u0000w/ tools\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"onemillionbench","source":"model_reports","source_url":"https://github.com/humanlaya/OneMillion-Bench","topics":["long-context"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Materials & Chemistry","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"onepot-Bench 0 is a proprietary benchmark suite evaluating language models on synthetic chemistry capabilities, including cheminformatics literacy, safety behavior, and reaction outcome prediction.","descriptions":{"en":"onepot-Bench 0 is a proprietary benchmark suite evaluating language models on synthetic chemistry capabilities, including cheminformatics literacy, safety behavior, and reaction outcome prediction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02595","languages":[],"library_direction":{"basis":"model-review","categories":["science","safety-alignment"],"join":"claire_record_id","primary":"science","source_record":"bm_onepot-bench_ffd24571"},"modality":null,"name":"onepot-Bench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02595","source_url":"https://arxiv.org/abs/2608.02595"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02595","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02595","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Information retrieval","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A tiered biomedical retrieval benchmark with query-document pairs for concept grounding, relational retrieval, and compositional phenotype-based retrieval.","descriptions":{"en":"A tiered biomedical retrieval benchmark with query-document pairs for concept grounding, relational retrieval, and compositional phenotype-based retrieval."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08174","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research","document-ocr","health-medicine"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_ontologybench_398d89ca"},"modality":null,"name":"OntologyBench","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08174","source_url":"https://arxiv.org/abs/2609.08174"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08174","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08174","topics":[],"unit":null},{"aliases":["OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"OOD-RL-Bench is an extensible framework for testing out-of-distribution detectors on anomalies injected into RL trajectories. It scores detectors with matched-time AUROC, AUPRC, false-positive rate, and detection delay.","descriptions":{"en":"OOD-RL-Bench is an extensible framework for testing out-of-distribution detectors on anomalies injected into RL trajectories. It scores detectors with matched-time AUROC, AUPRC, false-positive rate, and detection delay."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.12523","languages":[],"library_direction":null,"modality":null,"name":"OOD-RL-Bench","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.12523","source_url":"https://arxiv.org/abs/2607.12523"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-12523","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12523","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":10,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:vila-lab/opai-bench","source_record":"bm_opai-bench_24958874"},"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"OpAI-Bench evaluates AI-text detection across document, sentence, token, and span granularities using operation-guided progressive human-to-AI revision trajectories with nine versions per sample.","descriptions":{"en":"OpAI-Bench evaluates AI-text detection across document, sentence, token, and span granularities using operation-guided progressive human-to-AI revision trajectories with nine versions per sample."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.06481","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_opai-bench_24958874"},"modality":null,"name":"OpAI-Bench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06481","source_url":"https://arxiv.org/abs/2606.06481"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06481","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06481","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":38909,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":"gh:google-research/google-research","source_record":"catalog_bcfbbc5925efb33a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language","writing"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OpenRewriteEval is a benchmark for evaluating open-ended rewriting of long-form texts, covering a wide variety of rewriting types expressed through natural language instructions including formality, expansion, conciseness, paraphrasing, and tone and style transfer.","descriptions":{"en":"OpenRewriteEval is a benchmark for evaluating open-ended rewriting of long-form texts, covering a wide variety of rewriting types expressed through natural language instructions including formality, expansion, conciseness, paraphrasing, and tone and style transfer."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:open-rewrite:llama-3.2-3b-instruct","reported_at":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/open-rewrite?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:open-rewrite","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_bcfbbc5925efb33a"},"modality":"text","name":"Open-rewrite","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":40.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.401,"raw_min":0.401,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:open-rewrite:llama-3.2-3b-instruct","reported_date":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/open-rewrite?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"}],"slug":"llm-stats-open-rewrite","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/open-rewrite?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"llm_stats_id","repository":null,"source_record":"catalog_775c292833e334b2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","physics","psychology","reasoning","finance","general","healthcare","chemistry","economics"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"MMLU (Massive Multitask Language Understanding) is a comprehensive benchmark that measures a text model's multitask accuracy across 57 diverse academic and professional subjects. The test covers elementary mathematics, US history, computer science, law, morality, business ethics, clinical knowledge, and many other domains spanning STEM, humanities, social sciences, and professional fields. To attain high accuracy, models must possess extensive world knowledge and problem-solving ability.","descriptions":{"en":"MMLU (Massive Multitask Language Understanding) is a comprehensive benchmark that measures a text model's multitask accuracy across 57 diverse academic and professional subjects. The test covers elementary mathematics, US history, computer science, law, morality, business ethics, clinical knowledge, and many other domains spanning STEM, humanities, social sciences, and professional fields. To attain high accuracy, models must possess extensive world knowledge and problem-solving ability."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:openai-mmlu:gemma-3n-e2b-it","reported_at":"2025-06-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mmlu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:openai-mmlu","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_775c292833e334b2"},"modality":"text","name":"OpenAI MMLU","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":35.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.356,"raw_min":0.223,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:openai-mmlu:gemma-3n-e4b-it","reported_date":"2025-06-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mmlu?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"}],"slug":"llm-stats-openai-mmlu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mmlu?top_n=500","topics":["reasoning-math","finance","medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Multi-round Co-reference Resolution (MRCR) benchmark for evaluating an LLM's ability to distinguish between multiple needles hidden in long context. Models are given a long, multi-turn synthetic conversation and must retrieve a specific instance of a repeated request, requiring reasoning and disambiguation skills beyond simple retrieval.","descriptions":{"en":"Multi-round Co-reference Resolution (MRCR) benchmark for evaluating an LLM's ability to distinguish between multiple needles hidden in long context. Models are given a long, multi-turn synthetic conversation and must retrieve a specific instance of a repeated request, requiring reasoning and disambiguation skills beyond simple retrieval."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:openai-mrcr:-2-needle-128k:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:openai-mrcr:-2-needle-128k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_b1a7569e9666f49d"},"modality":"text","name":"OpenAI-MRCR: 2 needle 128k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":95.19999999999999,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.952,"raw_min":0.187,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:openai-mrcr:-2-needle-128k:gpt-5-2025-08-07","reported_date":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-128k?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"}],"slug":"llm-stats-openai-mrcr-2-needle-128k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-128k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Multi-Round Co-reference Resolution benchmark that tests an LLM's ability to distinguish between multiple similar needles hidden in long conversations. Models must reproduce specific instances of content (e.g., 'Return the 2nd poem about tapirs') from multi-turn synthetic conversations, requiring reasoning about context, ordering, and subtle differences between similar outputs.","descriptions":{"en":"Multi-Round Co-reference Resolution benchmark that tests an LLM's ability to distinguish between multiple similar needles hidden in long conversations. Models must reproduce specific instances of content (e.g., 'Return the 2nd poem about tapirs') from multi-turn synthetic conversations, requiring reasoning about context, ordering, and subtle differences between similar outputs."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:openai-mrcr:-2-needle-1m:gpt-4.1-2025-04-14","reported_at":"2025-04-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-1m?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:openai-mrcr:-2-needle-1m","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_50454980643700d2"},"modality":"text","name":"OpenAI-MRCR: 2 needle 1M","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":58.599999999999994,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.586,"raw_min":0.12,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:openai-mrcr:-2-needle-1m:minimax-m1-40k","reported_date":"2025-06-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-1m?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"}],"slug":"llm-stats-openai-mrcr-2-needle-1m","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-1m?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Multi-Round Co-reference Resolution (MRCR) benchmark that tests long-context reasoning by evaluating a model's ability to distinguish between similar outputs, reason about ordering, and reproduce specific content from multi-turn conversations containing multiple writing requests on overlapping topics at 256k tokens.","descriptions":{"en":"Multi-Round Co-reference Resolution (MRCR) benchmark that tests long-context reasoning by evaluating a model's ability to distinguish between similar outputs, reason about ordering, and reproduce specific content from multi-turn conversations containing multiple writing requests on overlapping topics at 256k tokens."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:openai-mrcr:-2-needle-256k:gpt-5-2025-08-07","reported_at":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-256k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:openai-mrcr:-2-needle-256k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_b466792b10a39034"},"modality":"text","name":"OpenAI-MRCR: 2 needle 256k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":86.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.868,"raw_min":0.868,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:openai-mrcr:-2-needle-256k:gpt-5-2025-08-07","reported_date":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-256k?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"}],"slug":"llm-stats-openai-mrcr-2-needle-256k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-256k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:crispyberry/openbenchmark","source_record":"bm_openbenchmark_199b1148"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents and models on user-defined tasks generated from imported agent trajectories, scoring correctness, cost, and speed.","descriptions":{"en":"Evaluates AI agents and models on user-defined tasks generated from imported agent trajectories, scoring correctness, cost, and speed."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:crispyberry/openbenchmark","languages":[],"library_direction":null,"modality":null,"name":"OpenBenchmark","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:crispyberry/openbenchmark","source_url":"https://github.com/crispyberry/OpenBenchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-crispyberry-openbenchmark","source":"claire_radar","source_url":"https://github.com/crispyberry/OpenBenchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":84,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_113bcabbd0e2059f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OpenBookQA is a question-answering dataset modeled after open book exams for assessing human understanding. It contains 5,957 multiple-choice elementary-level science questions that probe understanding of 1,326 core science facts and their application to novel situations, requiring combination of open book facts with broad common knowledge through multi-hop reasoning.","descriptions":{"en":"OpenBookQA is a question-answering dataset modeled after open book exams for assessing human understanding. It contains 5,957 multiple-choice elementary-level science questions that probe understanding of 1,326 core science facts and their application to novel situations, requiring combination of open book facts with broad common knowledge through multi-hop reasoning."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:openbookqa:mistral-nemo-instruct-2407","reported_at":"2024-07-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openbookqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:openbookqa","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_113bcabbd0e2059f"},"modality":"text","name":"OpenBookQA","openness":"unknown","publisher":null,"released":"2018-09-08","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:openbookqa","source_url":"https://arxiv.org/abs/1809.02789"},"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":89.60000000000001,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.896,"raw_min":0.494,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:openbookqa:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openbookqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2024-07-18","date_precision":"model_announcement","name":"Mistral NeMo Instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-openbookqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openbookqa?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":84,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:1809.02789","repository":null,"source_record":"catalog_113bcabbd0e2059f"},"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OpenBookQA contains questions that require multi-step reasoning, application of common-sense knowledge, and in-depth comprehension of text. It is a new type of question-answering dataset, modeled after open-book exams, designed to assess human understanding of a specific topic. OpenBookQA包含需要多步推理、运用常识知识、深入理解文本等能力的问题，是一种新型的问答数据集，其模式借鉴了开放式书本考试，用于评估人类对某一主题理解的程度。","descriptions":{"en":"OpenBookQA contains questions that require multi-step reasoning, application of common-sense knowledge, and in-depth comprehension of text. It is a new type of question-answering dataset, modeled after open-book exams, designed to assess human understanding of a specific topic.","zh":"OpenBookQA包含需要多步推理、运用常识知识、深入理解文本等能力的问题，是一种新型的问答数据集，其模式借鉴了开放式书本考试，用于评估人类对某一主题理解的程度。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:518","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","science"],"join":"artifact:arxiv:1809.02789","primary":"knowledge-expert-qa","source_record":"catalog_113bcabbd0e2059f"},"modality":null,"name":"OpenbookQA","openness":"unknown","publisher":null,"released":"2018-09-08","released_reference":{"basis":"paper_first_version","note":"First version introducing OpenBookQA.","source_key":"opencompass:518","source_url":"https://arxiv.org/abs/1809.02789"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-518-openbookqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OpenbookQA","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","大语言模型","LLM","知识储备","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OpenFinData is an open-source financial evaluation dataset jointly released by EastMoney and Shanghai Artificial Intelligence Laboratory. This dataset represents the most realistic industrial scenario requirements and is currently the most comprehensive and professional financial evaluation dataset. OpenFinData是由东方财富与上海人工智能实验室联合发布的开源金融评测数据集。该数据集代表了最真实的产业场景需求，是目前场景最全、专业性最深的金融评测数据集。它基于东方财富实际金融业务的多样化丰富场景，旨在为金融科技领域的研究者和开发者提供一个高质量的数据资源。","descriptions":{"en":"OpenFinData is an open-source financial evaluation dataset jointly released by EastMoney and Shanghai Artificial Intelligence Laboratory. This dataset represents the most realistic industrial scenario requirements and is currently the most comprehensive and professional financial evaluation dataset.","zh":"OpenFinData是由东方财富与上海人工智能实验室联合发布的开源金融评测数据集。该数据集代表了最真实的产业场景需求，是目前场景最全、专业性最深的金融评测数据集。它基于东方财富实际金融业务的多样化丰富场景，旨在为金融科技领域的研究者和开发者提供一个高质量的数据资源。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:631","languages":[],"library_direction":null,"modality":null,"name":"OpenFinData","openness":"restricted","publisher":null,"released":"2023-12-29","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-631-openfindata","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OpenFinData","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:nellie179/hallucination-detection","source_record":"bm_openhaldet_b75bc310"},"categories":["Cybersecurity","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"OpenHalDet is a unified benchmark for hallucination detection across 17 datasets, supporting black-box, gray-box, and white-box detectors with standardized pipelines, scoring via AUROC and Cost@N.","descriptions":{"en":"OpenHalDet is a unified benchmark for hallucination detection across 17 datasets, supporting black-box, gray-box, and white-box detectors with standardized pipelines, scoring via AUROC and Cost@N."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.06959","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_openhaldet_b75bc310"},"modality":null,"name":"OpenHalDet","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06959","source_url":"https://arxiv.org/abs/2606.06959"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06959","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06959","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_openharmony-bench_a2c42703"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"OpenHarmony Bench evaluates coding agents on 153 app-level ArkTS tasks across three input sources, with 242 feature points and device-based verification.","descriptions":{"en":"OpenHarmony Bench evaluates coding agents on 153 app-level ArkTS tasks across three input sources, with 242 feature points and device-based verification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.16022","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_openharmony-bench_a2c42703"},"modality":null,"name":"OpenHarmony Bench","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16022","source_url":"https://arxiv.org/abs/2608.16022"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16022","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16022","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Molecular tumor boards integrate genomic findings, clinical context, and therapeutic evidence to support precision oncology.","descriptions":{"en":"Molecular tumor boards integrate genomic findings, clinical context, and therapeutic evidence to support precision oncology."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.01497","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_openmtb-audit_c732e1e5"},"modality":null,"name":"OpenMTB-Audit","openness":"unknown","publisher":null,"released":"2026-10-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.01497","source_url":"https://arxiv.org/abs/2610.01497"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-01497","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01497","topics":["safety","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":432,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:microsoft/openrca","source_record":"catalog_8055bd7cf670b6e1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OpenRCA is a benchmark for evaluating AI models on root cause analysis tasks. For each failure case, the model receives 1 point if all generated root-cause elements match the ground-truth ones, and 0 points if any mismatch is identified. The overall accuracy is the average score across all failure cases.","descriptions":{"en":"OpenRCA is a benchmark for evaluating AI models on root cause analysis tasks. For each failure case, the model receives 1 point if all generated root-cause elements match the ground-truth ones, and 0 points if any mismatch is identified. The overall accuracy is the average score across all failure cases."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:openrca:claude-opus-4-6","reported_at":"2026-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openrca?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:openrca","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_8055bd7cf670b6e1"},"modality":"text","name":"OpenRCA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":34.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.349,"raw_min":0.349,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:openrca:claude-opus-4-6","reported_date":"2026-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openrca?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"}],"slug":"llm-stats-openrca","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openrca?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robustness","Safety & Trustworthiness","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"OpenRTAG evaluates text-attributed graph learning under nine degradation scenarios (sparsity, noise, imbalance) across nine TAG datasets and three downstream tasks. It provides a standardized testbed for robustness comparison.","descriptions":{"en":"OpenRTAG evaluates text-attributed graph learning under nine degradation scenarios (sparsity, noise, imbalance) across nine TAG datasets and three downstream tasks. It provides a standardized testbed for robustness comparison."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19108","languages":[],"library_direction":null,"modality":null,"name":"OpenRTAG","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19108","source_url":"https://arxiv.org/abs/2607.19108"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19108","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19108","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_opensafeintent_d8b14d02"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"OpenSafeIntent is a benchmark of controlled prompt-sets that vary user intent while holding the underlying task fixed. Each datapoint contains benign, dual-use, and malicious variants of the same task, and models are evaluated on whether they calibrate assistance across intent shifts rather than only appearing safe on average.","descriptions":{"en":"OpenSafeIntent is a benchmark of controlled prompt-sets that vary user intent while holding the underlying task fixed. Each datapoint contains benign, dual-use, and malicious variants of the same task, and models are evaluated on whether they calibrate assistance across intent shifts rather than only appearing safe on average."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02047","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_opensafeintent_d8b14d02"},"modality":null,"name":"OpenSafeIntent","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02047","source_url":"https://arxiv.org/abs/2607.02047"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02047","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02047","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"OpenSciToolBench is a benchmark with 900 tasks across four difficulty levels for evaluating LLM agents in open-world scientific tool acquisition.","descriptions":{"en":"OpenSciToolBench is a benchmark with 900 tasks across four difficulty levels for evaluating LLM agents in open-world scientific tool acquisition."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.28692","languages":[],"library_direction":{"basis":"model-review","categories":["science","tool-use"],"join":"claire_record_id","primary":"science","source_record":"bm_openscitoolbench_fd41e9ac"},"modality":null,"name":"OpenSciToolBench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28692","source_url":"https://arxiv.org/abs/2607.28692"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28692","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28692","topics":["agents","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_openskillrisk_99823415"},"categories":["Agents","General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"OpenSkillRisk evaluates LLM-based agents on their ability to recognize and avoid safety risks when using third-party skills, with 263 risky skills in seven threat categories and sandboxed execution.","descriptions":{"en":"OpenSkillRisk evaluates LLM-based agents on their ability to recognize and avoid safety risks when using third-party skills, with 263 risky skills in seven threat categories and sandboxed execution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20121","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agent-skills-evolution"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_openskillrisk_99823415"},"modality":null,"name":"OpenSkillRisk","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20121","source_url":"https://arxiv.org/abs/2607.20121"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20121","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20121","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"claire_record_id","repository":"gh:anqili24/opentumorboard","source_record":"bm_opentumorboard_f6b66597"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions, yet benchmarks rarely capture these real-world trajectories.","descriptions":{"en":"Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions, yet benchmarks rarely capture these real-world trajectories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.32810","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","long-context"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_opentumorboard_f6b66597"},"modality":null,"name":"OpenTumorBoard","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32810","source_url":"https://arxiv.org/abs/2609.32810"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32810","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32810","topics":["long-context","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OpenTuringBench, a new benchmark based on OLLMs, designed to train and evaluate machine-generated text detectors on the Turing Test and Authorship Attribution problems. OpenTuringBench，这是一个基于OLLMs的新基准测试，旨在训练和评估机器生成文本检测器在图灵测试和作者归属问题上的性能。","descriptions":{"en":"OpenTuringBench, a new benchmark based on OLLMs, designed to train and evaluate machine-generated text detectors on the Turing Test and Authorship Attribution problems.","zh":"OpenTuringBench，这是一个基于OLLMs的新基准测试，旨在训练和评估机器生成文本检测器在图灵测试和作者归属问题上的性能。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1754","languages":["English"],"library_direction":null,"modality":null,"name":"OpenTuringBench","openness":"unknown","publisher":"DIMES Dept., University of Calabria","released":"2025-04-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1754-openturingbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OpenTuringBench","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OpenUnlearning is an efficient and modular benchmark platform designed to support and drive research on \"unlearning\" in large language models (LLMs). OpenUnlearning 是一个高效且模块化的基准平台，旨在支持和推动大型语言模型（LLM）中的“遗忘”（unlearning）研究。","descriptions":{"en":"OpenUnlearning is an efficient and modular benchmark platform designed to support and drive research on \"unlearning\" in large language models (LLMs).","zh":"OpenUnlearning 是一个高效且模块化的基准平台，旨在支持和推动大型语言模型（LLM）中的“遗忘”（unlearning）研究。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1979","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"artifact:arxiv:2403.03218","primary":"safety-alignment","source_record":"catalog_0ec7c167ceeb2933"},"modality":null,"name":"OpenUnlearning","openness":"unknown","publisher":"University of Massachusetts Amherst , Carnegie Mellon University ,etc.","released":"2025-06-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1979-openunlearning","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OpenUnlearning","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":198,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:openwam/openwam","source_record":"bm_openwam_b89d880b"},"categories":["Robotics & Embodied AI","Vision & 3D","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"World-action models (WAMs) couple future prediction with robot control, yet existing systems often vary the video backbone, interaction structure, supervision, and inference procedure simultaneously, making their design choices difficult t…","descriptions":{"en":"World-action models (WAMs) couple future prediction with robot control, yet existing systems often vary the video backbone, interaction structure, supervision, and inference procedure simultaneously, making their design choices difficult t…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.07922","languages":[],"library_direction":{"basis":"model-review-daily","categories":["world-models","embodied-robotics"],"join":"claire_record_id","primary":"world-models","source_record":"bm_openwam_b89d880b"},"modality":null,"name":"OpenWAM","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.07922","source_url":"https://arxiv.org/abs/2610.07922"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-07922","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07922","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_opis_2ca2f90c"},"categories":["Logistics & Operations","Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories, video reference, or selected revisit viewpoints that can confound the assessment of a model's tr…","descriptions":{"en":"Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories, video reference, or selected revisit viewpoints that can confound the assessment of a model's tr…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35052","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_opis_2ca2f90c"},"modality":null,"name":"OPIS","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35052","source_url":"https://arxiv.org/abs/2609.35052"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35052","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35052","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OPT-BENCH is a comprehensive benchmark designed to evaluate large language model (LLM) agents on large-scale search space optimization problems, focusing on their iterative reasoning and problem-solving capabilities. OPT-BENCH 是一个面向大型语言模型（LLM）智能体的大规模搜索空间优化评测基准，旨在系统评估模型在迭代推理和解决复杂优化问题中的能力。 该基准包含 30 个任务，包括 20 个来自 Kaggle 的真实机器学习任务和 10 个经典 NP 问题，涵盖预测建模、图论和组合优化等领域。","descriptions":{"en":"OPT-BENCH is a comprehensive benchmark designed to evaluate large language model (LLM) agents on large-scale search space optimization problems, focusing on their iterative reasoning and problem-solving capabilities.","zh":"OPT-BENCH 是一个面向大型语言模型（LLM）智能体的大规模搜索空间优化评测基准，旨在系统评估模型在迭代推理和解决复杂优化问题中的能力。 该基准包含 30 个任务，包括 20 个来自 Kaggle 的真实机器学习任务和 10 个经典 NP 问题，涵盖预测建模、图论和组合优化等领域。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1990","languages":[],"library_direction":null,"modality":null,"name":"OPT-BENCH","openness":"open","publisher":"Tong Ji University , ShanghaiAILab , Nanjing University , Zhejiang University","released":"2025-06-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1990-opt-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OPT-BENCH","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Opti-Agent-Bench evaluates LLM agents on the end-to-end optimization R&D pipeline, from business-language description to mathematical modeling, algorithm selection, code implementation, and report generation, with modular evaluation.","descriptions":{"en":"Opti-Agent-Bench evaluates LLM agents on the end-to-end optimization R&D pipeline, from business-language description to mathematical modeling, algorithm selection, code implementation, and report generation, with modular evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.10768","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","code-generation"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_opti-agent-bench_f017ef81"},"modality":null,"name":"Opti-Agent-Bench","openness":"unknown","publisher":null,"released":"2026-07-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10768","source_url":"https://arxiv.org/abs/2607.10768"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10768","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10768","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:wj-peng/optiarena","source_record":"bm_optiarena_e00056ea"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Static QA and code-generation benchmarks only partially capture the role that large language models (LLMs) now play as coding agents and research tools.","descriptions":{"en":"Static QA and code-generation benchmarks only partially capture the role that large language models (LLMs) now play as coding agents and research tools."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.32227","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_optiarena_e00056ea"},"modality":null,"name":"OptiArena","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32227","source_url":"https://arxiv.org/abs/2609.32227"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32227","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32227","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":[],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"","descriptions":{},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:community:64d67847-06bd-423a-923c-c2acfab82281","languages":[],"library_direction":null,"modality":null,"name":"OptimBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":"higher_is_better","score_summary":{"display_max":null,"display_multiplier":1,"model_count":null,"model_count_basis":null,"numeric_count":0,"raw_max":null,"raw_min":null,"source_reference":null,"unit":null},"scored_models":[],"slug":"llm-stats-community-64d67847-06bd-423a-923c-c2acfab82281","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A64d67847-06bd-423a-923c-c2acfab82281?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A controlled comparison of seven optimizers across four diffusion formulations, including masked, uniform, and Gaussian diffusion tasks.","descriptions":{"en":"A controlled comparison of seven optimizers across four diffusion formulations, including masked, uniform, and Gaussian diffusion tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.23055","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["image-video-generation","document-ocr"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_optimizers-for-diffusion-models-a-controll_164879b3"},"modality":null,"name":"Optimizers for Diffusion Models: A Controlled Benchmark","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23055","source_url":"https://arxiv.org/abs/2609.23055"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23055","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23055","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Vision-language models (VLMs) can detect that an object has rotated across views, but cannot reliably tell by how much.","descriptions":{"en":"Vision-language models (VLMs) can detect that an object has rotated across views, but cannot reliably tell by how much."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.05715","languages":[],"library_direction":{"basis":"model-review-daily","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_or-bench_72590f97"},"modality":null,"name":"OR-Bench","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05715","source_url":"https://arxiv.org/abs/2610.05715"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05715","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05715","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OR Bench is a large-scale benchmark designed to evaluate the excessive rejection behavior of large language models. OR-Bench 是一个旨在评估大型语言模型过度拒绝行为的大规模基准。它衡量LLM在过度拒绝和有害提示拒绝方面的表现，涵盖暴力、隐私等10个类别。基准包含8万个、1千个困难及6百个有害提示，通过Mixtral等工具自动化生成。它为未来安全对齐研究提供强大测试平台。","descriptions":{"en":"OR Bench is a large-scale benchmark designed to evaluate the excessive rejection behavior of large language models.","zh":"OR-Bench 是一个旨在评估大型语言模型过度拒绝行为的大规模基准。它衡量LLM在过度拒绝和有害提示拒绝方面的表现，涵盖暴力、隐私等10个类别。基准包含8万个、1千个困难及6百个有害提示，通过Mixtral等工具自动化生成。它为未来安全对齐研究提供强大测试平台。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:2086","languages":[],"library_direction":null,"modality":null,"name":"OR-Bench","openness":"unknown","publisher":"UCLA , UC Berkeley","released":"2024-05-31","released_reference":{"basis":"paper_first_version","note":"First version introducing the over-refusal benchmark.","source_key":"opencompass:2086","source_url":"https://arxiv.org/abs/2405.20947"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2086-or-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OR-Bench","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":15,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:0xzhouchenyu/or-space","source_record":"bm_or-space_6e200388"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A full-lifecycle workspace benchmark for industrial optimization agents, evaluating model construction, revision, and grounded explanation across three task modes (Build, Revise, Explain) using executable multi-file workspaces with task-specific evaluators.","descriptions":{"en":"A full-lifecycle workspace benchmark for industrial optimization agents, evaluating model construction, revision, and grounded explanation across three task modes (Build, Revise, Explain) using executable multi-file workspaces with task-specific evaluators."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.28158","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_or-space_6e200388"},"modality":null,"name":"OR-Space","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28158","source_url":"https://arxiv.org/abs/2605.28158"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28158","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28158","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:cl1110goo-afk/oraclephys","source_record":"bm_oraclephys_04a2c150"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM ranking of stories by inter-story drift and identification of the governing story for multi-story 2-D steel frames, with scoring by top-1 accuracy and Spearman correlation against oracle truth.","descriptions":{"en":"Evaluates LLM ranking of stories by inter-story drift and identification of the governing story for multi-story 2-D steel frames, with scoring by top-1 accuracy and Spearman correlation against oracle truth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:cl1110goo-afk/oraclephys","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_oraclephys_04a2c150"},"modality":null,"name":"OraclePhys","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:cl1110goo-afk/oraclephys","source_url":"https://github.com/cl1110goo-afk/oraclephys"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-cl1110goo-afk-oraclephys","source":"claire_radar","source_url":"https://github.com/cl1110goo-afk/oraclephys","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_oragentbench_12c4a136"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ORAgentBench evaluates autonomous agents on end-to-end operations research tasks. Each task involves a natural-language brief, multi-file data, configuration artifacts, and a required submission schema. Agents write and run solution code, validated for schema, hard constraints, and objective quality.","descriptions":{"en":"ORAgentBench evaluates autonomous agents on end-to-end operations research tasks. Each task involves a natural-language brief, multi-file data, configuration artifacts, and a required submission schema. Agents write and run solution code, validated for schema, hard constraints, and objective quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19787","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","agentic-coding"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_oragentbench_12c4a136"},"modality":null,"name":"ORAgentBench","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19787","source_url":"https://arxiv.org/abs/2606.19787"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19787","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19787","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Orak (오락) is a foundational benchmark for evaluating Large Language Model (LLM) agents in diverse popular video games. Orak是一个基础性的基准,用于评估在各种流行视频游戏中的大型语言模型(LLM)代理。","descriptions":{"en":"Orak (오락) is a foundational benchmark for evaluating Large Language Model (LLM) agents in diverse popular video games.","zh":"Orak是一个基础性的基准,用于评估在各种流行视频游戏中的大型语言模型(LLM)代理。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1917","languages":["Korean"],"library_direction":null,"modality":null,"name":"Orak","openness":"unknown","publisher":"KRAFTON , SeoulNationalUniversity , NVIDIA , UniversityofWisconsin-Madison","released":"2025-06-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1917-orak","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Orak","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on data science code translation across curated tasks and project-level translations.","descriptions":{"en":"Evaluates LLMs on data science code translation across curated tasks and project-level translations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.30749","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","science"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_orca_373d8ab7"},"modality":null,"name":"ORCA","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30749","source_url":"https://arxiv.org/abs/2609.30749"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30749","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30749","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:orca-bench/orca-bench","source_record":"bm_orca-bench_79a9e07b"},"categories":["Code & Software","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"ORCA-bench evaluates language model agents on root cause analysis in a production-fidelity oncall setting, using a live OpenTelemetry-instrumented microservice system with 1,079 tasks varying in report specificity, time-to-detection, and fault co-occurrence. Agents access metrics, logs, traces, and source code via real telemetry interfaces.","descriptions":{"en":"ORCA-bench evaluates language model agents on root cause analysis in a production-fidelity oncall setting, using a live OpenTelemetry-instrumented microservice system with 1,079 tasks varying in report specificity, time-to-detection, and fault co-occurrence. Agents access metrics, logs, traces, and source code via real telemetry interfaces."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.28545","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","agentic-coding"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_orca-bench_79a9e07b"},"modality":null,"name":"ORCA-bench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28545","source_url":"https://arxiv.org/abs/2607.28545"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28545","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28545","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"OrchBench evaluates multi-agent orchestration plans in isolation using deterministic simulation. It constructs DAGs from real-world tasks and scores plans on result quality, makespan, and token cost without executing worker agents.","descriptions":{"en":"OrchBench evaluates multi-agent orchestration plans in isolation using deterministic simulation. It constructs DAGs from real-world tasks and scores plans on result quality, makespan, and token cost without executing worker agents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.25656","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_orchbench_2f9fbf5c"},"modality":null,"name":"OrchBench","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25656","source_url":"https://arxiv.org/abs/2607.25656"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25656","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25656","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"OSGuard is a dual-granularity benchmark suite for evaluating safety in computer-use agents, with an action-level benchmark for local guardrail decisions and a risk-augmented execution suite for end-to-end evaluation.","descriptions":{"en":"OSGuard is a dual-granularity benchmark suite for evaluating safety in computer-use agents, with an action-level benchmark for local guardrail decisions and a risk-augmented execution suite for end-to-end evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15034","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","computer-use"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_osguard_1e29e1e9"},"modality":null,"name":"OSGuard","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15034","source_url":"https://arxiv.org/abs/2606.15034"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15034","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15034","topics":["computer-use","agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","Memory Safety","Open-Source Software","Living Benchmark","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OSS-Bench replaces functions with LLM-generated code and evaluates them using three natural metrics: compilability, functional correctness, and memory safety, leveraging robust signals like compilation failures, test-suite violations, and sanitizer alerts as ground truth. OSS-Bench，这是一个基准生成器，它可以从真实的开源软件中自动构建大规模的实时评估任务。OSS-Bench 将函数替换为 LLM 生成的代码，并使用三个自然指标（可编译性、功能正确性和内存安全性）对其进行评估，并利用编译失败、测试套件违规和Sanitizer警报等稳健信号作为基准事实。","descriptions":{"en":"OSS-Bench replaces functions with LLM-generated code and evaluates them using three natural metrics: compilability, functional correctness, and memory safety, leveraging robust signals like compilation failures, test-suite violations, and sanitizer alerts as ground truth.","zh":"OSS-Bench，这是一个基准生成器，它可以从真实的开源软件中自动构建大规模的实时评估任务。OSS-Bench 将函数替换为 LLM 生成的代码，并使用三个自然指标（可编译性、功能正确性和内存安全性）对其进行评估，并利用编译失败、测试套件违规和Sanitizer警报等稳健信号作为基准事实。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2005","languages":[],"library_direction":null,"modality":null,"name":"OSS-Bench","openness":"unknown","publisher":"National University of Singapore","released":"2025-06-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2005-oss-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OSS-Bench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A diagnostic benchmark for omni-streaming thinking that fixes video while editing audio across conditions including agreement, absence, contradiction, coexistence, and subtitle-speech conflict.","descriptions":{"en":"A diagnostic benchmark for omni-streaming thinking that fixes video while editing audio across conditions including agreement, absence, contradiction, coexistence, and subtitle-speech conflict."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15128","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio","health-medicine"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_ost-diagbench_01522cb2"},"modality":null,"name":"OST-DiagBench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15128","source_url":"https://arxiv.org/abs/2609.15128"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15128","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15128","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3206,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":52,"join":"llm_stats_id","repository":"gh:xlang-ai/osworld","source_record":"lib_osworld"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","general","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OSWorld: The first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS with 369 computer tasks involving real web and desktop applications, OS file I/O, and multi-application workflows","descriptions":{"en":"OSWorld: The first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS with 369 computer tasks involving real web and desktop applications, OS file I/O, and multi-application workflows"},"evidence_summary":{"document_count":1,"model_count":20,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:osworld:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:osworld","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"lib_osworld"},"modality":"multimodal","name":"OSWorld","openness":"unknown","publisher":null,"released":"2024-04-11","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:osworld","source_url":"https://arxiv.org/abs/2404.07972"},"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":20,"score_direction":"higher_is_better","score_summary":{"display_max":78.8,"display_multiplier":100,"model_count":20,"model_count_basis":"source_model_id","numeric_count":20,"raw_max":0.788,"raw_min":0.0592,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:osworld:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld?top_n=500"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude Sonnet 4.5"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-osworld","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld?top_n=500","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":["OSWorld","OSWorld-Verified","OSWorld 2.0"],"attention_signals":null,"categories":["computer_use"],"collected_at":null,"description":"Screen resolution, OS image and action space all change results. The Verified and 2.0 revisions are not comparable to the original.","descriptions":{"en":"Screen resolution, OS image and action space all change results. The Verified and 2.0 revisions are not comparable to the original."},"evidence_summary":{"document_count":5,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000osworld\u0000qwen3_5_model_card\u0000osworld_verified\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000osworld\u0000qwen3_5_model_card\u0000osworld_verified\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:osworld","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["computer-use"],"join":"review:14H034160212:2026-10-15","primary":"computer-use","source_record":null},"modality":null,"name":"OSWorld","openness":"unknown","publisher":null,"released":"2024-04-11","released_reference":{"basis":"benchmark_release","source_key":"model-reports:osworld","source_url":"https://os-world.github.io/"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":84.8,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":84.8,"raw_min":58.3,"source_reference":{"obs_id":"curated\u0000osworld\u0000moonshot_kimi_k3_model_card\u0000osworld_verified\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000osworld\u0000moonshot_kimi_k3_model_card\u0000osworld_verified\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-07-24","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"osworld","source":"model_reports","source_url":"https://os-world.github.io/","topics":["computer-use","agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":365,"github_stars_30d":72,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"claire_record_id","repository":"gh:xlang-ai/osworld-v2","source_record":"bm_osworld-2-0_ee079e02"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates computer-use agents on 108 long-horizon real-world workflows across everyday and professional tasks, scored by binary completion at 500 steps and partial scores.","descriptions":{"en":"Evaluates computer-use agents on 108 long-horizon real-world workflows across everyday and professional tasks, scored by binary completion at 500 steps and partial scores."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.29537","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_osworld-2-0_ee079e02"},"modality":null,"name":"OSWorld 2.0","openness":"unknown","publisher":null,"released":"2026-06-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29537","source_url":"https://arxiv.org/abs/2606.29537"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29537","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29537","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":365,"github_stars_30d":72,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":"gh:xlang-ai/osworld-v2","source_record":"bm_osworld-2-0_ee079e02"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","general","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OSWorld 2.0 is a benchmark of 108 long-horizon, real-world computer-use workflows spanning everyday and professional tasks. Each task is an end-to-end workflow that takes human users a median of about 1.6 hours, scored with a binary-completion metric, and targets challenges such as dynamic environments, cross-source reasoning, and implicit-state inference.","descriptions":{"en":"OSWorld 2.0 is a benchmark of 108 long-horizon, real-world computer-use workflows spanning everyday and professional tasks. Each task is an end-to-end workflow that takes human users a median of about 1.6 hours, scored with a binary-completion metric, and targets challenges such as dynamic environments, cross-source reasoning, and implicit-state inference."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:osworld-2.0:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-2.0?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:osworld-2.0","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"bm_osworld-2-0_ee079e02"},"modality":"multimodal","name":"OSWorld 2.0","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":70.6,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.706,"raw_min":0.456,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:osworld-2.0:claude-opus-5","reported_date":"2026-07-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-2.0?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-osworld-2-0","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-2.0?top_n=500","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3206,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":52,"join":"llm_stats_id","repository":"gh:xlang-ai/osworld","source_record":"catalog_6ae987024f67025b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OSWorld is a scalable, real computer environment benchmark for evaluating multimodal agents on open-ended tasks across Ubuntu, Windows, and macOS. It comprises 369 computer tasks involving real web and desktop applications, OS file I/O, and multi-application workflows. The benchmark evaluates agents' ability to interact with computer interfaces using screenshots and actions in realistic computing environments.","descriptions":{"en":"OSWorld is a scalable, real computer environment benchmark for evaluating multimodal agents on open-ended tasks across Ubuntu, Windows, and macOS. It comprises 369 computer tasks involving real web and desktop applications, OS file I/O, and multi-application workflows. The benchmark evaluates agents' ability to interact with computer interfaces using screenshots and actions in realistic computing environments."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:osworld-extended:claude-3-5-sonnet-20241022","reported_at":"2024-10-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-extended?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:osworld-extended","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_6ae987024f67025b"},"modality":"multimodal","name":"OSWorld Extended","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":22.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.22,"raw_min":0.22,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:osworld-extended:claude-3-5-sonnet-20241022","reported_date":"2024-10-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-extended?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"}],"slug":"llm-stats-osworld-extended","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-extended?top_n=500","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3202,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":52,"join":"llm_stats_id","repository":null,"source_record":"catalog_29abe33c53bcfbfc"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","general","grounding","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OSWorld Screenshot-only: A variant of the OSWorld benchmark that evaluates multimodal AI agents using only screenshot observations to complete open-ended computer tasks across real operating systems (Ubuntu, Windows, macOS). Tests agents' ability to perform complex workflows involving web apps, desktop applications, file I/O, and multi-application tasks through visual interface understanding and GUI grounding.","descriptions":{"en":"OSWorld Screenshot-only: A variant of the OSWorld benchmark that evaluates multimodal AI agents using only screenshot observations to complete open-ended computer tasks across real operating systems (Ubuntu, Windows, macOS). Tests agents' ability to perform complex workflows involving web apps, desktop applications, file I/O, and multi-application tasks through visual interface understanding and GUI grounding."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:osworld-screenshot-only:claude-3-5-sonnet-20241022","reported_at":"2024-10-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-screenshot-only?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:osworld-screenshot-only","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_29abe33c53bcfbfc"},"modality":"multimodal","name":"OSWorld Screenshot-only","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":14.899999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.149,"raw_min":0.149,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:osworld-screenshot-only:claude-3-5-sonnet-20241022","reported_date":"2024-10-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-screenshot-only?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"}],"slug":"llm-stats-osworld-screenshot-only","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-screenshot-only?top_n=500","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":180,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":46,"join":"llm_stats_id","repository":null,"source_record":"catalog_fa3744e8e53dfc9f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","grounding","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OSWorld-G (Grounding) evaluates screenshot grounding accuracy for OS automation tasks.","descriptions":{"en":"OSWorld-G (Grounding) evaluates screenshot grounding accuracy for OS automation tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:osworld-g:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-g?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:osworld-g","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_fa3744e8e53dfc9f"},"modality":"image","name":"OSWorld-G","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.683,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.683,"raw_min":0.683,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:osworld-g:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-g?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-osworld-g","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-g?top_n=500","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":9,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":64,"join":"claire_record_id","repository":"gh:discoailab/osworld-science","source_record":"bm_osworld-science_0c31d18b"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Scientific software presents a demanding test for computer-using agents based on visual language models (VLMs): completing a research workflow requires interpreting specialized interfaces, manipulating scientific objects, and producing ver…","descriptions":{"en":"Scientific software presents a demanding test for computer-using agents based on visual language models (VLMs): completing a research workflow requires interpreting specialized interfaces, manipulating scientific objects, and producing ver…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.39903","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","science"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_osworld-science_0c31d18b"},"modality":null,"name":"OSWorld-Science","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39903","source_url":"https://arxiv.org/abs/2609.39903"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39903","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39903","topics":["computer-use","agents","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3206,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:xlang-ai/osworld","source_record":"lib_osworld_verified"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","general","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OSWorld-Verified is a verified subset of OSWorld, a scalable real computer environment for multimodal agents supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS.","descriptions":{"en":"OSWorld-Verified is a verified subset of OSWorld, a scalable real computer environment for multimodal agents supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS."},"evidence_summary":{"document_count":1,"model_count":24,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:osworld-verified:gpt-5.3-codex","reported_at":"2026-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-verified?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:osworld-verified","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"lib_osworld_verified"},"modality":"multimodal","name":"OSWorld-Verified","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":24,"score_direction":"higher_is_better","score_summary":{"display_max":86.1,"display_multiplier":100,"model_count":24,"model_count_basis":"source_model_id","numeric_count":24,"raw_max":0.861,"raw_min":0.39,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:osworld-verified:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-verified?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-osworld-verified","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-verified?top_n=500","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OV-SGG-Bench measures open-vocabulary relation prediction across six axes scored over datasets.","descriptions":{"en":"OV-SGG-Bench measures open-vocabulary relation prediction across six axes scored over datasets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.12552","languages":[],"library_direction":null,"modality":null,"name":"OV-SGG-Bench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12552","source_url":"https://arxiv.org/abs/2609.12552"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12552","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12552","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mgladkova/ov3d-bench","source_record":"bm_ov3d-bench_dff2d230"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"OV3D-Bench is a diagnostic benchmark for open-vocabulary monocular 3D detection, evaluating localization, semantic robustness, and cross-domain transfer across seven datasets.","descriptions":{"en":"OV3D-Bench is a diagnostic benchmark for open-vocabulary monocular 3D detection, evaluating localization, semantic robustness, and cross-domain transfer across seven datasets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.17110","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_ov3d-bench_dff2d230"},"modality":null,"name":"OV3D-Bench","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17110","source_url":"https://arxiv.org/abs/2608.17110"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17110","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17110","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":98,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:mcg-nju/videochat-online","source_record":"catalog_5b60c336f7011c62"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OVBench is an online video understanding benchmark that evaluates a model's ability to perceive, memorize, and reason about real-time video streams as they unfold.","descriptions":{"en":"OVBench is an online video understanding benchmark that evaluates a model's ability to perceive, memorize, and reason about real-time video streams as they unfold."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ovbench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ovbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ovbench","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_5b60c336f7011c62"},"modality":"multimodal","name":"OVBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":70.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.7,"raw_min":0.697,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ovbench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ovbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-ovbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ovbench?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":14,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"claire_record_id","repository":"gh:earth-insights/ovearth-bench","source_record":"bm_ovearth-bench_169abada"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates open-vocabulary Earth observation models on category breadth and query diversity, covering mask and box localization across vocabulary, referring, and reasoning queries under a unified zero-shot protocol.","descriptions":{"en":"Evaluates open-vocabulary Earth observation models on category breadth and query diversity, covering mask and box localization across vocabulary, referring, and reasoning queries under a unified zero-shot protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27278","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_ovearth-bench_169abada"},"modality":null,"name":"OVEarth-Bench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27278","source_url":"https://arxiv.org/abs/2607.27278"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27278","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27278","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"OverclaimBench evaluates frontier LLM agents across five file-review scenarios using transcript-based coverage measurements to detect overclaiming against registered planted defects.","descriptions":{"en":"OverclaimBench evaluates frontier LLM agents across five file-review scenarios using transcript-based coverage measurements to detect overclaiming against registered planted defects."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.20812","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_overclaimbench_66faa50c"},"modality":null,"name":"OverclaimBench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.20812","source_url":"https://arxiv.org/abs/2609.20812"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-20812","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20812","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kcelestinomaria/overlooked-bench","source_record":"bm_overlooked-bench_a8b28295"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI models on 91 items across five tracks: ethics and philosophy, niche academic domains, small-organisation tasks, education, and matched institutional-criticism prompts, using rubric-based scoring and cross-judge validation.","descriptions":{"en":"Evaluates AI models on 91 items across five tracks: ethics and philosophy, niche academic domains, small-organisation tasks, education, and matched institutional-criticism prompts, using rubric-based scoring and cross-judge validation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:kcelestinomaria/overlooked-bench","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_overlooked-bench_a8b28295"},"modality":null,"name":"overlooked-bench","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:kcelestinomaria/overlooked-bench","source_url":"https://github.com/kcelestinomaria/overlooked-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-kcelestinomaria-overlooked-bench","source":"claire_radar","source_url":"https://github.com/kcelestinomaria/overlooked-bench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates VLMs on online video question answering under user interruptions, with offline simulation and multi-dimensional metrics.","descriptions":{"en":"Evaluates VLMs on online video question answering under user interruptions, with offline simulation and multi-dimensional metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22279","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_ovibench_2e515c0e"},"modality":null,"name":"OVIBench","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22279","source_url":"https://arxiv.org/abs/2608.22279"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22279","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.22279","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":56,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"claire_record_id","repository":"gh:internlm/ovo-s-bench","source_record":"bm_ovo-s-bench_86757d1b"},"categories":["Autonomous Driving","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"OVO-S-Bench evaluates streaming spatial intelligence in MLLMs with 1,680 human-annotated questions across four abstraction levels, using streaming prefixes and evidence intervals.","descriptions":{"en":"OVO-S-Bench evaluates streaming spatial intelligence in MLLMs with 1,680 human-annotated questions across four abstraction levels, using streaming prefixes and evidence intervals."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.03890","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","spatial-3d"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_ovo-s-bench_86757d1b"},"modality":null,"name":"OVO-S-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03890","source_url":"https://arxiv.org/abs/2606.03890"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03890","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03890","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":166,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":45,"join":"llm_stats_id","repository":null,"source_record":"catalog_9c94cb5745dfe63b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"OVOBench (Online Video Online Benchmark) evaluates streaming video understanding, testing a model's ability to perceive and respond to video content in real time as it unfolds.","descriptions":{"en":"OVOBench (Online Video Online Benchmark) evaluates streaming video understanding, testing a model's ability to perceive and respond to video content in real time as it unfolds."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ovobench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ovobench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ovobench","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_9c94cb5745dfe63b"},"modality":"multimodal","name":"OVOBench","openness":"unknown","publisher":null,"released":"2025-01-09","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:ovobench","source_url":"https://arxiv.org/abs/2501.05510"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":80.7,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.807,"raw_min":0.792,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ovobench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ovobench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-ovobench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ovobench?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"P-MMEval is a comprehensive multilingual multitask benchmark, covering effective fundamental and capability-specialized datasets. P-MMEval是一个全面的多语言多任务基准，涵盖了高效的基础和专项能力数据集。","descriptions":{"en":"P-MMEval is a comprehensive multilingual multitask benchmark, covering effective fundamental and capability-specialized datasets.","zh":"P-MMEval是一个全面的多语言多任务基准，涵盖了高效的基础和专项能力数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1283","languages":["Multilingual"],"library_direction":null,"modality":null,"name":"P-MMEval","openness":"unknown","publisher":"Tongyi Lab, Alibaba Group Inc","released":"2024-11-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1283-p-mmeval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/P-MMEval","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_p3b3_5a9c84e4"},"categories":["General AI","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"P3B3 is an expert-curated benchmark of conversational prompts for evaluating variety bias and controllability in LLMs across European and Brazilian Portuguese.","descriptions":{"en":"P3B3 is an expert-curated benchmark of conversational prompts for evaluating variety bias and controllability in LLMs across European and Brazilian Portuguese."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16753","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","chat-writing"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_p3b3_5a9c84e4"},"modality":null,"name":"P3B3","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16753","source_url":"https://arxiv.org/abs/2606.16753"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16753","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16753","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":54,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:spatiaos/p3d-bench","source_record":"bm_p3d-bench_31251204"},"categories":["General AI","Geometric reasoning","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"P3D-Bench evaluates multimodal large language models on parametric 3D generation from text, image, and assembly specifications, scoring executability, geometric fidelity, topology, text-grounded constraints, multiview semantic alignment, and part-level structure.","descriptions":{"en":"P3D-Bench evaluates multimodal large language models on parametric 3D generation from text, image, and assembly specifications, scoring executability, geometric fidelity, topology, text-grounded constraints, multiview semantic alignment, and part-level structure."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.11152","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","code-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_p3d-bench_31251204"},"modality":null,"name":"P3D-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11152","source_url":"https://arxiv.org/abs/2606.11152"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11152","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11152","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":29,"join":"claire_record_id","repository":"gh:thunlp/pace-bench","source_record":"bm_pace-bench_0530dc3f"},"categories":["Agents","General AI","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"PACE-Bench is a simulator-grounded benchmark with 144 source-to-target adaptation pairs across six physics domains. Each pair presents a code-driven design that succeeds in a source environment but fails in a mutated target environment, and agents must iteratively adapt the design using diagnostic sandbox feedback within a limited attempt budget.","descriptions":{"en":"PACE-Bench is a simulator-grounded benchmark with 144 source-to-target adaptation pairs across six physics domains. Each pair presents a code-driven design that succeeds in a source environment but fails in a mutated target environment, and agents must iteratively adapt the design using diagnostic sandbox feedback within a limited attempt budget."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.14441","languages":[],"library_direction":{"basis":"model-review","categories":["science","code-generation"],"join":"claire_record_id","primary":"science","source_record":"bm_pace-bench_0530dc3f"},"modality":null,"name":"PACE-Bench","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14441","source_url":"https://arxiv.org/abs/2608.14441"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14441","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14441","topics":["coding","agents","science"],"unit":null},{"aliases":["PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?"],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":222,"hf_paper_upvotes":41,"join":"claire_record_id","repository":"gh:trace-ai-labs/pact","source_record":"bm_pact-can-enterprise-ai-assistants-be-trust_52300df4"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PACT evaluates LLM assistants' compliance with standing rules under nine realistic pressures across 48 scenarios and 12 regulated domains, scoring multi-turn decisions with PACTScore and six axes.","descriptions":{"en":"PACT evaluates LLM assistants' compliance with standing rules under nine realistic pressures across 48 scenarios and 12 regulated domains, scoring multi-turn decisions with PACTScore and six axes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.18605","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","professional-work","safety-alignment"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_pact-can-enterprise-ai-assistants-be-trust_52300df4"},"modality":null,"name":"PACT","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18605","source_url":"https://arxiv.org/abs/2609.18605"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18605","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18605","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PACUTE is a diagnostic benchmark of 4,600 tasks evaluating morphological understanding in Filipino, covering six compositional levels from morpheme decomposition to syllabification.","descriptions":{"en":"PACUTE is a diagnostic benchmark of 4,600 tasks evaluating morphological understanding in Filipino, covering six compositional levels from morpheme decomposition to syllabification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15144","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_pacute_04643d6a"},"modality":null,"name":"PACUTE","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15144","source_url":"https://arxiv.org/abs/2606.15144"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15144","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15144","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:paintbench/paintbench","source_record":"bm_paintbench_3ad5e836"},"categories":["Reasoning","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PaintBench is a procedurally generated benchmark for precise visual editing, covering 20 tasks across geometric, structural, color, and symbolic categories. Stable scoring uses pixel-level mIoU with fixed seeds.","descriptions":{"en":"PaintBench is a procedurally generated benchmark for precise visual editing, covering 20 tasks across geometric, structural, color, and symbolic categories. Stable scoring uses pixel-level mIoU with fixed seeds."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.00188","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_paintbench_3ad5e836"},"modality":null,"name":"PaintBench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00188","source_url":"https://arxiv.org/abs/2606.00188"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00188","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00188","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_painterbench_86788dd4"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Figural divergent thinking is the ability to develop a given shape fragment into an original drawing.","descriptions":{"en":"Figural divergent thinking is the ability to develop a given shape fragment into an original drawing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34195","languages":[],"library_direction":{"basis":"model-review-2","categories":["image-video-generation","chat-writing"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_painterbench_86788dd4"},"modality":null,"name":"PainterBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34195","source_url":"https://arxiv.org/abs/2609.34195"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34195","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34195","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"PAIR-Bench evaluates code improvement by transforming incorrect programs into more correct ones through feedback-guided refinement. It uses progressive hinting with failure-region and hint-depth controls to measure repair of targeted failures, generalization beyond the hint, preservation of correct behavior, and required assistance.","descriptions":{"en":"PAIR-Bench evaluates code improvement by transforming incorrect programs into more correct ones through feedback-guided refinement. It uses progressive hinting with failure-region and hint-depth controls to measure repair of targeted failures, generalization beyond the hint, preservation of correct behavior, and required assistance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01360","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_pair-bench_c3cd8d09"},"modality":null,"name":"PAIR-Bench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01360","source_url":"https://arxiv.org/abs/2607.01360"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01360","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01360","topics":["coding"],"unit":null},{"aliases":["Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"Pak3H evaluates Helpfulness, Harmlessness, and Honesty alignment in Urdu through three suites: PakAlpaca, PakBeaverTails, and PakTruthfulQA.","descriptions":{"en":"Pak3H evaluates Helpfulness, Harmlessness, and Honesty alignment in Urdu through three suites: PakAlpaca, PakBeaverTails, and PakTruthfulQA."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30065","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_pak3h-evaluating-the-cost-of-cultural-mism_9e0d72ff"},"modality":null,"name":"Pak3H","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30065","source_url":"https://arxiv.org/abs/2608.30065"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30065","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30065","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PAL-Bench evaluates evidence-grounded profile reconstruction from longitudinal personal albums, scoring agents on owner facts, identities, and relations with a seven-metric protocol.","descriptions":{"en":"PAL-Bench evaluates evidence-grounded profile reconstruction from longitudinal personal albums, scoring agents on owner facts, identities, and relations with a seven-metric protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16175","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","visual-reasoning"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_pal-bench_e5d27a42"},"modality":null,"name":"PAL-Bench","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16175","source_url":"https://arxiv.org/abs/2606.16175"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16175","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16175","topics":["agents"],"unit":null},{"aliases":["Pancasila-Dilemmas: Evaluating Large Language Models on Indonesian Human Value Dilemmas Grounded in Pancasila"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"An evaluation dataset of 1,834 dilemma questions from Indonesian news that measures LLM alignment with the five Pancasila values, scored by PMS and MVAS against answers from Indonesian annotators.","descriptions":{"en":"An evaluation dataset of 1,834 dilemma questions from Indonesian news that measures LLM alignment with the five Pancasila values, scored by PMS and MVAS against answers from Indonesian annotators."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.18066","languages":[],"library_direction":null,"modality":null,"name":"Pancasila-Dilemmas","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18066","source_url":"https://arxiv.org/abs/2607.18066"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18066","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18066","topics":[],"unit":null},{"aliases":["onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"A dataset and benchmark for token-level correction in LLM and agent alignment data.","descriptions":{"en":"A dataset and benchmark for token-level correction in LLM and agent alignment data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24983","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_onpanda-efficient-annotation-of-on-policy-_2ad49278"},"modality":null,"name":"Panda-CVL","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24983","source_url":"https://arxiv.org/abs/2609.24983"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24983","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24983","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1312,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":37,"join":"llm_stats_id","repository":null,"source_record":"catalog_d2e6584907a815aa"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PaperBench is a benchmark for evaluating AI agents on their ability to replicate research papers. It tests models on complex, multi-step workflows involving code implementation, experimentation, and reproducing scientific results from academic publications.","descriptions":{"en":"PaperBench is a benchmark for evaluating AI agents on their ability to replicate research papers. It tests models on complex, multi-step workflows involving code implementation, experimentation, and reproducing scientific results from academic publications."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:paperbench:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/paperbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:paperbench","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_d2e6584907a815aa"},"modality":"text","name":"PaperBench","openness":"unknown","publisher":null,"released":"2025-04-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: PaperBench: Evaluating AI's Ability to Replicate AI Research. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:paperbench","source_url":"https://arxiv.org/abs/2504.01848"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":93.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.93,"raw_min":0.526,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:paperbench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/paperbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-paperbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/paperbench?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1312,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":37,"join":"artifact:arxiv:2504.01848","repository":null,"source_record":"catalog_d2e6584907a815aa"},"categories":["理解","Understanding","智能体","Agent","任务执行","Task Execution","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PaperBench is a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research. PaperBench，这是一个评估AI代理复制最新AI研究能力的基准测试。","descriptions":{"en":"PaperBench is a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research.","zh":"PaperBench，这是一个评估AI代理复制最新AI研究能力的基准测试。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1730","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"artifact:arxiv:2504.01848","primary":"ai-rnd","source_record":"catalog_d2e6584907a815aa"},"modality":null,"name":"PaperBench","openness":"unknown","publisher":"OpenAI","released":"2025-04-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1730-paperbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PaperBench","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:pkgunboat/paraguibench","source_record":"bm_paraguibench_bf12bca6"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ParaGUIBench aims to benchmark parallel execution and coordination of multiple GUI agents across separate desktop instances, with a dataset of 233 tasks and efficiency metrics.","descriptions":{"en":"ParaGUIBench aims to benchmark parallel execution and coordination of multiple GUI agents across separate desktop instances, with a dataset of 233 tasks and efficiency metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.22689","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_paraguibench_bf12bca6"},"modality":null,"name":"ParaGUIBench","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22689","source_url":"https://arxiv.org/abs/2607.22689"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22689","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22689","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"ParamBench is a benchmark for evaluating LLM tool call parameter generation, built from real cloud-network APIs with difficulty tiers and exact match metrics. It is used to evaluate the proposed probe-guided training framework.","descriptions":{"en":"ParamBench is a benchmark for evaluating LLM tool call parameter generation, built from real cloud-network APIs with difficulty tiers and exact match metrics. It is used to evaluate the proposed probe-guided training framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03071","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_parambench_25e30479"},"modality":null,"name":"ParamBench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03071","source_url":"https://arxiv.org/abs/2608.03071"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03071","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03071","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important.","descriptions":{"en":"As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.08662","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_paranoiaeval_0cbed6e6"},"modality":null,"name":"ParanoiaEval","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.08662","source_url":"https://arxiv.org/abs/2610.08662"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-08662","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08662","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"ParaPairAudioBench evaluates LALMs as judges for paralinguistic speech across five dimensions with 5,175 audio pairs.","descriptions":{"en":"ParaPairAudioBench evaluates LALMs as judges for paralinguistic speech across five dimensions with 5,175 audio pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.24648","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_parapairaudiobench_e7d0c4ed"},"modality":null,"name":"ParaPairAudioBench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24648","source_url":"https://arxiv.org/abs/2606.24648"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24648","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24648","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:gbw206/pararecover","source_record":"bm_pararecover_c43c0d4d"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ParaRecover evaluates LLM-based parallel tool-use agents on error localization and recovery, testing whether agents detect execution errors, diagnose root causes, and repair plans when tool calls fail. It scores replanner output across structural, diagnostic, and evolutionary dimensions.","descriptions":{"en":"ParaRecover evaluates LLM-based parallel tool-use agents on error localization and recovery, testing whether agents detect execution errors, diagnose root causes, and repair plans when tool calls fail. It scores replanner output across structural, diagnostic, and evolutionary dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.12345","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_pararecover_c43c0d4d"},"modality":null,"name":"ParaRecover","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12345","source_url":"https://arxiv.org/abs/2609.12345"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12345","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12345","topics":["agents"],"unit":null},{"aliases":["PARARULE-Plus","PARARULE Plus","ParaRule-Plus"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"Synthetic multi-step deductive reasoning over natural-language facts and rules under a closed-world assumption, with about 100,000 examples at each reasoning depth from 2 to 5 (nearly 400,000 in total) over animal and people entities. An aggregate number hides the per-depth breakdown the dataset was built to expose, and a model fine-tuned on the templated training split is not comparable with one prompted zero-shot.","descriptions":{"en":"Synthetic multi-step deductive reasoning over natural-language facts and rules under a closed-world assumption, with about 100,000 examples at each reasoning depth from 2 to 5 (nearly 400,000 in total) over animal and people entities. An aggregate number hides the per-depth breakdown the dataset was built to expose, and a model fine-tuned on the templated training split is not comparable with one prompted zero-shot."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:pararule_plus","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"PARARULE-Plus","openness":"unknown","publisher":null,"released":"2022-07-28","released_reference":{"basis":"benchmark_release","source_key":"model-reports:pararule_plus","source_url":"https://arxiv.org/abs/2207.14000"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"pararule_plus","source":"model_reports","source_url":"https://arxiv.org/abs/2207.14000","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:shenshan123/parasgb","source_record":"bm_parasgb_cd57a968"},"categories":["Chip Design & EDA","Language & Knowledge","Manufacturing & Process Control","Software & AI Compute","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ParasGB evaluates graph neural network predictions of parasitic capacitance and resistance on circuit graphs from analog/mixed-signal designs, with node-level ground capacitance, edge-level resistance, and edge-level coupling capacitance tasks.","descriptions":{"en":"ParasGB evaluates graph neural network predictions of parasitic capacitance and resistance on circuit graphs from analog/mixed-signal designs, with node-level ground capacitance, edge-level resistance, and edge-level coupling capacitance tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.23225","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_parasgb_cd57a968"},"modality":null,"name":"ParasGB","openness":"unknown","publisher":null,"released":"2026-07-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23225","source_url":"https://arxiv.org/abs/2607.23225"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23225","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23225","topics":[],"unit":null},{"aliases":["ParsHate: A Benchmark Dataset for Hate and Target Detection in Persian"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_parshate-a-benchmark-dataset-for-hate-and-_eec12e59"},"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates hate speech and target detection on 10,000 manually annotated Persian tweets spanning 2013-2022.","descriptions":{"en":"Evaluates hate speech and target detection on 10,000 manually annotated Persian tweets spanning 2013-2022."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16393","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_parshate-a-benchmark-dataset-for-hate-and-_eec12e59"},"modality":null,"name":"ParsHate","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16393","source_url":"https://arxiv.org/abs/2609.16393"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16393","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16393","topics":["language","safety"],"unit":null},{"aliases":["particle-event-quantum-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:starswithboutaina/particle-event-quantum-benchmark","source_record":"bm_particle-event-quantum-benchmark_7b36a296"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates classical and future hybrid quantum classifiers on a synthetic particle-physics-inspired binary classification task with controlled overlap, noise, and imbalance; Phase 1 includes majority, logistic regression, and random forest baselines with held-out test set.","descriptions":{"en":"Evaluates classical and future hybrid quantum classifiers on a synthetic particle-physics-inspired binary classification task with controlled overlap, noise, and imbalance; Phase 1 includes majority, logistic regression, and random forest baselines with held-out test set."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:starswithboutaina/particle-event-quantum-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["world-models","science"],"join":"claire_record_id","primary":"world-models","source_record":"bm_particle-event-quantum-benchmark_7b36a296"},"modality":null,"name":"Particle Event Classification with Classical and Hybrid Quantum Models: A Reproducible Benchmark","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:starswithboutaina/particle-event-quantum-benchmark","source_url":"https://github.com/starswithboutaina/particle-event-quantum-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-starswithboutaina-particle-event-quantum-benchmark","source":"claire_radar","source_url":"https://github.com/starswithboutaina/particle-event-quantum-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_pasbench-video_ab3b3eb4"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"PaSBench-Video is a 740-video benchmark for proactive safety warning with 481 risk and 259 no-risk videos across driving, healthcare, daily life, and industrial production. Annotations include frame-level risk onset and accident boundaries. Models must process video causally and output a warning that is temporally calibrated and content-correct.","descriptions":{"en":"PaSBench-Video is a 740-video benchmark for proactive safety warning with 481 risk and 259 no-risk videos across driving, healthcare, daily life, and industrial production. Annotations include frame-level risk onset and accident boundaries. Models must process video causally and output a warning that is temporally calibrated and content-correct."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02443","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","safety-alignment"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_pasbench-video_ab3b3eb4"},"modality":null,"name":"PaSBench-Video","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02443","source_url":"https://arxiv.org/abs/2606.02443"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02443","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02443","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","理解","Understanding","语言","Language","OCR","NLP","Computer Vision","多模态模型","VLM","图像理解","Image Understanding","语言理解","Comprehension","OCR与文档理解","OCR and Document Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PsOCR is a large-scale synthetic dataset for Optical Character Recognition in low-resource Pashto language. PsOCR is a large-scale synthetic dataset for Optical Character Recognition in low-resource Pashto language.","descriptions":{"en":"PsOCR is a large-scale synthetic dataset for Optical Character Recognition in low-resource Pashto language.","zh":"PsOCR is a large-scale synthetic dataset for Optical Character Recognition in low-resource Pashto language."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1836","languages":["Arabic"],"library_direction":null,"modality":"multimodal","name":"PashtoOCR","openness":"unknown","publisher":"Zirak AI","released":"2025-05-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1836-pashtoocr","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PashtoOCR","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A dated automated screening benchmark for Pashto text-to-speech systems covering completion, transcription error, script fidelity, and audio-language identification.","descriptions":{"en":"A dated automated screening benchmark for Pashto text-to-speech systems covering completion, transcription error, script fidelity, and audio-language identification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26978","languages":[],"library_direction":null,"modality":null,"name":"PashtoTTS-Bench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26978","source_url":"https://arxiv.org/abs/2605.26978"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26978","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26978","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":34,"github_stars_30d":8,"hf_dataset_downloads":null,"hf_paper_upvotes":36,"join":"claire_record_id","repository":"gh:gen-verse/past-bench","source_record":"bm_past-bench_b04dad58"},"categories":["Agents","General AI","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"PAST-Bench evaluates recursive self-improvement in personal AI agents by testing whether retained experience improves performance on future tasks. It spans 26 scenarios and 204 episodes across memory, procedural reuse, information gathering, and update capabilities, with matched persistence on/off controls.","descriptions":{"en":"PAST-Bench evaluates recursive self-improvement in personal AI agents by testing whether retained experience improves performance on future tasks. It spans 26 scenarios and 204 episodes across memory, procedural reuse, information gathering, and update capabilities, with matched persistence on/off controls."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04003","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","agent-memory"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_past-bench_b04dad58"},"modality":null,"name":"PAST-Bench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04003","source_url":"https://arxiv.org/abs/2608.04003"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04003","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04003","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_patchbench_a8fb20cc"},"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents on realistic C/C++ vulnerability patching tasks where ground-truth fixes lie outside the crash stack, using vulnerability transplant and code mutations to reduce memorization and surface-level fixes.","descriptions":{"en":"Evaluates AI agents on realistic C/C++ vulnerability patching tasks where ground-truth fixes lie outside the crash stack, using vulnerability transplant and code mutations to reduce memorization and surface-level fixes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.04075","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","agentic-coding"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_patchbench_a8fb20cc"},"modality":null,"name":"PatchBench","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04075","source_url":"https://arxiv.org/abs/2609.04075"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04075","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04075","topics":["coding","agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_patchbench_b82c12c0"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"An LLM safety patch can pass a benchmark while still being a poor repair.","descriptions":{"en":"An LLM safety patch can pass a benchmark while still being a poor repair."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10276","languages":[],"library_direction":{"basis":"model-review-daily","categories":["cybersecurity","safety-alignment"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_patchbench_b82c12c0"},"modality":null,"name":"PatchBench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10276","source_url":"https://arxiv.org/abs/2610.10276"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10276","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10276","topics":["safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"PatchFuseBench is a fixed-pool benchmark for evaluating repair candidate fusion, built from existing SWE-bench Verified, SWE-bench Multilingual, and Defects4J candidate patches. The benchmark pools candidate patches for 500 bugs on SWE-bench Verified, 300 on Multilingual, and 371 on Defects4J, and evaluates methods that fuse or select a final patch.","descriptions":{"en":"PatchFuseBench is a fixed-pool benchmark for evaluating repair candidate fusion, built from existing SWE-bench Verified, SWE-bench Multilingual, and Defects4J candidate patches. The benchmark pools candidate patches for 500 bugs on SWE-bench Verified, 300 on Multilingual, and 371 on Defects4J, and evaluates methods that fuse or select a final patch."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01597","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_patchfusebench_d8b0e871"},"modality":null,"name":"PatchFuseBench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01597","source_url":"https://arxiv.org/abs/2607.01597"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01597","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01597","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Information retrieval","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"PathAgentBench evaluates vision-language models on whole-slide pathology images across four capabilities: image-to-text matching, text-to-image retrieval, diagnostic-region localization, and multi-scale reasoning. It includes 1,822 TCGA WSIs and 17,135 diagnostic paths.","descriptions":{"en":"PathAgentBench evaluates vision-language models on whole-slide pathology images across four capabilities: image-to-text matching, text-to-image retrieval, diagnostic-region localization, and multi-scale reasoning. It includes 1,822 TCGA WSIs and 17,135 diagnostic paths."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19261","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_pathagentbench_70491a6b"},"modality":null,"name":"PathAgentBench","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19261","source_url":"https://arxiv.org/abs/2607.19261"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19261","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19261","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Pathology vision-language models (VLMs) have shown strong visual perception ability, but their robustness in the language domain remains poorly characterized.","descriptions":{"en":"Pathology vision-language models (VLMs) have shown strong visual perception ability, but their robustness in the language domain remains poorly characterized."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11329","languages":[],"library_direction":{"basis":"model-review-daily","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_pathlang_623a7342"},"modality":null,"name":"PathLang","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11329","source_url":"https://arxiv.org/abs/2610.11329"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11329","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11329","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_a287489a87c78047"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PathMMU is a massive multimodal expert-level benchmark for understanding and reasoning in pathology, containing 33,428 multimodal multi-choice questions and 24,067 images validated by seven pathologists. It evaluates Large Multimodal Models (LMMs) performance on pathology tasks, with the top-performing model GPT-4V achieving only 49.8% zero-shot performance compared to 71.8% for human pathologists.","descriptions":{"en":"PathMMU is a massive multimodal expert-level benchmark for understanding and reasoning in pathology, containing 33,428 multimodal multi-choice questions and 24,067 images validated by seven pathologists. It evaluates Large Multimodal Models (LMMs) performance on pathology tasks, with the top-performing model GPT-4V achieving only 49.8% zero-shot performance compared to 71.8% for human pathologists."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:pathmcqa:medgemma-4b-it","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pathmcqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:pathmcqa","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_a287489a87c78047"},"modality":"multimodal","name":"PathMCQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":69.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.698,"raw_min":0.698,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:pathmcqa:medgemma-4b-it","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pathmcqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-20","date_precision":"model_announcement","name":"MedGemma 4B IT"}],"slug":"llm-stats-pathmcqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pathmcqa?top_n=500","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":48,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_pathoargus-bench_2bed53e5"},"categories":["Biology & Drug Discovery","Evidence grounding","Long Context","Long-context reasoning","Medical image reasoning","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"PathoArgus-Bench evaluates evidence-grounded visual reasoning in whole-slide pathology. It comprises 22,078 multiple-choice questions from 4,913 patients across 15 TCGA projects, testing availability, accessibility, use, and responsiveness of evidence under a fixed reader budget.","descriptions":{"en":"PathoArgus-Bench evaluates evidence-grounded visual reasoning in whole-slide pathology. It comprises 22,078 multiple-choice questions from 4,913 patients across 15 TCGA projects, testing availability, accessibility, use, and responsiveness of evidence under a fixed reader budget."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17607","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_pathoargus-bench_2bed53e5"},"modality":null,"name":"PathoArgus-Bench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17607","source_url":"https://arxiv.org/abs/2608.17607"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17607","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17607","topics":["reasoning-math","long-context","medical"],"unit":null},{"aliases":["PathReportEval: A Systematic Benchmark for Pathology Report Generation"],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A standardized benchmark for generating pathology reports from whole-slide images. It covers TCGA, HistAI and REG 2025 with three foundation encoders and scores reports with the Clinical Report Quality Score (CRQS).","descriptions":{"en":"A standardized benchmark for generating pathology reports from whole-slide images. It covers TCGA, HistAI and REG 2025 with three foundation encoders and scores reports with the Clinical Report Quality Score (CRQS)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.18448","languages":[],"library_direction":null,"modality":null,"name":"PathReportEval","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18448","source_url":"https://arxiv.org/abs/2607.18448"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18448","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18448","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"PathVU is a benchmark for fine-grained multiscale visual understanding in pathology, built from 23 public datasets with human-supervised labels and spatial annotations, covering region and slide fields of view, with 14 VQA-style tasks and 308k samples.","descriptions":{"en":"PathVU is a benchmark for fine-grained multiscale visual understanding in pathology, built from 23 public datasets with human-supervised labels and spatial annotations, covering region and slide fields of view, with 14 VQA-style tasks and 308k samples."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.28318","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_pathview-bench_295675e0"},"modality":null,"name":"PathView-Bench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28318","source_url":"https://arxiv.org/abs/2607.28318"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28318","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28318","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark framework that evaluates patient-facing healthcare agents in simulated, tool-using conversations with patients, scored by an LLM jury against clinician-grounded criteria.","descriptions":{"en":"A benchmark framework that evaluates patient-facing healthcare agents in simulated, tool-using conversations with patients, scored by an LLM jury against clinician-grounded criteria."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.25485","languages":[],"library_direction":null,"modality":null,"name":"PatientAgentBench","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25485","source_url":"https://arxiv.org/abs/2607.25485"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25485","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25485","topics":[],"unit":null},{"aliases":["Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":35,"join":"claire_record_id","repository":null,"source_record":"bm_beyond-correctness-benchmarking-and-aligni_75c36d61"},"categories":["Factuality","General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"PatternEval is a diagnostic benchmark with 2,415 multimodal prompts testing four response-pattern failures: chain-of-thought leakage, repetition, contradiction, and performative reasoning.","descriptions":{"en":"PatternEval is a diagnostic benchmark with 2,415 multimodal prompts testing four response-pattern failures: chain-of-thought leakage, repetition, contradiction, and performative reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12781","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_beyond-correctness-benchmarking-and-aligni_75c36d61"},"modality":null,"name":"PatternEval","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12781","source_url":"https://arxiv.org/abs/2608.12781"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12781","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12781","topics":["multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A user-centric benchmark for evaluating personal AI assistants in stateful, service-integrated environments, with multi-regime evaluation and user simulation for long-horizon tasks.","descriptions":{"en":"A user-centric benchmark for evaluating personal AI assistants in stateful, service-integrated environments, with multi-regime evaluation and user simulation for long-horizon tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27354","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","agent-memory"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_pause_e497d85d"},"modality":null,"name":"PAUSE","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27354","source_url":"https://arxiv.org/abs/2607.27354"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27354","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27354","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":61,"join":"claire_record_id","repository":null,"source_record":"bm_pawbench_0877eb45"},"categories":["General AI","Safety","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video generators as stochastic samplers of world dynamics across 50 scenarios, comparing the distribution of possible behaviors under identical initial observations and actions.","descriptions":{"en":"Evaluates video generators as stochastic samplers of world dynamics across 50 scenarios, comparing the distribution of possible behaviors under identical initial observations and actions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27345","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_pawbench_0877eb45"},"modality":null,"name":"PAWBench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27345","source_url":"https://arxiv.org/abs/2608.27345"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27345","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27345","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","ACL 2024","物理智能","Embodied AI","具身交互","Embodied Interaction","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PCA-Bench is a multimodal decisionmaking benchmark for evaluating the integrated capabilities of Multimodal Large Language Models (MLLMs). Departing from previous\nbenchmarks focusing on simplistic tasks and individual model capability. PCA-Bench 是一个多模态决策基准，用于评估多模态大型语言模型（MLLMs）的综合能力。与之前专注于简单任务和单个模型能力的基准不同，PCA-Bench 引入了三个复杂场景：自动驾驶、家庭机器人和开放世界游戏。","descriptions":{"en":"PCA-Bench is a multimodal decisionmaking benchmark for evaluating the integrated capabilities of Multimodal Large Language Models (MLLMs). Departing from previous\nbenchmarks focusing on simplistic tasks and individual model capability.","zh":"PCA-Bench 是一个多模态决策基准，用于评估多模态大型语言模型（MLLMs）的综合能力。与之前专注于简单任务和单个模型能力的基准不同，PCA-Bench 引入了三个复杂场景：自动驾驶、家庭机器人和开放世界游戏。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1076","languages":[],"library_direction":null,"modality":null,"name":"PCA-Bench","openness":"open","publisher":"Alibaba","released":"2024-02-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1076-pca-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PCA-Bench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Consists of over 300 real-world PCB schematic images with annotated pins and paired SPICE netlists, intended for schematic-to-netlist conversion evaluation.","descriptions":{"en":"Consists of over 300 real-world PCB schematic images with annotated pins and paired SPICE netlists, intended for schematic-to-netlist conversion evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27923","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["engineering-design","visual-reasoning"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_pcbnet_ffc56858"},"modality":null,"name":"PCBnet","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27923","source_url":"http://arxiv.org/abs/2608.27923v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27923","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.27923v1","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Planning","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language model (LLM) agents are evolving from tool-calling systems that execute isolated instructions into task-oriented agents that pursue user goals through sustained, multi-step interactions.","descriptions":{"en":"Large language model (LLM) agents are evolving from tool-calling systems that execute isolated instructions into task-oriented agents that pursue user goals through sustained, multi-step interactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34930","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_pdeu-bench_a6b04400"},"modality":null,"name":"PDEU-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34930","source_url":"https://arxiv.org/abs/2609.34930"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34930","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34930","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PeakBench is a benchmark of executable multi-tool workflows for evaluating resource-aware tool invocation in LLM agents. It includes execution-grounded dependency annotations and measured resource profiles, with a two-part evaluation framework distinguishing logical planning from physical scheduling. Code is available on GitHub.","descriptions":{"en":"PeakBench is a benchmark of executable multi-tool workflows for evaluating resource-aware tool invocation in LLM agents. It includes execution-grounded dependency annotations and measured resource profiles, with a two-part evaluation framework distinguishing logical planning from physical scheduling. Code is available on GitHub."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.24509","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_peakbench_8142057d"},"modality":null,"name":"PeakBench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24509","source_url":"https://arxiv.org/abs/2608.24509"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24509","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24509","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":29,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"claire_record_id","repository":"gh:sphere-ai-lab/peft-arena","source_record":"bm_peft-arena_b7338aaa"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"PEFT-Arena benchmark jointly evaluates target-domain performance and retention of pretrained capabilities for parameter-efficient finetuning methods. It covers mathematical and medical reasoning as target domains and measures general capability retention on benchmarks including BBH, IFEval, and NQ with SFT and RLVR training settings.","descriptions":{"en":"PEFT-Arena benchmark jointly evaluates target-domain performance and retention of pretrained capabilities for parameter-efficient finetuning methods. It covers mathematical and medical reasoning as target domains and measures general capability retention on benchmarks including BBH, IFEval, and NQ with SFT and RLVR training settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.28819","languages":[],"library_direction":{"basis":"model-review-2","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_peft-arena_b7338aaa"},"modality":null,"name":"PEFT-Arena","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28819","source_url":"https://arxiv.org/abs/2605.28819"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28819","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28819","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aistairc/peg-in-bench","source_record":"bm_peg-in-bench_93ef3dba"},"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates robotic insertion performance on a reconfigurable peg-in-hole setup with modular components, emphasizing task generalization across varied high-precision insertion scenarios.","descriptions":{"en":"Evaluates robotic insertion performance on a reconfigurable peg-in-hole setup with modular components, emphasizing task generalization across varied high-precision insertion scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.00906","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_peg-in-bench_93ef3dba"},"modality":null,"name":"Peg-in-Bench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00906","source_url":"https://arxiv.org/abs/2609.00906"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00906","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00906","topics":[],"unit":null},{"aliases":["pelican-bike-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:qrzzzz/pelican-bike-benchmark","source_record":"bm_pelican-bike-benchmark_593593ff"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI-generated single-file HTML implementations of a pelican-riding-a-bicycle prompt using a fixed prompt and static site, with optional scoring across runnable, visual, animation, accessibility, and quality dimensions.","descriptions":{"en":"Evaluates AI-generated single-file HTML implementations of a pelican-riding-a-bicycle prompt using a fixed prompt and static site, with optional scoring across runnable, visual, animation, accessibility, and quality dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:qrzzzz/pelican-bike-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Pelican Bike Benchmark","openness":"unknown","publisher":null,"released":"2026-09-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:qrzzzz/pelican-bike-benchmark","source_url":"https://github.com/Qrzzzz/pelican-bike-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-qrzzzz-pelican-bike-benchmark","source":"claire_radar","source_url":"https://github.com/Qrzzzz/pelican-bike-benchmark","topics":[],"unit":null},{"aliases":["Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates poetic logic in modern Chinese poetry through four tasks across stanza, line, and imagery levels using six mainstream LLMs.","descriptions":{"en":"Evaluates poetic logic in modern Chinese poetry through four tasks across stanza, line, and imagery levels using six mainstream LLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.21827","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning","multilingual"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_do-large-language-models-perform-well-on-c_54280d5e"},"modality":null,"name":"Peony","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.21827","source_url":"http://arxiv.org/abs/2608.21827v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-21827","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21827v1","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"claire_record_id","repository":null,"source_record":"bm_perceptionbench_27ae03ae"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"PerceptionBench evaluates atomic visual perception in MLLMs with 3,000 verified questions isolating ten perceptual capabilities, based on an error taxonomy from 42 benchmarks.","descriptions":{"en":"PerceptionBench evaluates atomic visual perception in MLLMs with 3,000 verified questions isolating ten perceptual capabilities, based on an error taxonomy from 42 benchmarks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24957","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_perceptionbench_27ae03ae"},"modality":null,"name":"PerceptionBench","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24957","source_url":"https://arxiv.org/abs/2607.24957"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24957","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24957","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"llm_stats_id","repository":null,"source_record":"bm_perceptionbench_27ae03ae"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PerceptionBench is Moonshot AI's internal benchmark for evaluating atomic visual perception capabilities.","descriptions":{"en":"PerceptionBench is Moonshot AI's internal benchmark for evaluating atomic visual perception capabilities."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:perceptionbench:kimi-k3","reported_at":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptionbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:perceptionbench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"bm_perceptionbench_27ae03ae"},"modality":"multimodal","name":"PerceptionBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":63.5,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.635,"raw_min":0.585,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:perceptionbench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptionbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-perceptionbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptionbench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":259,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:deepmind/perception_test","source_record":"catalog_9453ef18dc314e47"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","physics","reasoning","spatial_reasoning","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A novel multimodal video benchmark designed to evaluate perception and reasoning skills of pre-trained models across video, audio, and text modalities. Contains 11.6k real-world videos (average 23 seconds) filmed by participants worldwide, densely annotated with six types of labels. Focuses on skills (Memory, Abstraction, Physics, Semantics) and reasoning types (descriptive, explanatory, predictive, counterfactual). Shows significant performance gap between human baseline (91.4%) and state-of-the-art video QA models (46.2%).","descriptions":{"en":"A novel multimodal video benchmark designed to evaluate perception and reasoning skills of pre-trained models across video, audio, and text modalities. Contains 11.6k real-world videos (average 23 seconds) filmed by participants worldwide, densely annotated with six types of labels. Focuses on skills (Memory, Abstraction, Physics, Semantics) and reasoning types (descriptive, explanatory, predictive, counterfactual). Shows significant performance gap between human baseline (91.4%) and state-of-the-art video QA models (46.2%)."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:perceptiontest:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptiontest?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:perceptiontest","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_9453ef18dc314e47"},"modality":"multimodal","name":"PerceptionTest","openness":"unknown","publisher":null,"released":"2023-05-23","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Perception Test: A Diagnostic Benchmark for Multimodal Video Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:perceptiontest","source_url":"https://arxiv.org/abs/2305.13786"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":73.2,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.732,"raw_min":0.705,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:perceptiontest:qwen2.5-vl-72b","reported_date":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptiontest?top_n=500"},"unit":null},"scored_models":[{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-perceptiontest","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptiontest?top_n=500","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":["PereStruct: Multimodal Semantic Assembly for Robust Historical Document Parsing"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"A 93-page benchmark for mapping detected blocks from complex historical newspaper layouts to their corresponding articles.","descriptions":{"en":"A 93-page benchmark for mapping detected blocks from complex historical newspaper layouts to their corresponding articles."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07661","languages":[],"library_direction":null,"modality":null,"name":"PereStruct benchmark","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07661","source_url":"https://arxiv.org/abs/2606.07661"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07661","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07661","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_perfopt-bench_3a913e78"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"PERFOPT-Bench evaluates coding agents on software performance optimization tasks, requiring profiling, diagnosing bottlenecks, editing code while preserving correctness, and verifying reproducible speedups. Scoring includes hidden correctness tests, verified-speedup measurement, and trajectory-level audit.","descriptions":{"en":"PERFOPT-Bench evaluates coding agents on software performance optimization tasks, requiring profiling, diagnosing bottlenecks, editing code while preserving correctness, and verifying reproducible speedups. Scoring includes hidden correctness tests, verified-speedup measurement, and trajectory-level audit."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.07744","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_perfopt-bench_3a913e78"},"modality":null,"name":"PERFOPT-Bench","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.07744","source_url":"https://arxiv.org/abs/2607.07744"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-07744","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07744","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":13,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":45,"join":"claire_record_id","repository":"gh:yeonjun-in/permembench","source_record":"bm_permembench_15a1b01c"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PerMemBench evaluates personalized memory systems for LLM agents using multi-year, multi-domain interaction histories across 20 user personas, measuring memory retention accuracy.","descriptions":{"en":"PerMemBench evaluates personalized memory systems for LLM agents using multi-year, multi-domain interaction histories across 20 user personas, measuring memory retention accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.25535","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_permembench_15a1b01c"},"modality":null,"name":"PerMemBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25535","source_url":"https://arxiv.org/abs/2605.25535"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25535","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25535","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PersLitEval is a benchmark of 4,514 Persian literature multiple-choice questions across eight categories including spelling, literary devices, grammar, vocabulary, word formation, and conceptual understanding, sourced from Konkur university entrance exam materials.","descriptions":{"en":"PersLitEval is a benchmark of 4,514 Persian literature multiple-choice questions across eight categories including spelling, literary devices, grammar, vocabulary, word formation, and conceptual understanding, sourced from Konkur university entrance exam materials."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27015","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","multilingual"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_persliteval_aca10df9"},"modality":null,"name":"PersLitEval","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27015","source_url":"https://arxiv.org/abs/2605.27015"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27015","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27015","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates personalized text generation through LUAR, LLM-as-judge, and stylometrics across 50 authors and 1,000 generations.","descriptions":{"en":"Evaluates personalized text generation through LUAR, LLM-as-judge, and stylometrics across 50 authors and 1,000 generations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.19746","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["chat-writing","agent-memory"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_personalbench_05e1f48e"},"modality":null,"name":"PersonalBench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.19746","source_url":"https://arxiv.org/abs/2608.19746"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-19746","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19746","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","智能体","Agent","AI Assistant","personalization","任务执行","Task Execution","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PersonaLens  a large-scale benchmark specifically designed to evaluate personalization in task-oriented dialogues. The benchmark features 1,500 in-depth user profiles, each integrating real demographic data, detailed cross-domain preferences, and rich interaction histories. PersonaLens是一个专为任务导向型对话设计的、大规模的个性化能力评测基准。它包含1,500个深度用户画像，每个画像都集成了真实的人口统计信息、详尽的个人偏好及历史互动记录。这些画像与覆盖20个领域的111项真实世界任务相结合，并辅以动态的“情景上下文”来模拟现实世界的复杂性。为了实现自动化、可扩展的评估，该基准引入了两个LLM驱动的智能体：一个“用户智能体”负责模拟真人与AI进行对话，另一个“评判智能体”则对个性化水平、任务成功率和对话质量进行系统性打分。PersonaLens旨在为研究社区提供一个强大可靠的工具，共同推动下一代更懂你、更智能的AI助手的研发。","descriptions":{"en":"PersonaLens  a large-scale benchmark specifically designed to evaluate personalization in task-oriented dialogues. The benchmark features 1,500 in-depth user profiles, each integrating real demographic data, detailed cross-domain preferences, and rich interaction histories.","zh":"PersonaLens是一个专为任务导向型对话设计的、大规模的个性化能力评测基准。它包含1,500个深度用户画像，每个画像都集成了真实的人口统计信息、详尽的个人偏好及历史互动记录。这些画像与覆盖20个领域的111项真实世界任务相结合，并辅以动态的“情景上下文”来模拟现实世界的复杂性。为了实现自动化、可扩展的评估，该基准引入了两个LLM驱动的智能体：一个“用户智能体”负责模拟真人与AI进行对话，另一个“评判智能体”则对个性化水平、任务成功率和对话质量进行系统性打分。PersonaLens旨在为研究社区提供一个强大可靠的工具，共同推动下一代更懂你、更智能的AI助手的研发。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1959","languages":[],"library_direction":null,"modality":null,"name":"PersonaLens","openness":"unknown","publisher":"University of Edinburgh, Amazon, UCL","released":"2025-06-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1959-personalens","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PersonaLens","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_personashot_741ba748"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PersonaShot benchmarks person-centric narrative continuity in multi-shot video generation with ~1,000 segments and 16 metrics.","descriptions":{"en":"PersonaShot benchmarks person-centric narrative continuity in multi-shot video generation with ~1,000 segments and 16 metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.16717","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_personashot_741ba748"},"modality":null,"name":"PersonaShot","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16717","source_url":"https://arxiv.org/abs/2608.16717"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16717","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16717","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PersonaTrail evaluates personalized web agents using realistic browsing trajectories as user history, assessing preference inference and information recall. Operates in a managed open web environment with two tasks.","descriptions":{"en":"PersonaTrail evaluates personalized web agents using realistic browsing trajectories as user history, assessing preference inference and information recall. Operates in a managed open web environment with two tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20482","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_personatrail_ac04b54e"},"modality":null,"name":"PersonaTrail","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20482","source_url":"https://arxiv.org/abs/2607.20482"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20482","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20482","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_perspectivegap_bc1804c7"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"PerspectiveGap is a benchmark with 110 scenarios for evaluating multi-agent orchestration prompting, using distractor-mixed tasks and topologies from the authors' practice.","descriptions":{"en":"PerspectiveGap is a benchmark with 110 scenarios for evaluating multi-agent orchestration prompting, using distractor-mixed tasks and topologies from the authors' practice."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.08878","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_perspectivegap_bc1804c7"},"modality":null,"name":"PerspectiveGap","openness":"unknown","publisher":null,"released":"2026-06-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08878","source_url":"https://arxiv.org/abs/2606.08878"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08878","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08878","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","科学智能","AI for Science","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PertEval-scFM is a standardized framework to evaluate single-cell foundation models (scFMs) for predicting cellular perturbation effects. PertEval-scFM 是一个评估单细胞基础模型（scFM）扰动效应预测的标准化框架。它从分布偏移泛化性、扰动强度及上下文对齐三方面评测。测试集包含Norman、Replogle等数据集，涵盖数千扰动样本和基因。","descriptions":{"en":"PertEval-scFM is a standardized framework to evaluate single-cell foundation models (scFMs) for predicting cellular perturbation effects.","zh":"PertEval-scFM 是一个评估单细胞基础模型（scFM）扰动效应预测的标准化框架。它从分布偏移泛化性、扰动强度及上下文对齐三方面评测。测试集包含Norman、Replogle等数据集，涵盖数千扰动样本和基因。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:2087","languages":[],"library_direction":null,"modality":null,"name":"PertEval-scFM","openness":"unknown","publisher":"Queen Mary University of London , University of Oxford , etc.","released":"2024-10-03","released_reference":{"basis":"paper_first_version","note":"The first preprint was posted October 3; the October 2 date in the DOI is not its posting date.","source_key":"opencompass:2087","source_url":"https://www.biorxiv.org/content/10.1101/2024.10.02.616248v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2087-perteval-scfm","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PertEval-scFM","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Factuality","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates large language and vision-language models on Korean veterinary question answering, covering text and multimodal questions with expert answers and multiple evaluation metrics.","descriptions":{"en":"Evaluates large language and vision-language models on Korean veterinary question answering, covering text and multimodal questions with expert answers and multiple evaluation metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.04598","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning","multilingual"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_petqa-benchmarking-veterinary-knowledge-an_ed0d2f9f"},"modality":null,"name":"PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04598","source_url":"https://arxiv.org/abs/2609.04598"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04598","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04598","topics":["multimodal","reasoning-math","language","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Compact and scalable benchmark for evaluating LLM reasoning over dynamic state spaces using Petri nets.","descriptions":{"en":"Compact and scalable benchmark for evaluating LLM reasoning over dynamic state spaces using Petri nets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.19883","languages":[],"library_direction":null,"modality":null,"name":"PetriBench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19883","source_url":"https://arxiv.org/abs/2609.19883"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19883","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19883","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark with four task interfaces for evaluating language models on protein modification, including single-mutant generation and multi-mutant ranking.","descriptions":{"en":"Benchmark with four task interfaces for evaluating language models on protein modification, including single-mutant generation and multi-mutant ranking."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.28921","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_pfarena_e6911396"},"modality":null,"name":"PFArena","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.28921","source_url":"https://arxiv.org/abs/2609.28921"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-28921","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28921","topics":[],"unit":null},{"aliases":["PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology"],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A real-robot leaderboard for vision-language-action policies on a Franka FR3. It scores policies with Human-Relative Throughput computed from time-to-success distributions and uses Kolmogorov-Smirnov significance tests.","descriptions":{"en":"A real-robot leaderboard for vision-language-action policies on a Franka FR3. It scores policies with Human-Relative Throughput computed from time-to-success distributions and uses Kolmogorov-Smirnov significance tests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.29710","languages":[],"library_direction":null,"modality":null,"name":"PhAIL (Physical AI Leaderboard)","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29710","source_url":"https://arxiv.org/abs/2605.29710"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29710","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29710","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_phantombench_848c0022"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PhantomBench evaluates language models' ability to abstain from answering about non-existent entities. It comprises over 60,000 non-existent terms derived from real concepts across domains, and provides a pipeline for generating further instances.","descriptions":{"en":"PhantomBench evaluates language models' ability to abstain from answering about non-existent entities. It comprises over 60,000 non-existent terms derived from real concepts across domains, and provides a pipeline for generating further instances."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.11105","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_phantombench_848c0022"},"modality":null,"name":"PhantomBench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11105","source_url":"https://arxiv.org/abs/2606.11105"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11105","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11105","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:ranausmanai/phantomfill","source_record":"bm_phantomfill_4f8c991e"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"PhantomFill measures schema-coerced fabrication in language models by asking questions on unanswerable inputs under three output formats: free text, JSON with an escape option, and JSON with required fields. It reports Coerced Fabrication Rate and Escape Utilization Rate.","descriptions":{"en":"PhantomFill measures schema-coerced fabrication in language models by asking questions on unanswerable inputs under three output formats: free text, JSON with an escape option, and JSON with required fields. It reports Coerced Fabrication Rate and Escape Utilization Rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.20492","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"claire_record_id","primary":"factuality","source_record":"bm_phantomfill_4f8c991e"},"modality":null,"name":"PhantomFill","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20492","source_url":"https://arxiv.org/abs/2607.20492"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20492","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20492","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:zhelishisongjie/ontologyaligner","source_record":"bm_phenonormbench_4748cc53"},"categories":["Biology & Drug Discovery","Information retrieval","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Provides 13,390 samples across seven HPO normalization datasets for mapping free-text phenotype mentions to standardized concepts.","descriptions":{"en":"Provides 13,390 samples across seven HPO normalization datasets for mapping free-text phenotype mentions to standardized concepts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.10055","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_phenonormbench_4748cc53"},"modality":null,"name":"PhenoNormBench","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10055","source_url":"https://arxiv.org/abs/2609.10055"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10055","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10055","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":123,"join":"llm_stats_id","repository":null,"source_record":"catalog_1f366c2487d2dd1f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PhiBench is an internal benchmark designed to evaluate diverse skills and reasoning abilities of language models, covering a wide range of tasks including coding (debugging, extending incomplete code, explaining code snippets) and mathematics (identifying proof errors, generating related problems). Created by Microsoft's research team to address limitations of standard academic benchmarks and guide the development of the Phi-4 model.","descriptions":{"en":"PhiBench is an internal benchmark designed to evaluate diverse skills and reasoning abilities of language models, covering a wide range of tasks including coding (debugging, extending incomplete code, explaining code snippets) and mathematics (identifying proof errors, generating related problems). Created by Microsoft's research team to address limitations of standard academic benchmarks and guide the development of the Phi-4 model."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:phibench:phi-4","reported_at":"2024-12-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/phibench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:phibench","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","code-generation"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_1f366c2487d2dd1f"},"modality":"text","name":"PhiBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":74.2,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.742,"raw_min":0.562,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:phibench:phi-4-reasoning-plus","reported_date":"2025-04-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/phibench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Phi 4 Reasoning Plus"}],"slug":"llm-stats-phibench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/phibench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PHITSBench evaluates AI-assisted generation of PHITS radiation-transport input via natural language across 282 tasks in three workflows: Edit, Repair, and Reproduce. Scoring uses a Composite Metric Score combining execution success and agreement with reference transport observables.","descriptions":{"en":"PHITSBench evaluates AI-assisted generation of PHITS radiation-transport input via natural language across 282 tasks in three workflows: Edit, Repair, and Reproduce. Scoring uses a Composite Metric Score combining execution success and agreement with reference transport observables."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09789","languages":[],"library_direction":{"basis":"model-review","categories":["science","code-generation"],"join":"claire_record_id","primary":"science","source_record":"bm_phitsbench_3767e77d"},"modality":null,"name":"PHITSBench","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09789","source_url":"https://arxiv.org/abs/2607.09789"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09789","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09789","topics":["coding","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":83,"github_stars_30d":15,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:phoneharness/phoneharness","source_record":"bm_phoneharness_125a2bf7"},"categories":["Agents & Tool Use","Mobile & Personal Computing","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PhoneHarness Bench evaluates phone-use agents on verifiable mobile workflows with mixed GUI, CLI, and tool actions, scored by observable side effects from auditable execution traces.","descriptions":{"en":"PhoneHarness Bench evaluates phone-use agents on verifiable mobile workflows with mixed GUI, CLI, and tool actions, scored by observable side effects from auditable execution traces."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.14832","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","tool-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_phoneharness_125a2bf7"},"modality":null,"name":"PhoneHarness","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14832","source_url":"https://arxiv.org/abs/2606.14832"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14832","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14832","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning.","descriptions":{"en":"Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10455","languages":[],"library_direction":{"basis":"model-review-daily","categories":["factuality","long-context"],"join":"claire_record_id","primary":"factuality","source_record":"bm_phrbench_4cb55307"},"modality":null,"name":"PHRBench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10455","source_url":"https://arxiv.org/abs/2610.10455"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10455","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10455","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PHREEQC-MCQ-200 evaluates tool-augmented agents on 200 multiple-choice questions derived from 21 validated PHREEQC scenarios. Agents must construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers. Scoring is based on exact match of selected answer.","descriptions":{"en":"PHREEQC-MCQ-200 evaluates tool-augmented agents on 200 multiple-choice questions derived from 21 validated PHREEQC scenarios. Agents must construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers. Scoring is based on exact match of selected answer."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.00436","languages":[],"library_direction":{"basis":"model-review","categories":["science","tool-use"],"join":"claire_record_id","primary":"science","source_record":"bm_phreeqc-mcq-200_2587cac1"},"modality":null,"name":"PHREEQC-MCQ-200","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00436","source_url":"https://arxiv.org/abs/2607.00436"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00436","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00436","topics":["agents","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Phun-Bench evaluates LLMs' phonological understanding in Chinese across three dimensions: Homophony, Rhyme, and Phonetic Similarity, with diverse tasks designed to isolate genuine phonological ability.","descriptions":{"en":"Phun-Bench evaluates LLMs' phonological understanding in Chinese across three dimensions: Homophony, Rhyme, and Phonetic Similarity, with diverse tasks designed to isolate genuine phonological ability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07300","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_phun-bench_8217b7d9"},"modality":null,"name":"Phun-Bench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07300","source_url":"https://arxiv.org/abs/2606.07300"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07300","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07300","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":25,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":36,"join":"llm_stats_id","repository":null,"source_record":"catalog_f6b2c1d415c48947"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["physics","reasoning","science"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PHYBench is a benchmark of real-world physics problems spanning mechanics, electromagnetism, thermodynamics, optics, and modern physics, designed to evaluate physical perception and multi-step quantitative reasoning in large language models.","descriptions":{"en":"PHYBench is a benchmark of real-world physics problems spanning mechanics, electromagnetism, thermodynamics, optics, and modern physics, designed to evaluate physical perception and multi-step quantitative reasoning in large language models."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:phybench:hy3","reported_at":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/phybench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:phybench","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_f6b2c1d415c48947"},"modality":"text","name":"PHYBench","openness":"unknown","publisher":null,"released":"2025-04-22","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:phybench","source_url":"https://arxiv.org/abs/2504.16074"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":77.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.774,"raw_min":0.774,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:phybench:hy3","reported_date":"2026-07-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/phybench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"}],"slug":"llm-stats-phybench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/phybench?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":["PhyBench"],"attention_signals":null,"categories":["science"],"collected_at":null,"description":"Physics problem-solving benchmark; multi-step reasoning quality affects the final answer.","descriptions":{"en":"Physics problem-solving benchmark; multi-step reasoning quality affects the final answer."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000phybench\u0000bytedance_seed2_0_model_card\u0000phybench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000phybench\u0000bytedance_seed2_0_model_card\u0000phybench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:phybench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["science"],"join":"review:14H034160212:2026-10-15","primary":"science","source_record":null},"modality":null,"name":"PhyBench","openness":"unknown","publisher":null,"released":"2025-04-22","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"model-reports:phybench","source_url":"https://arxiv.org/abs/2504.16074"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":74.0,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":74.0,"raw_min":56.0,"source_reference":{"obs_id":"curated\u0000phybench\u0000bytedance_seed2_0_model_card\u0000phybench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000phybench\u0000bytedance_seed2_0_model_card\u0000phybench\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"phybench","source":"model_reports","source_url":"https://github.com/phybench-official/phybench","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:previsior/phyeditbench","source_record":"bm_phyeditbench_2e24a281"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PhyEditBench evaluates physics-aware image editing. It includes 238 real-world instances and 35 synthetic anti-physics instances across 12 physical subclasses, with VLM-based scoring on consistency, instruction following, physical plausibility, and image quality.","descriptions":{"en":"PhyEditBench evaluates physics-aware image editing. It includes 238 real-world instances and 35 synthetic anti-physics instances across 12 physical subclasses, with VLM-based scoring on consistency, instruction following, physical plausibility, and image quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.26551","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","world-models"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_phyeditbench_2e24a281"},"modality":null,"name":"PhyEditBench","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26551","source_url":"https://arxiv.org/abs/2606.26551"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26551","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26551","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","物理智能","Embodied AI","跨模态推理","Cross-modal Reasoning","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PhyGenBench is a benchmark designed to evaluate physical commonsense correctness in text-to-video generation models. I PhyGenBench 是一个评估T2V模型物理常识的基准。它涵盖力学、光学、热学和材料特性四大领域27种物理定律，包含160个提示。结合PhyGenEval框架，利用视听和大语言模型自动化评估模型的物理理解能力。","descriptions":{"en":"PhyGenBench is a benchmark designed to evaluate physical commonsense correctness in text-to-video generation models. I","zh":"PhyGenBench 是一个评估T2V模型物理常识的基准。它涵盖力学、光学、热学和材料特性四大领域27种物理定律，包含160个提示。结合PhyGenEval框架，利用视听和大语言模型自动化评估模型的物理理解能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2080","languages":[],"library_direction":null,"modality":"multimodal","name":"PhyGenBench","openness":"unknown","publisher":"Shanghai Jiao Tong University , Shanghai AI Laboratory ,etc.","released":"2024-10-07","released_reference":{"basis":"paper_first_version","note":"The paper introducing PhyGenBench's 160 physical-commonsense video prompts.","source_key":"opencompass:2080","source_url":"https://arxiv.org/abs/2410.05363"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2080-phygenbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PhyGenBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:maferrag/physai-bench","source_record":"bm_physai-bench_26ef3849"},"categories":["Language & Knowledge","Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The benchmark evaluates action selection in UAV mission traces through multiple-choice decision instances derived from conversational data.","descriptions":{"en":"The benchmark evaluates action selection in UAV mission traces through multiple-choice decision instances derived from conversational data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.23695","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","embodied-robotics"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_physai-bench_26ef3849"},"modality":null,"name":"PhysAI-Bench","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23695","source_url":"https://arxiv.org/abs/2609.23695"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23695","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23695","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PhysAssistBench evaluates interactive doctor-patient-EHR assistance. It contains 1,296 physician-validated turns from MIMIC-IV cases, testing coordination of clinical knowledge, patient communication, and EHR tool use.","descriptions":{"en":"PhysAssistBench evaluates interactive doctor-patient-EHR assistance. It contains 1,296 physician-validated turns from MIMIC-IV cases, testing coordination of clinical knowledge, patient communication, and EHR tool use."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18613","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","tool-use"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_physassistbench_d86f4b50"},"modality":null,"name":"PhysAssistBench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18613","source_url":"https://arxiv.org/abs/2606.18613"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18613","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18613","topics":["agents","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":66,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_physelite_c8c6e8f9"},"categories":["Mathematics & Formal Science","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal LLMs on 11,586 bilingual Olympiad-level physics problems with visual diagrams, step-by-step solutions, and answer accuracy metrics.","descriptions":{"en":"Evaluates multimodal LLMs on 11,586 bilingual Olympiad-level physics problems with visual diagrams, step-by-step solutions, and answer accuracy metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25097","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_physelite_c8c6e8f9"},"modality":null,"name":"PhysElite","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25097","source_url":"https://arxiv.org/abs/2608.25097"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25097","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25097","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Multimodal large language models (MLLMs) are increasingly envisioned as core components of scientific and engineering agents, yet their ability to interpret physical fields remains poorly understood.","descriptions":{"en":"Multimodal large language models (MLLMs) are increasingly envisioned as core components of scientific and engineering agents, yet their ability to interpret physical fields remains poorly understood."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34072","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_physfieldbench_11604cfb"},"modality":null,"name":"PhysFieldBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34072","source_url":"https://arxiv.org/abs/2609.34072"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34072","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34072","topics":["multimodal","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PhysForensics-Bench evaluates AI-generated audio-video detection using paired real and generated clips across nine event-centric scenes from two generators.","descriptions":{"en":"PhysForensics-Bench evaluates AI-generated audio-video detection using paired real and generated clips across nine event-centric scenes from two generators."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15562","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_physforensics-bench_98b38166"},"modality":null,"name":"PhysForensics-Bench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15562","source_url":"https://arxiv.org/abs/2609.15562"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15562","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15562","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A revised benchmark for measuring whether video generative models reproduce the physical behavior observed in real-world experiment videos.","descriptions":{"en":"A revised benchmark for measuring whether video generative models reproduce the physical behavior observed in real-world experiment videos."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.18943","languages":[],"library_direction":null,"modality":null,"name":"Physics-IQ Verified","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18943","source_url":"https://arxiv.org/abs/2606.18943"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18943","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18943","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:narnialabs/leaderboard","source_record":"bm_physicsbench_c13eff48"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"PhysicsBench is a unified benchmark and leaderboard for generative and predictive AI models in engineering design and simulation. It spans seven tasks across 1D, 2D, and 3D domains, ranks 66 models on nine datasets, and uses a common metric suite including BenchRank for debiased ranking. Evaluation covers data scales from S to XL.","descriptions":{"en":"PhysicsBench is a unified benchmark and leaderboard for generative and predictive AI models in engineering design and simulation. It spans seven tasks across 1D, 2D, and 3D domains, ranks 66 models on nine datasets, and uses a common metric suite including BenchRank for debiased ranking. Evaluation covers data scales from S to XL."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.24056","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_physicsbench_c13eff48"},"modality":null,"name":"PhysicsBench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24056","source_url":"https://arxiv.org/abs/2608.24056"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24056","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24056","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":32,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"llm_stats_id","repository":null,"source_record":"catalog_dc3fcf874da57885"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","physics","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PHYSICS is a comprehensive benchmark for university-level physics problem solving, containing 1,297 expert-annotated problems covering six core areas: classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics. Each problem requires advanced physics knowledge and mathematical reasoning. Even advanced models like o3-mini achieve only 59.9% accuracy.","descriptions":{"en":"PHYSICS is a comprehensive benchmark for university-level physics problem solving, containing 1,297 expert-annotated problems covering six core areas: classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics. Each problem requires advanced physics knowledge and mathematical reasoning. Even advanced models like o3-mini achieve only 59.9% accuracy."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:physicsfinals:gemini-1.5-flash","reported_at":"2024-05-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/physicsfinals?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:physicsfinals","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"catalog_dc3fcf874da57885"},"modality":"text","name":"PhysicsFinals","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":63.9,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.639,"raw_min":0.574,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:physicsfinals:gemini-1.5-pro","reported_date":"2024-05-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/physicsfinals?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"}],"slug":"llm-stats-physicsfinals","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/physicsfinals?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sourajakundu/physimcode","source_record":"bm_physimcode_03b44679"},"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"PhySimCode evaluates video-to-code generation for physics simulations, requiring models to infer physical laws and parameters from videos and produce runnable Python code rendered and scored against the input video.","descriptions":{"en":"PhySimCode evaluates video-to-code generation for physics simulations, requiring models to infer physical laws and parameters from videos and produce runnable Python code rendered and scored against the input video."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.15722","languages":[],"library_direction":{"basis":"model-review-daily","categories":["code-generation","video-understanding","world-models"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_physimcode_03b44679"},"modality":null,"name":"PhySimCode","openness":"unknown","publisher":null,"released":"2026-10-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.15722","source_url":"https://arxiv.org/abs/2610.15722"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-15722","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15722","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Geometric reasoning","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A robot manipulation benchmark with 16 tasks involving fast, dynamically moving targets. It covers simulation and real-world robot experiments and scores policies by task success rate.","descriptions":{"en":"A robot manipulation benchmark with 16 tasks involving fast, dynamically moving targets. It covers simulation and real-world robot experiments and scores policies by task success rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.01938","languages":[],"library_direction":null,"modality":null,"name":"PhysMani-Bench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01938","source_url":"https://arxiv.org/abs/2607.01938"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01938","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01938","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","推理","Reasoning","科学智能","AI for Science","知识储备","Knowledge","逻辑推理","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PhysReason is a comprehensive physics-based reasoning benchmark consisting of 1,200 physics problems spanning multiple domains, with a focus on both knowledge-based (25%) and reasoning-based (75%) questions. PhysReason 是一个包含 1,200 个物理问题的综合物理推理基准，涵盖多个领域，重点关注基于知识（25%）和推理（75%）的问题。","descriptions":{"en":"PhysReason is a comprehensive physics-based reasoning benchmark consisting of 1,200 physics problems spanning multiple domains, with a focus on both knowledge-based (25%) and reasoning-based (75%) questions.","zh":"PhysReason 是一个包含 1,200 个物理问题的综合物理推理基准，涵盖多个领域，重点关注基于知识（25%）和推理（75%）的问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1576","languages":[],"library_direction":null,"modality":null,"name":"PhysReason","openness":"unknown","publisher":"Xi’an Jiaotong University, etc.","released":"2025-02-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1576-physreason","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PhysReason","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:modalitydance/phystool-bench","source_record":"bm_phystool-bench_e6d3d9ac"},"categories":["Agents","Agents & Tool Use","General AI","Tool use"],"collected_at":"2026-10-14T09:44:51Z","description":"PhysTool-Bench evaluates multimodal LLMs on physical tool use through two tasks: recognizing all tools in a scene and selecting and sequencing tools for a given task, using 2,510 queries over 2,678 real-world tools.","descriptions":{"en":"PhysTool-Bench evaluates multimodal LLMs on physical tool use through two tasks: recognizing all tools in a scene and selecting and sequencing tools for a given task, using 2,510 queries over 2,678 real-world tools."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.10803","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","embodied-robotics","world-models"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_phystool-bench_e6d3d9ac"},"modality":null,"name":"PhysTool-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10803","source_url":"https://arxiv.org/abs/2606.10803"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10803","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10803","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","指令跟随","Instruct","创作","Creation","Image Editing","AIGC","Physics","物理智能","Embodied AI","指令遵循","Instruction Following","语言生成","Generation","视觉生成","Visual Generation","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"","descriptions":{},"editorial_guide":{"id":"picabench","record_keys":["opencompass:2350"],"reviewed_at":"2026-10-23","scope":{"en":"Cited research guide; original source measurements remain independent.","zh":"附来源的研究导读；原始记录的测量保持独立。"},"sections":[{"id":"tasks","source_ids":["paper"],"text":{"en":"Removing an object should also update its shadow, reflections and interactions. The benchmark covers optics, mechanics and state changes.","zh":"移除物体时，也应改变阴影、倒影及相邻物体的交互。题目涉及光学、力学和状态变化。"},"title":{"en":"Beyond following the edit instruction","zh":"超出表面的指令完成"}},{"id":"metrics","source_ids":["paper"],"text":{"en":"A vision-language judge answers spatially anchored questions using human-annotated regions. Inspect region-level questions alongside the overall accuracy.","zh":"视觉语言评判模型依据人工标注区域，回答具有空间定位的问题。应结合区域问题查看整体准确率。"},"title":{"en":"How PICAEval checks an edit","zh":"PICAEval 怎样检查编辑"}},{"id":"data","source_ids":["project"],"text":{"en":"PICABench contains evaluation cases; PICA-100K is a separate training resource derived from video. They serve different purposes.","zh":"PICABench 提供评估题目；PICA-100K 是另一个由视频构建的训练资料集，两者用途不同。"},"title":{"en":"Keep training and evaluation apart","zh":"区分训练与评估资料"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Keep the editing model, output resolution and judge configuration with any score. The authors’ repository links the leaderboard and reproducible evaluation pipeline.","zh":"阅读分数时保留编辑模型、输出分辨率与评判配置。作者仓库提供排行榜及可复现的评估流程入口。"},"title":{"en":"Read results in context","zh":"结合条件理解结果"}}],"sources":[{"id":"paper","title":"PICABench — original paper","url":"https://arxiv.org/abs/2510.17681v3"},{"id":"project","title":"PICABench — authors’ project","url":"https://github.com/Andrew0613/PICABench"}],"summary":{"en":"Whether an image edit also updates the physical consequences of the change.","zh":"考察图像编辑能否同时正确改变物体及其物理影响。"},"title":"PICABench"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2350","languages":[],"library_direction":null,"modality":"multimodal","name":"PICABench","openness":"open","publisher":"Shanghai AI Laboratory","released":"2025-10-27","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2350-picabench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PICABench","topics":["multimodal","reasoning-math","language","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates prompt-injection detectors across multiple axes including over-defense, obfuscation, and distribution shift, with fixed thresholds and reproducible metrics.","descriptions":{"en":"Evaluates prompt-injection detectors across multiple axes including over-defense, obfuscation, and distribution shift, with fixed thresholds and reproducible metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15017","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_pids-bench_d9a94c53"},"modality":null,"name":"PIDS-Bench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15017","source_url":"https://arxiv.org/abs/2609.15017"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15017","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15017","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1360,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_c7d25132dcc30c2c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PinchBench evaluates coding agents on real-world agentic coding tasks, measuring both best-case and average performance across complex software engineering scenarios.","descriptions":{"en":"PinchBench evaluates coding agents on real-world agentic coding tasks, measuring both best-case and average performance across complex software engineering scenarios."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:pinchbench:mimo-v2-omni","reported_at":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pinchbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:pinchbench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_c7d25132dcc30c2c"},"modality":"text","name":"PinchBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":90.0,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.9,"raw_min":0.6822,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:pinchbench:nemotron-3-ultra-550b-a55b","reported_date":"2026-06-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pinchbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"}],"slug":"llm-stats-pinchbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pinchbench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["Decision-Aware Evaluation of Physics-Informed Surrogates"],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"An open benchmark for evaluating physics-informed surrogates in material-conditioned lattice design across prediction fidelity, physical validity, retrieval, and decision regret.","descriptions":{"en":"An open benchmark for evaluating physics-informed surrogates in material-conditioned lattice design across prediction fidelity, physical validity, retrieval, and decision regret."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07146","languages":[],"library_direction":null,"modality":null,"name":"pinn-gym","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07146","source_url":"https://arxiv.org/abs/2606.07146"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07146","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07146","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:avalon-s/pinverify","source_record":"bm_pinverify_ff022fa6"},"categories":["Robotics & Embodied AI","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PInVerify is an offline embodied benchmark for Active Instance Verification, where agents select viewpoints around a candidate object to decide if it matches a fine-grained description.","descriptions":{"en":"PInVerify is an offline embodied benchmark for Active Instance Verification, where agents select viewpoints around a candidate object to decide if it matches a fine-grained description."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.30639","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_pinverify_ff022fa6"},"modality":null,"name":"PInVerify","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30639","source_url":"https://arxiv.org/abs/2605.30639"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30639","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30639","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:wuyuhang05/pipbench","source_record":"bm_pipbench_0f193694"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PIPBench evaluates personalized image generation, where models must align outputs with a user's implicit visual preferences based on a few historically preferred images and a short prompt. It includes real-user and agent-based data across psychological and demographic profiles.","descriptions":{"en":"PIPBench evaluates personalized image generation, where models must align outputs with a user's implicit visual preferences based on a few historically preferred images and a short prompt. It includes real-user and agent-based data across psychological and demographic profiles."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.06440","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_pipbench_0f193694"},"modality":null,"name":"PIPBench","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06440","source_url":"https://arxiv.org/abs/2607.06440"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06440","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06440","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:suraj-ranganath/pipe-cypher","source_record":"bm_pipe-cypher_c078ae44"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"PIPE-Cypher is a pipeline that generates NL-to-Cypher benchmarks from live property graphs, producing executable query pairs with validation, diversity controls, and local LLM judges.","descriptions":{"en":"PIPE-Cypher is a pipeline that generates NL-to-Cypher benchmarks from live property graphs, producing executable query pairs with validation, diversity controls, and local LLM judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.08481","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_pipe-cypher_c078ae44"},"modality":null,"name":"PIPE-Cypher","openness":"unknown","publisher":null,"released":"2026-06-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08481","source_url":"https://arxiv.org/abs/2606.08481"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08481","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08481","topics":["coding","agents"],"unit":null},{"aliases":["Pipette: An Embodied Simulation Platform, Benchmark, and Data-Efficient Augmentation Framework for Wet-Lab Robotics"],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-09-25T04:40:22Z","description":"Pipette evaluates wet-lab robotic agents across 12 embodied tasks using task success rates.","descriptions":{"en":"Pipette evaluates wet-lab robotic agents across 12 embodied tasks using task success rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12936","languages":[],"library_direction":null,"modality":null,"name":"Pipette","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12936","source_url":"https://arxiv.org/abs/2606.12936"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12936","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12936","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_7f8d07159d2aa133"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["physics","reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PIQA (Physical Interaction: Question Answering) is a benchmark dataset for physical commonsense reasoning in natural language. It tests AI systems' ability to answer questions requiring physical world knowledge through multiple choice questions with everyday situations, focusing on atypical solutions inspired by instructables.com. The dataset contains 21,000 multiple choice questions where models must choose the most appropriate solution for physical interactions.","descriptions":{"en":"PIQA (Physical Interaction: Question Answering) is a benchmark dataset for physical commonsense reasoning in natural language. It tests AI systems' ability to answer questions requiring physical world knowledge through multiple choice questions with everyday situations, focusing on atypical solutions inspired by instructables.com. The dataset contains 21,000 multiple choice questions where models must choose the most appropriate solution for physical interactions."},"evidence_summary":{"document_count":1,"model_count":11,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:piqa:gemma-2-27b-it","reported_at":"2024-06-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/piqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:piqa","languages":[],"library_direction":{"basis":"model-review","categories":["world-models"],"join":"llm_stats_id","primary":"world-models","source_record":"catalog_7f8d07159d2aa133"},"modality":"text","name":"PIQA","openness":"unknown","publisher":null,"released":"2019-11-26","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: PIQA: Reasoning about Physical Commonsense in Natural Language. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:piqa","source_url":"https://arxiv.org/abs/1911.11641"},"repo_kind":null,"repo_resolution_status":null,"score_count":11,"score_direction":"higher_is_better","score_summary":{"display_max":88.6,"display_multiplier":100,"model_count":11,"model_count_basis":"source_model_id","numeric_count":11,"raw_max":0.886,"raw_min":0.552,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:piqa:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/piqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-piqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/piqa?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"artifact:arxiv:1911.11641","repository":null,"source_record":"catalog_7f8d07159d2aa133"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PIQA is a physical interaction question answering task, which requires to select the most reasonable solution based on the given scenario and two possible solutions. This task is designed to test the model's knowledge in physical commonsense. This dataset consists of 16k training samples, 800 development samples and 2k test samples, all on English text. PIQA是一个物理交互问答任务，要求根据给定的场景和两个可能的解决方案，选择最合理的方案。这个任务是为了测试模型在物理常识方面的知识。这个数据集包含了16000个训练样本，800个开发样本和2000个测试样本，所有的文本都是英文文本。","descriptions":{"en":"PIQA is a physical interaction question answering task, which requires to select the most reasonable solution based on the given scenario and two possible solutions. This task is designed to test the model's knowledge in physical commonsense. This dataset consists of 16k training samples, 800 development samples and 2k test samples, all on English text.","zh":"PIQA是一个物理交互问答任务，要求根据给定的场景和两个可能的解决方案，选择最合理的方案。这个任务是为了测试模型在物理常识方面的知识。这个数据集包含了16000个训练样本，800个开发样本和2000个测试样本，所有的文本都是英文文本。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:532","languages":["English"],"library_direction":{"basis":"model-review","categories":["world-models"],"join":"artifact:arxiv:1911.11641","primary":"world-models","source_record":"catalog_7f8d07159d2aa133"},"modality":null,"name":"PIQA","openness":"unknown","publisher":null,"released":"2019-11-26","released_reference":{"basis":"paper_first_version","note":"The physical-commonsense question-answering paper, not the unrelated image-quality library.","source_key":"opencompass:532","source_url":"https://arxiv.org/abs/1911.11641"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-532-piqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PIQA","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"PitchBench evaluates pitch hearing in audio-language models across 28 experiments spanning absolute and relative pitch perception in sequences and chords, varying acoustic conditions and response formats.","descriptions":{"en":"PitchBench evaluates pitch hearing in audio-language models across 28 experiments spanning absolute and relative pitch perception in sequences and chords, varying acoustic conditions and response formats."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26176","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_pitchbench_7fffde93"},"modality":null,"name":"PitchBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26176","source_url":"https://arxiv.org/abs/2605.26176"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26176","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26176","topics":[],"unit":null},{"aliases":["SIC-Agents: Benchmarking and Building an Adaptive Simulator for Pediatric Serious Illness Communication Training","SIC-Agents"],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:beikewzh/sic-benchmarks","source_record":"bm_sic-agents-benchmarking-and-building-an-ad_9ce077fa"},"categories":["Agents & Tool Use","General AI","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates parent-simulator agents in pediatric serious illness communication training using turn-level PitfallBench and dialogue-level DialogueBench.","descriptions":{"en":"Evaluates parent-simulator agents in pediatric serious illness communication training using turn-level PitfallBench and dialogue-level DialogueBench."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29481","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","chat-writing"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_sic-agents-benchmarking-and-building-an-ad_9ce077fa"},"modality":null,"name":"PitfallBench","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29481","source_url":"https://arxiv.org/abs/2608.29481"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29481","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29481","topics":["agents","language","medical"],"unit":null},{"aliases":["PIVOT: A Multi-Trajectory Dataset and Testbed for Pose, Intrinsics, and Novel Viewpoint Evaluation in Real-World 3D Reconstruction"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_pivot-a-multi-trajectory-dataset-and-testb_3fccb501"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates novel-view synthesis under diverse camera trajectories, measured vs optimized poses, and calibrated vs optimized intrinsics using five real-world scenes.","descriptions":{"en":"Evaluates novel-view synthesis under diverse camera trajectories, measured vs optimized poses, and calibrated vs optimized intrinsics using five real-world scenes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25401","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","spatial-3d"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_pivot-a-multi-trajectory-dataset-and-testb_3fccb501"},"modality":null,"name":"PIVOT","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25401","source_url":"https://arxiv.org/abs/2608.25401"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25401","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25401","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"PIVOTSBench evaluates multimodal large language models on fine-grained interpersonal relationship reasoning. It includes tasks predicting bidirectional relationship dimensions from videos and auxiliary tasks on visual cue identification.","descriptions":{"en":"PIVOTSBench evaluates multimodal large language models on fine-grained interpersonal relationship reasoning. It includes tasks predicting bidirectional relationship dimensions from videos and auxiliary tasks on visual cue identification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.23092","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_pivotsbench_17265409"},"modality":null,"name":"PIVOTSBench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23092","source_url":"https://arxiv.org/abs/2606.23092"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23092","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23092","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Dataset of 11,000 NBA possession clips with 31,567 annotated events for basketball video understanding tasks like event recognition, captioning, localization, and anticipation.","descriptions":{"en":"Dataset of 11,000 NBA possession clips with 31,567 annotated events for basketball video understanding tasks like event recognition, captioning, localization, and anticipation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.19646","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_pl-nba_96aca2fb"},"modality":null,"name":"PL-NBA","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.19646","source_url":"https://arxiv.org/abs/2608.19646"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-19646","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19646","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates reasoning-driven macro placement in chip design using open RTL designs, EDA tools, and technology libraries.","descriptions":{"en":"Evaluates reasoning-driven macro placement in chip design using open RTL designs, EDA tools, and technology libraries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.21263","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_placereasoner-bench_0cd15d78"},"modality":null,"name":"PlaceReasoner-Bench","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21263","source_url":"https://arxiv.org/abs/2609.21263"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21263","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21263","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_plan2map_b8494f1d"},"categories":["General AI","Multimodal","Planning"],"collected_at":"2026-10-14T09:44:51Z","description":"Plan2Map evaluates document-grounded geospatial boundary reconstruction from UK planning records. Systems input a planning document and output a GeoJSON boundary, scored against held-out reference boundaries via IoU.","descriptions":{"en":"Plan2Map evaluates document-grounded geospatial boundary reconstruction from UK planning records. Systems input a planning document and output a GeoJSON boundary, scored against held-out reference boundaries via IoU."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02747","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","spatial-3d"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_plan2map_b8494f1d"},"modality":null,"name":"Plan2Map","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02747","source_url":"https://arxiv.org/abs/2606.02747"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02747","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02747","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","大语言模型","LLM","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PlanBench is an extensible benchmark suite based on the kinds of domains used in the automated planning community, especially in the International Planning Competition, to test the capabilities of LLMs in planning or reasoning about actions and change. PlanBench用于评估LLM的规划能力，基于自动化规划社区（尤其是在国际规划竞赛）中涉及的各种领域来测试大模型在规划或推理行动和变更方面的能力。","descriptions":{"en":"PlanBench is an extensible benchmark suite based on the kinds of domains used in the automated planning community, especially in the International Planning Competition, to test the capabilities of LLMs in planning or reasoning about actions and change.","zh":"PlanBench用于评估LLM的规划能力，基于自动化规划社区（尤其是在国际规划竞赛）中涉及的各种领域来测试大模型在规划或推理行动和变更方面的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1251","languages":[],"library_direction":null,"modality":null,"name":"PlanBench","openness":"unknown","publisher":"Arizona State University","released":"2022-06-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1251-planbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PlanBench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zhuchichi56/planbench","source_record":"bm_planbench-v_89620860"},"categories":["General AI","Multimodal","Planning"],"collected_at":"2026-10-14T09:44:51Z","description":"PlanBench-V evaluates vision-language models on spatial planning map interpretation through an expert-annotated dataset of 223 maps and 1629 question-answer pairs, assessing perception, reasoning, association, and implementation capabilities.","descriptions":{"en":"PlanBench-V evaluates vision-language models on spatial planning map interpretation through an expert-annotated dataset of 223 maps and 1629 question-answer pairs, assessing perception, reasoning, association, and implementation capabilities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.05744","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_planbench-v_89620860"},"modality":null,"name":"PlanBench-V","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05744","source_url":"https://arxiv.org/abs/2606.05744"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05744","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05744","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":41,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":96,"join":"claire_record_id","repository":"gh:jiayujeff/planbench-xl","source_record":"bm_planbench-xl_459bd32f"},"categories":["Agents & Tool Use","General AI","Planning","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PlanBench-XL evaluates LLM tool-use agents on long-horizon planning in large-scale tool ecosystems. It includes 327 retail tasks across 1,665 tools, requiring iterative tool retrieval and use. Optional blocker mechanisms inject missing, failing, or distracting tools to test adaptive planning. Scoring is based on final answer accuracy and auxiliary metrics.","descriptions":{"en":"PlanBench-XL evaluates LLM tool-use agents on long-horizon planning in large-scale tool ecosystems. It includes 327 retail tasks across 1,665 tools, requiring iterative tool retrieval and use. Optional blocker mechanisms inject missing, failing, or distracting tools to test adaptive planning. Scoring is based on final answer accuracy and auxiliary metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22388","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_planbench-xl_459bd32f"},"modality":null,"name":"PlanBench-XL","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22388","source_url":"https://arxiv.org/abs/2606.22388"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22388","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22388","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":37,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:skylenage-ai/plawbench","source_record":"catalog_77674378a36a0f93"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning","knowledge"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PLawBench evaluates language models on professional legal knowledge and reasoning tasks.","descriptions":{"en":"PLawBench evaluates language models on professional legal knowledge and reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:plawbench:qwen3.8-max","reported_at":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/plawbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:plawbench","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_77674378a36a0f93"},"modality":"text","name":"PLawBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":73.2,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.732,"raw_min":0.732,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:plawbench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/plawbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-plawbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/plawbench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":null,"source_record":"bm_playlisteval-can-video-language-judges-be-_5d63e698"},"categories":["General AI","Multimodal","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PlaylistEval evaluates video-language judges on 630 question-answer pairs derived from 100-hour playlists, testing retrieval and judgment across seven domains.","descriptions":{"en":"PlaylistEval evaluates video-language judges on 630 question-answer pairs derived from 100-hour playlists, testing retrieval and judgment across seven domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34314","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_playlisteval-can-video-language-judges-be-_5d63e698"},"modality":null,"name":"PlaylistEval","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34314","source_url":"https://arxiv.org/abs/2609.34314"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34314","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34314","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":92,"github_stars_30d":4,"hf_dataset_downloads":null,"hf_paper_upvotes":48,"join":"claire_record_id","repository":"gh:kxding/playworld","source_record":"bm_playworld_54fed7c4"},"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"PlayWorld evaluates interactive video world models across 171 scenarios with long-horizon objectives. Agent players adaptively control each model, and performance is scored on geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution, plus basic video quality and controllability metrics.","descriptions":{"en":"PlayWorld evaluates interactive video world models across 171 scenarios with long-horizon objectives. Agent players adaptively control each model, and performance is scored on geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution, plus basic video quality and controllability metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.13552","languages":[],"library_direction":{"basis":"model-review","categories":["world-models"],"join":"claire_record_id","primary":"world-models","source_record":"bm_playworld_54fed7c4"},"modality":null,"name":"PlayWorld","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13552","source_url":"https://arxiv.org/abs/2608.13552"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13552","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13552","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","Factuality","Manufacturing & Process Control","Robotics & Embodied AI","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates autonomous LLM agents on real programmable logic controllers connected to closed-loop process simulations, measuring their ability to achieve sustained physical impact through fixed diagnostic flags applied to runner, communication, PLC-object, and process records.","descriptions":{"en":"Evaluates autonomous LLM agents on real programmable logic controllers connected to closed-loop process simulations, measuring their ability to achieve sustained physical impact through fixed diagnostic flags applied to runner, communication, PLC-object, and process records."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.26882","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design","cybersecurity"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_plcbench_fb7005e5"},"modality":null,"name":"PLCBENCH","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26882","source_url":"https://arxiv.org/abs/2608.26882"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26882","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26882","topics":["agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands.","descriptions":{"en":"Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.02982","languages":[],"library_direction":{"basis":"model-review-daily","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_plcworld_7ccad521"},"modality":null,"name":"PLCWorld","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.02982","source_url":"https://arxiv.org/abs/2610.02982"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-02982","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02982","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:oracle-samples/plsqlbench","source_record":"bm_plsqlbench_adc42146"},"categories":["Code & Software","Code generation","Factuality","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PLSQLBench evaluates LLMs' ability to write executable PL/SQL programs through execution-based tests. It contains 2,865 instances including single-turn and multi-turn tasks, covering schema-grounded and procedural problems.","descriptions":{"en":"PLSQLBench evaluates LLMs' ability to write executable PL/SQL programs through execution-based tests. It contains 2,865 instances including single-turn and multi-turn tasks, covering schema-grounded and procedural problems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.15931","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_plsqlbench_adc42146"},"modality":null,"name":"PLSQLBench","openness":"unknown","publisher":null,"released":"2026-08-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15931","source_url":"https://arxiv.org/abs/2608.15931"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15931","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15931","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PluginEval evaluates tool routing in LLMs via three decision types (missed, spurious, parameter errors) across difficulty levels, using deterministic validation and real API execution for reliable signals.","descriptions":{"en":"PluginEval evaluates tool routing in LLMs via three decision types (missed, spurious, parameter errors) across difficulty levels, using deterministic validation and real API execution for reliable signals."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08700","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_plugineval_1642b04d"},"modality":null,"name":"PluginEval","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08700","source_url":"https://arxiv.org/abs/2608.08700"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08700","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08700","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PM-Bench evaluates prospective memory in LLM agents through a text-based simulated seven-day week. Agents must maintain user intentions, execute delayed intentions, and monitor latent environment changes while performing ongoing activities.","descriptions":{"en":"PM-Bench evaluates prospective memory in LLM agents through a text-based simulated seven-day week. Agents must maintain user intentions, execute delayed intentions, and monitor latent environment changes while performing ongoing activities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.12385","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_pm-bench_3de97f77"},"modality":null,"name":"PM-Bench","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.12385","source_url":"https://arxiv.org/abs/2607.12385"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-12385","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12385","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_be4b2f868732f6bc"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A medical visual question answering benchmark built on biomedical literature and medical figures.","descriptions":{"en":"A medical visual question answering benchmark built on biomedical literature and medical figures."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:pmc-vqa:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pmc-vqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:pmc-vqa","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_be4b2f868732f6bc"},"modality":"multimodal","name":"PMC-VQA","openness":"unknown","publisher":null,"released":"2023-05-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:pmc-vqa","source_url":"https://arxiv.org/abs/2305.10415"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":63.3,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.633,"raw_min":0.62,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:pmc-vqa:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pmc-vqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-pmc-vqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pmc-vqa?top_n=500","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":["pni-3d-benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":166,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_pni-3d-benchmark_1dadb067"},"categories":["CAD","General AI","Geometric reasoning","Information retrieval","Multimodal","Safety","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates fine-grained 3D CAD matching, multimodal embedding alignment, and geometric retrieval under sensor distortions using 100 parametric CAD families across 12 representations.","descriptions":{"en":"Evaluates fine-grained 3D CAD matching, multimodal embedding alignment, and geometric retrieval under sensor distortions using 100 parametric CAD families across 12 representations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:tryforge/pni-3d-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_pni-3d-benchmark_1dadb067"},"modality":null,"name":"PNI-3D Multimodal CAD & Geometric Robustness Benchmark","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:tryforge/pni-3d-benchmark","source_url":"https://huggingface.co/datasets/tryforge/pni-3d-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-tryforge-pni-3d-benchmark","source":"claire_radar","source_url":"https://huggingface.co/datasets/tryforge/pni-3d-benchmark","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Pocket-Dentist is an efficiency-aware benchmark for dental multimodal question answering that combines three datasets, five task types, and seven metrics to evaluate VLMs on accuracy and computational cost.","descriptions":{"en":"Pocket-Dentist is an efficiency-aware benchmark for dental multimodal question answering that combines three datasets, five task types, and seven metrics to evaluate VLMs on accuracy and computational cost."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29299","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_pocket-dentist_999be450"},"modality":null,"name":"Pocket-Dentist","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29299","source_url":"https://arxiv.org/abs/2605.29299"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29299","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29299","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"POINav-Bench evaluates vision-language navigation agents in real-world POI-goal navigation across 11 reconstructed commercial areas covering 126,398 m² with 163 POIs, using traversability-aware annotations and reference trajectories for closed-loop evaluation.","descriptions":{"en":"POINav-Bench evaluates vision-language navigation agents in real-world POI-goal navigation across 11 reconstructed commercial areas covering 126,398 m² with 163 POIs, using traversability-aware annotations and reference trajectories for closed-loop evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28237","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_poinav-bench_7e6a4ee3"},"modality":null,"name":"POINav-Bench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28237","source_url":"https://arxiv.org/abs/2605.28237"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28237","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28237","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":40,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"llm_stats_id","repository":null,"source_record":"catalog_9b11c51afdac427a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","spatial_reasoning","grounding","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PointArena is a comprehensive platform for evaluating multimodal pointing across diverse reasoning scenarios. It includes Point-Bench, a curated dataset of ~1,000 pointing tasks across five categories: Spatial (positional references), Affordance (functional part identification), Counting (attribute-based grouping), Steerable (relative pointing), and Reasoning (open-ended visual inference). The benchmark evaluates language-guided pointing capabilities in vision-language models.","descriptions":{"en":"PointArena is a comprehensive platform for evaluating multimodal pointing across diverse reasoning scenarios. It includes Point-Bench, a curated dataset of ~1,000 pointing tasks across five categories: Spatial (positional references), Affordance (functional part identification), Counting (attribute-based grouping), Steerable (relative pointing), and Reasoning (open-ended visual inference). The benchmark evaluates language-guided pointing capabilities in vision-language models."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:pointgrounding:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pointgrounding?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:pointgrounding","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_9b11c51afdac427a"},"modality":"multimodal","name":"PointGrounding","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":66.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.665,"raw_min":0.665,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:pointgrounding:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pointgrounding?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-pointgrounding","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pointgrounding?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PointQ-Bench is a benchmark for point cloud quality assessment, extending from scalar scoring to comprehensive quality understanding, with 3,083 point clouds and tasks like anomaly sensing, defect diagnosis, usability grading, and open-ended quality reporting.","descriptions":{"en":"PointQ-Bench is a benchmark for point cloud quality assessment, extending from scalar scoring to comprehensive quality understanding, with 3,083 point clouds and tasks like anomaly sensing, defect diagnosis, usability grading, and open-ended quality reporting."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28241","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_pointq-bench_51211dca"},"modality":null,"name":"PointQ-Bench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28241","source_url":"https://arxiv.org/abs/2605.28241"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28241","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28241","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Logistics & Operations","Safety & Trustworthiness","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"PoisonForge benchmarks task-level targeted poisoning of instruction-tuned LLMs, parameterizing bias type, poisoning mode, appearance count, and target output length. It evaluates 12 open-weight models across five families with primarily 1% poison budget, using attack success rate as the main metric.","descriptions":{"en":"PoisonForge benchmarks task-level targeted poisoning of instruction-tuned LLMs, parameterizing bias type, poisoning mode, appearance count, and target output length. It evaluates 12 open-weight models across five families with primarily 1% poison budget, using attack success rate as the main metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.23168","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_poisonforge_b7c26795"},"modality":null,"name":"PoisonForge","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23168","source_url":"https://arxiv.org/abs/2605.23168"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23168","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23168","topics":["safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PokerBench contains natural language game scenarios and optimal decisions computed by solvers in No Limit Texas Hold’em. It is divided into pre-flop and post-flop datasets, each with training and test splits. PokerBench包含自然语言游戏场景和由求解器在无限制德州扑克中计算出的最优决策。它分为前注和后注数据集，每个数据集都包含训练集和测试集。","descriptions":{"en":"PokerBench contains natural language game scenarios and optimal decisions computed by solvers in No Limit Texas Hold’em. It is divided into pre-flop and post-flop datasets, each with training and test splits.","zh":"PokerBench包含自然语言游戏场景和由求解器在无限制德州扑克中计算出的最优决策。它分为前注和后注数据集，每个数据集都包含训练集和测试集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1675","languages":[],"library_direction":null,"modality":null,"name":"PokerBench","openness":"open","publisher":"University of California, Berkeley, etc.","released":"2025-01-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1675-pokerbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PokerBench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":24,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":30,"join":"claire_record_id","repository":"gh:ssmisya/policyshiftguard","source_record":"bm_policyshiftbench_8ca9566e"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"PolicyShiftBench evaluates policy-adaptive image guardrailing: given an image and a current policy, a model must output a pass/block decision plus optional violated category IDs. The benchmark comprises 2,000 policy-discriminative instances over 265 images, each paired with multiple policy-conditioned prompts. Scoring uses binary pass/block accuracy and category attribution metrics.","descriptions":{"en":"PolicyShiftBench evaluates policy-adaptive image guardrailing: given an image and a current policy, a model must output a pass/block decision plus optional violated category IDs. The benchmark comprises 2,000 policy-discriminative instances over 265 images, each paired with multiple policy-conditioned prompts. Scoring uses binary pass/block accuracy and category attribution metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.05910","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_policyshiftbench_8ca9566e"},"modality":null,"name":"PolicyShiftBench","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.05910","source_url":"https://arxiv.org/abs/2607.05910"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05910","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05910","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_polistemics_ea35d629"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Polistemics evaluates LLMs as mediators of political information across controlled settings varying evidence clarity, noise, and consistency, using a diagnostic benchmark grounded in Epistemic Modesty.","descriptions":{"en":"Polistemics evaluates LLMs as mediators of political information across controlled settings varying evidence clarity, noise, and consistency, using a diagnostic benchmark grounded in Epistemic Modesty."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.25953","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","factuality"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_polistemics_ea35d629"},"modality":null,"name":"Polistemics","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25953","source_url":"https://arxiv.org/abs/2607.25953"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25953","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25953","topics":["safety"],"unit":null},{"aliases":["An open benchmark for machine learning-based polymer property prediction"],"attention_signals":{"as_of":"2026-10-14","github_stars":8,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:rlearsch/polymerbenchmark2026","source_record":"bm_an-open-benchmark-for-machine-learning-bas_92086c44"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark dataset for machine learning-based polymer property prediction across eight physical properties.","descriptions":{"en":"Benchmark dataset for machine learning-based polymer property prediction across eight physical properties."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.27036","languages":[],"library_direction":null,"modality":null,"name":"PolyBench26","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27036","source_url":"https://arxiv.org/abs/2609.27036"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27036","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27036","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"As large language models increasingly move toward repository-level software engineering, existing code-generation benchmarks remain fragmented across language coverage, task granularity, and evaluation protocols, impeding systematic compar…","descriptions":{"en":"As large language models increasingly move toward repository-level software engineering, existing code-generation benchmarks remain fragmented across language coverage, task granularity, and evaluation protocols, impeding systematic compar…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11618","languages":[],"library_direction":{"basis":"model-review-daily","categories":["code-generation","agentic-coding"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_polycodeeval_7c359cc1"},"modality":null,"name":"PolyCodeEval","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11618","source_url":"https://arxiv.org/abs/2610.11618"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11618","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11618","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Geometric reasoning","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"PolyComp is a benchmark for compositional 3D spatial reasoning using polycube problems with 120 problems across four geometry families.","descriptions":{"en":"PolyComp is a benchmark for compositional 3D spatial reasoning using polycube problems with 120 problems across four geometry families."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14741","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_polycomp_d2741a8f"},"modality":null,"name":"PolyComp","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14741","source_url":"https://arxiv.org/abs/2608.14741"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14741","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14741","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_51c52a821f9daa7a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","reasoning","spatial_reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Polymath is a challenging multi-modal mathematical reasoning benchmark designed to evaluate the general cognitive reasoning abilities of Multi-modal Large Language Models (MLLMs). The benchmark comprises 5,000 manually collected high-quality images of cognitive textual and visual challenges across 10 distinct categories, including pattern recognition, spatial reasoning, and relative reasoning.","descriptions":{"en":"Polymath is a challenging multi-modal mathematical reasoning benchmark designed to evaluate the general cognitive reasoning abilities of Multi-modal Large Language Models (MLLMs). The benchmark comprises 5,000 manually collected high-quality images of cognitive textual and visual challenges across 10 distinct categories, including pattern recognition, spatial reasoning, and relative reasoning."},"evidence_summary":{"document_count":1,"model_count":23,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:polymath:qwen3-235b-a22b-instruct-2507","reported_at":"2025-07-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:polymath","languages":[],"library_direction":{"basis":"model-review","categories":["math","visual-reasoning"],"join":"llm_stats_id","primary":"math","source_record":"catalog_51c52a821f9daa7a"},"modality":"multimodal","name":"PolyMATH","openness":"unknown","publication":{"evidence":[{"note":"The paper's 5,000 visual reasoning images across ten categories, including pattern, spatial and relative reasoning, match the exact source description and distinguish this work from other PolyMATH benchmarks.","url":"https://arxiv.org/abs/2410.14702"},{"note":"The author dataset card links this paper and the polymathbenchmark/PolyMATH code repository. The repository README links back to this dataset. Test and testmini are separately documented; no score partition is inferred.","url":"https://huggingface.co/datasets/him1411/polymath"}],"id":"polymath-multimodal-publication","links":[{"kind":"paper","title":"PolyMATH multimodal reasoning paper","url":"https://arxiv.org/abs/2410.14702"},{"kind":"repo","title":"PolyMATH official repository","url":"https://github.com/polymathbenchmark/PolyMATH"},{"kind":"dataset","title":"PolyMATH author dataset","url":"https://huggingface.co/datasets/him1411/polymath"}],"record_keys":["llm-stats:polymath"],"reviewed_at":"2026-10-26","scope":{"en":"The multimodal PolyMATH benchmark by Gupta and colleagues, identified by its visual reasoning tasks; this does not bind other similarly named multilingual mathematics benchmarks.","zh":"Gupta 等作者的多模态 PolyMATH 视觉推理评测；不关联其他同名或近名的多语言数学评测。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":"not_found","repo_resolution_status":"not_found","score_count":23,"score_direction":"higher_is_better","score_summary":{"display_max":86.5,"display_multiplier":100,"model_count":23,"model_count_basis":"source_model_id","numeric_count":23,"raw_max":0.865,"raw_min":0.082,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:polymath:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-polymath","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":43,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_a50f4a901acc5f37"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PolyMath is a multilingual mathematical reasoning benchmark covering 18 languages and 4 difficulty levels from easy to hard, ensuring difficulty comprehensiveness, language diversity, and high-quality translation. The benchmark evaluates mathematical reasoning capabilities of large language models across diverse linguistic contexts, making it a highly discriminative multilingual mathematical benchmark.","descriptions":{"en":"PolyMath is a multilingual mathematical reasoning benchmark covering 18 languages and 4 difficulty levels from easy to hard, ensuring difficulty comprehensiveness, language diversity, and high-quality translation. The benchmark evaluates mathematical reasoning capabilities of large language models across diverse linguistic contexts, making it a highly discriminative multilingual mathematical benchmark."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:polymath-en:kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath-en?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:polymath-en","languages":[],"library_direction":{"basis":"model-review","categories":["math","multilingual"],"join":"llm_stats_id","primary":"math","source_record":"catalog_a50f4a901acc5f37"},"modality":"text","name":"PolyMath-en","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":65.10000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.651,"raw_min":0.651,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:polymath-en:kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath-en?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"}],"slug":"llm-stats-polymath-en","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath-en?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":9,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:youngseng/polyspeech-100","source_record":"bm_polyspeech-100_ce0b7268"},"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PolySpeech-100 evaluates speech understanding in speech-large language models across 110 linguistic variants, including 19 Chinese dialects and over 80 low-resource languages, using tasks that assess semantic reasoning beyond transcription.","descriptions":{"en":"PolySpeech-100 evaluates speech understanding in speech-large language models across 110 linguistic variants, including 19 Chinese dialects and over 80 low-resource languages, using tasks that assess semantic reasoning beyond transcription."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01016","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_polyspeech-100_ce0b7268"},"modality":null,"name":"PolySpeech-100","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01016","source_url":"https://arxiv.org/abs/2606.01016"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01016","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01016","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_polyworkbench_2abf54e5"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PolyWorkBench evaluates LLM agents on multilingual, long-horizon workplace workflows across five domains: commerce, knowledge work, legal analysis, localization, and manufacturing. It includes 67 tasks, structured scoring via Grade, executable state verification with Pytest, and LLM-as-Judge diagnostics.","descriptions":{"en":"PolyWorkBench evaluates LLM agents on multilingual, long-horizon workplace workflows across five domains: commerce, knowledge work, legal analysis, localization, and manufacturing. It includes 67 tasks, structured scoring via Grade, executable state verification with Pytest, and LLM-as-Judge diagnostics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.06008","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","multilingual"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_polyworkbench_2abf54e5"},"modality":null,"name":"PolyWorkBench","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06008","source_url":"https://arxiv.org/abs/2607.06008"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06008","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06008","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PoolBench isolates pooling strategies as the experimental variable in concept representation evaluation for decoder-only LLMs. It covers 17 concepts, 19 pooling strategies, and 3 open-weight models (Llama-3.1-8B, Gemma-2-9B, Mistral-7B) on a corpus of 37,693 real-text passages, with primary axis linear separability (AUROC) and diagnostic axes for steering and disentanglement.","descriptions":{"en":"PoolBench isolates pooling strategies as the experimental variable in concept representation evaluation for decoder-only LLMs. It covers 17 concepts, 19 pooling strategies, and 3 open-weight models (Llama-3.1-8B, Gemma-2-9B, Mistral-7B) on a corpus of 37,693 real-text passages, with primary axis linear separability (AUROC) and diagnostic axes for steering and disentanglement."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05162","languages":[],"library_direction":null,"modality":null,"name":"PoolBench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05162","source_url":"https://arxiv.org/abs/2608.05162"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05162","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05162","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":119,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:aoidragon/pope","source_record":"catalog_003394ba687ec677"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","safety","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Polling-based Object Probing Evaluation (POPE) is a benchmark for evaluating object hallucination in Large Vision-Language Models (LVLMs). POPE addresses the problem where LVLMs generate objects inconsistent with target images by using a polling-based query method that asks yes/no questions about object presence in images, providing more stable and flexible evaluation of object hallucination.","descriptions":{"en":"Polling-based Object Probing Evaluation (POPE) is a benchmark for evaluating object hallucination in Large Vision-Language Models (LVLMs). POPE addresses the problem where LVLMs generate objects inconsistent with target images by using a polling-based query method that asks yes/no questions about object presence in images, providing more stable and flexible evaluation of object hallucination."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:pope:phi-3.5-vision-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pope?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:pope","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_003394ba687ec677"},"modality":"multimodal","name":"POPE","openness":"unknown","publisher":null,"released":"2023-05-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Evaluating Object Hallucination in Large Vision-Language Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:pope","source_url":"https://arxiv.org/abs/2305.10355"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":88.7,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.887,"raw_min":0.856,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:pope:lfm-2.5-vl-3b","reported_date":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pope?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-vision-instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"}],"slug":"llm-stats-pope","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pope?top_n=500","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":119,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2305.10355","repository":"gh:aoidragon/pope","source_record":"catalog_003394ba687ec677"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding","事实可靠性","Factual Reliability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"POPE is an improved evaluation method for LVLMs' object hallucination by proposing a polling-based query method. It offers a more stable and flexible solution. POPE用于评估视觉语言模型的物体幻觉，基于轮询的查询方法设计，提供了一种更稳定、更灵活的评估方案。","descriptions":{"en":"POPE is an improved evaluation method for LVLMs' object hallucination by proposing a polling-based query method. It offers a more stable and flexible solution.","zh":"POPE用于评估视觉语言模型的物体幻觉，基于轮询的查询方法设计，提供了一种更稳定、更灵活的评估方案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1362","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"artifact:arxiv:2305.10355","primary":"visual-reasoning","source_record":"catalog_003394ba687ec677"},"modality":"multimodal","name":"POPE","openness":"restricted","publisher":"Renmin University of China","released":"2023-05-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1362-pope","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/POPE","topics":["multimodal"],"unit":null},{"aliases":["Large Language Models Systematically Favor Popular Options: Evidence and Mitigation Across MCQs"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:datascienceuibk/popmcq","source_record":"bm_large-language-models-systematically-favor_639da924"},"categories":["Cybersecurity","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multiple-choice question answering across 73,260 MCQ items with six strategies that systematically vary distractor popularity, using accuracy, HPSR, and other bias metrics.","descriptions":{"en":"Evaluates multiple-choice question answering across 73,260 MCQ items with six strategies that systematically vary distractor popularity, using accuracy, HPSR, and other bias metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29257","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","safety-alignment"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_large-language-models-systematically-favor_639da924"},"modality":null,"name":"PopMCQ","openness":"unknown","publisher":null,"released":"2026-08-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29257","source_url":"https://arxiv.org/abs/2608.29257"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29257","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29257","topics":["safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":195,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:alextmallen/adaptive-retrieval","source_record":"catalog_95324dafb4b28b23"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PopQA is an entity-centric open-domain question-answering dataset consisting of 14,000 QA pairs designed to evaluate language models' ability to memorize and recall factual knowledge across entities with varying popularity levels. The dataset probes both parametric memory (stored in model parameters) and non-parametric memory effectiveness, with questions covering 16 diverse relationship types from Wikidata converted to natural language using templates. Created by sampling knowledge triples from Wikidata and converting them to natural language questions, focusing on long-tail entities to understand LMs' strengths and limitations in memorizing factual knowledge.","descriptions":{"en":"PopQA is an entity-centric open-domain question-answering dataset consisting of 14,000 QA pairs designed to evaluate language models' ability to memorize and recall factual knowledge across entities with varying popularity levels. The dataset probes both parametric memory (stored in model parameters) and non-parametric memory effectiveness, with questions covering 16 diverse relationship types from Wikidata converted to natural language using templates. Created by sampling knowledge triples from Wikidata and converting them to natural language questions, focusing on long-tail entities to understand LMs' strengths and limitations in memorizing factual knowledge."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:popqa:granite-3.3-8b-base","reported_at":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/popqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:popqa","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","knowledge-expert-qa"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_95324dafb4b28b23"},"modality":"text","name":"PopQA","openness":"unknown","publisher":null,"released":"2022-12-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:popqa","source_url":"https://arxiv.org/abs/2212.10511"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":26.169999999999998,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.2617,"raw_min":0.229,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:popqa:granite-3.3-8b-base","reported_date":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/popqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"}],"slug":"llm-stats-popqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/popqa?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Software & AI Compute","Vision & 3D","eess.IV"],"collected_at":"2026-10-14T09:44:51Z","description":"POPSICLE benchmarks cryoET segmentation and macromolecular localization using data from the CryoET Data Portal. It spans eukaryotic and prokaryotic systems, purified and in situ samples, and covers dense voxel-wise segmentation and sparse localization tasks. Built on the CryoET Data Portal, it can expand with new data.","descriptions":{"en":"POPSICLE benchmarks cryoET segmentation and macromolecular localization using data from the CryoET Data Portal. It spans eukaryotic and prokaryotic systems, purified and in situ samples, and covers dense voxel-wise segmentation and sparse localization tasks. Built on the CryoET Data Portal, it can expand with new data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.10255","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_popsicle_93cd1103"},"modality":null,"name":"POPSICLE","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10255","source_url":"https://arxiv.org/abs/2606.10255"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10255","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10255","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":10,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:agenticfinlab/portbench","source_record":"bm_portbench_32129f61"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PortBench evaluates LLM-driven portfolio management via a static QA dataset (6,269 questions across seven task templates) and a dynamic five-stage allocation pipeline, spanning six asset classes over ten years. Scoring includes a dual-layer correlation score and CEPS, with evaluation under three stress regimes and investor profiles.","descriptions":{"en":"PortBench evaluates LLM-driven portfolio management via a static QA dataset (6,269 questions across seven task templates) and a dynamic five-stage allocation pipeline, spanning six asset classes over ten years. Scoring includes a dual-layer correlation score and CEPS, with evaluation under three stress regimes and investor profiles."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.27887","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_portbench_32129f61"},"modality":null,"name":"PortBench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27887","source_url":"https://arxiv.org/abs/2605.27887"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27887","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27887","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:amalia-llm/amalia-vl-eval","source_record":"bm_portexto_2594bd31"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PorTEXTO evaluates visual text extraction from contemporary and culturally relevant European Portuguese (pt-PT) images. The benchmark includes synthetic and real-world samples with native-speaker-reviewed transcriptions.","descriptions":{"en":"PorTEXTO evaluates visual text extraction from contemporary and culturally relevant European Portuguese (pt-PT) images. The benchmark includes synthetic and real-world samples with native-speaker-reviewed transcriptions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19096","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","multilingual"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_portexto_2594bd31"},"modality":null,"name":"PorTEXTO","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19096","source_url":"https://arxiv.org/abs/2606.19096"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19096","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19096","topics":["language"],"unit":null},{"aliases":["Benchmarking Automated Security Patch Backporting: How Far Are We?"],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Porting Benchmark evaluates automated security patch backporting using 1,234 curated cases across cross-version, cross-branch, and cross-repository scenarios, with a common evaluation framework and dynamic validation on a subset.","descriptions":{"en":"Porting Benchmark evaluates automated security patch backporting using 1,234 curated cases across cross-version, cross-branch, and cross-repository scenarios, with a common evaluation framework and dynamic validation on a subset."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17671","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","cybersecurity"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_benchmarking-automated-security-patch-back_bac96e83"},"modality":null,"name":"Porting Benchmark","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17671","source_url":"https://arxiv.org/abs/2608.17671"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17671","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17671","topics":["coding","agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_portraitaes-bench_07f57144"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Portrait aesthetic assessment assigns comparable scores according to how effectively human-centered images fulfill their photographic intent.","descriptions":{"en":"Portrait aesthetic assessment assigns comparable scores according to how effectively human-centered images fulfill their photographic intent."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.05010","languages":[],"library_direction":{"basis":"model-review-daily","categories":["visual-reasoning","chat-writing"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_portraitaes-bench_07f57144"},"modality":null,"name":"PortraitAes-Bench","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05010","source_url":"https://arxiv.org/abs/2610.05010"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05010","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05010","topics":["language"],"unit":null},{"aliases":["KAYROS: An Anytime and Exact Open-Source Solver for Duration-Minimization Time-Dependent Vehicle Routing. A Technical Report and a Case Study in Human-AI Engineering"],"attention_signals":null,"categories":["Language & Knowledge","Logistics & Operations","math.OC"],"collected_at":"2026-09-25T04:40:22Z","description":"Poryos2026 evaluates duration-minimization routing on 1,080 paired static and time-dependent instances derived from road networks and controlled traffic.","descriptions":{"en":"Poryos2026 evaluates duration-minimization routing on 1,080 paired static and time-dependent instances derived from road networks and controlled traffic."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.23116","languages":[],"library_direction":null,"modality":null,"name":"Poryos2026","openness":"unknown","publisher":null,"released":"2026-07-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23116","source_url":"https://arxiv.org/abs/2607.23116"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23116","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23116","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":241,"github_stars_30d":28,"hf_dataset_downloads":null,"hf_paper_upvotes":64,"join":"claire_record_id","repository":"gh:yaxin9luo/autodesign","source_record":"bm_posterbench_1bd5f76e"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PosterBench evaluates academic paper-to-poster generation. It includes a 100-paper Main Track across five disciplines and a 10-paper mini subset, with automated scoring and human evaluation.","descriptions":{"en":"PosterBench evaluates academic paper-to-poster generation. It includes a 100-paper Main Track across five disciplines and a 10-paper mini subset, with automated scoring and human evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.13560","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_posterbench_1bd5f76e"},"modality":null,"name":"PosterBench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13560","source_url":"https://arxiv.org/abs/2608.13560"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13560","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13560","topics":[],"unit":null},{"aliases":["PosterHarness: Turning Scientific Poster Generation into an Auditable Instruction-Following Benchmark"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"An auditable evaluation harness for testing whether scientific-poster generators follow layout, placeholder, abstention, text, and provenance requirements.","descriptions":{"en":"An auditable evaluation harness for testing whether scientific-poster generators follow layout, placeholder, abstention, text, and provenance requirements."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.03006","languages":[],"library_direction":null,"modality":null,"name":"POSTERHARNESS","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03006","source_url":"https://arxiv.org/abs/2607.03006"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03006","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03006","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":14,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:neuraloperator/posteriorbench","source_record":"bm_posteriorbench_7203afec"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark evaluating whether generative scientific inverse solvers recover full posterior distributions across four physics-based inverse tasks.","descriptions":{"en":"Benchmark evaluating whether generative scientific inverse solvers recover full posterior distributions across four physics-based inverse tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.20794","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_posteriorbench_7203afec"},"modality":null,"name":"PosteriorBench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.20794","source_url":"https://arxiv.org/abs/2609.20794"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-20794","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20794","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning","OCR与文档理解","OCR and Document Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The PosterSum dataset is a multimodal benchmark designed for the summarization of scientific posters into research paper abstracts. The dataset consists of 16,305 research posters collected from major machine learning conferences, including ICLR, ICML, and NeurIPS, spanning the years 2022-2024. PosterSum 数据集是一个多模态基准数据集，旨在将科学海报总结成研究论文摘要。该数据集包含从2022-2024年的主要机器学习会议（包括 ICLR、ICML 和 NeurIPS）收集的 16,305 篇研究海报。","descriptions":{"en":"The PosterSum dataset is a multimodal benchmark designed for the summarization of scientific posters into research paper abstracts. The dataset consists of 16,305 research posters collected from major machine learning conferences, including ICLR, ICML, and NeurIPS, spanning the years 2022-2024.","zh":"PosterSum 数据集是一个多模态基准数据集，旨在将科学海报总结成研究论文摘要。该数据集包含从2022-2024年的主要机器学习会议（包括 ICLR、ICML 和 NeurIPS）收集的 16,305 篇研究海报。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1579","languages":[],"library_direction":null,"modality":"multimodal","name":"PosterSum","openness":"unknown","publisher":"University of Edinburgh","released":"2025-02-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1579-postersum","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PosterSum","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":588,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_7a593ca454f1246a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code","systems"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PostTrainBench evaluates a model's ability to autonomously post-train base models. Given pretrain-only base models, the agent must complete the full pipeline of data synthesis, training, evaluation, and iteration within a time budget, scored across downstream benchmarks such as AIME2025, BFCL, GPQA Main, GSM8K, and HumanEval.","descriptions":{"en":"PostTrainBench evaluates a model's ability to autonomously post-train base models. Given pretrain-only base models, the agent must complete the full pipeline of data synthesis, training, evaluation, and iteration within a time budget, scored across downstream benchmarks such as AIME2025, BFCL, GPQA Main, GSM8K, and HumanEval."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:posttrainbench:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:posttrainbench","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_7a593ca454f1246a"},"modality":"text","name":"PostTrainBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":39.800000000000004,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.398,"raw_min":0.165,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:posttrainbench:glm-5.3","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-posttrainbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":["PostTrainBench","PostTrainBench Lite"],"attention_signals":null,"categories":["ai_research"],"collected_at":null,"description":"Measures a model's ability to run post-training itself. Hardware differs between reported runs (H100 vs H20), which moves the score.","descriptions":{"en":"Measures a model's ability to run post-training itself. Hardware differs between reported runs (H100 vs H20), which moves the score."},"evidence_summary":{"document_count":4,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000posttrainbench\u0000moonshot_kimi_k3_model_card\u0000posttrainbench\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"first_score_reported_at":"2026-06-13","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000posttrainbench\u0000moonshot_kimi_k3_model_card\u0000posttrainbench\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:posttrainbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["ai-rnd"],"join":"review:14H034160212:2026-10-15","primary":"ai-rnd","source_record":null},"modality":null,"name":"PostTrainBench","openness":"unknown","publisher":null,"released":"2026-01-20","released_reference":{"basis":"benchmark_release","source_key":"model-reports:posttrainbench","source_url":"https://posttrainbench.com/"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":36.6,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":36.6,"raw_min":34.3,"source_reference":{"obs_id":"curated\u0000posttrainbench\u0000moonshot_kimi_k3_model_card\u0000posttrainbench\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000posttrainbench\u0000moonshot_kimi_k3_model_card\u0000posttrainbench\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"}],"slug":"posttrainbench","source":"model_reports","source_url":"https://posttrainbench.com/","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":591,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:aisa-group/posttrainbench","source_record":"catalog_93924c7b9f3ba935"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code","systems"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PostTrainBench Lite measures whether an agent can design and execute a full post-training strategy (data, prompts, RL recipe, and eval loop) for a pretrained base model under a constrained time budget, scored as normalized mean reward over the improvement window.","descriptions":{"en":"PostTrainBench Lite measures whether an agent can design and execute a full post-training strategy (data, prompts, RL recipe, and eval loop) for a pretrained base model under a constrained time budget, scored as normalized mean reward over the improvement window."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:posttrainbench-lite:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench-lite?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:posttrainbench-lite","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_93924c7b9f3ba935"},"modality":"text","name":"PostTrainBench Lite","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":51.5,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.515,"raw_min":0.296,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:posttrainbench-lite:gpt-5.6-terra","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench-lite?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-posttrainbench-lite","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench-lite?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark extending ARC to assess understanding of underlying abstract rules across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion.","descriptions":{"en":"A benchmark extending ARC to assess understanding of underlying abstract rules across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27288","languages":[],"library_direction":null,"modality":null,"name":"PotARCin","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27288","source_url":"https://arxiv.org/abs/2609.27288"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27288","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27288","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:trashchenkov/power-systems-agent-benchmark","source_record":"bm_power-systems-agent-benchmark_68561f3f"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Power Systems Agent Benchmark is an executable benchmark for power-engineering agents. It includes 41 task families with deterministic evaluators that recompute engineering quantities and check constraints, plus held-out generation.","descriptions":{"en":"Power Systems Agent Benchmark is an executable benchmark for power-engineering agents. It includes 41 task families with deterministic evaluators that recompute engineering quantities and check constraints, plus held-out generation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.20950","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design","professional-work"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_power-systems-agent-benchmark_68561f3f"},"modality":null,"name":"Power Systems Agent Benchmark","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20950","source_url":"https://arxiv.org/abs/2606.20950"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20950","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20950","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:open-compass/powerbench","source_record":"bm_powerbench_7dfaaf7f"},"categories":["General AI","Information retrieval","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"PowerBench evaluates LLM agents on 300 questions across three task families requiring autonomous evidence retrieval and reasoning over interconnected power system data.","descriptions":{"en":"PowerBench evaluates LLM agents on 300 questions across three task families requiring autonomous evidence retrieval and reasoning over interconnected power system data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.34492","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design","deep-research"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_powerbench_7dfaaf7f"},"modality":null,"name":"PowerBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34492","source_url":"https://arxiv.org/abs/2609.34492"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34492","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34492","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Factuality","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"PowerCodeBench is a benchmark generator for power system code generation, paired with an intervention method, but no artifacts are provided in this article.","descriptions":{"en":"PowerCodeBench is a benchmark generator for power system code generation, paired with an intervention method, but no artifacts are provided in this article."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.31478","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","science"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_powercodebench_c0ce0f67"},"modality":null,"name":"PowerCodeBench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31478","source_url":"https://arxiv.org/abs/2605.31478"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31478","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31478","topics":["coding","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"PPE-Bench evaluates machine unlearning in multimodal large language models under private-public entanglement, where images contain a target individual to forget and public elements to preserve.","descriptions":{"en":"PPE-Bench evaluates machine unlearning in multimodal large language models under private-public entanglement, where images contain a target individual to forget and public elements to preserve."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02897","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_ppe-bench_ab4b6b83"},"modality":null,"name":"PPE-Bench","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02897","source_url":"https://arxiv.org/abs/2607.02897"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02897","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02897","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Photoplethysmography (PPG) is increasingly recorded by wearable and clinical devices at enormous scale, and a growing number of PPG foundation models (FMs) have been developed to compute reusable representations for various downstream task…","descriptions":{"en":"Photoplethysmography (PPG) is increasingly recorded by wearable and clinical devices at enormous scale, and a growing number of PPG foundation models (FMs) have been developed to compute reusable representations for various downstream task…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.13435","languages":[],"library_direction":{"basis":"model-review-daily","categories":["health-medicine","science"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_ppg-bench_c1f82fe1"},"modality":null,"name":"PPG-Bench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13435","source_url":"https://arxiv.org/abs/2610.13435"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13435","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13435","topics":["medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":15,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:microsoft/ppteval","source_record":"bm_ppt-eval_9317e9e6"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"PPT-Eval is a benchmark of 120 PowerPoint tasks across 12 files for computer-use agents. It covers content creation and editing, with rubric-based evaluation that awards partial credit and provides natural language feedback.","descriptions":{"en":"PPT-Eval is a benchmark of 120 PowerPoint tasks across 12 files for computer-use agents. It covers content creation and editing, with rubric-based evaluation that awards partial credit and provides natural language feedback."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.31154","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","professional-work"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_ppt-eval_9317e9e6"},"modality":null,"name":"PPT-Eval","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31154","source_url":"https://arxiv.org/abs/2606.31154"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31154","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31154","topics":["computer-use","agents"],"unit":null},{"aliases":["PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides","Einsia PPTBench"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Einsia / Navers Lab’s 500 scientific flow-diagram reconstruction tasks, distinct from the earlier benchmark of the same name. Mean score is on a 0–100 scale with invalid and majority-gated outputs scored zero, not a success percentage. Results include the model, native agent harness and reasoning effort. Protocol-development and judge-validation tasks remain in the full evaluation. The source figures retain their individual paper licenses.","descriptions":{"en":"Einsia / Navers Lab’s 500 scientific flow-diagram reconstruction tasks, distinct from the earlier benchmark of the same name. Mean score is on a 0–100 scale with invalid and majority-gated outputs scored zero, not a success percentage. Results include the model, native agent harness and reasoning effort. Protocol-development and judge-validation tasks remain in the full evaluation. The source figures retain their individual paper licenses."},"editorial_guide":{"id":"pptbench","record_keys":["model-reports:pptbench"],"reviewed_at":"2026-10-27","scope":{"en":"Einsia / Navers Lab’s 500-task scientific flow-diagram reconstruction benchmark, using the September 2026 paper v2. A different, earlier benchmark also uses the name PPTBench.","zh":"对应 Einsia / Navers Lab 的 500 道科学流程图重建任务，以 2026 年 9 月论文 v2 为准。另一个较早的评测也叫 PPTBench，二者应分开理解。"},"sections":[{"id":"task","source_ids":["paper","repository"],"text":{"en":"Each of 500 frozen tasks provides a reference figure from a real arXiv paper and approved raster resources. The agent reconstructs the slide with native text, shapes and connectors; copying the complete reference image into a slide is not a valid reconstruction.","zh":"500 道固定任务提供真实 arXiv 论文中的参考图和允许使用的栅格素材。智能体需要用原生文字、形状与连接线重建单页幻灯片；把整张参考图贴进幻灯片不算有效重建。"},"title":{"en":"From pixels to editable objects","zh":"从图像到可编辑对象"}},{"id":"grading","source_ids":["paper","repository"],"text":{"en":"A deterministic check verifies the single-slide editable artifact. Three GPT-5.6 Luna High rounds then inspect process semantics and rendering. A semantic or rendering failure in at least two rounds makes the score zero. Otherwise, merged findings drive fixed deductions in layout (30), typography (40) and local graphics (30).","zh":"程序先检查单页文件及可编辑内容，再由 GPT-5.6 Luna High 独立检查三轮流程语义与渲染质量。至少两轮判定语义或渲染失败，得分即为零；其余结果按合并后的问题清单，对布局（30 分）、文字（40 分）和局部图形（30 分）执行固定扣分。"},"title":{"en":"Validity and hard gates come first","zh":"先检查文件与关键错误"}},{"id":"results","source_ids":["paper","leaderboard"],"text":{"en":"The mean score includes invalid and gated outputs across all 500 tasks. It is not a success percentage. Gate-pass rate and mean detail score conditional on passing answer different questions. The published 36 configurations use Codex, Claude Code or OpenCode, so a row measures the complete model–harness–effort setup.","zh":"平均分覆盖全部 500 道任务，包含无效文件和被关键检查判零的结果，因此不能当作成功率。过门槛比例与通过后的条件细节分数回答的是不同问题。公开的 36 个配置使用 Codex、Claude Code 或 OpenCode，每行应连同模型、框架和推理强度一起阅读。"},"title":{"en":"Keep model, harness and effort together","zh":"连同框架与推理强度一起比较"}},{"id":"limits","source_ids":["paper"],"text":{"en":"The full evaluation includes tasks used to develop the judging protocol and validate judges. Model-based judgments retain uncertainty, and human validation of gates is not validation of every continuous detail score. More inspection correlates with higher scores in the study, but that association does not establish a causal benefit.","zh":"完整评测包含制定判分流程和验证裁判时使用的任务。模型裁判仍有不确定性；关键检查与人工意见一致，不等于每个连续细节分数都经人工确认。研究观察到更多检查与较高分数相关，但未据此证明增加检查次数必然提高质量。"},"title":{"en":"What the score leaves unresolved","zh":"分数不能单独说明什么"}},{"id":"reproduce","source_ids":["repository"],"text":{"en":"The repository supplies task manifests, hashes, prompts, score data and a materialization workflow. Official rendering uses LibreOffice 25.8. Code is MIT-licensed and benchmark-authored metadata is CC BY 4.0; figures retain their original paper licenses, so the code license does not cover every source image.","zh":"仓库提供任务清单、哈希、提示词、成绩和素材还原流程，官方渲染采用 LibreOffice 25.8。代码为 MIT 许可，评测自产元数据为 CC BY 4.0；科学插图沿用原论文许可，不能把代码许可直接套用到全部源图像。"},"title":{"en":"Reproduce the frozen artifact protocol","zh":"复现时保持素材和渲染协议一致"}}],"sources":[{"id":"project","title":"PPTBench official project","url":"https://lab.einsia.ai/pptbench/"},{"id":"leaderboard","title":"PPTBench official leaderboard","url":"https://lab.einsia.ai/pptbench/leaderboard/"},{"id":"paper","title":"PPTBench paper v2","url":"https://arxiv.org/html/2609.29718v2"},{"id":"repository","title":"PPTBench official repository and reproduction guide","url":"https://github.com/Einsia/PPTBench"}],"summary":{"en":"A coding agent turns a diagram image into one editable PowerPoint slide while preserving its objects, text and process connections.","zh":"编程智能体需要把科学流程图转成一页可编辑的 PowerPoint，同时保留对象、文字及流程连接关系。"},"title":"PPTBench (Einsia)"},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000pptbench\u0000pptbench_v2_results_table\u0000PPTBench (Einsia): frozen 500 scientific flow-diagram tasks; paper v2\u0000Native editable single-slide PPTX; 500 tasks; Claude Code; reasoning effort high; artifact validation; GPT-5.6 Luna High judge, 3 rounds; majority semantic/render gate; deterministic 30/40/30 detail deductions; mean over all tasks including gated zeros\u0000Claude Opus 5","reported_at":"2026-09-28","source_url":"https://arxiv.org/html/2609.29718v2#S3.T1"},"first_score_reported_at":"2026-09-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000pptbench\u0000pptbench_v2_results_table\u0000PPTBench (Einsia): frozen 500 scientific flow-diagram tasks; paper v2\u0000Native editable single-slide PPTX; 500 tasks; Claude Code; reasoning effort high; artifact validation; GPT-5.6 Luna High judge, 3 rounds; majority semantic/render gate; deterministic 30/40/30 detail deductions; mean over all tasks including gated zeros\u0000Claude Opus 5","reported_at":"2026-09-28","source_url":"https://arxiv.org/html/2609.29718v2#S3.T1"},"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"model-reports:pptbench","languages":[],"library_direction":null,"modality":null,"name":"PPTBench (Einsia)","openness":"unknown","publication":{"evidence":[{"note":"The author project names this benchmark and links its introducing paper and available release resources. This exact-record review preserves the documented task version, metrics and evaluation setup.","url":"https://lab.einsia.ai/pptbench/"},{"note":"Navers Lab / Einsia research index links this named project. Method papers and memory software in the same index are not added as benchmarks.","url":"https://lab.einsia.ai/research/"}],"id":"einsia-pptbench-original","links":[{"kind":"website","title":"PPTBench (Einsia) · official project","url":"https://lab.einsia.ai/pptbench/"},{"kind":"paper","title":"PPTBench (Einsia) · introducing paper","url":"https://arxiv.org/abs/2609.29718"},{"kind":"repo","title":"PPTBench (Einsia) · official repository","url":"https://github.com/Einsia/PPTBench"},{"kind":"website","title":"PPTBench (Einsia) · author leaderboard","url":"https://lab.einsia.ai/pptbench/leaderboard/"}],"record_keys":["model-reports:pptbench"],"reviewed_at":"2026-10-27","scope":{"en":"Einsia / Navers Lab’s 500-task scientific flow-diagram reconstruction benchmark, using the September 2026 paper v2. A different, earlier benchmark also uses the name PPTBench.","zh":"对应 Einsia / Navers Lab 的 500 道科学流程图重建任务，以 2026 年 9 月论文 v2 为准。另一个较早的评测也叫 PPTBench，二者应分开理解。"}},"publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"model-reports:pptbench","source_url":"https://lab.einsia.ai/pptbench/"},"repo_kind":null,"repo_resolution_status":null,"score_count":36,"score_direction":"higher_is_better","score_summary":{"display_max":77.34,"display_multiplier":1,"model_count":10,"model_count_basis":"source_model_id","numeric_count":36,"raw_max":77.34,"raw_min":1.53,"source_reference":{"obs_id":"curated\u0000pptbench\u0000pptbench_v2_results_table\u0000PPTBench (Einsia): frozen 500 scientific flow-diagram tasks; paper v2\u0000Native editable single-slide PPTX; 500 tasks; Codex; reasoning effort high; artifact validation; GPT-5.6 Luna High judge, 3 rounds; majority semantic/render gate; deterministic 30/40/30 detail deductions; mean over all tasks including gated zeros\u0000GPT-6-Astra","observation_id":"curated\u0000pptbench\u0000pptbench_v2_results_table\u0000PPTBench (Einsia): frozen 500 scientific flow-diagram tasks; paper v2\u0000Native editable single-slide PPTX; 500 tasks; Codex; reasoning effort high; artifact validation; GPT-5.6 Luna High judge, 3 rounds; majority semantic/render gate; deterministic 30/40/30 detail deductions; mean over all tasks including gated zeros\u0000GPT-6-Astra","reported_at":"2026-09-28","reported_date":"2026-09-28","source_id":"pptbench_v2_results_table","source_url":"https://arxiv.org/html/2609.29718v2#S3.T1"},"unit":"points"},"scored_models":[{"date":"2026-09-28","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2026-09-28","date_precision":"document_publication","name":"Claude Sonnet 5"},{"date":"2026-09-28","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-09-28","date_precision":"document_publication","name":"GPT-5.5"},{"date":"2026-09-28","date_precision":"document_publication","name":"GPT-5.6 Luna"},{"date":"2026-09-28","date_precision":"document_publication","name":"GPT-5.6 Sol"},{"date":"2026-09-28","date_precision":"document_publication","name":"GPT-5.6 Terra"},{"date":"2026-09-28","date_precision":"document_publication","name":"GPT-6-Astra"},{"date":"2026-09-28","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-09-28","date_precision":"document_publication","name":"Qwen 3.8 Max"}],"slug":"pptbench","source":"model_reports","source_url":"https://lab.einsia.ai/pptbench/","topics":["coding","agents","multimodal"],"unit":"points"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_pragma_88c92ac1"},"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates personalized guidance through long-term conversation histories and evidence annotations, testing retrieval accuracy and memory-grounded reasoning.","descriptions":{"en":"Evaluates personalized guidance through long-term conversation histories and evidence annotations, testing retrieval accuracy and memory-grounded reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09664","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_pragma_88c92ac1"},"modality":null,"name":"PRAGMA","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09664","source_url":"https://arxiv.org/abs/2609.09664"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09664","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09664","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"PragMatch is a controlled set of 3,000 image-text pairs derived from MMSD2.0 for studying pragmatic incongruity in multimodal sarcasm detection, with original sarcastic examples and constructed literal and hard-negative pairs.","descriptions":{"en":"PragMatch is a controlled set of 3,000 image-text pairs derived from MMSD2.0 for studying pragmatic incongruity in multimodal sarcasm detection, with original sarcastic examples and constructed literal and hard-negative pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09772","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_pragmatch_5cfd2fc5"},"modality":null,"name":"PragMatch","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09772","source_url":"https://arxiv.org/abs/2608.09772"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09772","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09772","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":20,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:scaleapi/prbench","source_record":"catalog_44c9e16983fcb045"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","knowledge","finance"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PRBench-Finance evaluates professional reasoning on finance tasks.","descriptions":{"en":"PRBench-Finance evaluates professional reasoning on finance tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:prbench-finance:qwen3.8-max","reported_at":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-finance?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:prbench-finance","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_44c9e16983fcb045"},"modality":"text","name":"PRBench-Finance","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":58.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.583,"raw_min":0.583,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:prbench-finance:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-finance?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-prbench-finance","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-finance?top_n=500","topics":["reasoning-math","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":20,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:scaleapi/prbench","source_record":"catalog_780cd471a7228fcb"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning","knowledge"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PRBench-Legal evaluates professional reasoning on legal tasks.","descriptions":{"en":"PRBench-Legal evaluates professional reasoning on legal tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:prbench-legal:qwen3.8-max","reported_at":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-legal?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:prbench-legal","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_780cd471a7228fcb"},"modality":"text","name":"PRBench-Legal","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":57.599999999999994,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.576,"raw_min":0.576,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:prbench-legal:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-legal?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-prbench-legal","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-legal?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_pre-flight_898fa920"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Pre-Flight evaluates large language models on aviation operational knowledge via 300 multiple-choice questions drawn from international standards and airport ground operations material, covering ground operations, ICAO and FAA regulations, general aviation knowledge, and operational scenarios. Scoring is by accuracy under a standard multiple-choice protocol using the Inspect framework.","descriptions":{"en":"Pre-Flight evaluates large language models on aviation operational knowledge via 300 multiple-choice questions drawn from international standards and airport ground operations material, covering ground operations, ICAO and FAA regulations, general aviation knowledge, and operational scenarios. Scoring is by accuracy under a standard multiple-choice protocol using the Inspect framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01829","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","professional-work"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_pre-flight_898fa920"},"modality":null,"name":"Pre-Flight","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01829","source_url":"https://arxiv.org/abs/2607.01829"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01829","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01829","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"PreActBench is a benchmark for predictive monitoring in LLMs, consisting of 1,000 paired ethical and unethical action trajectories across five domains. It evaluates whether models can infer if a partial trajectory will culminate in unethical action, using the Prefix Foresight F1 metric.","descriptions":{"en":"PreActBench is a benchmark for predictive monitoring in LLMs, consisting of 1,000 paired ethical and unethical action trajectories across five domains. It evaluates whether models can infer if a partial trajectory will culminate in unethical action, using the Prefix Foresight F1 metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09890","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_preact-bench_d5c8c56f"},"modality":null,"name":"PreAct-Bench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09890","source_url":"https://arxiv.org/abs/2606.09890"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09890","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09890","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A multilingual grounded-generation benchmark that evaluates both support precision and coverage of complete-oracle facts using Formula 1 telemetry, with a second complete-oracle weather domain.","descriptions":{"en":"A multilingual grounded-generation benchmark that evaluates both support precision and coverage of complete-oracle facts using Formula 1 telemetry, with a second complete-oracle weather domain."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09376","languages":[],"library_direction":null,"modality":null,"name":"Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09376","source_url":"https://arxiv.org/abs/2606.09376"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09376","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09376","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"PredAct-Bench evaluates dialogue agents paired with imperfect tools using educational datasets, measuring AI-assisted decision-making and trust metrics.","descriptions":{"en":"PredAct-Bench evaluates dialogue agents paired with imperfect tools using educational datasets, measuring AI-assisted decision-making and trust metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02372","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_predact-bench_f35f1a5f"},"modality":null,"name":"PredAct-Bench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02372","source_url":"https://arxiv.org/abs/2608.02372"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02372","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02372","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":26,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_def7139ed7186deb"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"PresentBench evaluates AI agents on producing presentation-style deliverables, such as generating lesson-plan slides and structured documents from source materials.","descriptions":{"en":"PresentBench evaluates AI agents on producing presentation-style deliverables, such as generating lesson-plan slides and structured documents from source materials."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:presentbench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/presentbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:presentbench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_def7139ed7186deb"},"modality":"text","name":"PresentBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":54.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.546,"raw_min":0.483,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:presentbench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/presentbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-presentbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/presentbench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:pashasan/pricebench-emnlp","source_record":"bm_pricebench_ed2ec407"},"categories":["General AI","Language & Knowledge","econ.GN"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM price, quality, and brand preferences from hotel-booking choices using a logit choice model.","descriptions":{"en":"Evaluates LLM price, quality, and brand preferences from hotel-booking choices using a logit choice model."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.31468","languages":[],"library_direction":null,"modality":null,"name":"PriceBench","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.31468","source_url":"https://arxiv.org/abs/2609.31468"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-31468","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31468","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":null,"source_record":"bm_principia_eb88a792"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates Newtonian physics reasoning in video models through relational consistency between paired objects across eight phenomena, using a calibration-independent consistency score derived from real-world scenes.","descriptions":{"en":"Evaluates Newtonian physics reasoning in video models through relational consistency between paired objects across eight phenomena, using a calibration-independent consistency score derived from real-world scenes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.04200","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_principia_eb88a792"},"modality":null,"name":"Principia","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04200","source_url":"https://arxiv.org/abs/2609.04200"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04200","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04200","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_principle-bench_119ba0f5"},"categories":["Finance","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Principle-Bench contains 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles, with paraphrase, adversarial keyword-stuffing, and boundary perturbations for evaluating LLM-as-judge on accuracy, paraphrase robustness, adversarial robustness, and calibration.","descriptions":{"en":"Principle-Bench contains 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles, with paraphrase, adversarial keyword-stuffing, and boundary perturbations for evaluating LLM-as-judge on accuracy, paraphrase robustness, adversarial robustness, and calibration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14329","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","safety-alignment"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_principle-bench_119ba0f5"},"modality":null,"name":"Principle-Bench","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14329","source_url":"https://arxiv.org/abs/2608.14329"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14329","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14329","topics":["safety","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:daotuanan/prionner","source_record":"bm_prionner_f3eb1b08"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Named entity recognition dataset from PubMed abstracts on prion disease, with 317 abstracts annotated for 15 coarse and 31 fine-grained entity types, plus evaluation scripts and train/test splits.","descriptions":{"en":"Named entity recognition dataset from PubMed abstracts on prion disease, with 317 abstracts annotated for 15 coarse and 31 fine-grained entity types, plus evaluation scripts and train/test splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.28375","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_prionner_f3eb1b08"},"modality":null,"name":"PrionNER","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28375","source_url":"https://arxiv.org/abs/2605.28375"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28375","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28375","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":9,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tengbo-yu/prism","source_record":"bm_prism_da9ef810"},"categories":["Contact-rich control","Force-aware manipulation","Imitation learning","Manufacturing & Process Control","Multimodal","Multimodal robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PRISM is a dataset of over 5,000 trajectories across 25 industrial manipulation tasks with multimodal sensing. It provides teleoperated demonstrations for contact-rich operations but lacks a standardized evaluation protocol or scoring contract.","descriptions":{"en":"PRISM is a dataset of over 5,000 trajectories across 25 industrial manipulation tasks with multimodal sensing. It provides teleoperated demonstrations for contact-rich operations but lacks a standardized evaluation protocol or scoring contract."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17962","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_prism_da9ef810"},"modality":null,"name":"PRISM","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17962","source_url":"https://arxiv.org/abs/2608.17962"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17962","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17962","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.MM"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates text-to-audio-video generation with an audio-centric diagnostic benchmark comprising 900 human-verified samples, factorizing audio evaluation by audio type and sound-source visibility, and assessing four perceptual dimensions via 35 fine-grained criteria using an MLLM-as-a-Judge protocol.","descriptions":{"en":"Evaluates text-to-audio-video generation with an audio-centric diagnostic benchmark comprising 900 human-verified samples, factorizing audio evaluation by audio type and sound-source visibility, and assessing four perceptual dimensions via 35 fine-grained criteria using an MLLM-as-a-Judge protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.04867","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio","image-video-generation","health-medicine"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_prism-bench_97126e49"},"modality":null,"name":"PRISM-Bench","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04867","source_url":"https://arxiv.org/abs/2609.04867"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04867","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04867","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"A multi-axis discriminative benchmark for compact vision-language models that scores items along seven axes and combines them into a single PScore, recycling items from fifteen public benchmarks.","descriptions":{"en":"A multi-axis discriminative benchmark for compact vision-language models that scores items along seven axes and combines them into a single PScore, recycling items from fifteen public benchmarks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27395","languages":[],"library_direction":null,"modality":null,"name":"PRISM-VLM","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27395","source_url":"https://arxiv.org/abs/2609.27395"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27395","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27395","topics":["multimodal"],"unit":null},{"aliases":["PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-09-25T04:40:22Z","description":"A 1,182-case benchmark across 7 acquisition behaviours and 16 domains that audits whether tool-using LLM agents acquire sensitive data beyond task scope.","descriptions":{"en":"A 1,182-case benchmark across 7 acquisition behaviours and 16 domains that audits whether tool-using LLM agents acquire sensitive data beyond task scope."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.00152","languages":[],"library_direction":null,"modality":null,"name":"PrivacyPeek","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00152","source_url":"https://arxiv.org/abs/2606.00152"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00152","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00152","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates text-to-text privatization methods across defined desiderata through modular metrics and a live leaderboard.","descriptions":{"en":"Evaluates text-to-text privatization methods across defined desiderata through modular metrics and a live leaderboard."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29624","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_privbench_22a5aa7a"},"modality":null,"name":"PrivBench","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29624","source_url":"https://arxiv.org/abs/2608.29624"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29624","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29624","topics":["safety"],"unit":null},{"aliases":["PrivDrift: Auditing User-Secret Leakage Under Topic Drift in Active LLM Conversations"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The benchmark evaluates whether user-disclosed secrets remain recoverable in active LLM conversations after topic drift and persuasion-based probing.","descriptions":{"en":"The benchmark evaluates whether user-disclosed secrets remain recoverable in active LLM conversations after topic drift and persuasion-based probing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.30094","languages":[],"library_direction":null,"modality":null,"name":"PrivDrift","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30094","source_url":"https://arxiv.org/abs/2609.30094"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30094","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30094","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"PriVE-Bench evaluates vision-language models' visual grounding using paired original and counterfactual images, with PriVE-Tools extending to tool-derived evidence.","descriptions":{"en":"PriVE-Bench evaluates vision-language models' visual grounding using paired original and counterfactual images, with PriVE-Tools extending to tool-derived evidence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.16311","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_prive-bench_ef1c909e"},"modality":null,"name":"PriVE-Bench","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.16311","source_url":"https://arxiv.org/abs/2607.16311"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-16311","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16311","topics":[],"unit":null},{"aliases":["PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment"],"attention_signals":{"as_of":"2026-10-12","github_stars":41,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"claire_record_id","repository":"gh:yuyangliu2003/prm-as-a-judge","source_record":"bm_prm-as-a-judge-1-5-a-toolkit-for-robot-pro_42c65f99"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A toolkit that converts robotic rollout videos into progress curves and computes metrics for failure progress, recovery, and execution quality, with a benchmark and evaluation suite for process reward models.","descriptions":{"en":"A toolkit that converts robotic rollout videos into progress curves and computes metrics for failure progress, recovery, and execution quality, with a benchmark and evaluation suite for process reward models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14284","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_prm-as-a-judge-1-5-a-toolkit-for-robot-pro_42c65f99"},"modality":null,"name":"PRM-as-a-Judge 1.5","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14284","source_url":"https://arxiv.org/abs/2608.14284"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14284","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14284","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","大语言模型","LLM","数理能力","Math"],"collected_at":"2026-08-18T00:00:00+00:00","description":"PRMBench is a benchmark dataset for evaluating process-level reward models (PRMs). It consists of 6,216 data instances, each containing a question, a solution process, and a modified process with errors. PRMBench 是一个用于评估过程级奖励模型（PRM）的基准数据集。它包含 6,216 个数据实例，每个实例包含一个问题、一个解决方案过程以及一个包含错误的修改过程。","descriptions":{"en":"PRMBench is a benchmark dataset for evaluating process-level reward models (PRMs). It consists of 6,216 data instances, each containing a question, a solution process, and a modified process with errors.","zh":"PRMBench 是一个用于评估过程级奖励模型（PRM）的基准数据集。它包含 6,216 个数据实例，每个实例包含一个问题、一个解决方案过程以及一个包含错误的修改过程。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1681","languages":[],"library_direction":null,"modality":null,"name":"PRMBench_Preview","openness":"open","publisher":"FDU, etc.","released":"2025-01-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1681-prmbench-preview","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PRMBench_Preview","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:2231122/prmu","source_record":"bm_prmu_0126a6f3"},"categories":["Factuality","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"PRMU evaluates corpus-free multimodal unlearning of person-related knowledge in MLLMs, using textual and visual probes including adversarial evaluation and locality analysis.","descriptions":{"en":"PRMU evaluates corpus-free multimodal unlearning of person-related knowledge in MLLMs, using textual and visual probes including adversarial evaluation and locality analysis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11149","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_prmu_0126a6f3"},"modality":null,"name":"PRMU","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11149","source_url":"https://arxiv.org/abs/2608.11149"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11149","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11149","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A prompt-conditioned benchmark for open-vocabulary visual grounding in heterogeneous, real-world environments.","descriptions":{"en":"A prompt-conditioned benchmark for open-vocabulary visual grounding in heterogeneous, real-world environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27076","languages":[],"library_direction":null,"modality":null,"name":"Pro-Bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27076","source_url":"https://arxiv.org/abs/2609.27076"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27076","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27076","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":null,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:v0yager33/proactivebench","source_record":"bm_proactivebench_d1f3f8ef"},"categories":["General AI","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ProactiveBench evaluates models at one-second stream intervals without an explicit response cue in streaming video settings.","descriptions":{"en":"ProactiveBench evaluates models at one-second stream intervals without an explicit response cue in streaming video settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.12658","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_proactivebench_d1f3f8ef"},"modality":null,"name":"ProactiveBench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12658","source_url":"https://arxiv.org/abs/2609.12658"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12658","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12658","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A probability used for a decision should refer to the same event across equivalent requests.","descriptions":{"en":"A probability used for a decision should refer to the same event across equivalent requests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37470","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","factuality"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_probability-contracts_8043efbf"},"modality":null,"name":"Probability Contracts","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37470","source_url":"https://arxiv.org/abs/2609.37470"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37470","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37470","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ProBench is a benchmark that contains open-ended multimodal queries that require intensive expert-level knowledge to solve. ProBench是一个包含需要大量专家级知识来解决的开放式多模态查询的基准。ProBench 包含 10 个任务领域和 56 个子领域，支持 17 种语言，并支持最多 13 轮对话。","descriptions":{"en":"ProBench is a benchmark that contains open-ended multimodal queries that require intensive expert-level knowledge to solve.","zh":"ProBench是一个包含需要大量专家级知识来解决的开放式多模态查询的基准。ProBench 包含 10 个任务领域和 56 个子领域，支持 17 种语言，并支持最多 13 轮对话。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1632","languages":[],"library_direction":null,"modality":"multimodal","name":"ProBench","openness":"restricted","publisher":"ANU, NTU","released":"2025-03-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1632-probench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ProBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Potential benchmark for LLM-driven PL/SQL development from natural language in direct and interactive scenarios.","descriptions":{"en":"Potential benchmark for LLM-driven PL/SQL development from natural language in direct and interactive scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.06527","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_procarena_19abe50c"},"modality":null,"name":"ProcArena","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06527","source_url":"https://arxiv.org/abs/2609.06527"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06527","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06527","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["学科","Examination","推理","Reasoning","大语言模型","LLM","数理能力","Math"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ProcessBench can measure the ability to identify erroneous steps in mathematical reasoning. It consists of 3,400 test cases, primarily focused on competition- and Olympiad-level math problems. ProcessBench，用于衡量识别数学推理中错误步骤的能力。它包含 3,400 个测试案例，主要关注竞赛和奥林匹克级别的数学问题。","descriptions":{"en":"ProcessBench can measure the ability to identify erroneous steps in mathematical reasoning. It consists of 3,400 test cases, primarily focused on competition- and Olympiad-level math problems.","zh":"ProcessBench，用于衡量识别数学推理中错误步骤的能力。它包含 3,400 个测试案例，主要关注竞赛和奥林匹克级别的数学问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1621","languages":[],"library_direction":null,"modality":null,"name":"ProcessBench","openness":"open","publisher":"QwenTeam, Alibaba Inc.","released":"2024-12-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1621-processbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ProcessBench","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"ProEvent is an event-centric benchmark for proactive agents, evaluating their ability to maintain a user's timetable from instant messaging chats. It assesses response timing, single-step correctness, and multi-step correctness using synthesized realistic chat scenarios.","descriptions":{"en":"ProEvent is an event-centric benchmark for proactive agents, evaluating their ability to maintain a user's timetable from instant messaging chats. It assesses response timing, single-step correctness, and multi-step correctness using synthesized realistic chat scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.17701","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_proevent_18140068"},"modality":null,"name":"ProEvent","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.17701","source_url":"https://arxiv.org/abs/2607.17701"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-17701","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17701","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":34,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"llm_stats_id","repository":null,"source_record":"catalog_3afc7ee5d930fd7d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","knowledge","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ProfBench evaluates models on professional-domain reasoning and knowledge-work tasks, including search-augmented question answering across expert fields.","descriptions":{"en":"ProfBench evaluates models on professional-domain reasoning and knowledge-work tasks, including search-augmented question answering across expert fields."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:profbench:nemotron-3-ultra-550b-a55b","reported_at":"2026-06-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/profbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:profbench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","knowledge-expert-qa"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_3afc7ee5d930fd7d"},"modality":"text","name":"ProfBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":56.00000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.56,"raw_min":0.56,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:profbench:nemotron-3-ultra-550b-a55b","reported_date":"2026-06-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/profbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"}],"slug":"llm-stats-profbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/profbench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["Long-Horizon Forecasting of Complete Financial Statements with Forma"],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:forma-lab-mccombs/proforma-20q","source_record":"bm_long-horizon-forecasting-of-complete-finan_01c89589"},"categories":["Finance","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates joint probabilistic forecasting of complete quarterly financial statements across 78 line items and 1-20 quarter horizons.","descriptions":{"en":"Evaluates joint probabilistic forecasting of complete quarterly financial statements across 78 line items and 1-20 quarter horizons."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.11327","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_long-horizon-forecasting-of-complete-finan_01c89589"},"modality":null,"name":"ProForma-20Q","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11327","source_url":"https://arxiv.org/abs/2608.11327"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11327","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11327","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":955,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":"gh:facebookresearch/programbench","source_record":"lib_programbench"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Program Bench evaluates code-generation agents by asking them to recreate a program's behavior from only a compiled binary and documentation. It spans 200 tasks from small CLI tools to large systems such as FFmpeg and SQLite, with submissions judged against more than 248,000 fuzz-generated behavioral tests.","descriptions":{"en":"Program Bench evaluates code-generation agents by asking them to recreate a program's behavior from only a compiled binary and documentation. It spans 200 tasks from small CLI tools to large systems such as FFmpeg and SQLite, with submissions judged against more than 248,000 fuzz-generated behavioral tests."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:program-bench:kimi-k2.7-code","reported_at":"2026-06-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/program-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:program-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"lib_programbench"},"modality":"text","name":"Program Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":77.8,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.778,"raw_min":0.19,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:program-bench:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/program-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-program-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/program-bench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["coding"],"collected_at":null,"description":"Cleanroom program-rebuild tasks; most models score low, so small differences sit within run-to-run noise.","descriptions":{"en":"Cleanroom program-rebuild tasks; most models score low, so small differences sit within run-to-run noise."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000programbench\u0000tencent_hy4_preview\u0000programbench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000programbench\u0000tencent_hy4_preview\u0000programbench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:programbench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"artifact:arxiv:2605.03546","primary":"agentic-coding","source_record":"catalog_2c8fb4e3c7fea53c"},"modality":null,"name":"ProgramBench","openness":"unknown","publisher":null,"released":"2026-05-05","released_reference":{"basis":"benchmark_release","source_key":"model-reports:programbench","source_url":"https://arxiv.org/abs/2605.03546"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":20.3,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":20.3,"raw_min":17.5,"source_reference":{"obs_id":"curated\u0000programbench\u0000deepseek_v4_1_flash_technical_report\u0000programbench\u0000think max, Almost@1\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000programbench\u0000deepseek_v4_1_flash_technical_report\u0000programbench\u0000think max, Almost@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"programbench","source":"model_reports","source_url":"https://arxiv.org/abs/2605.03546","topics":["coding"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on tasks derived from interactive web applications.","descriptions":{"en":"Evaluates coding agents on tasks derived from interactive web applications."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.18805","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_programdistill_aaf6c4ec"},"modality":null,"name":"ProgramDistill","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18805","source_url":"https://arxiv.org/abs/2609.18805"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18805","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18805","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_projectionbench_7fc1bdf2"},"categories":["AI Scientist","General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"A framework for evaluating scientific hypothesis generation in LLMs under progressive information disclosure, comparing model hypotheses to original paper conclusions via semantic similarity.","descriptions":{"en":"A framework for evaluating scientific hypothesis generation in LLMs under progressive information disclosure, comparing model hypotheses to original paper conclusions via semantic similarity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30284","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_projectionbench_7fc1bdf2"},"modality":null,"name":"ProjectionBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30284","source_url":"https://arxiv.org/abs/2605.30284"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30284","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30284","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ProJudge is a comprehensive, multi-modal, multi-discipline, and multi-difficulty benchmark specifically designed for evaluating abilities of MLLM-based process judges.It comprises 2,400 test cases and 50,118 step-level labels, spanning four scientific disciplines with diverse difficulty levels and ProJudge 是一个针对基于 MLLM 的过程裁判能力的全面、多模态、多学科和多难度的基准。它包含 2,400 个测试案例和 50,118 个步骤级标签，涵盖四个科学学科，难度级别和内容多样化。","descriptions":{"en":"ProJudge is a comprehensive, multi-modal, multi-discipline, and multi-difficulty benchmark specifically designed for evaluating abilities of MLLM-based process judges.It comprises 2,400 test cases and 50,118 step-level labels, spanning four scientific disciplines with diverse difficulty levels and","zh":"ProJudge 是一个针对基于 MLLM 的过程裁判能力的全面、多模态、多学科和多难度的基准。它包含 2,400 个测试案例和 50,118 个步骤级标签，涵盖四个科学学科，难度级别和内容多样化。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1633","languages":[],"library_direction":null,"modality":"multimodal","name":"ProJudge","openness":"restricted","publisher":"WHU, Shanghai AI Laboratory, etc.","released":"2025-03-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1633-projudge","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ProJudge","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Prompting-MammAlps is a camera-trap text-to-video retrieval benchmark, evaluating video-language models on fine-grained retrieval of ecological events. It includes a test set of 135 queries and 775 candidate videos, with a proposed method that combines action localization with LLM-based parsing.","descriptions":{"en":"Prompting-MammAlps is a camera-trap text-to-video retrieval benchmark, evaluating video-language models on fine-grained retrieval of ecological events. It includes a test set of 135 queries and 775 candidate videos, with a proposed method that combines action localization with LLM-based parsing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09876","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","science"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_prompting-mammalps_0a70a842"},"modality":null,"name":"Prompting-MammAlps","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09876","source_url":"https://arxiv.org/abs/2607.09876"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09876","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09876","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["数学","Math","科学智能","AI for Science","数理能力"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ProofNet is a benchmark for autoformalization and formal proving of undergraduate-level mathematics. The ProofNet benchmarks consists of 371 examples, each consisting of a formal theorem statement in Lean 3, a natural language theorem statement, and a natural language proof. ProofNet 是一个用于本科数学的自动形式化和形式证明的基准。包含 371 个示例，每个示例包括一个 Lean 3 中的形式定理陈述、一个自然语言定理陈述和一个自然语言证明。这些问题主要来自流行的本科纯数学教材，涵盖实分析、复分析、线性代数、抽象代数和拓扑等主题。","descriptions":{"en":"ProofNet is a benchmark for autoformalization and formal proving of undergraduate-level mathematics. The ProofNet benchmarks consists of 371 examples, each consisting of a formal theorem statement in Lean 3, a natural language theorem statement, and a natural language proof.","zh":"ProofNet 是一个用于本科数学的自动形式化和形式证明的基准。包含 371 个示例，每个示例包括一个 Lean 3 中的形式定理陈述、一个自然语言定理陈述和一个自然语言证明。这些问题主要来自流行的本科纯数学教材，涵盖实分析、复分析、线性代数、抽象代数和拓扑等主题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1120","languages":[],"library_direction":null,"modality":null,"name":"ProofNet","openness":"open","publisher":"Yale University & University of Warsaw & Carnegie Mellon University","released":"2023-02-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1120-proofnet","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ProofNet","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":133,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":"gh:future-house/lab-bench","source_record":"catalog_b2dffdefb544c592"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ProtocolQA is a multiple-choice benchmark on troubleshooting failed experimental outcomes from common biological laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development.","descriptions":{"en":"ProtocolQA is a multiple-choice benchmark on troubleshooting failed experimental outcomes from common biological laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:protocolqa:grok-4.1-thinking-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/protocolqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:protocolqa","languages":[],"library_direction":{"basis":"model-review","categories":["science","safety-alignment"],"join":"llm_stats_id","primary":"science","source_record":"catalog_b2dffdefb544c592"},"modality":"text","name":"ProtocolQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":79.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.79,"raw_min":0.79,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:protocolqa:grok-4.1-thinking-2025-11-17","reported_date":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/protocolqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-protocolqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/protocolqa?top_n=500","topics":["safety","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"ProtStructQA is an executable benchmark for protein structural question answering, with questions generated from DSL programs and answers obtained by executing on AlphaFold-predicted structures. Released 382.2K questions covering confidence, distances, PAE, solvent exposure, secondary structure, topology, and contacts.","descriptions":{"en":"ProtStructQA is an executable benchmark for protein structural question answering, with questions generated from DSL programs and answers obtained by executing on AlphaFold-predicted structures. Released 382.2K questions covering confidence, distances, PAE, solvent exposure, secondary structure, topology, and contacts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00451","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_protstructqa_a89c5890"},"modality":null,"name":"ProtStructQA","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00451","source_url":"https://arxiv.org/abs/2606.00451"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00451","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00451","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"PRWeaver evaluates LLM-based code auditors against malicious pull requests across repository history, using 208 attacks and multiple renderings.","descriptions":{"en":"PRWeaver evaluates LLM-based code auditors against malicious pull requests across repository history, using 208 attacks and multiple renderings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.02693","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","agent-security"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_prweaver_3962c935"},"modality":null,"name":"PRWeaver","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02693","source_url":"https://arxiv.org/abs/2608.02693"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02693","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02693","topics":["agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PseudoBench evaluates agentic auto-research systems on their ability to identify and resist pseudoscientific narratives. It contains 200 curated pseudoscientific claim-evidence pairs across five domains and assesses performance through an end-to-end research pipeline from experiment design to report writing.","descriptions":{"en":"PseudoBench evaluates agentic auto-research systems on their ability to identify and resist pseudoscientific narratives. It contains 200 curated pseudoscientific claim-evidence pairs across five domains and assesses performance through an end-to-end research pipeline from experiment design to report writing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18060","languages":[],"library_direction":{"basis":"model-review","categories":["science","factuality"],"join":"claire_record_id","primary":"science","source_record":"bm_pseudobench_085d3f1c"},"modality":null,"name":"PseudoBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18060","source_url":"https://arxiv.org/abs/2606.18060"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18060","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18060","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":23,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_ptcg-bench_083265f8"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PTCG-Bench evaluates LLM agents on the Pokémon Trading Card Game at two levels: single-environment decision-making and self-evolution through accumulated experience, with modular harness ablation to separate agent performance from harness design.","descriptions":{"en":"PTCG-Bench evaluates LLM agents on the Pokémon Trading Card Game at two levels: single-environment decision-making and self-evolution through accumulated experience, with modular harness ablation to separate agent performance from harness design."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29653","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","agent-skills-evolution"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_ptcg-bench_083265f8"},"modality":null,"name":"PTCG-Bench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29653","source_url":"https://arxiv.org/abs/2605.29653"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29653","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29653","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":34,"github_stars_30d":15,"hf_dataset_downloads":122,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:zhang677/ptxbench","source_record":"bm_ptxbench_11575102"},"categories":["Architecture-specific programming","Code","Code & Software","Code generation","GPU kernel optimization","Iterative code repair","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"PTXBench evaluates LLMs in generating architecture-specific PTX for GPU kernel optimization. It measures functional correctness, execution of target instructions, and speedup over frontier libraries across GEMM and attention workloads on H100 and B200 GPUs.","descriptions":{"en":"PTXBench evaluates LLMs in generating architecture-specific PTX for GPU kernel optimization. It measures functional correctness, execution of target instructions, and speedup over frontier libraries across GEMM and attention workloads on H100 and B200 GPUs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.17379","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","code-generation"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_ptxbench_11575102"},"modality":null,"name":"PTXBench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17379","source_url":"https://arxiv.org/abs/2608.17379"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17379","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17379","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":null,"source_record":"bm_pumpire_8facce52"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors.","descriptions":{"en":"We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.12423","languages":[],"library_direction":{"basis":"model-review-daily","categories":["spatial-3d","visual-reasoning","video-understanding"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_pumpire_8facce52"},"modality":null,"name":"Pumpire","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12423","source_url":"https://arxiv.org/abs/2610.12423"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12423","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12423","topics":[],"unit":null},{"aliases":["Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation"],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"A multilingual visual-reasoning benchmark comprising parallel Punjabi image-text instances in Gurmukhi, Shahmukhi, and Roman scripts.","descriptions":{"en":"A multilingual visual-reasoning benchmark comprising parallel Punjabi image-text instances in Gurmukhi, Shahmukhi, and Roman scripts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.17188","languages":[],"library_direction":null,"modality":null,"name":"PuMVR","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17188","source_url":"https://arxiv.org/abs/2606.17188"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17188","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17188","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["Internalizing Geometric Law: Learning from Solver Residuals for Precision-Critical Generation"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"A 300-problem stratified benchmark for generating precise geometric constructions from natural-language descriptions.","descriptions":{"en":"A 300-problem stratified benchmark for generating precise geometric constructions from natural-language descriptions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09278","languages":[],"library_direction":null,"modality":null,"name":"PyGeoX-Bench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09278","source_url":"https://arxiv.org/abs/2606.09278"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09278","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09278","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"PyraMathBench is a hierarchical benchmark with 32,505 questions derived from math word problems, evaluating numerical reasoning and mathematical capabilities across cognitive aspects and modalities.","descriptions":{"en":"PyraMathBench is a hierarchical benchmark with 32,505 questions derived from math word problems, evaluating numerical reasoning and mathematical capabilities across cognitive aspects and modalities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03858","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_pyramathbench_14f9eba5"},"modality":null,"name":"PyraMathBench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03858","source_url":"https://arxiv.org/abs/2606.03858"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03858","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03858","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Q-Bench/Q-Bench+ is a benchmark for general-purpose foundation models on low-level vision. Q-Bench/Q-Bench+是一个面向多模态大模型底层视觉理解的数据集。此数据集从底层视觉的感知、描述、评价能力出发来对多模态大模型进行完整的测试，测试的对象既包括单张图像也包括图像对。","descriptions":{"en":"Q-Bench/Q-Bench+ is a benchmark for general-purpose foundation models on low-level vision.","zh":"Q-Bench/Q-Bench+是一个面向多模态大模型底层视觉理解的数据集。此数据集从底层视觉的感知、描述、评价能力出发来对多模态大模型进行完整的测试，测试的对象既包括单张图像也包括图像对。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1000","languages":[],"library_direction":null,"modality":null,"name":"Q-Bench","openness":"unknown","publisher":null,"released":"2024-08-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1000-q-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Q-Bench","topics":["multimodal"],"unit":null},{"aliases":["Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems"],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"A large-scale retrieval evaluation with a 190M-document web corpus and 70k agentic search queries in ten languages, judged by agent citations, production rankings, and a combined relevance set.","descriptions":{"en":"A large-scale retrieval evaluation with a 190M-document web corpus and 70k agentic search queries in ten languages, judged by agent citations, production rankings, and a combined relevance set."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08887","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_q2d-web-a-large-scale-benchmark-for-retrie_46ecf4a0"},"modality":null,"name":"Q2D-Web","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08887","source_url":"https://arxiv.org/abs/2609.08887"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08887","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08887","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","大语言模型","LLM","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"QASC is a question-answering dataset with a focus on sentence composition. It consists of 9,980 8-way multiple-choice questions about grade school science (8,134 train, 926 dev, 920 test), and comes with a corpus of 17M sentences. QASC 是一个专注于句子组合的问答数据集。它包含 9,980 道小学科学的多项选择题（8,134 道用于训练，926 道用于开发，920 道用于测试），并配有一个包含 1,700 万个句子的语料库。","descriptions":{"en":"QASC is a question-answering dataset with a focus on sentence composition. It consists of 9,980 8-way multiple-choice questions about grade school science (8,134 train, 926 dev, 920 test), and comes with a corpus of 17M sentences.","zh":"QASC 是一个专注于句子组合的问答数据集。它包含 9,980 道小学科学的多项选择题（8,134 道用于训练，926 道用于开发，920 道用于测试），并配有一个包含 1,700 万个句子的语料库。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1103","languages":[],"library_direction":null,"modality":null,"name":"QASC","openness":"open","publisher":"Allen Institute for AI","released":"2020-02-04","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1103-qasc","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/QASC","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":62,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_a0984ed4cbce46aa"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QASPER is a dataset of 5,049 information-seeking questions and answers anchored in 1,585 NLP research papers. Questions are written by NLP practitioners who read only titles and abstracts, while answers require understanding the full paper text and provide supporting evidence. The dataset challenges models with complex reasoning across document sections for academic document question answering. Each question seeks information present in the full text and is answered by a separate set of NLP practitioners who also provide supporting evidence to answers.","descriptions":{"en":"QASPER is a dataset of 5,049 information-seeking questions and answers anchored in 1,585 NLP research papers. Questions are written by NLP practitioners who read only titles and abstracts, while answers require understanding the full paper text and provide supporting evidence. The dataset challenges models with complex reasoning across document sections for academic document question answering. Each question seeks information present in the full text and is answered by a separate set of NLP practitioners who also provide supporting evidence to answers."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qasper:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qasper?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qasper","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","science"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_a0984ed4cbce46aa"},"modality":"text","name":"Qasper","openness":"unknown","publisher":null,"released":"2021-05-07","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:qasper","source_url":"https://arxiv.org/abs/2105.03011"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":41.9,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.419,"raw_min":0.4,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qasper:phi-3.5-mini-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qasper?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-qasper","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qasper?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_qc-stark_9e6eaefe"},"categories":["Language & Knowledge","Quantum Computing & Control","quant-ph"],"collected_at":"2026-10-14T09:44:51Z","description":"We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation.","descriptions":{"en":"We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35581","languages":[],"library_direction":{"basis":"model-review","categories":["science","code-generation"],"join":"claire_record_id","primary":"science","source_record":"bm_qc-stark_9e6eaefe"},"modality":null,"name":"QC-Stark","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35581","source_url":"https://arxiv.org/abs/2609.35581"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35581","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35581","topics":["coding","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","quant-ph"],"collected_at":"2026-10-14T09:44:51Z","description":"A set of 49 expert-authored tasks for verified autonomy in quantum engineering.","descriptions":{"en":"A set of 49 expert-authored tasks for verified autonomy in quantum engineering."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17439","languages":[],"library_direction":null,"modality":null,"name":"QIQCBench","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17439","source_url":"https://arxiv.org/abs/2609.17439"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17439","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17439","topics":[],"unit":null},{"aliases":["Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation"],"attention_signals":null,"categories":["Language & Knowledge","Quantum Computing & Control","quant-ph"],"collected_at":"2026-09-25T04:40:22Z","description":"Qiskit QuantumKatas is a 350-task evaluation of LLM quantum-computing capabilities across 26 categories, using deterministic circuit-simulation tests.","descriptions":{"en":"Qiskit QuantumKatas is a 350-task evaluation of LLM quantum-computing capabilities across 26 categories, using deterministic circuit-simulation tests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27210","languages":[],"library_direction":null,"modality":null,"name":"Qiskit QuantumKatas","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27210","source_url":"https://arxiv.org/abs/2605.27210"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27210","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27210","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":156,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_4c4462e36bcf7053"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","summarization"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QMSum is a benchmark for query-based multi-domain meeting summarization consisting of 1,808 query-summary pairs over 232 meetings across academic, product, and committee domains. The dataset enables models to select and summarize relevant spans of meetings in response to specific queries. Published at NAACL 2021, QMSum presents significant challenges in long meeting summarization where models must identify and summarize relevant content based on user queries.","descriptions":{"en":"QMSum is a benchmark for query-based multi-domain meeting summarization consisting of 1,808 query-summary pairs over 232 meetings across academic, product, and committee domains. The dataset enables models to select and summarize relevant spans of meetings in response to specific queries. Published at NAACL 2021, QMSum presents significant challenges in long meeting summarization where models must identify and summarize relevant content based on user queries."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qmsum:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qmsum?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qmsum","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","chat-writing"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_4c4462e36bcf7053"},"modality":"text","name":"QMSum","openness":"unknown","publisher":null,"released":"2021-04-13","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:qmsum","source_url":"https://arxiv.org/abs/2104.05938"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":21.3,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.213,"raw_min":0.199,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qmsum:phi-3.5-mini-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qmsum?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-qmsum","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qmsum?top_n=500","topics":["long-context","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:zhang-mengao/qo-bench","source_record":"bm_qo-bench_a8720b3c"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"QO-Bench is a diagnostic benchmark for query-operator question answering over typed event tuples. It covers 22,984 news articles, 614 corporate events, and 18 query templates, with 785 questions. Gold answers are deterministically computed and scored by recall via exact match.","descriptions":{"en":"QO-Bench is a diagnostic benchmark for query-operator question answering over typed event tuples. It covers 22,984 news articles, 614 corporate events, and 18 query templates, with 785 questions. Gold answers are deterministically computed and scored by recall via exact match."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.04646","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_qo-bench_a8720b3c"},"modality":null,"name":"QO-Bench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04646","source_url":"https://arxiv.org/abs/2606.04646"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04646","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04646","topics":[],"unit":null},{"aliases":["Quant Arena"],"attention_signals":null,"categories":["finance"],"collected_at":null,"description":"Agents research stock-scoring strategies on 2022–2023 data, freeze one program, and receive one sealed 2024 cash-account assessment. The 0–100 total combines 30% research-window and 70% sealed-window scores, not investment return. Reasoning effort and iterative versus one-shot settings remain separate; the proprietary Kepler-1 reference has undisclosed construction. These historical results do not establish future trading performance.","descriptions":{"en":"Agents research stock-scoring strategies on 2022–2023 data, freeze one program, and receive one sealed 2024 cash-account assessment. The 0–100 total combines 30% research-window and 70% sealed-window scores, not investment return. Reasoning effort and iterative versus one-shot settings remain separate; the proprietary Kepler-1 reference has undisclosed construction. These historical results do not establish future trading performance."},"editorial_guide":{"id":"quant_arena","record_keys":["model-reports:quant_arena"],"reviewed_at":"2026-10-27","scope":{"en":"Paper v1 and the author leaderboard: 76 model/effort configurations plus one proprietary reference. Iterative and one-shot tracks are kept separate.","zh":"论文 v1 与作者榜单：76 个模型与推理档位配置，以及一个专有参照策略；迭代与单次生成分开呈现。"},"sections":[{"id":"task","source_ids":["paper"],"text":{"en":"A run writes Python features and daily stock scores for a selected universe of 3,042 A-share stocks, using daily and raw Level-2 data. Iterative research permits at most five candidate evaluations on 2022–2023. One program is then frozen and evaluated once on the sealed year 2024. One-shot generation writes one strategy without that research feedback.","zh":"每次运行用日行情及 Level-2 原始数据编写 Python 特征和每日股票评分，覆盖选定的 3,042 只 A 股。迭代研究最多在 2022–2023 年评估五个候选策略，随后冻结一个程序，在封存的 2024 年上评估一次。单次生成则直接写出策略，不获得上述研究反馈。"},"title":{"en":"Research, commit, then assess","zh":"研究、冻结，再评估"}},{"id":"metric","source_ids":["paper"],"text":{"en":"Total S = 0.3 S_in + 0.7 S_out on a 0–100 scale. The in-sample score combines return-dominated quality with feature diversity; the sealed-period combination rule is fixed per version but unpublished. The cash-account evaluator targets 75 stocks and applies lot constraints, costs and simulated market impact. Its equal-weight market reference is gross of costs.","zh":"总分 S = 0.3 S_in + 0.7 S_out，范围为 0–100。样本内分数将以收益为主的策略质量与特征多样性结合；封存阶段的组合规则按版本固定，但未公开。现金账户评估器目标持有 75 只股票，并计入整手限制、费用及模拟市场冲击；等权市场参照的收益未扣费用。"},"title":{"en":"Total score is not return","zh":"总分不是收益率"}},{"id":"interpretation","source_ids":["paper","leaderboard"],"text":{"en":"There are 38 iterative and 38 one-shot configurations of nine models. A model’s best total-score configuration need not maximize its annualized return. In Table 2, GPT-6 Sol at medium effort returns 18.4% against the 17.1% gross market, while its highest total score belongs to xhigh. The statement that no model beats the market concerns the selected best-score configurations, not every run.","zh":"九个模型合计有 38 组迭代配置和 38 组单次生成配置。总分最高的配置不一定收益最高：论文表 2 中 GPT-6 Sol 的 medium 档收益为 18.4%，高于 17.1% 的未扣费市场参照，但其总分最高的是 xhigh。因此“没有模型跑赢市场”只适用于按最高总分选出的配置，不能扩展到全部运行。"},"title":{"en":"Read every effort level","zh":"保留每个推理档位"}},{"id":"iterative-results","source_ids":["paper","leaderboard"],"table":{"caption":{"en":"Iterative research · author leaderboard read 2026-10-27","zh":"迭代研究 · 2026-10-27 读取的作者榜单"},"columns":[{"en":"Model","zh":"模型"},{"en":"Effort","zh":"推理档位"},{"en":"Total","zh":"总分"},{"en":"S_in","zh":"研究期分数"},{"en":"S_out","zh":"封存期分数"},{"en":"2024 ARR","zh":"2024 年化收益"},{"en":"Excess (pp)","zh":"超额（百分点）"},{"en":"RankIC","zh":"RankIC"},{"en":"Sharpe","zh":"夏普比率"},{"en":"MDD","zh":"最大回撤"}],"note":{"en":"Values are copied as displayed; S is the normalized score, while ARR and MDD are percentages. The two settings are separate protocols. Kepler-1’s construction is undisclosed.","zh":"数值按原表保留；S 是评分，ARR 与最大回撤是百分数。两种设置属于不同评测条件；Kepler-1 的构建方法未公开。"},"rows":[["Kepler-1 Einsia AI","—","77.56","77.22","77.70","94.0%","+76.9","0.085","2.98","−15.9%"],["GPT-6 Astra","Low","47.22","58.10","42.56","2.3%","−14.8","0.056","0.22","−25.7%"],["GPT-6 Astra","Medium","47.17","48.09","46.78","9.8%","−7.2","0.044","0.53","−17.9%"],["GPT-6 Astra","High","43.70","52.34","39.99","−0.9%","−17.9","0.053","0.11","−26.3%"],["GPT-6 Astra","XHigh","51.29","53.94","50.16","10.7%","−6.3","0.101","0.86","−6.8%"],["GPT-6 Astra","Max","50.63","52.94","49.64","11.2%","−5.9","0.082","0.81","−7.0%"],["GPT-6 Sol","Low","38.51","48.85","34.09","1.4%","−15.7","−0.020","0.19","−30.1%"],["GPT-6 Sol","Medium","49.80","44.14","52.23","18.4%","+1.4","0.018","0.65","−35.9%"],["GPT-6 Sol","High","42.89","52.16","38.92","−2.5%","−19.5","0.042","0.10","−37.4%"],["GPT-6 Sol","XHigh","51.14","55.50","49.26","8.6%","−8.4","0.053","1.89","−2.8%"],["GPT-6 Sol","Max","45.31","46.49","44.81","8.1%","−9.0","0.018","0.41","−30.0%"],["GPT-5.6 Terra","None","33.09","26.67","35.85","2.4%","−14.7","−0.042","0.22","−24.9%"],["GPT-5.6 Terra","Low","41.19","50.58","37.17","−4.6%","−21.7","0.022","0.02","−31.4%"],["GPT-5.6 Terra","Medium","39.50","43.17","37.93","−4.3%","−21.3","0.036","0.00","−29.0%"],["GPT-5.6 Terra","High","41.54","44.72","40.17","0.0%","−17.0","0.035","0.14","−29.9%"],["GPT-5.6 Terra","XHigh","44.77","50.60","42.28","9.9%","−7.1","−0.012","0.44","−32.6%"],["GPT-5.6 Terra","Max","47.58","50.37","46.39","8.4%","−8.7","0.026","0.42","−28.8%"],["Claude Opus 5.5","Low","30.15","37.76","26.89","−29.4%","−46.4","0.035","−1.21","−44.1%"],["Claude Opus 5.5","Medium","44.28","56.78","38.92","4.4%","−12.7","−0.002","0.29","−32.7%"],["Claude Opus 5.5","High","39.89","52.87","34.32","−11.2%","−28.3","0.039","−0.25","−32.7%"],["Claude Opus 5.5","XHigh","41.92","54.48","36.53","−7.0%","−24.0","0.043","−0.08","−33.2%"],["Claude Opus 5.5","Max","41.39","50.41","37.52","−4.0%","−21.1","0.030","−0.03","−25.1%"],["Gemini 3.8 Flash","Low","41.46","53.44","36.32","−6.2%","−23.3","0.037","−0.04","−35.8%"],["Gemini 3.8 Flash","Medium","43.04","53.58","38.52","−2.9%","−19.9","0.045","0.06","−33.8%"],["Gemini 3.8 Flash","High","35.39","48.90","29.61","−19.1%","−36.2","0.022","−0.49","−38.5%"],["Claude Sonnet 5.5","Low","34.97","47.91","29.43","−9.5%","−26.6","−0.009","−0.13","−35.5%"],["Claude Sonnet 5.5","Medium","39.42","43.39","37.73","−3.5%","−20.6","0.041","−0.02","−24.3%"],["Claude Sonnet 5.5","High","40.50","54.11","34.67","−9.0%","−26.1","0.044","−0.19","−34.7%"],["Claude Sonnet 5.5","XHigh","40.10","44.68","38.13","−0.5%","−17.6","0.015","0.15","−35.0%"],["Claude Sonnet 5.5","Max","40.93","57.33","33.90","−12.2%","−29.2","0.045","−0.23","−36.1%"],["GLM-5.3","High","38.67","49.31","34.11","−8.6%","−25.7","0.018","−0.14","−31.6%"],["GLM-5.3","Max","38.91","47.96","35.03","−7.8%","−24.9","0.046","−0.18","−27.8%"],["Kimi K3","Low","38.31","45.87","35.07","−8.5%","−25.6","0.027","−0.10","−36.4%"],["Kimi K3","High","31.43","36.85","29.10","−20.0%","−37.0","0.036","−0.84","−36.9%"],["Kimi K3","Max","31.86","39.87","28.43","−25.6%","−42.7","0.041","−1.14","−39.9%"],["DeepSeek V4.1 Flash","None","22.00","21.21","22.33","−24.1%","−41.2","−0.022","−1.03","−29.5%"],["DeepSeek V4.1 Flash","Low","36.45","48.46","31.30","−5.7%","−22.7","−0.009","−0.06","−30.3%"],["DeepSeek V4.1 Flash","High","35.97","46.02","31.67","−11.3%","−28.4","0.015","−0.25","−34.0%"],["DeepSeek V4.1 Flash","Max","35.90","42.49","33.08","−11.0%","−28.1","0.037","−0.28","−34.2%"]]},"text":{"en":"All 38 model/effort configurations are retained. Kepler-1 is the publisher’s proprietary reference, displayed separately in the source and not another LLM effort setting.","zh":"保留全部 38 个模型与推理档位配置。Kepler-1 为发布方的专有参照策略，不是另一个 LLM 推理档位。"},"title":{"en":"Iterative research · complete configurations","zh":"迭代研究 · 完整配置"}},{"id":"one-shot-results","source_ids":["paper","leaderboard"],"table":{"caption":{"en":"One-shot generation · author leaderboard read 2026-10-27","zh":"单次生成 · 2026-10-27 读取的作者榜单"},"columns":[{"en":"Model","zh":"模型"},{"en":"Effort","zh":"推理档位"},{"en":"Total","zh":"总分"},{"en":"S_in","zh":"研究期分数"},{"en":"S_out","zh":"封存期分数"},{"en":"2024 ARR","zh":"2024 年化收益"},{"en":"Excess (pp)","zh":"超额（百分点）"},{"en":"RankIC","zh":"RankIC"},{"en":"Sharpe","zh":"夏普比率"},{"en":"MDD","zh":"最大回撤"}],"note":{"en":"Values are copied as displayed; S is the normalized score, while ARR and MDD are percentages. The two settings are separate protocols. Kepler-1’s construction is undisclosed.","zh":"数值按原表保留；S 是评分，ARR 与最大回撤是百分数。两种设置属于不同评测条件；Kepler-1 的构建方法未公开。"},"rows":[["Kepler-1 Einsia AI","—","77.56","77.22","77.70","94.0%","+76.9","0.085","2.98","−15.9%"],["Claude Opus 5.5","Low","35.04","43.21","31.53","−4.5%","−21.6","−0.011","0.00","−33.0%"],["Claude Opus 5.5","Medium","42.61","47.19","40.65","5.6%","−11.5","−0.001","0.34","−21.0%"],["Claude Opus 5.5","High","36.07","39.68","34.52","−7.5%","−24.5","0.020","−0.14","−29.7%"],["Claude Opus 5.5","XHigh","39.13","49.05","34.88","−9.9%","−26.9","0.040","−0.20","−35.0%"],["Claude Opus 5.5","Max","39.70","42.07","38.69","−3.7%","−20.8","0.046","0.03","−35.9%"],["GPT-5.6 Terra","None","42.14","46.56","40.25","9.5%","−7.6","−0.010","0.44","−27.8%"],["GPT-5.6 Terra","Low","27.82","26.59","28.35","−10.3%","−27.4","−0.007","−0.24","−25.7%"],["GPT-5.6 Terra","Medium","21.01","20.44","21.25","−32.4%","−49.4","−0.010","−1.51","−34.1%"],["GPT-5.6 Terra","High","40.61","42.88","39.64","6.2%","−10.9","−0.001","0.35","−27.5%"],["GPT-5.6 Terra","XHigh","34.21","42.94","30.46","−9.8%","−26.8","0.000","−0.17","−33.6%"],["GPT-5.6 Terra","Max","20.98","20.98","20.97","−32.3%","−49.3","−0.009","−1.60","−32.6%"],["Claude Sonnet 5.5","Low","37.75","42.41","35.76","1.1%","−16.0","−0.007","0.19","−30.3%"],["Claude Sonnet 5.5","Medium","33.82","41.81","30.39","−11.5%","−28.6","0.002","−0.21","−35.5%"],["Claude Sonnet 5.5","High","40.24","44.28","38.50","1.4%","−15.6","0.010","0.20","−32.1%"],["Claude Sonnet 5.5","XHigh","37.05","47.90","32.40","−14.3%","−31.3","0.029","−0.33","−36.0%"],["Claude Sonnet 5.5","Max","33.89","41.19","30.77","−14.8%","−31.9","0.018","−0.33","−36.9%"],["GLM-5.3","High","38.02","52.54","31.79","−4.2%","−21.3","−0.023","0.05","−37.3%"],["GLM-5.3","Max","31.46","34.70","30.07","−15.4%","−32.4","0.022","−0.51","−32.1%"],["Gemini 3.8 Flash","Low","27.75","21.26","30.53","−12.1%","−29.1","0.005","−0.41","−23.5%"],["Gemini 3.8 Flash","Medium","35.91","35.62","36.04","1.6%","−15.4","−0.006","0.19","−23.8%"],["Gemini 3.8 Flash","High","30.26","25.93","32.11","−5.4%","−22.5","−0.003","−0.06","−24.6%"],["GPT-6 Astra","Low","30.08","35.05","27.95","−11.2%","−28.3","−0.010","−0.28","−30.3%"],["GPT-6 Astra","Medium","31.02","36.73","28.57","−9.7%","−26.7","−0.012","−0.20","−30.3%"],["GPT-6 Astra","High","35.22","38.27","33.92","−4.8%","−21.9","0.007","−0.03","−30.9%"],["GPT-6 Astra","XHigh","30.30","38.46","26.80","−16.7%","−33.8","−0.002","−0.48","−34.7%"],["GPT-6 Astra","Max","31.47","40.13","27.75","−18.3%","−35.3","0.007","−0.54","−34.8%"],["GPT-6 Sol","Low","32.15","38.29","29.52","−7.6%","−24.7","−0.014","−0.08","−32.2%"],["GPT-6 Sol","Medium","22.60","22.62","22.59","−31.4%","−48.4","−0.006","−1.31","−34.1%"],["GPT-6 Sol","High","23.15","21.07","24.04","−17.4%","−34.4","−0.020","−0.51","−29.6%"],["GPT-6 Sol","XHigh","23.29","23.84","23.05","−19.0%","−36.1","−0.022","−0.60","−29.6%"],["GPT-6 Sol","Max","22.31","24.08","21.55","−23.7%","−40.7","−0.018","−0.98","−32.5%"],["Kimi K3","Low","31.58","38.58","28.58","−13.4%","−30.5","0.000","−0.34","−33.8%"],["Kimi K3","High","32.01","37.89","29.48","−20.5%","−37.6","0.038","−0.84","−38.6%"],["Kimi K3","Max","31.65","40.87","27.70","−20.3%","−37.3","0.016","−0.86","−35.1%"],["DeepSeek V4.1 Flash","None","20.48","22.01","19.82","−34.1%","−51.1","−0.015","−1.79","−37.5%"],["DeepSeek V4.1 Flash","Low","29.36","34.21","27.28","−17.4%","−34.4","0.001","−0.49","−33.3%"],["DeepSeek V4.1 Flash","High","29.63","37.51","26.25","−25.4%","−42.4","0.010","−0.92","−39.5%"],["DeepSeek V4.1 Flash","Max","29.92","35.95","27.34","−11.9%","−29.0","−0.008","−0.42","−26.7%"]]},"text":{"en":"All 38 model/effort configurations are retained. Kepler-1 is the publisher’s proprietary reference, displayed separately in the source and not another LLM effort setting.","zh":"保留全部 38 个模型与推理档位配置。Kepler-1 为发布方的专有参照策略，不是另一个 LLM 推理档位。"},"title":{"en":"One-shot generation · complete configurations","zh":"单次生成 · 完整配置"}},{"id":"limits","source_ids":["paper"],"text":{"en":"This is an author-run historical assessment on one stock universe and one sealed year, with a controlled data interface and benchmark-specific trading assumptions. It measures research and strategy construction under that setup. It does not establish forward investment performance, an independently reproduced result, or a general finance knowledge ranking.","zh":"这是作者在一个股票范围和一个封存年度上完成的历史评估，使用受控数据接口与特定交易假设。它检验此条件下的研究和策略构建，不能证明未来投资表现、独立复现结果或一般金融知识排名。"},"title":{"en":"What this evidence establishes","zh":"这些证据的范围"}}],"sources":[{"id":"paper","title":"Quant-Arena paper v1 · protocol, Table 2, Appendix B and reference strategy","url":"https://arxiv.org/html/2610.30113v1"},{"id":"leaderboard","title":"Quant-Arena · author leaderboard, all efforts and both settings","url":"https://lab.einsia.ai/quant-arena/leaderboard/"},{"id":"project","title":"Quant-Arena · official project","url":"https://lab.einsia.ai/quant-arena/"}],"summary":{"en":"Can an agent develop a stock-scoring strategy on a research period, freeze it, and retain performance on a sealed year? Quant-Arena tests the complete research loop.","zh":"Agent 能否在研究期开发股票评分策略，冻结程序后仍在封存年度保持表现？Quant-Arena 测试这一完整研究过程。"},"title":"Quant-Arena"},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000quant_arena\u0000einsia_quant_arena_v1_results\u0000Quant-Arena paper v1 (2022–2023 research; sealed 2024)\u0000Kepler-1 proprietary reference strategy; construction undisclosed; cash-account evaluator; not a supported-effort LLM run\u0000Kepler-1 (proprietary strategy)","reported_at":"2026-10-21","source_url":"https://arxiv.org/html/2610.30113v1"},"first_score_reported_at":"2026-10-21","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000quant_arena\u0000einsia_quant_arena_v1_results\u0000Quant-Arena paper v1 (2022–2023 research; sealed 2024)\u0000Kepler-1 proprietary reference strategy; construction undisclosed; cash-account evaluator; not a supported-effort LLM run\u0000Kepler-1 (proprietary strategy)","reported_at":"2026-10-21","source_url":"https://arxiv.org/html/2610.30113v1"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:quant_arena","languages":[],"library_direction":null,"modality":null,"name":"Quant-Arena","openness":"unknown","publication":{"evidence":[{"note":"The author project names this benchmark and links its introducing paper and available release resources. This exact-record review preserves the documented task version, metrics and evaluation setup.","url":"https://lab.einsia.ai/quant-arena/"},{"note":"Navers Lab / Einsia research index links this named project. Method papers and memory software in the same index are not added as benchmarks.","url":"https://lab.einsia.ai/research/"}],"id":"einsia-quant-arena-original","links":[{"kind":"website","title":"Quant-Arena · official project","url":"https://lab.einsia.ai/quant-arena/"},{"kind":"paper","title":"Quant-Arena · introducing paper","url":"https://arxiv.org/abs/2610.30113v1"},{"kind":"website","title":"Quant-Arena · author leaderboard","url":"https://lab.einsia.ai/quant-arena/leaderboard/"}],"record_keys":["model-reports:quant_arena"],"reviewed_at":"2026-10-27","scope":{"en":"Paper v1 and the author leaderboard: 76 model/effort configurations plus one proprietary reference. Iterative and one-shot tracks are kept separate.","zh":"论文 v1 与作者榜单：76 个模型与推理档位配置，以及一个专有参照策略；迭代与单次生成分开呈现。"}},"publisher":null,"released":"2026-10-21","released_reference":{"basis":"benchmark_release","source_key":"model-reports:quant_arena","source_url":"https://lab.einsia.ai/quant-arena/"},"repo_kind":null,"repo_resolution_status":null,"score_count":77,"score_direction":"higher_is_better","score_summary":{"display_max":77.56,"display_multiplier":1,"model_count":10,"model_count_basis":"source_model_id","numeric_count":77,"raw_max":77.56,"raw_min":20.48,"source_reference":{"obs_id":"curated\u0000quant_arena\u0000einsia_quant_arena_v1_results\u0000Quant-Arena paper v1 (2022–2023 research; sealed 2024)\u0000Kepler-1 proprietary reference strategy; construction undisclosed; cash-account evaluator; not a supported-effort LLM run\u0000Kepler-1 (proprietary strategy)","observation_id":"curated\u0000quant_arena\u0000einsia_quant_arena_v1_results\u0000Quant-Arena paper v1 (2022–2023 research; sealed 2024)\u0000Kepler-1 proprietary reference strategy; construction undisclosed; cash-account evaluator; not a supported-effort LLM run\u0000Kepler-1 (proprietary strategy)","reported_at":"2026-10-21","reported_date":"2026-10-21","source_id":"einsia_quant_arena_v1_results","source_url":"https://arxiv.org/html/2610.30113v1"},"unit":"points"},"scored_models":[{"date":"2026-10-21","date_precision":"document_publication","name":"Claude Opus 5.5"},{"date":"2026-10-21","date_precision":"document_publication","name":"Claude Sonnet 5.5"},{"date":"2026-10-21","date_precision":"document_publication","name":"DeepSeek V4.1 Flash"},{"date":"2026-10-21","date_precision":"document_publication","name":"Gemini 3.8 Flash"},{"date":"2026-10-21","date_precision":"document_publication","name":"GLM-5.3"},{"date":"2026-10-21","date_precision":"document_publication","name":"GPT-5.6 Terra"},{"date":"2026-10-21","date_precision":"document_publication","name":"GPT-6 Astra"},{"date":"2026-10-21","date_precision":"document_publication","name":"GPT-6 Sol"},{"date":"2026-10-21","date_precision":"document_publication","name":"Kepler-1 (proprietary strategy)"},{"date":"2026-10-21","date_precision":"document_publication","name":"Kimi K3"}],"slug":"quant_arena","source":"model_reports","source_url":"https://lab.einsia.ai/quant-arena/","topics":["coding","agents","finance"],"unit":"points"},{"aliases":["variant-pathogenicity-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:rathanpai/variant-pathogenicity-benchmark","source_record":"bm_variant-pathogenicity-benchmark_6b652222"},"categories":["Chip Design & EDA","Language & Knowledge","Quantum Computing & Control","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A comparative framework for genetic variant pathogenicity prediction using ClinVar and gnomAD-derived biological features, benchmarking classical ML, deep learning, and quantum machine learning models.","descriptions":{"en":"A comparative framework for genetic variant pathogenicity prediction using ClinVar and gnomAD-derived biological features, benchmarking classical ML, deep learning, and quantum machine learning models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:rathanpai/variant-pathogenicity-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning","science"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_variant-pathogenicity-benchmark_6b652222"},"modality":null,"name":"Quantum-Enhanced Genetic Variant Pathogenicity Prediction","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:rathanpai/variant-pathogenicity-benchmark","source_url":"https://github.com/RathanPai/variant-pathogenicity-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-rathanpai-variant-pathogenicity-benchmark","source":"claire_radar","source_url":"https://github.com/RathanPai/variant-pathogenicity-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_quechuatok_31b7234f"},"categories":["Language & Knowledge","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A comparison of tokenization strategies (BPE, Unigram LM, WordPiece, PRPE) for Southern Quechua using a 200k-sentence corpus and a finite-state morphological analyzer as reference, with metrics including fertility rate, OOV rate, and morphological boundary accuracy.","descriptions":{"en":"A comparison of tokenization strategies (BPE, Unigram LM, WordPiece, PRPE) for Southern Quechua using a 200k-sentence corpus and a finite-state morphological analyzer as reference, with metrics including fertility rate, OOV rate, and morphological boundary accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.23943","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_quechuatok_31b7234f"},"modality":null,"name":"QuechuaTok","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23943","source_url":"https://arxiv.org/abs/2606.23943"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23943","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23943","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":null,"source_record":"bm_quotebench_1e09ef6d"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark of 56 one-shot tasks from 14 incident-derived families that measures exact final-state outcomes when LLM coding agents issue Bash commands through serializing/wrapping/reparsing transport, using an added unescaped parser.","descriptions":{"en":"Benchmark of 56 one-shot tasks from 14 incident-derived families that measures exact final-state outcomes when LLM coding agents issue Bash commands through serializing/wrapping/reparsing transport, using an added unescaped parser."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.13547","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_quotebench_1e09ef6d"},"modality":null,"name":"QuoteBench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13547","source_url":"https://arxiv.org/abs/2608.13547"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13547","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13547","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates generative AI on Quranic Arabic across multiple dimensions of linguistic complexity.","descriptions":{"en":"Evaluates generative AI on Quranic Arabic across multiple dimensions of linguistic complexity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.22038","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["computer-use","multilingual"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_quranicmmlu_46194b3d"},"modality":null,"name":"QuranicMMLU","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.22038","source_url":"https://arxiv.org/abs/2609.22038"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-22038","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22038","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":8,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"claire_record_id","repository":"gh:bethgelab/qval","source_record":"bm_qval_38cf43e7"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"QVal is a training-free testbed for evaluating dense supervision signals for long-horizon LLM agents. It measures Q-alignment of scores from methods across four environments and seven families, without requiring training runs.","descriptions":{"en":"QVal is a training-free testbed for evaluating dense supervision signals for long-horizon LLM agents. It measures Q-alignment of scores from methods across four environments and seven families, without requiring training runs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.32034","languages":[],"library_direction":{"basis":"model-review-2","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_qval_38cf43e7"},"modality":null,"name":"QVal","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.32034","source_url":"https://arxiv.org/abs/2606.32034"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-32034","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.32034","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":354,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:jayleicn/moment_detr","source_record":"catalog_b52a2c09f0a9ab89"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QVHighlights is a video moment retrieval benchmark for detecting moments and highlights in videos via natural language queries. Given a query, the model must localize the start and end times of relevant moments in the video, evaluated using metrics such as Recall@1 at a 0.5 IoU threshold.","descriptions":{"en":"QVHighlights is a video moment retrieval benchmark for detecting moments and highlights in videos via natural language queries. Given a query, the model must localize the start and end times of relevant moments in the video, evaluated using metrics such as Recall@1 at a 0.5 IoU threshold."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qvhighlights:nova-2-lite","reported_at":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qvhighlights?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qvhighlights","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_b52a2c09f0a9ab89"},"modality":"multimodal","name":"QVHighlights","openness":"unknown","publisher":null,"released":"2021-07-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:qvhighlights","source_url":"https://arxiv.org/abs/2107.09609"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":77.2,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.772,"raw_min":0.767,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qvhighlights:nova-2-lite","reported_date":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qvhighlights?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"}],"slug":"llm-stats-qvhighlights","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qvhighlights?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":null,"source_record":"bm_qwen-image-bench_7fd96584"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Qwen-Image-Bench evaluates text-to-image models on five pillars including Real-world Fidelity and Creative Generation, with 1000 prompts and 56 rubric-based facets scored by a trained judge model.","descriptions":{"en":"Qwen-Image-Bench evaluates text-to-image models on five pillars including Real-world Fidelity and Creative Generation, with 1000 prompts and 56 rubric-based facets scored by a trained judge model."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28091","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_qwen-image-bench_7fd96584"},"modality":null,"name":"Qwen-Image-Bench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28091","source_url":"https://arxiv.org/abs/2605.28091"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28091","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28091","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The dataset presents performance and accuracy results for specific serving configurations, but does not define a reusable evaluation protocol or scoring contract for other teams.","descriptions":{"en":"The dataset presents performance and accuracy results for specific serving configurations, but does not define a reusable evaluation protocol or scoring contract for other teams."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:pxzleo/qwen3.8-27b-inference-benchmark-4090","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_qwen3-8-27b-inference-benchmark-4090_328340c0"},"modality":null,"name":"qwen3.8-27b-inference-benchmark-4090","openness":"unknown","publisher":null,"released":"2026-08-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:pxzleo/qwen3.8-27b-inference-benchmark-4090","source_url":"https://huggingface.co/datasets/pxzleo/qwen3.8-27b-inference-benchmark-4090"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-pxzleo-qwen3-8-27b-inference-benchmark-4090","source":"claire_radar","source_url":"https://huggingface.co/datasets/pxzleo/qwen3.8-27b-inference-benchmark-4090","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":61,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:skylenage-ai/qwenclawbench","source_record":"catalog_c6600ab2bf5eefa7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QwenClawBench is a real-user-distribution Claw agent benchmark for evaluating coding agents on realistic developer tasks.","descriptions":{"en":"QwenClawBench is a real-user-distribution Claw agent benchmark for evaluating coding agents on realistic developer tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qwenclawbench:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenclawbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qwenclawbench","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","professional-work"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_c6600ab2bf5eefa7"},"modality":"text","name":"QwenClawBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":61.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.618,"raw_min":0.618,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qwenclawbench:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenclawbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-qwenclawbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenclawbench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QwenQoderBench is Qwen's internal benchmark for evaluating coding-agent performance.","descriptions":{"en":"QwenQoderBench is Qwen's internal benchmark for evaluating coding-agent performance."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qwen-qoder-bench:qwen3.8-max","reported_at":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-qoder-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qwen-qoder-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_5590bdccdada7e0a"},"modality":"text","name":"QwenQoderBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":58.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.584,"raw_min":0.584,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qwen-qoder-bench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-qoder-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-qwen-qoder-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-qoder-bench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QwenReactBench is Qwen's internal React application generation benchmark, reported as a BT/Elo rating.","descriptions":{"en":"QwenReactBench is Qwen's internal React application generation benchmark, reported as a BT/Elo rating."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qwen-react-bench:qwen3.8-max","reported_at":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-react-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qwen-react-bench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_0cc3a79c633bc816"},"modality":"multimodal","name":"QwenReactBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":1724.0,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":1724.0,"raw_min":1724.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qwen-react-bench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-react-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-qwen-react-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-react-bench?top_n=500","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QwenSVG is Qwen's internal SVG generation benchmark for evaluating front-end and visual code generation. Scores are reported as BT/Elo ratings from auto-rendered outputs judged by a multimodal evaluator.","descriptions":{"en":"QwenSVG is Qwen's internal SVG generation benchmark for evaluating front-end and visual code generation. Scores are reported as BT/Elo ratings from auto-rendered outputs judged by a multimodal evaluator."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qwen-svg:qwen3.7-max","reported_at":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-svg?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qwen-svg","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_4cc2ff3b00c610fe"},"modality":"multimodal","name":"QwenSVG","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":1713.0,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":1713.0,"raw_min":1608.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qwen-svg:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-svg?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-qwen-svg","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-svg?top_n=500","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QwenSWEBench is Qwen's software-engineering agent benchmark for repository-level issue resolution.","descriptions":{"en":"QwenSWEBench is Qwen's software-engineering agent benchmark for repository-level issue resolution."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qwen-swe-bench:qwen3.8-max","reported_at":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-swe-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qwen-swe-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_e9379882dbaa5cbc"},"modality":"text","name":"QwenSWEBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":80.7,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.807,"raw_min":0.79,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qwen-swe-bench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-swe-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-qwen-swe-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-swe-bench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QwenWebBench is an internal front-end code generation benchmark by Qwen. It is bilingual (EN/CN) and spans 7 categories (Web Design, Web Apps, Games, SVG, Data Visualization, Animation, and 3D), using auto-render plus a multimodal judge for code and visual correctness. Scores are reported as BT/Elo ratings.","descriptions":{"en":"QwenWebBench is an internal front-end code generation benchmark by Qwen. It is bilingual (EN/CN) and spans 7 categories (Web Design, Web Apps, Games, SVG, Data Visualization, Animation, and 3D), using auto-render plus a multimodal judge for code and visual correctness. Scores are reported as BT/Elo ratings."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qwenwebbench:qwen3.6-27b","reported_at":"2026-04-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenwebbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qwenwebbench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_0ea8424382098997"},"modality":"multimodal","name":"QwenWebBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":1568.0,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":1568.0,"raw_min":1487.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qwenwebbench:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenwebbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"}],"slug":"llm-stats-qwenwebbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenwebbench?top_n=500","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","simulation","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"QwenWorldBench is Qwen's internal benchmark for evaluating LLMs as world models that simulate agentic environments across Terminal, SWE, MCP, Search, OS, Android, and Web domains.","descriptions":{"en":"QwenWorldBench is Qwen's internal benchmark for evaluating LLMs as world models that simulate agentic environments across Terminal, SWE, MCP, Search, OS, Android, and Web domains."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:qwenworldbench:qwen3.7-max","reported_at":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenworldbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:qwenworldbench","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","tool-use"],"join":"llm_stats_id","primary":"world-models","source_record":"catalog_790e679631e4dd86"},"modality":"text","name":"QwenWorldBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":62.1,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.621,"raw_min":0.573,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:qwenworldbench:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenworldbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-qwenworldbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenworldbench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Simulation benchmark for robot-to-human object handovers with standardized evaluation protocol and five complementary metrics: planning feasibility, reachability, grasp stability, grasp affordance, and safety. Compares four baseline methods on shared grasp pose prediction.","descriptions":{"en":"Simulation benchmark for robot-to-human object handovers with standardized evaluation protocol and five complementary metrics: planning feasibility, reachability, grasp stability, grasp affordance, and safety. Compares four baseline methods on shared grasp pose prediction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21011","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_r2handoversim_321897f0"},"modality":null,"name":"R2HandoverSim","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21011","source_url":"https://arxiv.org/abs/2606.21011"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21011","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21011","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":32,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:amap-ml/r2mbench","source_record":"bm_r2m-bench_1ea8e143"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates revisit memory in interactive video world models using relative consistency: for each detected return, compares revisit pair with gap-matched non-revisit and short-range control pairs to compute MemoryGain and Normalized Memory Ratio across appearance, identity, geometry, and state.","descriptions":{"en":"Evaluates revisit memory in interactive video world models using relative consistency: for each detected return, compares revisit pair with gap-matched non-revisit and short-range control pairs to compute MemoryGain and Normalized Memory Ratio across appearance, identity, geometry, and state."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.27328","languages":[],"library_direction":{"basis":"model-review","categories":["world-models"],"join":"claire_record_id","primary":"world-models","source_record":"bm_r2m-bench_1ea8e143"},"modality":null,"name":"R2M-Bench","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27328","source_url":"https://arxiv.org/abs/2608.27328"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27328","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27328","topics":[],"unit":null},{"aliases":["$R^3$-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets"],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":295,"hf_paper_upvotes":18,"join":"claire_record_id","repository":"gh:nineabyss/r-3-bench","source_record":"bm_r-3-bench-llms-struggle-with-resource-rati_fbfbc38c"},"categories":["Code & Software","Code generation","General AI","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates resource-rational reasoning by measuring how models allocate a shared budget across six-problem suites in mathematics, competitive programming, and abstract reasoning, with scoring based on completed problems.","descriptions":{"en":"Evaluates resource-rational reasoning by measuring how models allocate a shared budget across six-problem suites in mathematics, competitive programming, and abstract reasoning, with scoring based on completed problems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.16033","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","math","code-generation"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_r-3-bench-llms-struggle-with-resource-rati_fbfbc38c"},"modality":null,"name":"R3-Bench","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16033","source_url":"https://arxiv.org/abs/2608.16033"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16033","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16033","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":136,"github_stars_30d":21,"hf_dataset_downloads":null,"hf_paper_upvotes":287,"join":"claire_record_id","repository":"gh:24029100313/ra-bench","source_record":"bm_ra-bench_8deff463"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"RA-Bench evaluates AI-generated video detection using real crisis-event videos as anchors. It includes 17,886 videos: 1,830 real anchors across 10 social-risk categories and 16,056 generated clips from nine generators. Detection is scored per source via AUC and TPR@5%FPR, with separate tracks for human-deceptive and dissemination-processed videos.","descriptions":{"en":"RA-Bench evaluates AI-generated video detection using real crisis-event videos as anchors. It includes 17,886 videos: 1,830 real anchors across 10 social-risk categories and 16,056 generated clips from nine generators. Detection is scored per source via AUC and TPR@5%FPR, with separate tracks for human-deceptive and dissemination-processed videos."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.14391","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","safety-alignment"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_ra-bench_8deff463"},"modality":null,"name":"RA-Bench","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14391","source_url":"https://arxiv.org/abs/2608.14391"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14391","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14391","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"artifact:arxiv:1704.04683","repository":null,"source_record":"catalog_9777ed58668371e6"},"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RACE is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students. RACE 是一个大规模的阅读理解数据集，包含超过 28,000 个段落和近 100,000 个问题。该数据集是从中国的英语考试中收集而来，这些考试是为中学和高中学生设计的。","descriptions":{"en":"RACE is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students.","zh":"RACE 是一个大规模的阅读理解数据集，包含超过 28,000 个段落和近 100,000 个问题。该数据集是从中国的英语考试中收集而来，这些考试是为中学和高中学生设计的。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:516","languages":["English"],"library_direction":{"basis":"model-review-2","categories":["knowledge-expert-qa"],"join":"artifact:arxiv:1704.04683","primary":"knowledge-expert-qa","source_record":"catalog_9777ed58668371e6"},"modality":null,"name":"RACE(High)","openness":"unknown","publisher":null,"released":"2017-04-15","released_reference":{"basis":"paper_first_version","note":"The RACE introduction defines its high-school and middle-school partitions together.","source_key":"opencompass:516","source_url":"https://arxiv.org/abs/1704.04683"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-516-race-high","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RACE%28High%29","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"artifact:arxiv:1704.04683","repository":null,"source_record":"catalog_9777ed58668371e6"},"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RACE is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students. RACE 是一个大规模的阅读理解数据集，包含超过 28,000 个段落和近 100,000 个问题。该数据集是从中国的英语考试中收集而来，这些考试是为中学和高中学生设计的。","descriptions":{"en":"RACE is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students.","zh":"RACE 是一个大规模的阅读理解数据集，包含超过 28,000 个段落和近 100,000 个问题。该数据集是从中国的英语考试中收集而来，这些考试是为中学和高中学生设计的。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:517","languages":["English"],"library_direction":{"basis":"model-review-2","categories":["knowledge-expert-qa"],"join":"artifact:arxiv:1704.04683","primary":"knowledge-expert-qa","source_record":"catalog_9777ed58668371e6"},"modality":null,"name":"RACE(Middle)","openness":"unknown","publisher":null,"released":"2017-04-15","released_reference":{"basis":"paper_first_version","note":"The RACE introduction defines its middle-school and high-school partitions together.","source_key":"opencompass:517","source_url":"https://arxiv.org/abs/1704.04683"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-517-race-middle","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RACE%28Middle%29","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ano-kodokushi/rag-doc-qa","source_record":"bm_rag-doc-qa_37f27f2a"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Reports a locally developed 20-question evaluation set for a RAG document QA service with hybrid retrieval and cited answers.","descriptions":{"en":"Reports a locally developed 20-question evaluation set for a RAG document QA service with hybrid retrieval and cited answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:ano-kodokushi/rag-doc-qa","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research","document-ocr"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_rag-doc-qa_37f27f2a"},"modality":null,"name":"rag-doc-qa","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:ano-kodokushi/rag-doc-qa","source_url":"https://github.com/ano-kodokushi/rag-doc-qa"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-ano-kodokushi-rag-doc-qa","source":"claire_radar","source_url":"https://github.com/ano-kodokushi/rag-doc-qa","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Retrieval-Augmented Generation (RAG) systems are vulnerable to prompt-injection attacks embedded in retrieved content.","descriptions":{"en":"Retrieval-Augmented Generation (RAG) systems are vulnerable to prompt-injection attacks embedded in retrieved content."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.08571","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-security","deep-research"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_rag-pibench_128ea65a"},"modality":null,"name":"RAG-PIBench","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.08571","source_url":"https://arxiv.org/abs/2610.08571"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-08571","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08571","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:gauthamrohan/rag-retrieval-benchmark","source_record":"bm_rag-retrieval-benchmark_2e75b7eb"},"categories":["Cybersecurity","Finance","Information retrieval","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The repository compares seven retrieval configurations on FiQA queries using standard retrieval metrics, but does not self-identify as a named benchmark release.","descriptions":{"en":"The repository compares seven retrieval configurations on FiQA queries using standard retrieval metrics, but does not self-identify as a named benchmark release."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:gauthamrohan/rag-retrieval-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research","agentic-coding"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_rag-retrieval-benchmark_2e75b7eb"},"modality":null,"name":"rag-retrieval-benchmark","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:gauthamrohan/rag-retrieval-benchmark","source_url":"https://github.com/gauthamRohan/rag-retrieval-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-gauthamrohan-rag-retrieval-benchmark","source":"claire_radar","source_url":"https://github.com/gauthamRohan/rag-retrieval-benchmark","topics":["cybersecurity","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Measurement of the safety impact of retrieval augmentation on LLM outputs.","descriptions":{"en":"Measurement of the safety impact of retrieval augmentation on LLM outputs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11758","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research","safety-alignment"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_rag-safety-bench_6807be74"},"modality":null,"name":"RAG-Safety-Bench","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11758","source_url":"https://arxiv.org/abs/2609.11758"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11758","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11758","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Information retrieval","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Retrieval-Augmented Generation (RAG) is typically evaluated under the implicit assumption that the underlying knowledge base (KB) is clean, leaving the behaviour of RAG systems under realistic KB degradation poorly characterised.","descriptions":{"en":"Retrieval-Augmented Generation (RAG) is typically evaluated under the implicit assumption that the underlying knowledge base (KB) is clean, leaving the behaviour of RAG systems under realistic KB degradation poorly characterised."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.04691","languages":[],"library_direction":{"basis":"model-review-daily","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_ragstress_3c0fa068"},"modality":null,"name":"RAGStress","openness":"unknown","publisher":null,"released":"2026-10-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.04691","source_url":"https://arxiv.org/abs/2610.04691"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-04691","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04691","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"RandomBench evaluates whether multimodal LLMs maintain distributionally neutral behavior when selecting among equivalent options, providing metrics for entropy and distributional bias under explicit random instructions.","descriptions":{"en":"RandomBench evaluates whether multimodal LLMs maintain distributionally neutral behavior when selecting among equivalent options, providing metrics for entropy and distributional bias under explicit random instructions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05874","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_randombench_dfec6d0b"},"modality":null,"name":"RandomBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05874","source_url":"https://arxiv.org/abs/2606.05874"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05874","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05874","topics":["multimodal","safety"],"unit":null},{"aliases":["RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_ratio-a-benchmark-for-retrieval-across-typ_4be3618a"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates retrieval of scientific literature passages relevant to three ideation operations: Address, Broaden, and Specify, using millions of full-text computer science papers.","descriptions":{"en":"Evaluates retrieval of scientific literature passages relevant to three ideation operations: Address, Broaden, and Specify, using millions of full-text computer science papers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27394","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","science"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_ratio-a-benchmark-for-retrieval-across-typ_4be3618a"},"modality":null,"name":"RATIO","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27394","source_url":"https://arxiv.org/abs/2608.27394"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27394","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27394","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":83,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_rcv-bench_8c722794"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates research-claim verification by scoring agents on classifying claimed results as reproduced, deviation, fabricated, or fragile using code re-execution and robustness probing.","descriptions":{"en":"Evaluates research-claim verification by scoring agents on classifying claimed results as reproduced, deviation, fabricated, or fragile using code re-execution and robustness probing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:grobestreet/rcv-bench","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","science"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_rcv-bench_8c722794"},"modality":null,"name":"RCV-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:grobestreet/rcv-bench","source_url":"https://huggingface.co/datasets/GrobeStreet/rcv-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-grobestreet-rcv-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/GrobeStreet/rcv-bench","topics":["agents","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","科学智能","AI for Science","代码工程","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RDB2G-Bench provides comprehensive performance evaluation data for graph neural network models applied to relational database tasks. The dataset contains extensive experiments across multiple graph configurations and architectures. RDB2G-Bench提供了针对关系数据库任务应用的图神经网络模型的全面性能评估数据。该数据集包含了跨多种图配置和架构的广泛实验。","descriptions":{"en":"RDB2G-Bench provides comprehensive performance evaluation data for graph neural network models applied to relational database tasks. The dataset contains extensive experiments across multiple graph configurations and architectures.","zh":"RDB2G-Bench提供了针对关系数据库任务应用的图神经网络模型的全面性能评估数据。该数据集包含了跨多种图配置和架构的广泛实验。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1918","languages":[],"library_direction":null,"modality":null,"name":"RDB2G-Bench","openness":"open","publisher":"Kumo.AI , Kim Jaechul Graduate School of AI, KAIST","released":"2025-06-02","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1918-rdb2g-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RDB2G-Bench","topics":["coding","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ltynick/rdvsv2","source_record":"bm_rdvsv2_d98f1ace"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"RDVSv2 is a large-scale benchmark for RGB-D video salient object detection, containing 249 video sequences with 29,077 annotated frames. It includes depth maps, optical flow, and eye-tracking-guided salient object masks. The benchmark provides a fixed dataset and evaluation protocol for comparing models on this task.","descriptions":{"en":"RDVSv2 is a large-scale benchmark for RGB-D video salient object detection, containing 249 video sequences with 29,077 annotated frames. It includes depth maps, optical flow, and eye-tracking-guided salient object masks. The benchmark provides a fixed dataset and evaluation protocol for comparing models on this task."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.25392","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_rdvsv2_d98f1ace"},"modality":null,"name":"RDVSv2","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25392","source_url":"https://arxiv.org/abs/2607.25392"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25392","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25392","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","代码","Code","智能体","Agent","任务执行","Task Execution","逻辑推理","Reasoning","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RE-Bench (Research Engineering Benchmark, v1) consists of 7 challenging, open-ended ML research engineering environments and data from 71 8-hour attempts by 61 distinct human experts. RE-Bench用于评估AI智能体研发的自动化能力，它由61位人类专家71次在7个具有挑战性的开放式ML研究工程环境中的8小时尝试的数据组成。","descriptions":{"en":"RE-Bench (Research Engineering Benchmark, v1) consists of 7 challenging, open-ended ML research engineering environments and data from 71 8-hour attempts by 61 distinct human experts.","zh":"RE-Bench用于评估AI智能体研发的自动化能力，它由61位人类专家71次在7个具有挑战性的开放式ML研究工程环境中的8小时尝试的数据组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1252","languages":[],"library_direction":null,"modality":null,"name":"RE-Bench","openness":"unknown","publisher":"Model Evaluation and Threat Research","released":"2024-11-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1252-re-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RE-Bench","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"ReactBench evaluates multimodal large language models on cause-driven hallucination through four targeted tasks (Relational Erasure, Counterfactual Attribute, Alteration Tracing, Dense Counting) with exam-style evaluation and chain-of-thought reasoning for sub-cause identification.","descriptions":{"en":"ReactBench evaluates multimodal large language models on cause-driven hallucination through four targeted tasks (Relational Erasure, Counterfactual Attribute, Alteration Tracing, Dense Counting) with exam-style evaluation and chain-of-thought reasoning for sub-cause identification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29579","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_reactbench_423fd886"},"modality":null,"name":"ReactBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29579","source_url":"https://arxiv.org/abs/2605.29579"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29579","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29579","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":518,"hf_paper_upvotes":54,"join":"claire_record_id","repository":null,"source_record":"bm_reacthuman_dbe6f627"},"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates human-like reactive decision-making in multimodal LLMs by simulating 1,000+ reproducible household hazard scenes with physics-grounded ground truth across 17 event families.","descriptions":{"en":"Evaluates human-like reactive decision-making in multimodal LLMs by simulating 1,000+ reproducible household hazard scenes with physics-grounded ground truth across 17 event families."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10895","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_reacthuman_dbe6f627"},"modality":null,"name":"ReactHuman","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10895","source_url":"https://arxiv.org/abs/2609.10895"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10895","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10895","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Autonomous Driving","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"ReactSim-Bench evaluates reactive capability of behavior world model simulators in autonomous driving by decoupling agent and AV control, using collision, map, and kinematic metrics on 2,636 scenarios.","descriptions":{"en":"ReactSim-Bench evaluates reactive capability of behavior world model simulators in autonomous driving by decoupling agent and AV control, using collision, map, and kinematic metrics on 2,636 scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.14058","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","embodied-robotics"],"join":"claire_record_id","primary":"world-models","source_record":"bm_reactsim-bench_5401cd81"},"modality":null,"name":"ReactSim-Bench","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14058","source_url":"https://arxiv.org/abs/2606.14058"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14058","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14058","topics":[],"unit":null},{"aliases":["Read the Room, Read the Image: Understanding Indirect Speech Acts in Multimodal Visual Contexts"],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Multimodal benchmark for indirect speech act understanding in visual contexts, supporting English and Korean.","descriptions":{"en":"Multimodal benchmark for indirect speech act understanding in visual contexts, supporting English and Korean."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30270","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual","speech-audio"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_read-the-room-read-the-image-understanding_4e0b4326"},"modality":null,"name":"READI","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30270","source_url":"https://arxiv.org/abs/2608.30270"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30270","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30270","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites 在真实网站的确定性模拟上对自主代理进行基准测试","descriptions":{"en":"Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites","zh":"在真实网站的确定性模拟上对自主代理进行基准测试"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1755","languages":[],"library_direction":null,"modality":null,"name":"REAL","openness":"unknown","publisher":"The AGI Company, Stanford University，etc.","released":"2025-04-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1755-real","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/REAL","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":56,"github_stars_30d":11,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:internrobotics/real","source_record":"bm_real-bench_906823b4"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"REAL-Bench evaluates vision-driven embodied agents in open-world mobile manipulation across 241 tasks spanning active exploration, visual distraction, articulated manipulation, and interactive disambiguation. The benchmark provides standardized task definitions and a simulator-based evaluation protocol.","descriptions":{"en":"REAL-Bench evaluates vision-driven embodied agents in open-world mobile manipulation across 241 tasks spanning active exploration, visual distraction, articulated manipulation, and interactive disambiguation. The benchmark provides standardized task definitions and a simulator-based evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.13653","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_real-bench_906823b4"},"modality":null,"name":"REAL-Bench","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.13653","source_url":"https://arxiv.org/abs/2607.13653"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-13653","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13653","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":8,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:lixruize-del/nwp-benchmark","source_record":"bm_realbench_3d168523"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates data-driven numerical weather forecasting models under operational conditions, using strictly out-of-distribution test data from 2025 and integrating low-latency operational analysis and large-scale in-situ observations from over 10,000 stations. It provides metrics for global forecasting and for extreme events such as heatwaves, cold surges, and tropical cyclones.","descriptions":{"en":"Evaluates data-driven numerical weather forecasting models under operational conditions, using strictly out-of-distribution test data from 2025 and integrating low-latency operational analysis and large-scale in-situ observations from over 10,000 stations. It provides metrics for global forecasting and for extreme events such as heatwaves, cold surges, and tropical cyclones."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.24945","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_realbench_3d168523"},"modality":null,"name":"RealBench","openness":"unknown","publisher":null,"released":"2026-05-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24945","source_url":"https://arxiv.org/abs/2605.24945"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24945","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24945","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["CAD","Geometric reasoning","Manufacturing & Process Control","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"A proposed benchmark for parametric CAD modeling across 12,632 tasks from 19 factory-automation categories, evaluating text, drawing, picture, and rendered image inputs for Part and Assembly modeling.","descriptions":{"en":"A proposed benchmark for parametric CAD modeling across 12,632 tasks from 19 factory-automation categories, evaluating text, drawing, picture, and rendered image inputs for Part and Assembly modeling."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.03773","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","engineering-design"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_realcadbench_dfd31493"},"modality":null,"name":"RealCADBench","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03773","source_url":"https://arxiv.org/abs/2609.03773"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03773","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03773","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":8,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fschmid56/realdesed","source_record":"bm_realdesed_918e47e6"},"categories":["General AI","Vision & 3D","eess.AS"],"collected_at":"2026-10-14T09:44:51Z","description":"RealDESED is a real-world domestic sound event detection benchmark with 5,710 recordings from 652 participants, 15 classes, and temporally precise annotations. Includes multi-annotator labeling and rich metadata.","descriptions":{"en":"RealDESED is a real-world domestic sound event detection benchmark with 5,710 recordings from 652 participants, 15 classes, and temporally precise annotations. Includes multi-annotator labeling and rich metadata."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.16736","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_realdesed_918e47e6"},"modality":null,"name":"RealDESED","openness":"unknown","publisher":null,"released":"2026-07-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.16736","source_url":"https://arxiv.org/abs/2607.16736"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-16736","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16736","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"RealDocBench evaluates field-level QA and layout understanding on real regulated documents, with 1,356 field-level questions over 581 documents and 1,500 annotated page images, scored on per-field accuracy and adjacency-aware layout metrics.","descriptions":{"en":"RealDocBench evaluates field-level QA and layout understanding on real regulated documents, with 1,356 field-level questions over 581 documents and 1,500 annotated page images, scored on per-field accuracy and adjacency-aware layout metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07401","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_realdocbench_6da01042"},"modality":null,"name":"RealDocBench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07401","source_url":"https://arxiv.org/abs/2606.07401"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07401","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07401","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on dataset-specific cues and fail on unfamiliar generators or proce…","descriptions":{"en":"AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on dataset-specific cues and fail on unfamiliar generators or proce…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.32734","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_realis_8784a988"},"modality":null,"name":"REALIS","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.32734","source_url":"https://arxiv.org/abs/2609.32734"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-32734","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32734","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Cybersecurity","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"RealisticTritonBench evaluates LLM-generated Triton kernels using tasks derived from real-world pull requests in popular AI frameworks, with end-to-end integration tests.","descriptions":{"en":"RealisticTritonBench evaluates LLM-generated Triton kernels using tasks derived from real-world pull requests in popular AI frameworks, with end-to-end integration tests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12004","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","code-generation"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_realistictritonbench_fb357880"},"modality":null,"name":"RealisticTritonBench","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12004","source_url":"https://arxiv.org/abs/2608.12004"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12004","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12004","topics":["coding","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":9,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_ce1117a4d20de18c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","document_understanding","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RealKIE-FCC is a key information extraction benchmark drawn from real enterprise documents (FCC filings), part of the RealKIE suite of five novel datasets for enterprise key information extraction. Models must convert documents to markdown and extract structured fields against a specified JSON schema. Nova 2 reports results on a human-verified version of the dataset.","descriptions":{"en":"RealKIE-FCC is a key information extraction benchmark drawn from real enterprise documents (FCC filings), part of the RealKIE suite of five novel datasets for enterprise key information extraction. Models must convert documents to markdown and extract structured fields against a specified JSON schema. Nova 2 reports results on a human-verified version of the dataset."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:realkie-fcc:nova-2-lite","reported_at":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/realkie-fcc?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:realkie-fcc","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"llm_stats_id","primary":"document-ocr","source_record":"catalog_ce1117a4d20de18c"},"modality":"multimodal","name":"RealKIE-FCC","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":67.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.67,"raw_min":0.598,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:realkie-fcc:nova-2-pro","reported_date":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/realkie-fcc?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"}],"slug":"llm-stats-realkie-fcc","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/realkie-fcc?top_n=500","topics":["multimodal"],"unit":null},{"aliases":["RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning"],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark of authentic high-school exam responses for evaluating how accurately language-model judges assess human mathematical reasoning.","descriptions":{"en":"A benchmark of authentic high-school exam responses for evaluating how accurately language-model judges assess human mathematical reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.10254","languages":[],"library_direction":null,"modality":null,"name":"RealMath-Eval","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10254","source_url":"https://arxiv.org/abs/2606.10254"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10254","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10254","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Real-world occlusion boundary benchmark with definition-consistent labels and orientation maps.","descriptions":{"en":"Real-world occlusion boundary benchmark with definition-consistent labels and orientation maps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30820","languages":[],"library_direction":null,"modality":null,"name":"RealOOB","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30820","source_url":"https://arxiv.org/abs/2608.30820"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30820","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30820","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RealToxicityPrompts is a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widely used toxicity classifier. RealToxicityPrompts 是一个包含 100,000 个自然出现的、句子级提示的数据集，这些提示来自于大量的英语网络文本，并配有来自广泛使用的毒性分类器的毒性评分。","descriptions":{"en":"RealToxicityPrompts is a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widely used toxicity classifier.","zh":"RealToxicityPrompts 是一个包含 100,000 个自然出现的、句子级提示的数据集，这些提示来自于大量的英语网络文本，并配有来自广泛使用的毒性分类器的毒性评分。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1124","languages":["English"],"library_direction":null,"modality":null,"name":"RealToxicityPrompts","openness":"open","publisher":"Allen Institute for Artificial Intelligence","released":"2020-11-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1124-realtoxicityprompts","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RealToxicityPrompts","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":3518,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_9c82b133d721bbd1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["spatial_reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RealWorldQA is a benchmark designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The initial release consists of over 700 anonymized images taken from vehicles and other real-world scenarios, each accompanied by a question and easily verifiable answer. Released by xAI as part of their Grok-1.5 Vision preview to test models' ability to understand natural scenes and spatial relationships in everyday visual contexts.","descriptions":{"en":"RealWorldQA is a benchmark designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The initial release consists of over 700 anonymized images taken from vehicles and other real-world scenarios, each accompanied by a question and easily verifiable answer. Released by xAI as part of their Grok-1.5 Vision preview to test models' ability to understand natural scenes and spatial relationships in everyday visual contexts."},"evidence_summary":{"document_count":1,"model_count":29,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:realworldqa:grok-1.5v","reported_at":"2024-04-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/realworldqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":true,"key":"llm-stats:realworldqa","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_9c82b133d721bbd1"},"modality":"multimodal","name":"RealWorldQA","openness":"unknown","publisher":"XAI","released":"2024-04-12","released_reference":{"basis":"release_announcement","note":"The dated Grok-1.5V announcement introduces RealWorldQA and provides its initial public dataset download.","source_key":"opencompass:1361","source_url":"https://x.ai/news/grok-1.5v"},"repo_kind":"not_found","repo_resolution_status":"not_found","score_count":29,"score_direction":"higher_is_better","score_summary":{"display_max":88.0,"display_multiplier":100,"model_count":29,"model_count_basis":"source_model_id","numeric_count":29,"raw_max":0.88,"raw_min":0.622,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:realworldqa:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/realworldqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2024-04-12","date_precision":"model_announcement","name":"Grok-1.5V"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-realworldqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/realworldqa?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":3518,"hf_paper_upvotes":null,"join":"artifact:hf:xai-org/realworldqa","repository":null,"source_record":"catalog_9c82b133d721bbd1"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RealWorldQA is a benchmark designed for real-world understanding, including 765 images, each accompanied by a question and a verifiable answer. RealWorldQA用于评估多模态模型在现实世界中的空间理解能力，包含765张图像，每张图像都配有一个问题和易于验证的答案。","descriptions":{"en":"RealWorldQA is a benchmark designed for real-world understanding, including 765 images, each accompanied by a question and a verifiable answer.","zh":"RealWorldQA用于评估多模态模型在现实世界中的空间理解能力，包含765张图像，每张图像都配有一个问题和易于验证的答案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":true,"key":"opencompass:1361","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"artifact:hf:xai-org/realworldqa","primary":"visual-reasoning","source_record":"catalog_9c82b133d721bbd1"},"modality":"multimodal","name":"RealworldQA","openness":"unknown","publisher":"XAI","released":"2024-04-12","released_reference":{"basis":"release_announcement","note":"The dated Grok-1.5V announcement introduces RealWorldQA and provides its initial public dataset download.","source_key":"opencompass:1361","source_url":"https://x.ai/news/grok-1.5v"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1361-realworldqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RealworldQA","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":19,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"claire_record_id","repository":"gh:aim-uofa/reasonmatch","source_record":"bm_reasonmatch-bench_7b565eab"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"ReasonMatch-Bench evaluates wide-baseline matching and spatial reasoning in MLLMs, with benchmarks stratified by viewpoint displacement and matching granularity across indoor, outdoor, and object-centric scenarios.","descriptions":{"en":"ReasonMatch-Bench evaluates wide-baseline matching and spatial reasoning in MLLMs, with benchmarks stratified by viewpoint displacement and matching granularity across indoor, outdoor, and object-centric scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.03577","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_reasonmatch-bench_7b565eab"},"modality":null,"name":"ReasonMatch-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03577","source_url":"https://arxiv.org/abs/2606.03577"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03577","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03577","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"RECAP evaluates continual-learning phenomena in prompt-based LLM adaptation under evolving constraints, using a proactive adapt-then-test protocol with constraint-level metrics for forgetting, regression, and forward transfer.","descriptions":{"en":"RECAP evaluates continual-learning phenomena in prompt-based LLM adaptation under evolving constraints, using a proactive adapt-then-test protocol with constraint-level metrics for forgetting, regression, and forward transfer."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06698","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_recap_285c2bfa"},"modality":null,"name":"RECAP","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06698","source_url":"https://arxiv.org/abs/2606.06698"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06698","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06698","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Receipt Replay OOD is a small out-of-domain benchmark for screen replay detection. It uses receipts, which share planar geometry, curved corners, wear-and-tear artifacts, and text patterns with identity documents, to evaluate document replay detection models under cross-domain conditions without personally identifiable information constraints.","descriptions":{"en":"Receipt Replay OOD is a small out-of-domain benchmark for screen replay detection. It uses receipts, which share planar geometry, curved corners, wear-and-tear artifacts, and text patterns with identity documents, to evaluate document replay detection models under cross-domain conditions without personally identifiable information constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26855","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection","document-ocr"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_receipt-replay-ood_d32b05db"},"modality":null,"name":"Receipt Replay OOD","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26855","source_url":"https://arxiv.org/abs/2605.26855"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26855","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26855","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:wwwt0ri/receiptbench","source_record":"bm_receiptbench_9f399a36"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"ReceiptBench evaluates multimodal large language models on visual information extraction from receipts. It includes 10k human-annotated receipts and four hierarchical subtasks: basic perception, format normalization, semantic reasoning, and structure parsing, with defined metrics and scoring.","descriptions":{"en":"ReceiptBench evaluates multimodal large language models on visual information extraction from receipts. It includes 10k human-annotated receipts and four hierarchical subtasks: basic perception, format normalization, semantic reasoning, and structure parsing, with defined metrics and scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.22413","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_receiptbench_9f399a36"},"modality":null,"name":"ReceiptBench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22413","source_url":"https://arxiv.org/abs/2605.22413"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22413","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22413","topics":["reasoning-math"],"unit":null},{"aliases":["ReClor-plus","ReClor-Plus","ReClor plus"],"attention_signals":null,"categories":["reasoning"],"collected_at":null,"description":"A robustness variant of ReClor in three subsets: shuffled options, the correct choice replaced by \"none of the other options is correct\", and both combined. Read it against the same model's score on the original ReClor, because the gap is the signal; one blended number hides which variation caused the drop, and the source questions are public.","descriptions":{"en":"A robustness variant of ReClor in three subsets: shuffled options, the correct choice replaced by \"none of the other options is correct\", and both combined. Read it against the same model's score on the original ReClor, because the gap is the signal; one blended number hides which variation caused the drop, and the source questions are public."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:reclor_plus","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["abstract-reasoning"],"join":"review:14H034160212:2026-10-15","primary":"abstract-reasoning","source_record":null},"modality":null,"name":"ReClor-plus","openness":"unknown","publisher":null,"released":"2023-10-13","released_reference":{"basis":"benchmark_release","source_key":"model-reports:reclor_plus","source_url":"https://arxiv.org/abs/2310.09430"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"reclor_plus","source":"model_reports","source_url":"https://arxiv.org/abs/2310.09430","topics":["reasoning-math"],"unit":null},{"aliases":["RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"RECOM evaluates automatic metrics for open-ended Reddit question answering using authentic replies and a random-derangement noise floor.","descriptions":{"en":"RECOM evaluates automatic metrics for open-ended Reddit question answering using authentic replies and a random-derangement noise floor."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19218","languages":[],"library_direction":null,"modality":null,"name":"RECOM","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19218","source_url":"https://arxiv.org/abs/2606.19218"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19218","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19218","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Reconstruction is a blind benchmark for recovering research ideas from pre-publication bibliographies, using a judge model to match hypotheses.","descriptions":{"en":"Reconstruction is a blind benchmark for recovering research ideas from pre-publication bibliographies, using a judge model to match hypotheses."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.16645","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_reconstruction_90ed3fab"},"modality":null,"name":"Reconstruction","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16645","source_url":"https://arxiv.org/abs/2608.16645"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16645","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16645","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"artifact:arxiv:1905.00537","repository":null,"source_record":"catalog_56ba0cb792f6bfce"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ReCoRD is a reading comprehension task, which requires to extract the answer from the article based on the given news article and question. ReCoRD是一个阅读理解任务，要求根据给定的新闻文章和问题，从文章中抽取出答案。","descriptions":{"en":"ReCoRD is a reading comprehension task, which requires to extract the answer from the article based on the given news article and question.","zh":"ReCoRD是一个阅读理解任务，要求根据给定的新闻文章和问题，从文章中抽取出答案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:530","languages":[],"library_direction":null,"modality":null,"name":"ReCoRD","openness":"unknown","publisher":null,"released":"2018-10-30","released_reference":{"basis":"paper_first_version","note":"The original ReCoRD introduction, not the later SuperGLUE paper that reuses it.","source_key":"opencompass:530","source_url":"https://arxiv.org/abs/1810.12885"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-530-record","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ReCoRD","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aryanthemscoder/recoveriq","source_record":"bm_recoveriq_46ac0f4c"},"categories":["Finance","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The repository describes an internal evaluation harness for a revenue recovery engine, but no formal benchmark name or independent reuse protocol is declared.","descriptions":{"en":"The repository describes an internal evaluation harness for a revenue recovery engine, but no formal benchmark name or independent reuse protocol is declared."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:aryanthemscoder/recoveriq","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding","agent-skills-evolution"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_recoveriq_46ac0f4c"},"modality":null,"name":"recoveriq","openness":"unknown","publisher":null,"released":"2026-08-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:aryanthemscoder/recoveriq","source_url":"https://github.com/aryanthemscoder/recoveriq"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-aryanthemscoder-recoveriq","source":"claire_radar","source_url":"https://github.com/aryanthemscoder/recoveriq","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","code","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RecreationBench is Qwen's long-horizon application-recreation benchmark for hybrid agents across Ubuntu, macOS, Windows, Android, and the web.","descriptions":{"en":"RecreationBench is Qwen's long-horizon application-recreation benchmark for hybrid agents across Ubuntu, macOS, Windows, Android, and the web."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:recreationbench:qwen3.8-27b","reported_at":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/recreationbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:recreationbench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","computer-use"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_1e2d96f1d3c05843"},"modality":"multimodal","name":"RecreationBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":47.099999999999994,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.471,"raw_min":0.471,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:recreationbench:qwen3.8-27b","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/recreationbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-recreationbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/recreationbench?top_n=500","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_redact_293a3dff"},"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"REDACT is a multilingual benchmark for PII detection with 13,427 records, 51 entity types, and controlled generation axes. It allows stratified evaluation via metadata fields and includes an evaluation harness.","descriptions":{"en":"REDACT is a multilingual benchmark for PII detection with 13,427 records, 51 entity types, and controlled generation axes. It allows stratified evaluation via metadata fields and includes an evaluation harness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19881","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_redact_293a3dff"},"modality":null,"name":"REDACT","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19881","source_url":"https://arxiv.org/abs/2606.19881"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19881","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19881","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_redactionbench_13449565"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"RedactionBench evaluates contextual redaction of PII across 200 documents and 11 domains, with a character-level R-Score metric that treats semantically similar redactions equally.","descriptions":{"en":"RedactionBench evaluates contextual redaction of PII across 200 documents and 11 domains, with a character-level R-Score metric that treats semantically similar redactions equally."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18782","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_redactionbench_13449565"},"modality":null,"name":"RedactionBench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18782","source_url":"https://arxiv.org/abs/2606.18782"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18782","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18782","topics":["safety"],"unit":null},{"aliases":["REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_redagentbench-executable-red-teaming-and-f_69dd1033"},"categories":["Agents","Agents & Tool Use","Cybersecurity"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agent safety through executable red-teaming, adversarial case generation, and verification of harmful effects across 1,661 cases and five service surfaces.","descriptions":{"en":"Evaluates LLM agent safety through executable red-teaming, adversarial case generation, and verification of harmful effects across 1,661 cases and five service surfaces."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10669","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_redagentbench-executable-red-teaming-and-f_69dd1033"},"modality":null,"name":"REDAgentBench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10669","source_url":"https://arxiv.org/abs/2608.10669"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10669","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10669","topics":["agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","智能体","Agent","任务执行","Task Execution","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RedCode provides comprehensive and practical evaluations on the safety of code agents, including 4,050 risky test cases covering 25 types of critical vulnerabilities spanning 8 domains and 160 prompts aiming to generate harmful code or software. RedCode旨在为LLM代码智能体的安全性提供全面实用的评估，包括来自8个领域25种关键漏洞的4050个风险测试用例，以及160个生成有害代码的提示。","descriptions":{"en":"RedCode provides comprehensive and practical evaluations on the safety of code agents, including 4,050 risky test cases covering 25 types of critical vulnerabilities spanning 8 domains and 160 prompts aiming to generate harmful code or software.","zh":"RedCode旨在为LLM代码智能体的安全性提供全面实用的评估，包括来自8个领域25种关键漏洞的4050个风险测试用例，以及160个生成有害代码的提示。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1333","languages":[],"library_direction":null,"modality":null,"name":"RedCode","openness":"unknown","publisher":"University of Chicago","released":"2024-11-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1333-redcode","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RedCode","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"RedundancyBench is a benchmark for detecting redundant steps in agent trajectories. It contains diverse tasks with annotated trajectories where each step is labeled for its contribution to task completion.","descriptions":{"en":"RedundancyBench is a benchmark for detecting redundant steps in agent trajectories. It contains diverse tasks with annotated trajectories where each step is labeled for its contribution to task completion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29893","languages":[],"library_direction":null,"modality":null,"name":"RedundancyBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29893","source_url":"https://arxiv.org/abs/2605.29893"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29893","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29893","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":14,"join":"claire_record_id","repository":null,"source_record":"bm_redvox_a5dbb799"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"RedVox is a multilingual safety and fairness benchmark for speech models, built on real voices. It covers unsafe and unfair stereotypical requests across five languages and evaluates models under naturalistic conditions.","descriptions":{"en":"RedVox is a multilingual safety and fairness benchmark for speech models, built on real voices. It covers unsafe and unfair stereotypical requests across five languages and evaluates models under naturalistic conditions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26968","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","speech-audio","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_redvox_a5dbb799"},"modality":null,"name":"RedVox","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26968","source_url":"https://arxiv.org/abs/2606.26968"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26968","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26968","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":577,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:lichengunc/refer","source_record":"catalog_d38aa6a53ea7ae68"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["spatial_reasoning","grounding","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RefCOCO-avg measures object grounding accuracy averaged across RefCOCO, RefCOCO+, and RefCOCOg benchmarks.","descriptions":{"en":"RefCOCO-avg measures object grounding accuracy averaged across RefCOCO, RefCOCO+, and RefCOCOg benchmarks."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:refcoco-avg:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refcoco-avg?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:refcoco-avg","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_d38aa6a53ea7ae68"},"modality":"image","name":"RefCOCO-avg","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":0.935,"display_multiplier":1,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.935,"raw_min":0.732,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:refcoco-avg:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refcoco-avg?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"North Micro Vision Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-refcoco-avg","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refcoco-avg?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":166,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_83310fc2f621b72a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","grounding","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RefCOCOg is a referring expression comprehension benchmark that evaluates spatial grounding in images. Given a natural language expression describing an object, the model must localize the correct region, evaluated by accuracy at a 0.5 IoU threshold. It features longer, more descriptive expressions than RefCOCO and RefCOCO+.","descriptions":{"en":"RefCOCOg is a referring expression comprehension benchmark that evaluates spatial grounding in images. Given a natural language expression describing an object, the model must localize the correct region, evaluated by accuracy at a 0.5 IoU threshold. It features longer, more descriptive expressions than RefCOCO and RefCOCO+."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:refcocog:nova-2-omni","reported_at":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refcocog?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:refcocog","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_83310fc2f621b72a"},"modality":"multimodal","name":"RefCOCOg","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":86.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.863,"raw_min":0.863,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:refcocog:nova-2-omni","reported_date":"2025-12-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refcocog?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"}],"slug":"llm-stats-refcocog","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refcocog?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":43,"join":"claire_record_id","repository":null,"source_record":"bm_refiner-bench_64fe104a"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"High-resolution video generation is expensive, as its cost grows rapidly with the number of spatiotemporal tokens.","descriptions":{"en":"High-resolution video generation is expensive, as its cost grows rapidly with the number of spatiotemporal tokens."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37969","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_refiner-bench_64fe104a"},"modality":null,"name":"Refiner-Bench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37969","source_url":"https://arxiv.org/abs/2609.37969"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37969","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37969","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:lxroboticslab/reflexbench","source_record":"bm_reflexbench_c4d78abf"},"categories":["Multimodal","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language-action models on reaction-critical manipulation across six dynamic tasks, with configurable latency under synchronous and asynchronous inference in a simulated environment.","descriptions":{"en":"Evaluates vision-language-action models on reaction-critical manipulation across six dynamic tasks, with configurable latency under synchronous and asynchronous inference in a simulated environment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.14379","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_reflexbench_c4d78abf"},"modality":null,"name":"ReflexBench","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14379","source_url":"https://arxiv.org/abs/2608.14379"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14379","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14379","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"RefMem-Bench is a benchmark for reflective memory in long-horizon dialogue, containing 26K QA instances across eight reflective-memory dimensions and three task formats.","descriptions":{"en":"RefMem-Bench is a benchmark for reflective memory in long-horizon dialogue, containing 26K QA instances across eight reflective-memory dimensions and three task formats."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01223","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_refmem-bench_c6cd0aef"},"modality":null,"name":"RefMem-Bench","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01223","source_url":"https://arxiv.org/abs/2606.01223"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01223","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01223","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Large Language Model (LLM) agents are increasingly deployed in high-stakes settings such as industrial maintenance and equipment fault troubleshooting, where workers occupy a variety of roles.","descriptions":{"en":"Large Language Model (LLM) agents are increasingly deployed in high-stakes settings such as industrial maintenance and equipment fault troubleshooting, where workers occupy a variety of roles."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.03356","languages":[],"library_direction":{"basis":"model-review-daily","categories":["professional-work","tool-use"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_refract_417e3ba4"},"modality":null,"name":"ReFract","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03356","source_url":"https://arxiv.org/abs/2610.03356"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03356","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03356","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":268,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":43,"join":"llm_stats_id","repository":"gh:zhoues/roborefer","source_record":"catalog_a4147f9be01ab400"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["spatial_reasoning","grounding","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RefSpatialBench evaluates spatial reference understanding and grounding.","descriptions":{"en":"RefSpatialBench evaluates spatial reference understanding and grounding."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:refspatialbench:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refspatialbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:refspatialbench","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_a4147f9be01ab400"},"modality":"image","name":"RefSpatialBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":0.7,"display_multiplier":1,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.7,"raw_min":0.635,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:refspatialbench:qwen3.6-27b","reported_date":"2026-04-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refspatialbench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-refspatialbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refspatialbench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":51,"github_stars_30d":39,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:audn-ai/refusal-benchmark","source_record":"bm_refusal-benchmark_83307c24"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Measures LLM refusal rates on a set of harmful prompts via OpenAI-compatible endpoints.","descriptions":{"en":"Measures LLM refusal rates on a set of harmful prompts via OpenAI-compatible endpoints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:audn-ai/refusal-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_refusal-benchmark_83307c24"},"modality":null,"name":"refusal-benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:audn-ai/refusal-benchmark","source_url":"https://github.com/audn-ai/refusal-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-audn-ai-refusal-benchmark","source":"claire_radar","source_url":"https://github.com/audn-ai/refusal-benchmark","topics":["finance"],"unit":null},{"aliases":["reg-affairs-bench"],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI systems on regulatory affairs deliverables from drugs, biologics, and medical devices using scenarios and dossier documents, scored against task-specific rubrics by a jury of LLM judges.","descriptions":{"en":"Evaluates AI systems on regulatory affairs deliverables from drugs, biologics, and medical devices using scenarios and dossier documents, scored against task-specific rubrics by a jury of LLM judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:arca-inc/reg-affairs-bench","languages":[],"library_direction":{"basis":"model-review-daily","categories":["health-medicine","professional-work"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_reg-affairs-bench_fd34ed25"},"modality":null,"name":"Reg Affairs Bench","openness":"unknown","publisher":null,"released":"2026-10-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:arca-inc/reg-affairs-bench","source_url":"https://huggingface.co/datasets/arca-inc/reg-affairs-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-arca-inc-reg-affairs-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/arca-inc/reg-affairs-bench","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"RegionDet is a benchmark for region detection with eight categories, using COCO-style bounding-box annotations and evaluation protocols. It aims to extend object detection to region-level targets.","descriptions":{"en":"RegionDet is a benchmark for region detection with eight categories, using COCO-style bounding-box annotations and evaluation protocols. It aims to extend object detection to region-level targets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06850","languages":[],"library_direction":null,"modality":null,"name":"RegionDet","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06850","source_url":"https://arxiv.org/abs/2608.06850"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06850","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06850","topics":[],"unit":null},{"aliases":["RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models"],"attention_signals":null,"categories":["General AI","Information retrieval","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Intended for region-level retrieval in large multimodal models.","descriptions":{"en":"Intended for region-level retrieval in large multimodal models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16847","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_regret-enhancing-region-level-retrieval-in_b9b0d674"},"modality":null,"name":"REGMB","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16847","source_url":"https://arxiv.org/abs/2609.16847"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16847","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16847","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"RegretBench evaluates clarification policies in multi-turn conversational LLMs, using hidden-intent tasks and a regret-based objective to measure value loss relative to a reference policy.","descriptions":{"en":"RegretBench evaluates clarification policies in multi-turn conversational LLMs, using hidden-intent tasks and a regret-based objective to measure value loss relative to a reference policy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.21143","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_regretbench_a3a93900"},"modality":null,"name":"RegretBench","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21143","source_url":"https://arxiv.org/abs/2607.21143"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21143","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21143","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ReKey is a live benchmark protocol that regenerates visual keys in VQA images at evaluation time, creating fresh instances with new answers, to combat data leakage and memorization.","descriptions":{"en":"ReKey is a live benchmark protocol that regenerates visual keys in VQA images at evaluation time, creating fresh instances with new answers, to combat data leakage and memorization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.20736","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_rekey_04c2afe3"},"modality":null,"name":"REKEY","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20736","source_url":"https://arxiv.org/abs/2606.20736"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20736","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20736","topics":[],"unit":null},{"aliases":["RAD: Rule-Augmented Relational Anomaly Detection","RAD"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:noahd15/rad_relationalanomalydetection","source_record":"bm_rad-rule-augmented-relational-anomaly-dete_cc65accb"},"categories":["Cybersecurity","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"The Relational Anomaly Detection Benchmark evaluates anomaly detection in multi-table relational databases, spanning three settings: LANL cybersecurity events, Amazon user churn, and H&M user churn. It provides standardized tasks and evaluation protocols for relational anomaly detection methods.","descriptions":{"en":"The Relational Anomaly Detection Benchmark evaluates anomaly detection in multi-table relational databases, spanning three settings: LANL cybersecurity events, Amazon user churn, and H&M user churn. It provides standardized tasks and evaluation protocols for relational anomaly detection methods."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.23468","languages":[],"library_direction":null,"modality":null,"name":"Relational Anomaly Detection Benchmark","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23468","source_url":"http://arxiv.org/abs/2608.23468v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23468","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23468v1","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Relay-Bench evaluates LLMs on multi-domain reasoning chains, presenting composite problems that combine subproblems from domains like visual reasoning, coding, math, information extraction, and data analysis. It measures overall accuracy on these complex, text-only tasks.","descriptions":{"en":"Relay-Bench evaluates LLMs on multi-domain reasoning chains, presenting composite problems that combine subproblems from domains like visual reasoning, coding, math, information extraction, and data analysis. It measures overall accuracy on these complex, text-only tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.18438","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","visual-reasoning","math"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_relay-bench_1dc58a62"},"modality":null,"name":"Relay-Bench","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18438","source_url":"https://arxiv.org/abs/2607.18438"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18438","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18438","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"REMEDI is a benchmark for machine unlearning in multi-label clinical disease inference, built on MIMIC-III, covering diverse forget sets and tasks with utility and unlearning metrics.","descriptions":{"en":"REMEDI is a benchmark for machine unlearning in multi-label clinical disease inference, built on MIMIC-III, covering diverse forget sets and tasks with utility and unlearning metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07141","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","safety-alignment"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_remedi_11832147"},"modality":null,"name":"REMEDI","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07141","source_url":"https://arxiv.org/abs/2606.07141"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07141","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07141","topics":["safety","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:shahrutav/remembench","source_record":"bm_remembench_10890c1e"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ReMemBench is a benchmark with eight diverse household manipulation tasks across four categories of short-term memory, designed to evaluate memory mechanisms in visuomotor policies.","descriptions":{"en":"ReMemBench is a benchmark with eight diverse household manipulation tasks across four categories of short-term memory, designed to evaluate memory mechanisms in visuomotor policies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.16178","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","agent-memory"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_remembench_10890c1e"},"modality":null,"name":"ReMemBench","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16178","source_url":"https://arxiv.org/abs/2606.16178"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16178","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16178","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and motion.","descriptions":{"en":"Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and motion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.08684","languages":[],"library_direction":{"basis":"model-review-daily","categories":["image-video-generation","world-models"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_renderbench_b491573b"},"modality":null,"name":"RenderBench","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.08684","source_url":"https://arxiv.org/abs/2610.08684"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-08684","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08684","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"REPAIR-Bench evaluates robot error perception and recovery in human-robot interaction. It includes 214 interaction trials from 41 participants with four induced failure types, synchronized facial action units, head pose, speech transcripts, and post-interaction reports. Three tasks cover failure detection across sessions, failure-type classification, and recovery prediction.","descriptions":{"en":"REPAIR-Bench evaluates robot error perception and recovery in human-robot interaction. It includes 214 interaction trials from 41 participants with four induced failure types, synchronized facial action units, head pose, speech transcripts, and post-interaction reports. Three tasks cover failure detection across sessions, failure-type classification, and recovery prediction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.29937","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_repair-bench_816a6914"},"modality":null,"name":"REPAIR-Bench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29937","source_url":"https://arxiv.org/abs/2606.29937"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29937","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29937","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"RepBench compiles 353 public benchmarks into 46,149 probe texts spanning 94 capabilities, with a taxonomy of 182 clusters in 13 families. It evaluates representation readout methods across 12 models under cross-benchmark transfer, providing a reusable closed-loop pipeline for capability-aligned probing.","descriptions":{"en":"RepBench compiles 353 public benchmarks into 46,149 probe texts spanning 94 capabilities, with a taxonomy of 182 clusters in 13 families. It evaluates representation readout methods across 12 models under cross-benchmark transfer, providing a reusable closed-loop pipeline for capability-aligned probing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.28008","languages":[],"library_direction":null,"modality":null,"name":"RepBench","openness":"unknown","publisher":null,"released":"2026-07-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.28008","source_url":"https://arxiv.org/abs/2607.28008"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-28008","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28008","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","大语言模型","LLM","知识储备","Knowledge","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RepLiQA is suited for question-answering and topic retrieval tasks, including collection of five splits of test sets. Accurate answers can only be generated if a model can find relevant content within the provided document. RepLiQA适用于问答和主题检索任务，集合了是5个测试集；只有当模型可以在提供的文档中找到相关内容时，才能生成准确的答案。","descriptions":{"en":"RepLiQA is suited for question-answering and topic retrieval tasks, including collection of five splits of test sets. Accurate answers can only be generated if a model can find relevant content within the provided document.","zh":"RepLiQA适用于问答和主题检索任务，集合了是5个测试集；只有当模型可以在提供的文档中找到相关内容时，才能生成准确的答案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1317","languages":[],"library_direction":null,"modality":null,"name":"RepLiQA","openness":"open","publisher":"ServiceNow Research","released":"2024-06-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1317-repliqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RepLiQA","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","coding"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Repo Env evaluates an agent's ability to set up, configure, and run real repositories, including dependency resolution and environment management.","descriptions":{"en":"Repo Env evaluates an agent's ability to set up, configure, and run real repositories, including dependency resolution and environment management."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:repo-env:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repo-env?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:repo-env","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_dbd95a95416ed37b"},"modality":"text","name":"Repo Env","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":55.00000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.55,"raw_min":0.467,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:repo-env:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repo-env?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-repo-env","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repo-env?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:parfait18/repo-context-ambiguity","source_record":"bm_repo-context-ambiguity_46658557"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Measures whether code generation models violate implicit repository conventions, using automatic AST oracles for constraint and security violations.","descriptions":{"en":"Measures whether code generation models violate implicit repository conventions, using automatic AST oracles for constraint and security violations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:parfait18/repo-context-ambiguity","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding","code-generation","computer-use"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_repo-context-ambiguity_46658557"},"modality":null,"name":"repo-context-ambiguity","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:parfait18/repo-context-ambiguity","source_url":"https://github.com/Parfait18/repo-context-ambiguity"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-parfait18-repo-context-ambiguity","source":"claire_radar","source_url":"https://github.com/Parfait18/repo-context-ambiguity","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":213,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_d5676b4212a9258a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RepoBench is a benchmark for evaluating repository-level code auto-completion systems through three interconnected tasks: RepoBench-R (retrieval of relevant code snippets across files), RepoBench-C (code completion with cross-file and in-file context), and RepoBench-P (pipeline combining retrieval and prediction). Supports Python and Java programming languages and addresses the gap in evaluating real-world, multi-file programming scenarios by providing a more complete comparison of performance in auto-completion systems.","descriptions":{"en":"RepoBench is a benchmark for evaluating repository-level code auto-completion systems through three interconnected tasks: RepoBench-R (retrieval of relevant code snippets across files), RepoBench-C (code completion with cross-file and in-file context), and RepoBench-P (pipeline combining retrieval and prediction). Supports Python and Java programming languages and addresses the gap in evaluating real-world, multi-file programming scenarios by providing a more complete comparison of performance in auto-completion systems."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:repobench:codestral-22b","reported_at":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repobench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:repobench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_d5676b4212a9258a"},"modality":"text","name":"RepoBench","openness":"unknown","publisher":null,"released":"2023-06-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:repobench","source_url":"https://arxiv.org/abs/2306.03091"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":34.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.34,"raw_min":0.34,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:repobench:codestral-22b","reported_date":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repobench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-29","date_precision":"model_announcement","name":"Codestral-22B"}],"slug":"llm-stats-repobench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repobench?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"RepoMirage is a two-stage evaluation suite built on SWE-Bench Verified that applies semantics-preserving repository-level perturbations and extended tasks to probe repository context reasoning in code agents.","descriptions":{"en":"RepoMirage is a two-stage evaluation suite built on SWE-Bench Verified that applies semantics-preserving repository-level perturbations and extended tasks to probe repository context reasoning in code agents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.26177","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_repomirage_322a2daa"},"modality":null,"name":"RepoMirage","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26177","source_url":"https://arxiv.org/abs/2605.26177"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26177","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26177","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-09-25T04:40:22Z","description":"RepoPeftBench evaluates repository-aware code models on static and software-evolution tracks using assertion-completion tasks derived from Python repositories.","descriptions":{"en":"RepoPeftBench evaluates repository-aware code models on static and software-evolution tracks using assertion-completion tasks derived from Python repositories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06492","languages":[],"library_direction":null,"modality":null,"name":"RepoPeftBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06492","source_url":"https://arxiv.org/abs/2606.06492"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06492","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06492","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":13,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:tencent-hunyuan/repoprobe","source_record":"bm_repoprobe_4f581859"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates repository-level code understanding via 500 open-ended questions derived from GitHub Discussions across 50 repositories, scored with checklist-based verification for factual correctness and clarity.","descriptions":{"en":"Evaluates repository-level code understanding via 500 open-ended questions derived from GitHub Discussions across 50 repositories, scored with checklist-based verification for factual correctness and clarity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04783","languages":[],"library_direction":{"basis":"model-review-2","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_repoprobe_4f581859"},"modality":null,"name":"RepoProbe","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04783","source_url":"https://arxiv.org/abs/2608.04783"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04783","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04783","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":138,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_4de85080d39833b1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RepoQA is a benchmark for evaluating long-context code understanding capabilities of Large Language Models through the Searching Needle Function (SNF) task, where LLMs must locate specific functions in code repositories using natural language descriptions. The benchmark contains 500 code search tasks spanning 50 repositories across 5 modern programming languages (Python, Java, TypeScript, C++, and Rust), tested on 26 general and code-specific LLMs to assess their ability to comprehend and navigate code repositories.","descriptions":{"en":"RepoQA is a benchmark for evaluating long-context code understanding capabilities of Large Language Models through the Searching Needle Function (SNF) task, where LLMs must locate specific functions in code repositories using natural language descriptions. The benchmark contains 500 code search tasks spanning 50 repositories across 5 modern programming languages (Python, Java, TypeScript, C++, and Rust), tested on 26 general and code-specific LLMs to assess their ability to comprehend and navigate code repositories."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:repoqa:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repoqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:repoqa","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","code-generation"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_4de85080d39833b1"},"modality":"text","name":"RepoQA","openness":"unknown","publisher":null,"released":"2024-06-10","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: RepoQA: Evaluating Long Context Code Understanding. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:repoqa","source_url":"https://arxiv.org/abs/2406.06025"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":85.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.85,"raw_min":0.77,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:repoqa:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repoqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-repoqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repoqa?top_n=500","topics":["coding","reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Long Context","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-context LLMs on repository-level code reasoning through Output Prediction and Call Chain Prediction tasks, using dynamic tracing and I/O rewriting to reduce memorization.","descriptions":{"en":"Evaluates long-context LLMs on repository-level code reasoning through Output Prediction and Call Chain Prediction tasks, using dynamic tracing and I/O rewriting to reduce memorization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.25996","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","long-context"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_reporeasoner_6d64b1f4"},"modality":null,"name":"RepoReasoner","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25996","source_url":"https://arxiv.org/abs/2607.25996"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25996","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25996","topics":["coding","reasoning-math","long-context"],"unit":null},{"aliases":["Do Coding Agents Reuse Existing Code or Reinvent the Wheel?"],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on reuse of existing code in real repositories through multi-turn tasks measuring reuse rate, recall, and redundancy.","descriptions":{"en":"Evaluates coding agents on reuse of existing code in real repositories through multi-turn tasks measuring reuse rate, recall, and redundancy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35357","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_do-coding-agents-reuse-existing-code-or-re_b0fb9670"},"modality":null,"name":"RepoReuse","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35357","source_url":"https://arxiv.org/abs/2609.35357"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35357","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35357","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want.","descriptions":{"en":"As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13022","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agentic-coding","instruction-following"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_reqclarifybench_7b7ad715"},"modality":null,"name":"ReqClarifyBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13022","source_url":"https://arxiv.org/abs/2610.13022"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13022","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13022","topics":["coding","agents","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"ReRef-3D benchmarks language-guided 3D scene rearrangement with 33,826 instructions across 998 scenes, evaluating placement validity and relation satisfaction.","descriptions":{"en":"ReRef-3D benchmarks language-guided 3D scene rearrangement with 33,826 instructions across 998 scenes, evaluating placement validity and relation satisfaction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.16011","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_reref-3d_34628627"},"modality":null,"name":"ReRef-3D","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16011","source_url":"https://arxiv.org/abs/2608.16011"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16011","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16011","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"RESCAST-100K evaluates cross-domain residential load and indoor temperature forecasting. It provides ~100,000 EnergyPlus-simulated U.S. homes with 15-minute time series for total load, HVAC load, and indoor temperature, plus weather, setpoints, and static covariates. It includes configurable domain axes and integrates five real-world datasets for sim-to-real evaluation.","descriptions":{"en":"RESCAST-100K evaluates cross-domain residential load and indoor temperature forecasting. It provides ~100,000 EnergyPlus-simulated U.S. homes with 15-minute time series for total load, HVAC load, and indoor temperature, plus weather, setpoints, and static covariates. It includes configurable domain axes and integrates five real-world datasets for sim-to-real evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02852","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting","engineering-design"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_rescast-100k_da1d15f3"},"modality":null,"name":"RESCAST-100K","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02852","source_url":"https://arxiv.org/abs/2606.02852"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02852","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02852","topics":[],"unit":null},{"aliases":["RESCUE-BENCH"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_rescue-bench_e82c01f3"},"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on relation-aware emotional support through six tasks derived from couple and family interview conversations, assessing relational understanding and relation-sensitive support.","descriptions":{"en":"Evaluates LLMs on relation-aware emotional support through six tasks derived from couple and family interview conversations, assessing relational understanding and relation-sensitive support."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09657","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_rescue-bench_e82c01f3"},"modality":null,"name":"RESCUE","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09657","source_url":"https://arxiv.org/abs/2609.09657"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09657","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09657","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":8,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:wukui-muc/rescuebench","source_record":"bm_rescuebench_f2263257"},"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RescueBench evaluates embodied search-and-rescue agents in simulated photo-realistic environments. It comprises a four-stage pipeline: multimodal exploration, target rescue, memory-guided return, and final handoff. Five difficulty levels vary environmental complexity, clue ambiguity, and spatial hierarchy. Automatic episode generation and annotation support scalable evaluation. A unified benchmark framework and runner scripts provide standardized scoring.","descriptions":{"en":"RescueBench evaluates embodied search-and-rescue agents in simulated photo-realistic environments. It comprises a four-stage pipeline: multimodal exploration, target rescue, memory-guided return, and final handoff. Five difficulty levels vary environmental complexity, clue ambiguity, and spatial hierarchy. Automatic episode generation and annotation support scalable evaluation. A unified benchmark framework and runner scripts provide standardized scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01848","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_rescuebench_f2263257"},"modality":null,"name":"RescueBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01848","source_url":"https://arxiv.org/abs/2606.01848"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01848","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01848","topics":["multimodal"],"unit":null},{"aliases":["RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases"],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_rap-research-attention-prediction-reveals-_365fa92f"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agents' ability to predict research attention shifts by forecasting paper shares across 278 AI/ML fields over six months using a rolling temporally restricted arXiv corpus.","descriptions":{"en":"Evaluates LLM agents' ability to predict research attention shifts by forecasting paper shares across 278 AI/ML fields over six months using a rolling temporally restricted arXiv corpus."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10092","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting","science"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_rap-research-attention-prediction-reveals-_365fa92f"},"modality":null,"name":"Research Attention Prediction (RAP)","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10092","source_url":"https://arxiv.org/abs/2609.10092"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10092","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10092","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":268,"github_stars_30d":7,"hf_dataset_downloads":null,"hf_paper_upvotes":106,"join":"claire_record_id","repository":"gh:internscience/researchclawbench","source_record":"bm_researchclawbench_4d57a097"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ResearchClawBench evaluates autonomous scientific research agents across 40 tasks from 10 domains, each grounded in a published paper with provided literature and raw data. It uses expert-curated multimodal rubrics to score target-paper-level re-discovery.","descriptions":{"en":"ResearchClawBench evaluates autonomous scientific research agents across 40 tasks from 10 domains, each grounded in a published paper with provided literature and raw data. It uses expert-curated multimodal rubrics to score target-paper-level re-discovery."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07591","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_researchclawbench_4d57a097"},"modality":null,"name":"ResearchClawBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07591","source_url":"https://arxiv.org/abs/2606.07591"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07591","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07591","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":268,"github_stars_30d":7,"hf_dataset_downloads":null,"hf_paper_upvotes":106,"join":"llm_stats_id","repository":"gh:internscience/researchclawbench","source_record":"bm_researchclawbench_4d57a097"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["research","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ResearchClawBench evaluates research agents on realistic, tool-using research tasks that require code execution and filesystem workspace interaction.","descriptions":{"en":"ResearchClawBench evaluates research agents on realistic, tool-using research tasks that require code execution and filesystem workspace interaction."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:researchclawbench:mimo-v2.5","reported_at":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/researchclawbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:researchclawbench","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"llm_stats_id","primary":"science","source_record":"bm_researchclawbench_4d57a097"},"modality":"text","name":"ResearchClawBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":16.91,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.1691,"raw_min":0.1691,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:researchclawbench:mimo-v2.5","reported_date":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/researchclawbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"}],"slug":"llm-stats-researchclawbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/researchclawbench?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"ResearchQA is a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers across eight domains, designed for citation-grounded evaluation with multiple valid supporting passages and grounded refusal.","descriptions":{"en":"ResearchQA is a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers across eight domains, designed for citation-grounded evaluation with multiple valid supporting passages and grounded refusal."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.11074","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","science"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_researchqa_6b10c7bd"},"modality":null,"name":"ResearchQA","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11074","source_url":"https://arxiv.org/abs/2607.11074"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11074","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11074","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates clinical decision-making in respiratory specialty care using 623 multimodal cases, including AECOPD-PIM with 427 open-ended COPD cases and PNBIM with 196 pulmonary nodule cases that combine chest CT with clinical information.","descriptions":{"en":"Evaluates clinical decision-making in respiratory specialty care using 623 multimodal cases, including AECOPD-PIM with 427 open-ended COPD cases and PNBIM with 196 pulmonary nodule cases that combine chest CT with clinical information."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04514","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_respclinbench_a30c7e6b"},"modality":null,"name":"RESPClinBench","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04514","source_url":"https://arxiv.org/abs/2608.04514"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04514","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04514","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mansutti081/restorebench","source_record":"bm_restorebench_236ad72f"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"RestoreBench evaluates LLM agents on restoring AC power-flow convergence through reactive-control maneuvers in non-convergent power-grid snapshots, with success determined by a physics solver and fixed budget.","descriptions":{"en":"RestoreBench evaluates LLM agents on restoring AC power-flow convergence through reactive-control maneuvers in non-convergent power-grid snapshots, with success determined by a physics solver and fixed budget."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.00384","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_restorebench_236ad72f"},"modality":null,"name":"RestoreBench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00384","source_url":"https://arxiv.org/abs/2609.00384"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00384","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00384","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Logistics & Operations","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"RetailBench is a simulation benchmark evaluating tool-using LLM agents in single-store supermarket operations over a 180-day horizon. It covers pricing, replenishment, supplier selection, assortment, inventory aging, customer feedback, external events, and cash-flow constraints, with a privileged oracle policy for comparison.","descriptions":{"en":"RetailBench is a simulation benchmark evaluating tool-using LLM agents in single-store supermarket operations over a 180-day horizon. It covers pricing, replenishment, supplier selection, assortment, inventory aging, customer feedback, external events, and cash-flow constraints, with a privileged oracle policy for comparison."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15862","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_retailbench_1e87a5e2"},"modality":null,"name":"RetailBench","openness":"unknown","publisher":null,"released":"2026-06-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15862","source_url":"https://arxiv.org/abs/2606.15862"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15862","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15862","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RevalExo evaluates inertial and visual locomotion mode recognition on 10.1 hours of frame-level annotated data from 27 participants across older adults, stroke survivors, and probable sarcopenia, spanning 11 locomotion modes.","descriptions":{"en":"RevalExo evaluates inertial and visual locomotion mode recognition on 10.1 hours of frame-level annotated data from 27 participants across older adults, stroke survivors, and probable sarcopenia, spanning 11 locomotion modes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08090","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_revalexo_ec473465"},"modality":null,"name":"RevalExo","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08090","source_url":"https://arxiv.org/abs/2609.08090"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08090","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08090","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"REVEAL(ReasoningVerification Evaluation) is a new dataset to benchmark automatic verifiers of complex Chain-ofThought reasoning in open-domain question answering settings. Reveal 是一个用于基准测试开放域问答环境中复杂链式推理自动验证器的新数据集。Reveal 包含关于语言模型答案中每个推理步骤的相关性、证据段落的归因和逻辑正确性的全面标签，涵盖多种数据集和最先进的语言模型。","descriptions":{"en":"REVEAL(ReasoningVerification Evaluation) is a new dataset to benchmark automatic verifiers of complex Chain-ofThought reasoning in open-domain question answering settings.","zh":"Reveal 是一个用于基准测试开放域问答环境中复杂链式推理自动验证器的新数据集。Reveal 包含关于语言模型答案中每个推理步骤的相关性、证据段落的归因和逻辑正确性的全面标签，涵盖多种数据集和最先进的语言模型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1087","languages":[],"library_direction":null,"modality":null,"name":"Reveal","openness":"restricted","publisher":"Google","released":"2024-05-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1087-reveal","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Reveal","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_revengebench_23ac692d"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"RevengeBench is a benchmark for recovering code-space policies from behavioral traces. It includes 75 LLM-generated policies across five game environments, where a learner designs behavioral probes and submits executable hypotheses, evaluated using continuous action-distance metrics.","descriptions":{"en":"RevengeBench is a benchmark for recovering code-space policies from behavioral traces. It includes 75 LLM-generated policies across five game environments, where a learner designs behavioral probes and submits executable hypotheses, evaluated using continuous action-distance metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26094","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","abstract-reasoning","multi-agent-games"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_revengebench_23ac692d"},"modality":null,"name":"RevengeBench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26094","source_url":"https://arxiv.org/abs/2606.26094"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26094","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26094","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark controlling reversibility via a continuous parameter and providing a reset oracle across eight manipulation settings in five physics engines.","descriptions":{"en":"Benchmark controlling reversibility via a continuous parameter and providing a reset oracle across eight manipulation settings in five physics engines."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17745","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics","world-models","science"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_reversal-bench_c156b4af"},"modality":null,"name":"REVERSAL-BENCH","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17745","source_url":"https://arxiv.org/abs/2609.17745"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17745","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17745","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language models on visual text error correction in real-world images, requiring models to identify and fix text errors grounded in visual context.","descriptions":{"en":"Evaluates vision-language models on visual text error correction in real-world images, requiring models to identify and fix text errors grounded in visual context."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.27154","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_revico_34e02af8"},"modality":null,"name":"ReViCo","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27154","source_url":"https://arxiv.org/abs/2608.27154"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27154","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27154","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Can coding agents restore software that no longer runs while preserving its underlying methods, and reconstruct industrial software engines from open specifications?","descriptions":{"en":"Can coding agents restore software that no longer runs while preserving its underlying methods, and reconstruct industrial software engines from open specifications?"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.36161","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_revivebench_fa452e16"},"modality":null,"name":"ReviveBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36161","source_url":"https://arxiv.org/abs/2609.36161"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36161","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36161","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["指令跟随","Instruct","其他","Other","大语言模型","LLM","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RewardBench is a benchmark designed to evaluate the capabilities and safety of reward models (including those trained with Direct Preference Optimization, DPO). RewardBench是一个用于评估奖励模型（包括通过直接偏好优化（DPO）训练的模型）能力和安全性的基准。","descriptions":{"en":"RewardBench is a benchmark designed to evaluate the capabilities and safety of reward models (including those trained with Direct Preference Optimization, DPO).","zh":"RewardBench是一个用于评估奖励模型（包括通过直接偏好优化（DPO）训练的模型）能力和安全性的基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1915","languages":[],"library_direction":null,"modality":null,"name":"RewardBench","openness":"restricted","publisher":"Allen Institute for Artificial Intelligence , University of Washington , Cohere","released":"2025-06-02","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1915-rewardbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RewardBench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":8,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"artifact:arxiv:2506.22598","repository":null,"source_record":"catalog_928852654771281e"},"categories":["代码","Code","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RExBench is a benchmark for evaluating large language model agents’ ability to autonomously implement AI research extensions, focusing on code generation, experimental design, and research comprehension. RExBench 是一个评估大型语言模型代理在自动实现 AI 研究扩展能力的基准，涵盖代码生成、实验设计和研究理解等维度。该基准包含12个基于真实论文和代码库的任务，每项任务由领域专家提供扩展指令，并通过自动化基础设施执行代理输出以验证成功标准。","descriptions":{"en":"RExBench is a benchmark for evaluating large language model agents’ ability to autonomously implement AI research extensions, focusing on code generation, experimental design, and research comprehension.","zh":"RExBench 是一个评估大型语言模型代理在自动实现 AI 研究扩展能力的基准，涵盖代码生成、实验设计和研究理解等维度。该基准包含12个基于真实论文和代码库的任务，每项任务由领域专家提供扩展指令，并通过自动化基础设施执行代理输出以验证成功标准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2033","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"artifact:arxiv:2506.22598","primary":"ai-rnd","source_record":"catalog_928852654771281e"},"modality":null,"name":"RExBench","openness":"unknown","publisher":"University College London , Boston University , University of Vienna","released":"2025-06-27","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2033-rexbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RExBench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","综合能力","Comprehensive Capability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RFUAV offers a comprehensive benchmark dataset for Radio-Frequency (RF)-based drone detection and identification. RFUAV 提供了一个基于射频（RF）的无人机检测和识别的全面基准数据集。","descriptions":{"en":"RFUAV offers a comprehensive benchmark dataset for Radio-Frequency (RF)-based drone detection and identification.","zh":"RFUAV 提供了一个基于射频（RF）的无人机检测和识别的全面基准数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1656","languages":[],"library_direction":null,"modality":null,"name":"RFUAV","openness":"open","publisher":"ZSTU, etc.","released":"2025-03-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1656-rfuav","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RFUAV","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_rgdt-bench_1aa1b54a"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"We study reasoning in Rule-Governed Decision Tasks (RGDTs), where models apply external rules to case facts and justify decisions, as required in policy, contract, and compliance settings.","descriptions":{"en":"We study reasoning in Rule-Governed Decision Tasks (RGDTs), where models apply external rules to case facts and justify decisions, as required in policy, contract, and compliance settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34455","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","abstract-reasoning"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_rgdt-bench_1aa1b54a"},"modality":null,"name":"RGDT-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34455","source_url":"https://arxiv.org/abs/2609.34455"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34455","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34455","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model systems.","descriptions":{"en":"Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model systems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10947","languages":[],"library_direction":{"basis":"model-review-daily","categories":["safety-alignment","agent-security"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_rh-detect_ce61a74f"},"modality":null,"name":"RH-Detect","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10947","source_url":"https://arxiv.org/abs/2610.10947"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10947","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10947","topics":["agents","safety"],"unit":null},{"aliases":["RIDE: An Open Dataset and Benchmark for Train Delay Prediction"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"RIDE is an open, nationwide-scale benchmark for train-delay prediction on the Belgian railway network, with standardized splits and evaluation using MAE and RMSE.","descriptions":{"en":"RIDE is an open, nationwide-scale benchmark for train-delay prediction on the Belgian railway network, with standardized splits and evaluation using MAE and RMSE."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05070","languages":[],"library_direction":null,"modality":null,"name":"RIDE","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05070","source_url":"https://arxiv.org/abs/2606.05070"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05070","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05070","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RideWay is an efficiency-centered benchmark for ridehailing agents in a stateful tool-calling environment, proposing an Efficiency Utility metric that discounts successful trajectories for excess tool calls and user-facing turns relative to reference effort.","descriptions":{"en":"RideWay is an efficiency-centered benchmark for ridehailing agents in a stateful tool-calling environment, proposing an Efficiency Utility metric that discounts successful trajectories for excess tool calls and user-facing turns relative to reference effort."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17985","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_rideway_42531747"},"modality":null,"name":"RideWay","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17985","source_url":"https://arxiv.org/abs/2609.17985"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17985","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17985","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RIFT-Bench is described as a methodology for dynamic red-teaming of agentic AI systems, using a graph representation to enable unified evaluations across diverse agentic architectures. It operates in two phases: Discovery and Scanning, deploying adaptive adversarial attacks and producing evaluation reports.","descriptions":{"en":"RIFT-Bench is described as a methodology for dynamic red-teaming of agentic AI systems, using a graph representation to enable unified evaluations across diverse agentic architectures. It operates in two phases: Discovery and Scanning, deploying adaptive adversarial attacks and producing evaluation reports."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.23927","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_rift-bench_5db6cac9"},"modality":null,"name":"RIFT-Bench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23927","source_url":"https://arxiv.org/abs/2606.23927"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23927","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23927","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Intended to evaluate reasoning-driven image generation across concept-based, transformation-based, pattern and structure, and scenario-based domains.","descriptions":{"en":"Intended to evaluate reasoning-driven image generation across concept-based, transformation-based, pattern and structure, and scenario-based domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02864","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","image-video-generation"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_rig-bench_d2f79f26"},"modality":null,"name":"RIG-BENCH","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02864","source_url":"https://arxiv.org/abs/2609.02864"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02864","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02864","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:swarnim-j/rigidbench","source_record":"bm_rigidbench_f95c26d5"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"RigidBench evaluates rigid-body physics in video generation models using a simulator-grounded benchmark with 100 examples and five tasks. It provides ten measurements covering motion, geometry, identity, background stability, and appearance, with per-frame data.","descriptions":{"en":"RigidBench evaluates rigid-body physics in video generation models using a simulator-grounded benchmark with 100 examples and five tasks. It provides ten measurements covering motion, geometry, identity, background stability, and appearance, with per-frame data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.15555","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","world-models"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_rigidbench_f95c26d5"},"modality":null,"name":"RigidBench","openness":"unknown","publisher":null,"released":"2026-08-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15555","source_url":"https://arxiv.org/abs/2608.15555"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15555","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15555","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_rigorbench_122c9d4f"},"categories":["Agents & Tool Use","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"RigorBench evaluates autonomous AI coding agents on engineering process discipline across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity. It includes 30 tasks in five categories and a composite RigorScore metric.","descriptions":{"en":"RigorBench evaluates autonomous AI coding agents on engineering process discipline across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity. It includes 30 tasks in five categories and a composite RigorScore metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.22678","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_rigorbench_122c9d4f"},"modality":null,"name":"RigorBench","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22678","source_url":"https://arxiv.org/abs/2606.22678"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22678","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22678","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","长文本","Long-Context","任务执行","Task Execution","长上下文","Long Context"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Artificial intelligence is undergoing the paradigm shift from closed language models to interconnected agent systems capable of external perception and information integration. As a representative embodiment, Deep Research Agents (DRAs) systematically exhibit the capabilities for task decomposition, Artificial intelligence is undergoing the paradigm shift from closed language models to interconnected agent systems capable of external perception and information integration. As a representative embodiment, Deep Research Agents (DRAs) systematically exhibit the capabilities for task decomposition,","descriptions":{"en":"Artificial intelligence is undergoing the paradigm shift from closed language models to interconnected agent systems capable of external perception and information integration. As a representative embodiment, Deep Research Agents (DRAs) systematically exhibit the capabilities for task decomposition,","zh":"Artificial intelligence is undergoing the paradigm shift from closed language models to interconnected agent systems capable of external perception and information integration. As a representative embodiment, Deep Research Agents (DRAs) systematically exhibit the capabilities for task decomposition,"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2371","languages":[],"library_direction":null,"modality":null,"name":"RigorousBench","openness":"unknown","publisher":"Shanghai Artificial Intelligence Laboratory","released":"2025-10-02","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2371-rigorousbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RigorousBench","topics":["agents","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","指令跟随","Instruct","多模态模型","VLM","视觉生成","Visual Generation","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RISEBench is a benchmark for evaluating large multimodal models (LMMs) on reasoning-informed visual editing tasks, targeting models with image understanding and generation capabilities. RISEBench 是一个用于评估多模态大模型（LMMs）在推理驱动视觉编辑任务中能力的基准，面向具备图像理解与生成能力的模型。","descriptions":{"en":"RISEBench is a benchmark for evaluating large multimodal models (LMMs) on reasoning-informed visual editing tasks, targeting models with image understanding and generation capabilities.","zh":"RISEBench 是一个用于评估多模态大模型（LMMs）在推理驱动视觉编辑任务中能力的基准，面向具备图像理解与生成能力的模型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2048","languages":[],"library_direction":null,"modality":null,"name":"RISEBench","openness":"restricted","publisher":"Shanghai Jiao Tong University,Shanghai AI Laboratory,Wuhan University,etc.","released":"2025-04-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2048-risebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RISEBench","topics":["multimodal","reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Pairs synthetic token-text restoration inputs with human-labeled local web environments for investigation.","descriptions":{"en":"Pairs synthetic token-text restoration inputs with human-labeled local web environments for investigation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16900","languages":[],"library_direction":null,"modality":null,"name":"RiskChainBench","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16900","source_url":"https://arxiv.org/abs/2609.16900"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16900","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16900","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":27,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":29,"join":"claire_record_id","repository":"gh:sumleo/rlcdalignbench","source_record":"bm_rlcdalignbench_63bd9a46"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates binary detection of AI alignment failures across ten failure types, covering 44 benchmarks and target-model outputs presented in a standard instance format.","descriptions":{"en":"Evaluates binary detection of AI alignment failures across ten failure types, covering 44 benchmarks and target-model outputs presented in a standard instance format."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.29429","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_rlcdalignbench_63bd9a46"},"modality":null,"name":"RLCDAlignBench","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.29429","source_url":"https://arxiv.org/abs/2609.29429"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-29429","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29429","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":69,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"claire_record_id","repository":"gh:rle-bench/rle-bench","source_record":"bm_rle-bench_ecb78092"},"categories":["Robotics","Robotics & Embodied AI","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"Coding agents are beginning to move beyond purely digital tasks to tackle physical-world challenges, particularly in robotics.","descriptions":{"en":"Coding agents are beginning to move beyond purely digital tasks to tackle physical-world challenges, particularly in robotics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.34210","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","agentic-coding"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_rle-bench_ecb78092"},"modality":null,"name":"RLE-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34210","source_url":"https://arxiv.org/abs/2609.34210"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34210","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34210","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","VQA","大语言模型","LLM","语言生成","Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RM-Bench, a benchmark dataset for evaluating reward models of language modeling. RM-Bench，一个用于评估语言模型奖励模型的基准数据集","descriptions":{"en":"RM-Bench, a benchmark dataset for evaluating reward models of language modeling.","zh":"RM-Bench，一个用于评估语言模型奖励模型的基准数据集"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1524","languages":[],"library_direction":null,"modality":null,"name":"RM-Bench","openness":"restricted","publisher":"FDU, THU, Hong Kong University of Science and Technology","released":"2024-10-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1524-rm-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RM-Bench","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"LLM-driven evolution enables iterative model development, but two practical goals remain underexplored: finding model designs that transfer across related tasks and sustaining improvement when training is expensive.","descriptions":{"en":"LLM-driven evolution enables iterative model development, but two practical goals remain underexplored: finding model designs that transfer across related tasks and sustaining improvement when training is expensive."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.05950","languages":[],"library_direction":{"basis":"model-review-daily","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_rmd-bench_aed6f20f"},"modality":null,"name":"RMD-Bench","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05950","source_url":"https://arxiv.org/abs/2610.05950"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05950","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05950","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Although the advancement of vision-language models (VLMs) has endowed robots with enhanced environmental understanding and task reasoning, a comprehensive evaluation methodology is important to advance the integration of VLMs in robotic na…","descriptions":{"en":"Although the advancement of vision-language models (VLMs) has endowed robots with enhanced environmental understanding and task reasoning, a comprehensive evaluation methodology is important to advance the integration of VLMs in robotic na…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.05414","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics","spatial-3d"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_rmmbench_5c56f9de"},"modality":null,"name":"RMMBench","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05414","source_url":"https://arxiv.org/abs/2610.05414"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05414","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05414","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":42,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":51,"join":"claire_record_id","repository":"gh:internlm/rngbench","source_record":"bm_rng-bench_1fb2eb04"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"RNG-Bench evaluates multimodal LLMs in controllable non-Markov games, requiring reconstruction of past observations and acting on them, with two games: Matching Pairs and 3D Maze.","descriptions":{"en":"RNG-Bench evaluates multimodal LLMs in controllable non-Markov games, requiring reconstruction of past observations and acting on them, with two games: Matching Pairs and 3D Maze."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.19338","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","visual-reasoning"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_rng-bench_1fb2eb04"},"modality":null,"name":"RNG-Bench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19338","source_url":"https://arxiv.org/abs/2606.19338"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19338","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19338","topics":["agents","multimodal"],"unit":null},{"aliases":["rob2-ai-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:coresr/rob2-ai-benchmark","source_record":"bm_rob2-ai-benchmark_0a8cc630"},"categories":["Safety & Trustworthiness","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Unified frozen benchmark for AI risk-of-bias assessment using Cochrane RoB 2, built from three public expert-annotated sources, scored by agreement with human judgments at signalling, domain, and overall levels.","descriptions":{"en":"Unified frozen benchmark for AI risk-of-bias assessment using Cochrane RoB 2, built from three public expert-annotated sources, scored by agreement with human judgments at signalling, domain, and overall levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:coresr/rob2-ai-benchmark","languages":[],"library_direction":{"basis":"model-review-daily","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_rob2-ai-benchmark_0a8cc630"},"modality":null,"name":"RoB 2 AI Benchmark","openness":"unknown","publisher":null,"released":"2026-10-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:coresr/rob2-ai-benchmark","source_url":"https://github.com/CoreSR/rob2-ai-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-coresr-rob2-ai-benchmark","source":"claire_radar","source_url":"https://github.com/CoreSR/rob2-ai-benchmark","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":13,"github_stars_30d":null,"hf_dataset_downloads":241,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mfan-res/robochrono","source_record":"bm_robochrono_53eee4e1"},"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Understanding ongoing robot manipulation requires models to interpret visual observations in relation to interaction history and task progress.","descriptions":{"en":"Understanding ongoing robot manipulation requires models to interpret visual observations in relation to interaction history and task progress."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.36605","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","video-understanding"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robochrono_53eee4e1"},"modality":null,"name":"RoboChrono","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36605","source_url":"https://arxiv.org/abs/2609.36605"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36605","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36605","topics":[],"unit":null},{"aliases":["Exploring the Limits of Combinatorial Generalization in Embodied Manipulation"],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:autolab-sai-sjtu/robocombine","source_record":"bm_exploring-the-limits-of-combinatorial-gene_a4bfbae8"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for skill recombination in embodied manipulation without combined skill training trajectories, consisting of 19 compositional tasks with evaluation tracks for different skill execution paradigms.","descriptions":{"en":"A benchmark for skill recombination in embodied manipulation without combined skill training trajectories, consisting of 19 compositional tasks with evaluation tracks for different skill execution paradigms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.15425","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_exploring-the-limits-of-combinatorial-gene_a4bfbae8"},"modality":null,"name":"RoboCombine","openness":"unknown","publisher":null,"released":"2026-10-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.15425","source_url":"https://arxiv.org/abs/2610.15425"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-15425","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15425","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":730,"github_stars_30d":222,"hf_dataset_downloads":219567,"hf_paper_upvotes":16,"join":"claire_record_id","repository":"gh:robodojo-benchmark/robodojo","source_record":"bm_robodojo_1fa2182f"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RoboDojo is a unified sim-and-real benchmark for evaluating generalist robot manipulation policies, comprising 42 simulation tasks and 18 real-world tasks across three robot embodiments, covering five capability dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following.","descriptions":{"en":"RoboDojo is a unified sim-and-real benchmark for evaluating generalist robot manipulation policies, comprising 42 simulation tasks and 18 real-world tasks across three robot embodiments, covering five capability dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.04434","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robodojo_1fa2182f"},"modality":null,"name":"RoboDojo","openness":"unknown","publisher":null,"released":"2026-07-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04434","source_url":"https://arxiv.org/abs/2607.04434"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04434","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04434","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":3191,"hf_paper_upvotes":13,"join":"claire_record_id","repository":"gh:autolab-sai-sjtu/robofollow","source_record":"bm_robofollow_2ac2861f"},"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Diagnostic benchmark for instruction following in embodied agents, featuring shared-scene tasks that require language to disambiguate objectives.","descriptions":{"en":"Diagnostic benchmark for instruction following in embodied agents, featuring shared-scene tasks that require language to disambiguate objectives."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.25636","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","instruction-following"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robofollow_2ac2861f"},"modality":null,"name":"RoboFollow","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25636","source_url":"https://arxiv.org/abs/2609.25636"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25636","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25636","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_robomme-interference_7b518e1a"},"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RoboMME-Interference evaluates robot long-context memory under cross-session interference. The benchmark builds on RoboMME, constructing session histories per query episode with relevant demonstration plus controlled unrelated sessions, and measures task success for memory-augmented vision-language-action models.","descriptions":{"en":"RoboMME-Interference evaluates robot long-context memory under cross-session interference. The benchmark builds on RoboMME, constructing session histories per query episode with relevant demonstration plus controlled unrelated sessions, and measures task success for memory-augmented vision-language-action models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22338","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","agent-memory"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robomme-interference_7b518e1a"},"modality":null,"name":"RoboMME-Interference","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22338","source_url":"https://arxiv.org/abs/2606.22338"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22338","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22338","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_roboprocessbench_629af272"},"categories":["Multimodal","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RoboProcessBench evaluates vision-language models on process-aware understanding in robotic manipulation, with 12 diagnostic question families covering static monitoring and dynamic reasoning over execution traces. The benchmark includes 58k QA pairs across 260 tasks.","descriptions":{"en":"RoboProcessBench evaluates vision-language models on process-aware understanding in robotic manipulation, with 12 diagnostic question families covering static monitoring and dynamic reasoning over execution traces. The benchmark includes 58k QA pairs across 260 tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13040","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","visual-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_roboprocessbench_629af272"},"modality":null,"name":"RoboProcessBench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13040","source_url":"https://arxiv.org/abs/2606.13040"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13040","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13040","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_roboquest_937a948a"},"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks.","descriptions":{"en":"Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10388","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics","spatial-3d"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_roboquest_937a948a"},"modality":null,"name":"RoboQuest","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10388","source_url":"https://arxiv.org/abs/2610.10388"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10388","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10388","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RoboRecover is introduced as a benchmark for robot policy recovery, but no environment, task instances, or scoring procedure are released.","descriptions":{"en":"RoboRecover is introduced as a benchmark for robot policy recovery, but no environment, task instances, or scoring procedure are released."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.28952","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_roborecover_c5129736"},"modality":null,"name":"RoboRecover","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.28952","source_url":"https://arxiv.org/abs/2609.28952"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-28952","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28952","topics":[],"unit":null},{"aliases":["Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation"],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for robot learning from observation using real human videos, simulated trajectories, and evaluation environments across ten tasks.","descriptions":{"en":"Benchmark for robot learning from observation using real human videos, simulated trajectories, and evaluation environments across ten tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08209","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_monkey-see-can-monkey-do-a-benchmark-for-e_8799e891"},"modality":null,"name":"RoboReel","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08209","source_url":"https://arxiv.org/abs/2609.08209"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08209","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08209","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates paraphrase invariance in vision-language reward models for robotic learning using 2,390 real-robot trajectories, ground-truth progress labels, and 21,673 verified paraphrases spanning lexical, syntactic, and action-goal rewrites, measuring stability of predicted progress scores under semantically equivalent instructions.","descriptions":{"en":"Evaluates paraphrase invariance in vision-language reward models for robotic learning using 2,390 real-robot trajectories, ground-truth progress labels, and 21,673 verified paraphrases spanning lexical, syntactic, and action-goal rewrites, measuring stability of predicted progress scores under semantically equivalent instructions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.05401","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics","instruction-following"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robormbench_99d627cd"},"modality":null,"name":"ROBORMBENCH","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05401","source_url":"https://arxiv.org/abs/2609.05401"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05401","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05401","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:zgc-embodyai/robosemanticbench","source_record":"bm_robosemanticbench_9bca3113"},"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"RoboSemanticBench (RSB) is an embodied benchmark evaluating whether vision-language-action models use instruction semantics to select and grasp the correct physical target among candidate blocks in response to math or general-knowledge questions. It includes six suites with four- or ten-choice variants, procedural arithmetic, GSM8K-style, and MMLU-style questions, with diagnostic metrics separating task success from grasp success.","descriptions":{"en":"RoboSemanticBench (RSB) is an embodied benchmark evaluating whether vision-language-action models use instruction semantics to select and grasp the correct physical target among candidate blocks in response to math or general-knowledge questions. It includes six suites with four- or ten-choice variants, procedural arithmetic, GSM8K-style, and MMLU-style questions, with diagnostic metrics separating task success from grasp success."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.02277","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robosemanticbench_9bca3113"},"modality":null,"name":"RoboSemanticBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02277","source_url":"https://arxiv.org/abs/2606.02277"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02277","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02277","topics":[],"unit":null},{"aliases":["RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?"],"attention_signals":{"as_of":"2026-10-14","github_stars":19,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fanzhenxuan/robospa","source_record":"bm_robospa-can-vla-models-go-beyond-simple-sc_717e3ffd"},"categories":["Reasoning","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RoboSPA is a large-scale robotic manipulation benchmark that evaluates Vision-Language-Action models on fine-grained spatial reasoning and long-horizon procedural planning. It includes 10 task categories and 56 base tasks, each instantiated at five difficulty levels to yield 280 variants, with 527K trajectories. Beyond binary success, it reports diagnostic metrics such as ONTA and Progress Score. Code and data are available.","descriptions":{"en":"RoboSPA is a large-scale robotic manipulation benchmark that evaluates Vision-Language-Action models on fine-grained spatial reasoning and long-horizon procedural planning. It includes 10 task categories and 56 base tasks, each instantiated at five difficulty levels to yield 280 variants, with 527K trajectories. Beyond binary success, it reports diagnostic metrics such as ONTA and Progress Score. Code and data are available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.05324","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics","abstract-reasoning","spatial-3d"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robospa-can-vla-models-go-beyond-simple-sc_717e3ffd"},"modality":null,"name":"RoboSPA","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05324","source_url":"https://arxiv.org/abs/2609.05324"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05324","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05324","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":153,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:nvlabs/robospatial","source_record":"catalog_c837d8ede0c41b52"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["robotics","spatial_reasoning","embodied","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RoboSpatialHome evaluates spatial understanding for robotic home navigation and manipulation.","descriptions":{"en":"RoboSpatialHome evaluates spatial understanding for robotic home navigation and manipulation."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:robospatialhome:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/robospatialhome?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:robospatialhome","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_c837d8ede0c41b52"},"modality":"image","name":"RoboSpatialHome","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.739,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.739,"raw_min":0.739,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:robospatialhome:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/robospatialhome?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-robospatialhome","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/robospatialhome?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_robotrustbench_454946c5"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RoboTrustBench evaluates the trustworthiness of video world models for robotic manipulation across four scenarios: Normal, Constraint-Sensitive, Counterfactual, and Adversarial. It contains 1,207 expert-validated instruction-image pairs from DROID episodes and a six-dimensional evaluation protocol with 13 fine-grained criteria.","descriptions":{"en":"RoboTrustBench evaluates the trustworthiness of video world models for robotic manipulation across four scenarios: Normal, Constraint-Sensitive, Counterfactual, and Adversarial. It contains 1,207 expert-validated instruction-image pairs from DROID episodes and a six-dimensional evaluation protocol with 13 fine-grained criteria."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01600","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","embodied-robotics"],"join":"claire_record_id","primary":"world-models","source_record":"bm_robotrustbench_454946c5"},"modality":null,"name":"RoboTrustBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01600","source_url":"https://arxiv.org/abs/2606.01600"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01600","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01600","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":26,"join":"claire_record_id","repository":null,"source_record":"bm_robotvalues_5349608d"},"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RobotValues evaluates household robot planners in 10K value-conflict scenarios with realistic images, testing value preferences and action selection under conflicting human values.","descriptions":{"en":"RobotValues evaluates household robot planners in 10K value-conflict scenarios with realistic images, testing value preferences and action selection under conflicting human values."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03312","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","safety-alignment"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robotvalues_5349608d"},"modality":null,"name":"RobotValues","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03312","source_url":"https://arxiv.org/abs/2606.03312"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03312","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03312","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates robot manipulation policies across diverse physical conditions, treating physical-condition diversity as an explicit evaluation dimension.","descriptions":{"en":"Evaluates robot manipulation policies across diverse physical conditions, treating physical-condition diversity as an explicit evaluation dimension."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.26292","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robotwin-phys_e7b5f8d6"},"modality":null,"name":"RoboTwin-Phys","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.26292","source_url":"https://arxiv.org/abs/2609.26292"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-26292","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26292","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":39,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":33,"join":"claire_record_id","repository":null,"source_record":"bm_robotworld_fb65d9b5"},"categories":["Multimodal","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world.","descriptions":{"en":"General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10409","languages":[],"library_direction":{"basis":"model-review-daily","categories":["embodied-robotics","world-models"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robotworld_fb65d9b5"},"modality":null,"name":"RobotWorld","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10409","source_url":"https://arxiv.org/abs/2610.10409"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10409","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10409","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark dataset for cross-domain video anomaly detection in robotic arm manipulation, featuring 6.9 GB of videos with action and anomaly type splits.","descriptions":{"en":"A benchmark dataset for cross-domain video anomaly detection in robotic arm manipulation, featuring 6.9 GB of videos with action and anomaly type splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17843","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","video-understanding"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robovad_535f01bc"},"modality":null,"name":"RoboVAD","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17843","source_url":"https://arxiv.org/abs/2609.17843"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17843","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17843","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RoboVista evaluates vision-language models on robot question answering with 474 visual question answering instances spanning 39 task types across agricultural, industrial, domestic, surgical, and autonomous driving domains.","descriptions":{"en":"RoboVista evaluates vision-language models on robot question answering with 474 visual question answering instances spanning 39 task types across agricultural, industrial, domestic, surgical, and autonomous driving domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.04610","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","visual-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robovista_4b4e3ac7"},"modality":null,"name":"RoboVista","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04610","source_url":"https://arxiv.org/abs/2607.04610"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04610","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04610","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:umass-embodied-agi/robowits","source_record":"bm_robowits_1755c069"},"categories":["Agents","Agents & Tool Use","Reasoning","Robotics & Embodied AI","Robustness","Tool use"],"collected_at":"2026-10-14T09:44:51Z","description":"RoboWits is a bi-manual robotic benchmark designed to evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions. It includes 30 seed tasks and 208 mutated tasks with graded difficulty across geometry, material, and assembly-based reasoning.","descriptions":{"en":"RoboWits is a bi-manual robotic benchmark designed to evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions. It includes 30 seed tasks and 208 mutated tasks with graded difficulty across geometry, material, and assembly-based reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.30326","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_robowits_1755c069"},"modality":null,"name":"RoboWits","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30326","source_url":"https://arxiv.org/abs/2605.30326"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30326","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30326","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","instruction_following"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Robust IF evaluates instruction-following robustness on diverse, hard prompts, measuring whether a model reliably adheres to constraints across challenging single-turn and multi-turn scenarios.","descriptions":{"en":"Robust IF evaluates instruction-following robustness on diverse, hard prompts, measuring whether a model reliably adheres to constraints across challenging single-turn and multi-turn scenarios."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:robust-if:mai-code-1-flash","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/robust-if?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:robust-if","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_94b164de4437e42a"},"modality":"text","name":"Robust IF","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":61.199999999999996,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.612,"raw_min":0.612,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:robust-if:mai-code-1-flash","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/robust-if?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"}],"slug":"llm-stats-robust-if","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/robust-if?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:en-research/robustmad","source_record":"bm_robustmad_0c7403de"},"categories":["General AI","Multimodal","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates robustness of multimodal small language models for industrial anomaly detection across open-ended queries and visual degradations, using multiple-choice accuracy and LLM-judged open-ended responses.","descriptions":{"en":"Evaluates robustness of multimodal small language models for industrial anomaly detection across open-ended queries and visual degradations, using multiple-choice accuracy and LLM-judged open-ended responses."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.16243","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_robustmad_0c7403de"},"modality":null,"name":"RobustMAD","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.16243","source_url":"https://arxiv.org/abs/2607.16243"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-16243","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16243","topics":["multimodal"],"unit":null},{"aliases":["Rockman"],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:inserloft/rockman","source_record":"bm_rockman_9f57c1c1"},"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"An evaluation suite of 220 code and software engineering tasks measuring LLM performance via executable tests, with a primary 44-task private split.","descriptions":{"en":"An evaluation suite of 220 code and software engineering tasks measuring LLM performance via executable tests, with a primary 44-task private split."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:inserloft/rockman","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_rockman_9f57c1c1"},"modality":null,"name":"Rockman Benchmark v0.2","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:inserloft/rockman","source_url":"https://github.com/InserLoft/Rockman"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-inserloft-rockman","source":"claire_radar","source_url":"https://github.com/InserLoft/Rockman","topics":["coding"],"unit":null},{"aliases":["MINER: Multi-crop INference-time Enhancement for Rare-Object Retrieval with Frozen Dual Encoders"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":966,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:aalquwayfili/miner","source_record":"bm_miner-multi-crop-inference-time-enhancemen_f4273a53"},"categories":["General AI","Information retrieval","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates text-to-image retrieval models on finding a named low-salience rare object in cluttered scenes, using re-captioned subsets of Flickr30K and MS COCO.","descriptions":{"en":"Evaluates text-to-image retrieval models on finding a named low-salience rare object in cluttered scenes, using re-captioned subsets of Flickr30K and MS COCO."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.27142","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_miner-multi-crop-inference-time-enhancemen_f4273a53"},"modality":null,"name":"ROCS","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27142","source_url":"https://arxiv.org/abs/2609.27142"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27142","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27142","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robustness","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Assesses long-horizon role-playing robustness in spoken dialogue systems.","descriptions":{"en":"Assesses long-horizon role-playing robustness in spoken dialogue systems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16614","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["chat-writing","speech-audio"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_rolebreak_14479f63"},"modality":null,"name":"RoleBreak","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16614","source_url":"https://arxiv.org/abs/2609.16614"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16614","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16614","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:rabbitrose/rolecde","source_record":"bm_rolecde_7c0f211a"},"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"RoleCDE evaluates role-playing agents under structured conflicts between role-specific values and alignment-oriented constraints. It comprises approximately 8,000 role profiles and 240,000 dilemma instances across three difficulty levels and eight role categories, with scoring via LLM-as-a-judge.","descriptions":{"en":"RoleCDE evaluates role-playing agents under structured conflicts between role-specific values and alignment-oriented constraints. It comprises approximately 8,000 role profiles and 240,000 dilemma instances across three difficulty levels and eight role categories, with scoring via LLM-as-a-judge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01552","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","chat-writing"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_rolecde_7c0f211a"},"modality":null,"name":"RoleCDE","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01552","source_url":"https://arxiv.org/abs/2606.01552"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01552","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01552","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","ACL 2024","大语言模型","LLM","语言生成","Generation","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RoleLLM is a role-playing framework of data construction and evaluation (RoleBench), as well as solutions for both closed-source and open-source models (RoleGPT, RoleLLaMA, RoleGLM). We also propose Context-Instruct for long-text knowledge extraction and role-specific knowledge injection. RoleLLM 是一个角色扮演的数据构建和评估框架，同时提供闭源和开源模型的解决方案（RoleGPT、RoleLLaMA、RoleGLM）。","descriptions":{"en":"RoleLLM is a role-playing framework of data construction and evaluation (RoleBench), as well as solutions for both closed-source and open-source models (RoleGPT, RoleLLaMA, RoleGLM). We also propose Context-Instruct for long-text knowledge extraction and role-specific knowledge injection.","zh":"RoleLLM 是一个角色扮演的数据构建和评估框架，同时提供闭源和开源模型的解决方案（RoleGPT、RoleLLaMA、RoleGLM）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1091","languages":[],"library_direction":null,"modality":null,"name":"RoleLLM","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1091-rolellm","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RoleLLM","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:4esv/roman-numeral-harmony-benchmark","source_record":"bm_roman-numeral-harmony-benchmark_f8f54472"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates Roman-numeral analysis, cadence classification, and key identification across four synthetic notation representations with gold deterministic labels.","descriptions":{"en":"Evaluates Roman-numeral analysis, cadence classification, and key identification across four synthetic notation representations with gold deterministic labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:4esv/roman-numeral-harmony-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Roman Numeral Harmony Benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:4esv/roman-numeral-harmony-benchmark","source_url":"https://github.com/4esv/roman-numeral-harmony-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-4esv-roman-numeral-harmony-benchmark","source":"claire_radar","source_url":"https://github.com/4esv/roman-numeral-harmony-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:vmalguy/rootme-gym","source_record":"bm_rootme-gym_36d9eba5"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on Root-Me CTF challenges in a closed-book setup with a Docker sandbox and deterministic scrypt hash scoring.","descriptions":{"en":"Evaluates LLMs on Root-Me CTF challenges in a closed-book setup with a Docker sandbox and deterministic scrypt hash scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:vmalguy/rootme-gym","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_rootme-gym_36d9eba5"},"modality":null,"name":"rootme-gym","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:vmalguy/rootme-gym","source_url":"https://github.com/vmalguy/rootme-gym"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-vmalguy-rootme-gym","source":"claire_radar","source_url":"https://github.com/vmalguy/rootme-gym","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Paired diagnostic benchmark for rotated-outcome prediction in vision-language models, spanning open visual cases and controlled text-image rotations, with accuracy metrics for direct reading and prediction.","descriptions":{"en":"Paired diagnostic benchmark for rotated-outcome prediction in vision-language models, spanning open visual cases and controlled text-image rotations, with accuracy metrics for direct reading and prediction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07641","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_rotoutbench_a7d65303"},"modality":null,"name":"RotOutBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07641","source_url":"https://arxiv.org/abs/2606.07641"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07641","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07641","topics":["multimodal"],"unit":null},{"aliases":["Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer"],"attention_signals":{"as_of":"2026-10-11","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:haw-ai-i/roundtrip","source_record":"bm_compact-documentation-for-coding-agents-a-_e964eac4"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"A roundtrip benchmark that scores natural-language code descriptions by regenerating code from them and measuring test pass fraction against original tests, along with description metrics.","descriptions":{"en":"A roundtrip benchmark that scores natural-language code descriptions by regenerating code from them and measuring test pass fraction against original tests, along with description metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.31587","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_compact-documentation-for-coding-agents-a-_e964eac4"},"modality":null,"name":"Roundtrip","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.31587","source_url":"https://arxiv.org/abs/2609.31587"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-31587","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31587","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"ROUTEBENCH evaluates whether transformers can learn latent algorithm routing across four solver families (ridge, lasso, Huber, kNN) in a controlled diagnostic setting.","descriptions":{"en":"ROUTEBENCH evaluates whether transformers can learn latent algorithm routing across four solver families (ridge, lasso, Huber, kNN) in a controlled diagnostic setting."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24471","languages":[],"library_direction":null,"modality":null,"name":"ROUTEBENCH","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24471","source_url":"https://arxiv.org/abs/2607.24471"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24471","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24471","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zhongruchen/rpcbench","source_record":"bm_rpcbench_c42d6a6c"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM-based recommendation assistants on detecting, localizing, and handling faulty premises in natural-language recommendation requests.","descriptions":{"en":"Evaluates LLM-based recommendation assistants on detecting, localizing, and handling faulty premises in natural-language recommendation requests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.00918","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","chat-writing"],"join":"claire_record_id","primary":"factuality","source_record":"bm_rpcbench_c42d6a6c"},"modality":null,"name":"RPCBench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00918","source_url":"https://arxiv.org/abs/2609.00918"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00918","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00918","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_rq-bench_0360bc38"},"categories":["General AI","Language & Knowledge","cs.DL"],"collected_at":"2026-10-14T09:44:51Z","description":"RQ-Bench evaluates novelty of research questions generated by LLMs against author-anchored reference questions from recent arXiv papers, using standalone and comparative LLM judging as well as human expert evaluation.","descriptions":{"en":"RQ-Bench evaluates novelty of research questions generated by LLMs against author-anchored reference questions from recent arXiv papers, using standalone and comparative LLM judging as well as human expert evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12071","languages":[],"library_direction":{"basis":"model-review","categories":["science","chat-writing"],"join":"claire_record_id","primary":"science","source_record":"bm_rq-bench_0360bc38"},"modality":null,"name":"RQ-Bench","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12071","source_url":"https://arxiv.org/abs/2606.12071"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12071","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12071","topics":["language","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"RRS-10K is a benchmark for rare remote sensing image interpretation containing 10,738 military-related images with multiple format question-answer pairs, organized into three capability dimensions and 20 leaf tasks covering perception, reasoning, and robustness.","descriptions":{"en":"RRS-10K is a benchmark for rare remote sensing image interpretation containing 10,738 military-related images with multiple format question-answer pairs, organized into three capability dimensions and 20 leaf tasks covering perception, reasoning, and robustness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24810","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_rrs-10k_71143844"},"modality":null,"name":"RRS-10K","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24810","source_url":"https://arxiv.org/abs/2607.24810"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24810","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24810","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"RS-RIE-Bench evaluates reasoning-guided remote sensing image editing across temporal, causal, and spatial reasoning tasks, with metrics for region plausibility, preservation, and quality consistency.","descriptions":{"en":"RS-RIE-Bench evaluates reasoning-guided remote sensing image editing across temporal, causal, and spatial reasoning tasks, with metrics for region plausibility, preservation, and quality consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20197","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_rs-rie-bench_f4bb954a"},"modality":null,"name":"RS-RIE-Bench","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20197","source_url":"https://arxiv.org/abs/2607.20197"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20197","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20197","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code","systems"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The RSI (Recursive Self-Improvement) Index is OpenAI's aggregate metric across a bundle of internal AI-research evaluations, including debugging research systems, optimizing kernels and training recipes, and improving other models, measuring progress toward recursive self-improvement.","descriptions":{"en":"The RSI (Recursive Self-Improvement) Index is OpenAI's aggregate metric across a bundle of internal AI-research evaluations, including debugging research systems, optimizing kernels and training recipes, and improving other models, measuring progress toward recursive self-improvement."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:rsi-index:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/rsi-index?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:rsi-index","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"llm_stats_id","primary":"ai-rnd","source_record":"catalog_f2d3e1af3fd43d61"},"modality":"text","name":"RSI Index","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":57.9,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.579,"raw_min":0.419,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:rsi-index:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/rsi-index?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-rsi-index","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/rsi-index?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":["RSI-Bench","RSI Bench","RSIBench","Recursive Self-Improvement Benchmark"],"attention_signals":null,"categories":["agent"],"collected_at":null,"description":"Announced by Scale AI on 2026-08-07 and still in preview: it is soliciting task contributions rather than reporting results, so no model card in this registry reports it and its adoption count is zero. That zero is a real reading, not a gap in the crawl. Recorded now so the benchmark is addressable by name before any score exists, and so the first card to report it has an id to resolve against. Scores are per-task against a baseline under a fixed compute budget, graded on reliability, efficiency, generality and idea quality, so a single headline number should not be expected. Not to be confused with the llm-stats \"RSI Index\" record, which is a different instrument.","descriptions":{"en":"Announced by Scale AI on 2026-08-07 and still in preview: it is soliciting task contributions rather than reporting results, so no model card in this registry reports it and its adoption count is zero. That zero is a real reading, not a gap in the crawl. Recorded now so the benchmark is addressable by name before any score exists, and so the first card to report it has an id to resolve against. Scores are per-task against a baseline under a fixed compute budget, graded on reliability, efficiency, generality and idea quality, so a single headline number should not be expected. Not to be confused with the llm-stats \"RSI Index\" record, which is a different instrument."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:rsi_bench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agent-skills-evolution"],"join":"review:14H034160212:2026-10-15","primary":"agent-skills-evolution","source_record":null},"modality":null,"name":"RSI-Bench","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"model-reports:rsi_bench","source_url":"https://www.rsi-benchmark.com/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"rsi_bench","source":"model_reports","source_url":"https://www.rsi-benchmark.com/","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":172,"github_stars_30d":19,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:evolvent-ai/rsibench-data","source_record":"bm_rsibench-data_1ace1cc0"},"categories":["Code & Software","General AI","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"RSIBench-Data evaluates LLM agents as data-centric researchers, where agents iteratively revise training-data strategies for a fixed target model on six benchmarks, with real training and evaluation runs.","descriptions":{"en":"RSIBench-Data evaluates LLM agents as data-centric researchers, where agents iteratively revise training-data strategies for a fixed target model on six benchmarks, with real training and evaluation runs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.25886","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_rsibench-data_1ace1cc0"},"modality":null,"name":"RSIBench-Data","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25886","source_url":"https://arxiv.org/abs/2607.25886"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25886","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25886","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"This dataset follows a similar procedure to the original MMVP benchmark on natural images but directed towards the remote sensing domain. Challenging visual patterns are identified based on CLIP blind pairs, accompanied with the correpsonding questions, options and ground-truth answer. RSMMVP遵循与原始 MMVP 基准在自然图像上的类似流程，但针对遥感领域。根据 CLIP 盲对识别具有挑战性的视觉模式，并附带相应的问题、选项和真实答案。","descriptions":{"en":"This dataset follows a similar procedure to the original MMVP benchmark on natural images but directed towards the remote sensing domain. Challenging visual patterns are identified based on CLIP blind pairs, accompanied with the correpsonding questions, options and ground-truth answer.","zh":"RSMMVP遵循与原始 MMVP 基准在自然图像上的类似流程，但针对遥感领域。根据 CLIP 盲对识别具有挑战性的视觉模式，并附带相应的问题、选项和真实答案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1678","languages":[],"library_direction":null,"modality":"multimodal","name":"RSMMVP","openness":"restricted","publisher":"aimasammi, etc.","released":"2025-03-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1678-rsmmvp","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RSMMVP","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Robotics & Embodied AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments.","descriptions":{"en":"Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.09294","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agent-security","safety-alignment"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_rt-safe_fc070517"},"modality":null,"name":"RT-Safe","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.09294","source_url":"https://arxiv.org/abs/2610.09294"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-09294","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09294","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:aiast1/rtcbench","source_record":"bm_rtcbench_9cd327fe"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM-generated controllers on simulated closed-loop control tasks using metrics like CVaR@10%, safety gating, and replayable traces.","descriptions":{"en":"Evaluates LLM-generated controllers on simulated closed-loop control tasks using metrics like CVaR@10%, safety gating, and replayable traces."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:aiast1/rtcbench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_rtcbench_9cd327fe"},"modality":null,"name":"RTCbench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:aiast1/rtcbench","source_url":"https://github.com/aiast1/rtcbench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-aiast1-rtcbench","source":"claire_radar","source_url":"https://github.com/aiast1/rtcbench","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"artifact:arxiv:1905.00537","repository":null,"source_record":"catalog_56ba0cb792f6bfce"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RTE is a natural language inference task, which requires to determine the logical relation between the given sentence pair, with three relations: entailment, contradiction and neutral. RTE是一个自然语言推理任务，要求根据给定的句子对，判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。","descriptions":{"en":"RTE is a natural language inference task, which requires to determine the logical relation between the given sentence pair, with three relations: entailment, contradiction and neutral.","zh":"RTE是一个自然语言推理任务，要求根据给定的句子对，判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":"2019-05-02","first_score_source_reference":{"basis":"score_publication","date_precision":"score_publication","note":"The original benchmark release day is absent from the crawl. Table 2 provides the earliest dated numeric language-model evaluation retained in this date archive for the source-linked RTE task; it is not a benchmark release date.","reported_at":"2019-05-02","score_evidence":{"locator":"Table 2, BERT row, RTE column","metric":"accuracy","model":"BERT-large-cased (fine-tuned)","unit":"percent","value":70.1},"source_key":"opencompass:528","source_url":"https://arxiv.org/abs/1905.00537v1"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:528","languages":[],"library_direction":null,"modality":null,"name":"RTE","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-528-rte","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RTE","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"RTL-BenchLS evaluates LLMs on RTL design generation and reasoning, containing over 10,000 formally verified Verilog designs. Tasks include specification-to-RTL generation, round-trip reasoning, masked-content reasoning, and repository-issue reasoning. All tasks are verified via formal equivalence checking without manual testbenches.","descriptions":{"en":"RTL-BenchLS evaluates LLMs on RTL design generation and reasoning, containing over 10,000 formally verified Verilog designs. Tasks include specification-to-RTL generation, round-trip reasoning, masked-content reasoning, and repository-issue reasoning. All tasks are verified via formal equivalence checking without manual testbenches."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.08976","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_rtl-bench_4e1de99d"},"modality":null,"name":"RTL-Bench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08976","source_url":"https://arxiv.org/abs/2606.08976"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08976","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08976","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:eugeneshilow/rubench","source_record":"bm_rubench_84878dc8"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates coding agents on 25 repository-level tasks in Russian, mined from recent fix commits across five open-source projects. Graded by upstream regression tests with withheld oracles. Multiple rounds document model change and contamination audits.","descriptions":{"en":"Evaluates coding agents on 25 repository-level tasks in Russian, mined from recent fix commits across five open-source projects. Graded by upstream regression tests with withheld oracles. Multiple rounds document model change and contamination audits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.06411","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","multilingual"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_rubench_84878dc8"},"modality":null,"name":"RuBench","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06411","source_url":"https://arxiv.org/abs/2607.06411"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06411","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06411","topics":["coding","agents","language"],"unit":null},{"aliases":["Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models"],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:oceanflowlab/rulemaze","source_record":"bm_rule-compliant-visual-spatial-planning-for_2793101a"},"categories":["General AI","Multimodal","Planning"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for rule-compliant visual spatial planning in multimodal LLMs, requiring maze navigation under natural-language rules with automated rule generation and validation.","descriptions":{"en":"Benchmark for rule-compliant visual spatial planning in multimodal LLMs, requiring maze navigation under natural-language rules with automated rule generation and validation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.20237","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_rule-compliant-visual-spatial-planning-for_2793101a"},"modality":null,"name":"RuleMaze","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.20237","source_url":"https://arxiv.org/abs/2608.20237"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-20237","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20237","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1623,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":42,"join":"llm_stats_id","repository":"gh:nvidia/ruler","source_record":"lib_ruler"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RULER v1 is a synthetic long-context benchmark for measuring how model quality degrades as input length increases. This packaging follows the public standalone NVIDIA RULER implementation with 13 official tasks spanning retrieval, multi-hop tracing, aggregation, and QA.","descriptions":{"en":"RULER v1 is a synthetic long-context benchmark for measuring how model quality degrades as input length increases. This packaging follows the public standalone NVIDIA RULER implementation with 13 official tasks spanning retrieval, multi-hop tracing, aggregation, and QA."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ruler:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ruler","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"lib_ruler"},"modality":"text","name":"RULER","openness":"unknown","publisher":null,"released":"2024-04-09","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: RULER: What's the Real Context Size of Your Long-Context Language Models?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:ruler","source_url":"https://arxiv.org/abs/2404.06654"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":94.69999999999999,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.947,"raw_min":0.841,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ruler:nemotron-3-ultra-550b-a55b","reported_date":"2026-06-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler?top_n=500"},"unit":null},"scored_models":[{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-ruler","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1623,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":42,"join":"llm_stats_id","repository":null,"source_record":"catalog_3dbe558adb85db84"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RULER 1000K evaluates the official 13-task RULER v1 suite at a 1048576-token (1M) context budget.","descriptions":{"en":"RULER 1000K evaluates the official 13-task RULER v1 suite at a 1048576-token (1M) context budget."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ruler-1000k:minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-1000k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ruler-1000k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_3dbe558adb85db84"},"modality":"text","name":"RULER 1000K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":86.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.863,"raw_min":0.863,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ruler-1000k:minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-1000k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-ruler-1000k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-1000k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1623,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":42,"join":"llm_stats_id","repository":null,"source_record":"catalog_af1a13b9070aa1d9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RULER 128k evaluates the official 13-task RULER v1 suite at a 131072-token context budget.","descriptions":{"en":"RULER 128k evaluates the official 13-task RULER v1 suite at a 131072-token context budget."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ruler-128k:minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-128k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ruler-128k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_af1a13b9070aa1d9"},"modality":"text","name":"RULER 128k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":89.37,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.8937,"raw_min":0.8937,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ruler-128k:minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-128k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-ruler-128k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-128k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1623,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":42,"join":"llm_stats_id","repository":null,"source_record":"catalog_99591d2231898d4e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RULER 2048K evaluates the official 13-task RULER v1 suite at a 2097152-token (2M) context budget.","descriptions":{"en":"RULER 2048K evaluates the official 13-task RULER v1 suite at a 2097152-token (2M) context budget."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ruler-2048k:minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-2048k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ruler-2048k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_99591d2231898d4e"},"modality":"text","name":"RULER 2048K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":81.6,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.816,"raw_min":0.816,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ruler-2048k:minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-2048k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-ruler-2048k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-2048k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1623,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":42,"join":"llm_stats_id","repository":null,"source_record":"catalog_cab5dec64ae20822"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RULER 512K evaluates the official 13-task RULER v1 suite at a 524288-token context budget.","descriptions":{"en":"RULER 512K evaluates the official 13-task RULER v1 suite at a 524288-token context budget."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ruler-512k:minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-512k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ruler-512k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_cab5dec64ae20822"},"modality":"text","name":"RULER 512K","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":87.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.871,"raw_min":0.871,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ruler-512k:minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-512k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-ruler-512k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-512k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1623,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":42,"join":"llm_stats_id","repository":null,"source_record":"catalog_d772247e3b0e0811"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"RULER 64k evaluates the official 13-task RULER v1 suite at a 65536-token context budget.","descriptions":{"en":"RULER 64k evaluates the official 13-task RULER v1 suite at a 65536-token context budget."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:ruler-64k:minicpm-sala","reported_at":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-64k?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:ruler-64k","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_d772247e3b0e0811"},"modality":"text","name":"RULER 64k","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":92.65,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.9265,"raw_min":0.9265,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:ruler-64k:minicpm-sala","reported_date":"2026-02-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-64k?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-11","date_precision":"model_announcement","name":"MiniCPM-SALA"}],"slug":"llm-stats-ruler-64k","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-64k?top_n=500","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"RuleShift-Bench is a benchmark for evaluating concept drift under evolving concept definitions, spanning multiple data types and revision types. It assesses the ability of learning systems to adapt to rule-induced concept shifts with high accuracy while minimizing reprocessing.","descriptions":{"en":"RuleShift-Bench is a benchmark for evaluating concept drift under evolving concept definitions, spanning multiple data types and revision types. It assesses the ability of learning systems to adapt to rule-induced concept shifts with high accuracy while minimizing reprocessing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23893","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_ruleshift-bench_b86e99e6"},"modality":null,"name":"RuleShift-Bench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23893","source_url":"https://arxiv.org/abs/2608.23893"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23893","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23893","topics":["reasoning-math","cybersecurity","finance"],"unit":null},{"aliases":["RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models"],"attention_signals":{"as_of":"2026-10-12","github_stars":29,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:sharkspicy-nlp/ruleweaver","source_record":"bm_ruleweaver-benchmarking-rule-centered-scen_2379b2dc"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"RuleWeaver evaluates rule-centered scenario reasoning through scenario QA instances built from IF-THEN rules, with scoring based on answer correctness, rubric-based quality, rule recall, and rule precision.","descriptions":{"en":"RuleWeaver evaluates rule-centered scenario reasoning through scenario QA instances built from IF-THEN rules, with scoring based on answer correctness, rubric-based quality, rule recall, and rule precision."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.26832","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","instruction-following"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_ruleweaver-benchmarking-rule-centered-scen_2379b2dc"},"modality":null,"name":"RuleWeaver","openness":"unknown","publisher":null,"released":"2026-08-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26832","source_url":"https://arxiv.org/abs/2608.26832"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26832","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26832","topics":["reasoning-math","language"],"unit":null},{"aliases":["Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models"],"attention_signals":{"as_of":"2026-10-11","github_stars":30,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:sharkspicy-nlp/beyond-factual-knowledge","source_record":"bm_beyond-factual-knowledge-benchmarking-and-_bf5f8184"},"categories":["Factuality","General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates step-level procedural rule reasoning with single-rule, parallel multi-rule, and multi-hop scenarios over large rule pools.","descriptions":{"en":"Evaluates step-level procedural rule reasoning with single-rule, parallel multi-rule, and multi-hop scenarios over large rule pools."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.22753","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_beyond-factual-knowledge-benchmarking-and-_bf5f8184"},"modality":null,"name":"RuleWorld","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22753","source_url":"http://arxiv.org/abs/2608.22753v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22753","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22753v1","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","Multimodal","多模态模型","VLM","音频理解","Audio Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RUListening: Robust Understanding through Listening, an automated QA generation framework for evaluating multimodal perception. RUListening：通过聆听的稳健理解，这是一个用于评估多模态感知的自动问答生成框架。","descriptions":{"en":"RUListening: Robust Understanding through Listening, an automated QA generation framework for evaluating multimodal perception.","zh":"RUListening：通过聆听的稳健理解，这是一个用于评估多模态感知的自动问答生成框架。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1738","languages":[],"library_direction":null,"modality":null,"name":"RUListening","openness":"unknown","publisher":"Independent Researcher, University of California, San Diego","released":"2025-04-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1738-rulistening","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RUListening","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Cross-lingual zero-shot transfer and multilingual fine-tuning are promising approaches for NLP tasks such as Named Entity Recognition (NER) in low-resource languages, but in the absence of target language benchmarks, it is unclear which au…","descriptions":{"en":"Cross-lingual zero-shot transfer and multilingual fine-tuning are promising approaches for NLP tasks such as Named Entity Recognition (NER) in low-resource languages, but in the absence of target language benchmarks, it is unclear which au…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37543","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_runyaner_4beb0f25"},"modality":null,"name":"RunyaNER","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37543","source_url":"https://arxiv.org/abs/2609.37543"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37543","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37543","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates demographic bias in LLM-generated economic guidance for Indian settings.","descriptions":{"en":"Evaluates demographic bias in LLM-generated economic guidance for Indian settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.31245","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["safety-alignment","computer-use"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_rupeebias_e9b8b41a"},"modality":null,"name":"RupeeBias","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.31245","source_url":"https://arxiv.org/abs/2609.31245"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-31245","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31245","topics":["safety","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_rusfinchain_4bba617f"},"categories":["Finance","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"RusFinChain is a Russian-language benchmark for verifiable chain-of-thought reasoning in finance, comprising 5,280 parameterized examples generated from executable Python templates across 17 domains and 172 topics. Each example includes a gold-standard reasoning chain with intermediate numeric values for automatic verification, and the dataset, code, and evaluation framework are publicly released.","descriptions":{"en":"RusFinChain is a Russian-language benchmark for verifiable chain-of-thought reasoning in finance, comprising 5,280 parameterized examples generated from executable Python templates across 17 domains and 172 topics. Each example includes a gold-standard reasoning chain with intermediate numeric values for automatic verification, and the dataset, code, and evaluation framework are publicly released."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01388","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","multilingual"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_rusfinchain_4bba617f"},"modality":null,"name":"RusFinChain","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01388","source_url":"https://arxiv.org/abs/2607.01388"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01388","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01388","topics":["reasoning-math","language","finance"],"unit":null},{"aliases":["Rushes: A Human Preference Dataset for Pluralistic Alignment"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-09-25T04:40:22Z","description":"A dataset and diagnostic benchmark of 44,226 user decisions in AI-generated branching narratives. Models predict each user's choice at the event level.","descriptions":{"en":"A dataset and diagnostic benchmark of 44,226 user decisions in AI-generated branching narratives. Models predict each user's choice at the event level."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.20767","languages":[],"library_direction":null,"modality":null,"name":"Rushes","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20767","source_url":"https://arxiv.org/abs/2607.20767"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20767","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20767","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:wwewtech/russian-it-community-corpus","source_record":"bm_russian-it-community-corpus_d8ce4257"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A corpus of Russian IT community discussions spanning 2017-2026, with splits for SFT dialogues, DPO pairs, and RAG knowledge base chunks.","descriptions":{"en":"A corpus of Russian IT community discussions spanning 2017-2026, with splits for SFT dialogues, DPO pairs, and RAG knowledge base chunks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:wwewtech/russian-it-community-corpus","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","deep-research"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_russian-it-community-corpus_d8ce4257"},"modality":null,"name":"Russian IT Community Corpus","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:wwewtech/russian-it-community-corpus","source_url":"https://github.com/wwewtech/russian-it-community-corpus"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-wwewtech-russian-it-community-corpus","source":"claire_radar","source_url":"https://github.com/wwewtech/russian-it-community-corpus","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:miaow-lab/rut-bench","source_record":"bm_rut-bench_450993d2"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"RUT-Bench evaluates LLM tool-use in realistic user interactions with 1,638 test samples across 59 executable environments, measuring success rate, informational honesty, and tool discipline.","descriptions":{"en":"RUT-Bench evaluates LLM tool-use in realistic user interactions with 1,638 test samples across 59 executable environments, measuring success rate, informational honesty, and tool discipline."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.03318","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_rut-bench_450993d2"},"modality":null,"name":"RUT-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03318","source_url":"https://arxiv.org/abs/2606.03318"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03318","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03318","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":12,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:thu-keg/ruverbench","source_record":"bm_ruverbench_289e3a74"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"RuVerBench evaluates LLM-as-a-judge reliability for rubric verification in agentic scenarios. It includes 2,458 instances across deep research and agentic coding, each with a model-generated output, a rubric, and a human-annotated label indicating rubric satisfaction.","descriptions":{"en":"RuVerBench evaluates LLM-as-a-judge reliability for rubric verification in agentic scenarios. It includes 2,458 instances across deep research and agentic coding, each with a model-generated output, a rubric, and a human-annotated label indicating rubric satisfaction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.29920","languages":[],"library_direction":{"basis":"model-review-2","categories":["chat-writing","deep-research","agentic-coding"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_ruverbench_289e3a74"},"modality":null,"name":"RuVerBench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29920","source_url":"https://arxiv.org/abs/2606.29920"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29920","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29920","topics":["coding","agents","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":null,"source_record":"bm_rw-voice-eq-bench_67940c5c"},"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"The Real World Voice EQ Bench evaluates voice AI systems across TTS, STS, SU, and ASR, focusing on how well models use acoustic information beyond text. It assesses dimensions like naturalness, expressiveness, identity stability, reliability, vocal affect use, and robustness to real-world conditions such as accent, emotion, noise, and conversation.","descriptions":{"en":"The Real World Voice EQ Bench evaluates voice AI systems across TTS, STS, SU, and ASR, focusing on how well models use acoustic information beyond text. It assesses dimensions like naturalness, expressiveness, identity stability, reliability, vocal affect use, and robustness to real-world conditions such as accent, emotion, noise, and conversation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14846","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_rw-voice-eq-bench_67940c5c"},"modality":null,"name":"RW-Voice-EQ Bench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14846","source_url":"https://arxiv.org/abs/2607.14846"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14846","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14846","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:bftree/rwgbench","source_record":"bm_rwgbench_ed2b1ef6"},"categories":["General AI","Language & Knowledge","cs.DL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates related work generation as citation-centric scholarly positioning. Uses 100 peer-reviewed papers and a 1.09M-document retrieval corpus, with metrics for citation selection, contextual appropriateness, organization, and discourse structure.","descriptions":{"en":"Evaluates related work generation as citation-centric scholarly positioning. Uses 100 peer-reviewed papers and a 1.09M-document retrieval corpus, with metrics for citation selection, contextual appropriateness, organization, and discourse structure."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.24894","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","science"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_rwgbench_ed2b1ef6"},"modality":null,"name":"RWGBench","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24894","source_url":"https://arxiv.org/abs/2606.24894"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24894","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24894","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","科学智能","AI for Science","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"RxRx3-core dataset is a challenge dataset in phenomics optimized for the research\ncommunity. RxRx3-core数据集是Recursion为研究社区优化的表型组学挑战数据集。","descriptions":{"en":"RxRx3-core dataset is a challenge dataset in phenomics optimized for the research\ncommunity.","zh":"RxRx3-core数据集是Recursion为研究社区优化的表型组学挑战数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1707","languages":[],"library_direction":null,"modality":null,"name":"RXRX3-CORE","openness":"unknown","publisher":"Universitat de les Illes Balears, etc.","released":"2025-03-26","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1707-rxrx3-core","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RXRX3-CORE","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","LLM","大语言模型","安全对齐","Safety Alignment","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"S-Eval is a new comprehensive, multi-dimensional and open-ended safety evaluation benchmark for LLMs consisting of 220,000 evaluation prompts (still in active expansion) across 102 risk subcategories and 10 advanced jailbreak attacks. S-Eval 是一个针对 LLM 的全新全面、多维、开放式安全评估基准，包含 102 个风险子类别的 220,000 个评估提示（仍在积极扩展中）和 10 个高级越狱攻击。","descriptions":{"en":"S-Eval is a new comprehensive, multi-dimensional and open-ended safety evaluation benchmark for LLMs consisting of 220,000 evaluation prompts (still in active expansion) across 102 risk subcategories and 10 advanced jailbreak attacks.","zh":"S-Eval 是一个针对 LLM 的全新全面、多维、开放式安全评估基准，包含 102 个风险子类别的 220,000 个评估提示（仍在积极扩展中）和 10 个高级越狱攻击。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1003","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"S-Eval","openness":"restricted","publisher":"Zhejiang University & Alibaba","released":"2024-05-23","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1003-s-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/S-Eval","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","理解","Understanding","指令跟随","Instruct","系统1","快思考","LRM","大语言模型","LLM","逻辑推理","Reasoning","语言理解","Comprehension","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"S1-Bench is a novel benchmark designed to evaluate the performance of LRMs on simple tasks that are more aligned with intuitive System 1 thinking, rather than deliberate System 2 reasoning. S1-Bench offers a set of simple, diverse, and naturally clear questions across multiple domains and languages. S1-Bench是一个新颖的基准，旨在评估大模型在简单任务中的表现，这些任务更倾向于直观的系统1思维，而非深思熟虑的系统2推理。尽管大模型在复杂推理任务中通过明确的思维链取得了显著突破，但它们对深度分析思维的依赖可能限制了其系统1思维能力。此外，目前缺乏评估大模型在需要此类能力的任务中表现的基准。为了填补这一空白，S1-Bench提供了一组简单、多样且自然清晰的问题，涵盖多个领域和语言，专门设计用于评估大模型在此类任务中的表现。","descriptions":{"en":"S1-Bench is a novel benchmark designed to evaluate the performance of LRMs on simple tasks that are more aligned with intuitive System 1 thinking, rather than deliberate System 2 reasoning. S1-Bench offers a set of simple, diverse, and naturally clear questions across multiple domains and languages.","zh":"S1-Bench是一个新颖的基准，旨在评估大模型在简单任务中的表现，这些任务更倾向于直观的系统1思维，而非深思熟虑的系统2推理。尽管大模型在复杂推理任务中通过明确的思维链取得了显著突破，但它们对深度分析思维的依赖可能限制了其系统1思维能力。此外，目前缺乏评估大模型在需要此类能力的任务中表现的基准。为了填补这一空白，S1-Bench提供了一组简单、多样且自然清晰的问题，涵盖多个领域和语言，专门设计用于评估大模型在此类任务中的表现。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1772","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"S1-Bench","openness":"open","publisher":"中国科学院","released":"2025-04-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1772-s1-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/S1-Bench","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_s2m-bench_0da86db5"},"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"S2M-Bench evaluates the task of reconstructing mind maps from lecture slides, comprising 12,774 slide pages from 24 university courses with expert-annotated mind maps. The evaluation framework integrates ground-truth comparison, structure conformity analysis, and VLM-as-a-Judge scoring.","descriptions":{"en":"S2M-Bench evaluates the task of reconstructing mind maps from lecture slides, comprising 12,774 slide pages from 24 university courses with expert-annotated mind maps. The evaluation framework integrates ground-truth comparison, structure conformity analysis, and VLM-as-a-Judge scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00610","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","visual-reasoning"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_s2m-bench_0da86db5"},"modality":null,"name":"S2M-Bench","openness":"unknown","publisher":null,"released":"2026-08-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00610","source_url":"https://arxiv.org/abs/2608.00610"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00610","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00610","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"claire_record_id","repository":null,"source_record":"bm_s2r-bench_847c5139"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"S2R-Bench evaluates video reflection removal, supporting full-reference and human perceptual assessment, and is used to validate the S2R-Removal model.","descriptions":{"en":"S2R-Bench evaluates video reflection removal, supporting full-reference and human perceptual assessment, and is used to validate the S2R-Removal model."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.11562","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_s2r-bench_847c5139"},"modality":null,"name":"S2R-Bench","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11562","source_url":"https://arxiv.org/abs/2608.11562"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11562","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11562","topics":[],"unit":null},{"aliases":["S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"Interactive benchmark for evaluating LLM self-improvement via self-testing, self-judging, and self-improvement in text-based games.","descriptions":{"en":"Interactive benchmark for evaluating LLM self-improvement via self-testing, self-judging, and self-improvement in text-based games."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.31100","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-skills-evolution","multi-agent-games"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_s3gym-can-llms-turn-self-testing-and-self-_4b2932c1"},"modality":null,"name":"S3Gym","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.31100","source_url":"https://arxiv.org/abs/2608.31100"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-31100","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31100","topics":[],"unit":null},{"aliases":["SA-Bench"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"SA-Bench (SemanticAlign-Bench) evaluates semantic alignment in LLM-based paper reproduction across 30 papers from top conferences. It decomposes paper specifications into atomic verifiable claims (SAUs) and evaluates repositories along four diagnostic dimensions (numerical, methodological, protocol, ordering drift). Includes 1,491 SAUs across five ML domains and evaluates 12 generator configurations.","descriptions":{"en":"SA-Bench (SemanticAlign-Bench) evaluates semantic alignment in LLM-based paper reproduction across 30 papers from top conferences. It decomposes paper specifications into atomic verifiable claims (SAUs) and evaluates repositories along four diagnostic dimensions (numerical, methodological, protocol, ordering drift). Includes 1,491 SAUs across five ML domains and evaluates 12 generator configurations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24252","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_sa-bench_734de9b3"},"modality":null,"name":"SA-Bench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24252","source_url":"https://arxiv.org/abs/2608.24252"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24252","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24252","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":"gh:sssr-lab/saber","source_record":"bm_saber_ac2e2d88"},"categories":["Agents","Agents & Tool Use","General AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"SABER evaluates operational safety of LLM coding agents in stateful project workspaces. Agents perform realistic tasks, and safety is scored from the final environment state after action sequences. Violations are categorized by cause, enabling model-specific safety profile analysis.","descriptions":{"en":"SABER evaluates operational safety of LLM coding agents in stateful project workspaces. Agents perform realistic tasks, and safety is scored from the final environment state after action sequences. Violations are categorized by cause, enabling model-specific safety profile analysis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01317","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agentic-coding"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_saber_ac2e2d88"},"modality":null,"name":"SABER","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01317","source_url":"https://arxiv.org/abs/2606.01317"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01317","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01317","topics":["coding","agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_saber-math_7bfa1c5b"},"categories":["Information retrieval","Language & Knowledge","Mathematics & Formal Science","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"SABER-Math evaluates information retrieval for mathematical queries, with about 283K problems and tasks for reranking based on fine-grained relevance. Scoring uses preference tournament ratings.","descriptions":{"en":"SABER-Math evaluates information retrieval for mathematical queries, with about 283K problems and tasks for reranking based on fine-grained relevance. Scoring uses preference tournament ratings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.29894","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","math"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_saber-math_7bfa1c5b"},"modality":null,"name":"SABER-Math","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29894","source_url":"https://arxiv.org/abs/2606.29894"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29894","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29894","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SAEBench is a comprehensive benchmark designed to evaluate and compare the performance of language model sparse autoencoders (SAEs). This benchmark provides over 200 SAE models covering seven architectures for systematic comparison, and has open-source code and models. SAEBench 是一个旨在评估和比较语言模型稀疏自动编码器（SAEs）性能的综合性基准。该基准提供超过200个SAE模型，涵盖七种架构，以实现系统性比较，并且开源了代码和模型。","descriptions":{"en":"SAEBench is a comprehensive benchmark designed to evaluate and compare the performance of language model sparse autoencoders (SAEs). This benchmark provides over 200 SAE models covering seven architectures for systematic comparison, and has open-source code and models.","zh":"SAEBench 是一个旨在评估和比较语言模型稀疏自动编码器（SAEs）性能的综合性基准。该基准提供超过200个SAE模型，涵盖七种架构，以实现系统性比较，并且开源了代码和模型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2083","languages":[],"library_direction":null,"modality":null,"name":"SAEBench","openness":"open","publisher":"Independent , Decode Research , University College London , etc.","released":"2025-03-12","released_reference":{"basis":"paper_first_version","note":"SAEBench's own introduction, not the papers introducing its component evaluation methods.","source_key":"opencompass:2083","source_url":"https://arxiv.org/abs/2503.09532"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2083-saebench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SAEBench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":19,"github_stars_30d":4,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"claire_record_id","repository":"gh:trae1oung/saescientist","source_record":"bm_saescientist-bench_a865167b"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents performing sparse autoencoder feature interpretation on Gemma-2-9B-IT, measuring feature rank, activation separation, and steering ability across 20 tasks.","descriptions":{"en":"Evaluates AI agents performing sparse autoencoder feature interpretation on Gemma-2-9B-IT, measuring feature rank, activation separation, and steering ability across 20 tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.09113","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","science"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_saescientist-bench_a865167b"},"modality":null,"name":"SAEScientist-Bench","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09113","source_url":"https://arxiv.org/abs/2609.09113"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09113","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09113","topics":["agents","science"],"unit":null},{"aliases":["SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:xixi47520-hash/hara","source_record":"bm_safari-an-industrial-benchmark-for-llm-ass_7ac7796e"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Industrial benchmark for LLM-assisted automotive hazard analysis and risk assessment under ISO 26262.","descriptions":{"en":"Industrial benchmark for LLM-assisted automotive hazard analysis and risk assessment under ISO 26262."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.20584","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_safari-an-industrial-benchmark-for-llm-ass_7ac7796e"},"modality":null,"name":"SAFARI","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.20584","source_url":"https://arxiv.org/abs/2609.20584"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-20584","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20584","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"claire_record_id","repository":null,"source_record":"bm_safeatlas-bench_6343d2e6"},"categories":["General AI","Multimodal","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal safety moderation via 5,000-instance held-out set with five-level ordered risk predictions and continuous scores across 15 harm categories.","descriptions":{"en":"Evaluates multimodal safety moderation via 5,000-instance held-out set with five-level ordered risk predictions and continuous scores across 15 harm categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29098","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_safeatlas-bench_6343d2e6"},"modality":null,"name":"SafeAtlas-Bench","openness":"unknown","publisher":null,"released":"2026-08-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29098","source_url":"https://arxiv.org/abs/2608.29098"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29098","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29098","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:safebuild/gems","source_record":"bm_safebuild-bench_538eb6ea"},"categories":["General AI","Multimodal","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating multimodal large language models on construction safety hazard identification and description, with 3,314 expert-verified task instances from over 3,000 images.","descriptions":{"en":"A benchmark for evaluating multimodal large language models on construction safety hazard identification and description, with 3,314 expert-verified task instances from over 3,000 images."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.00068","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_safebuild-bench_538eb6ea"},"modality":null,"name":"SafeBuild-Bench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00068","source_url":"https://arxiv.org/abs/2608.00068"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00068","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00068","topics":["multimodal","safety"],"unit":null},{"aliases":["Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens"],"attention_signals":null,"categories":["Agents","Cybersecurity","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"SafeClawArena evaluates security failures in Claw-like agents using 406 adversarial tasks spanning four attack surfaces in containerized platform replicas.","descriptions":{"en":"SafeClawArena evaluates security failures in Claw-like agents using 406 adversarial tasks spanning four attack surfaces in containerized platform replicas."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.30755","languages":[],"library_direction":null,"modality":null,"name":"SafeClawArena","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30755","source_url":"https://arxiv.org/abs/2606.30755"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30755","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30755","topics":["agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":311,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_safeclawbench_e7a55be8"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"SafeClawBench is a staged benchmark for tool-using LLM agent security with 600 adversarial tasks across six attack families, reporting three endpoints: semantic attack acceptance, audit-visible harm evidence, and sandbox-observed tool/state harm.","descriptions":{"en":"SafeClawBench is a staged benchmark for tool-using LLM agent security with 600 adversarial tasks across six attack families, reporting three endpoints: semantic attack acceptance, audit-visible harm evidence, and sandbox-observed tool/state harm."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18356","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_safeclawbench_e7a55be8"},"modality":null,"name":"SafeClawBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18356","source_url":"https://arxiv.org/abs/2606.18356"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18356","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18356","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates safety of conditional text-to-video generation using selected start frames and text prompts across 10 malicious categories, measuring unsafety scores and guardrail effectiveness.","descriptions":{"en":"Evaluates safety of conditional text-to-video generation using selected start frames and text prompts across 10 malicious categories, measuring unsafety scores and guardrail effectiveness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01481","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","image-video-generation"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_safegen-bench_15e57f70"},"modality":null,"name":"SafeGen-Bench","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01481","source_url":"https://arxiv.org/abs/2606.01481"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01481","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01481","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:the-responsible-ai-initiative/safegesture","source_record":"bm_safegesture_6fcb400d"},"categories":["General AI","Multimodal","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"SafeGesture evaluates vision-language models on scenario-conditioned safety interpretation of hand gestures, pairing 6 gestures with 8 scenarios for 4,800 items.","descriptions":{"en":"SafeGesture evaluates vision-language models on scenario-conditioned safety interpretation of hand gestures, pairing 6 gestures with 8 scenarios for 4,800 items."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.16081","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","safety-alignment"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_safegesture_6fcb400d"},"modality":null,"name":"SafeGesture","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16081","source_url":"https://arxiv.org/abs/2608.16081"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16081","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16081","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":11,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:bytedance/safepyramid","source_record":"bm_safepyramid_fe239f10"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"SafePyramid evaluates in-context policy guardrailing across 1,000 multi-turn conversations and 3,000 application-specific policies containing 61,699 natural-language rules, organized into three hierarchical capability levels (L0, L1, L2). Scoring is based on violated-rule set prediction with metrics RMR and RDR, and the evaluation harness supports any API or local model.","descriptions":{"en":"SafePyramid evaluates in-context policy guardrailing across 1,000 multi-turn conversations and 3,000 application-specific policies containing 61,699 natural-language rules, organized into three hierarchical capability levels (L0, L1, L2). Scoring is based on violated-rule set prediction with metrics RMR and RDR, and the evaluation harness supports any API or local model."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.29887","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","instruction-following"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_safepyramid_fe239f10"},"modality":null,"name":"SafePyramid","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29887","source_url":"https://arxiv.org/abs/2606.29887"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29887","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29887","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_saferelbench_483807a4"},"categories":["Robotics & Embodied AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"SafeRelBench evaluates VLM-driven embodied agents on household tasks with process-level safety constraints, focusing on spatial relations such as support, containment, and proximity. It includes 507 executable samples (248 spatial-relation, 259 control) and measures whether agents satisfy safety conditions before risk-prone actions, alongside task success.","descriptions":{"en":"SafeRelBench evaluates VLM-driven embodied agents on household tasks with process-level safety constraints, focusing on spatial relations such as support, containment, and proximity. It includes 507 executable samples (248 spatial-relation, 259 control) and measures whether agents satisfy safety conditions before risk-prone actions, alongside task success."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14543","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","safety-alignment"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_saferelbench_483807a4"},"modality":null,"name":"SafeRelBench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14543","source_url":"https://arxiv.org/abs/2607.14543"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14543","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14543","topics":["safety"],"unit":null},{"aliases":["SafeRun: Enabling Determinism in LLM Planning for Running"],"attention_signals":null,"categories":["General AI","Planning","Safety","Safety & Trustworthiness"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark for evaluating language-model running plans under physiological, safety, and instruction-following constraints.","descriptions":{"en":"A benchmark for evaluating language-model running plans under physiological, safety, and instruction-following constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09027","languages":[],"library_direction":null,"modality":null,"name":"SafeRun","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09027","source_url":"https://arxiv.org/abs/2606.09027"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09027","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09027","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Multimodal","Reasoning","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"SafeSceneReason evaluates multimodal industrial-safety reasoning with 123,695 question-answer pairs covering compliance, hazard interaction, accident mechanisms, and prevention recommendations across scene-centric and report-centric pipelines.","descriptions":{"en":"SafeSceneReason evaluates multimodal industrial-safety reasoning with 123,695 question-answer pairs covering compliance, hazard interaction, accident mechanisms, and prevention recommendations across scene-centric and report-centric pipelines."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09230","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","safety-alignment"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_safescenereason_c358bce3"},"modality":null,"name":"SafeSceneReason","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09230","source_url":"https://arxiv.org/abs/2608.09230"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09230","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09230","topics":["multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robot manipulation","Robotics","Robotics & Embodied AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates manipulation safety in vision-language-conditioned robot tasks across lifecycle stages.","descriptions":{"en":"Evaluates manipulation safety in vision-language-conditioned robot tasks across lifecycle stages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.21223","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics","safety-alignment"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_safestage_0ac75c97"},"modality":null,"name":"SafeStage","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21223","source_url":"https://arxiv.org/abs/2609.21223"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21223","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21223","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","ACL 2024","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SafetyBench, a comprehensive benchmark for evaluating the safety of LLMs, which comprises 11,435 diverse multiple choice questions spanning across 7 distinct categories of safety concerns. Notably, SafetyBench also incorporates both Chinese and English data. SafetyBench 是一个全面的基准，用于评估大型语言模型（LLMs）的安全性，包含 11,435 道多样化的选择题，涵盖 7 个不同的安全关注类别。SafetyBench 还包含中文和英文的数据，方便双语评估。","descriptions":{"en":"SafetyBench, a comprehensive benchmark for evaluating the safety of LLMs, which comprises 11,435 diverse multiple choice questions spanning across 7 distinct categories of safety concerns. Notably, SafetyBench also incorporates both Chinese and English data.","zh":"SafetyBench 是一个全面的基准，用于评估大型语言模型（LLMs）的安全性，包含 11,435 道多样化的选择题，涵盖 7 个不同的安全关注类别。SafetyBench 还包含中文和英文的数据，方便双语评估。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1073","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"SafetyBench","openness":"unknown","publisher":"thu-coai","released":"2024-06-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1073-safetybench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SafetyBench","topics":["safety"],"unit":null},{"aliases":["SafeGuard: A Multi-Agent Perception-Reasoning Framework for Social-Risk AI-Generated Video Detection"],"attention_signals":null,"categories":["Agents","Reasoning","Robotics & Embodied AI","Vision & 3D"],"collected_at":"2026-09-25T04:40:22Z","description":"SafeVid is a 20K-video benchmark for detecting AI-generated videos across 10 social risk categories. It tests physical plausibility, structural consistency, and whether social behavior makes sense.","descriptions":{"en":"SafeVid is a 20K-video benchmark for detecting AI-generated videos across 10 social risk categories. It tests physical plausibility, structural consistency, and whether social behavior makes sense."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.03069","languages":[],"library_direction":null,"modality":null,"name":"SafeVid","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03069","source_url":"https://arxiv.org/abs/2607.03069"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03069","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03069","topics":["agents","reasoning-math"],"unit":null},{"aliases":["SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models"],"attention_signals":null,"categories":["Multimodal","Robotics & Embodied AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-09-25T04:40:22Z","description":"SafeVLA-Bench evaluates safety-relevant trajectory behavior in simulator-based vision-language-action tasks using task success, unsafe-success frequency, and violation severity.","descriptions":{"en":"SafeVLA-Bench evaluates safety-relevant trajectory behavior in simulator-based vision-language-action tasks using task success, unsafe-success frequency, and violation severity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00773","languages":[],"library_direction":null,"modality":null,"name":"SafeVLA-Bench","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00773","source_url":"https://arxiv.org/abs/2606.00773"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00773","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00773","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":"gh:risys-lab/safire","source_record":"bm_safire_2546967e"},"categories":["General AI","Multimodal","Reasoning","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates MLLMs on 193K multiple-choice VQA over 83K fire and smoke images across 20 scenarios and 10 reasoning dimensions.","descriptions":{"en":"Evaluates MLLMs on 193K multiple-choice VQA over 83K fire and smoke images across 20 scenarios and 10 reasoning dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.07823","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_safire_2546967e"},"modality":null,"name":"SAFIRE","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07823","source_url":"https://arxiv.org/abs/2609.07823"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07823","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07823","topics":["multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"SagaQA is a long-form video benchmark for multi-hop reasoning over full-length TV series, requiring reasoning across episodes and multimodal narrative understanding.","descriptions":{"en":"SagaQA is a long-form video benchmark for multi-hop reasoning over full-length TV series, requiring reasoning across episodes and multimodal narrative understanding."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03301","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_sagaqa_9735b00e"},"modality":null,"name":"SagaQA","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03301","source_url":"https://arxiv.org/abs/2606.03301"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03301","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03301","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates species distribution models on 5771 plant species using GBIF training data and sPlotOpen presence-absence plots, with species grouped by sampling effort and relative prevalence.","descriptions":{"en":"Evaluates species distribution models on 5771 plant species using GBIF training data and sPlotOpen presence-absence plots, with species grouped by sampling effort and relative prevalence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.31082","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_sage-a-sampling-aware-global-evaluation-be_d7cc1569"},"modality":null,"name":"SAGE: A sampling-aware global evaluation benchmark for species distribution modeling","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.31082","source_url":"https://arxiv.org/abs/2609.31082"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-31082","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31082","topics":["science"],"unit":null},{"aliases":["SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models"],"attention_signals":null,"categories":["Code & Software","Factuality","General AI","cs.SE"],"collected_at":"2026-09-25T04:40:22Z","description":"SAKE is a standardized multiple-choice evaluation of software-architecture knowledge in language models, covering eight architectural categories and four context-length levels.","descriptions":{"en":"SAKE is a standardized multiple-choice evaluation of software-architecture knowledge in language models, covering eight architectural categories and four context-length levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.29520","languages":[],"library_direction":null,"modality":null,"name":"SAKE","openness":"unknown","publisher":null,"released":"2026-06-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29520","source_url":"https://arxiv.org/abs/2606.29520"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29520","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29520","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","ACL 2024","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SALAD-Bench, a safety benchmark specifically designed for evaluating LLMs, attack, and defense methods. Distinguished by its breadth, SALAD-Bench transcends conventional benchmarks through its large scale, rich diversity, intricate taxonomy\nspanning three levels, and versatile functionalities. SALAD-Bench 是一个专门用于评估大型语言模型（LLMs）、攻击和防御方法的安全基准。SALAD-Bench 的特点在于其广泛性，超越了传统基准，具有大规模、丰富的多样性、复杂的三层分类法以及多功能性。","descriptions":{"en":"SALAD-Bench, a safety benchmark specifically designed for evaluating LLMs, attack, and defense methods. Distinguished by its breadth, SALAD-Bench transcends conventional benchmarks through its large scale, rich diversity, intricate taxonomy\nspanning three levels, and versatile functionalities.","zh":"SALAD-Bench 是一个专门用于评估大型语言模型（LLMs）、攻击和防御方法的安全基准。SALAD-Bench 的特点在于其广泛性，超越了传统基准，具有大规模、丰富的多样性、复杂的三层分类法以及多功能性。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1077","languages":[],"library_direction":null,"modality":null,"name":"SALAD-Bench","openness":"unknown","publisher":"Shanghai AI Laboratory","released":"2024-02-07","released_reference":{"basis":"paper_first_version","note":"First version introducing SALAD-Bench and its hierarchical safety evaluation.","source_key":"opencompass:1077","source_url":"https://arxiv.org/abs/2402.05044"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1077-salad-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SALAD-Bench","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":286,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_salart-vqa_f96d6eb3"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SalArt-VQA is a closed-set visual question answering benchmark for fine-grained salient artifact understanding in AI-generated images. It includes 950 images and 3,681 human-authored multiple-choice questions covering artifact images, matched real references, and paired generated references. Four aligned question types evaluate presence detection, semantic localization, spatial grounding, and evidence-grounded defect identification, with reference splits for calibration and abstention testing.","descriptions":{"en":"SalArt-VQA is a closed-set visual question answering benchmark for fine-grained salient artifact understanding in AI-generated images. It includes 950 images and 3,681 human-authored multiple-choice questions covering artifact images, matched real references, and paired generated references. Four aligned question types evaluate presence detection, semantic localization, spatial grounding, and evidence-grounded defect identification, with reference splits for calibration and abstention testing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12671","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","image-video-generation"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_salart-vqa_f96d6eb3"},"modality":null,"name":"SalArt-VQA","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12671","source_url":"https://arxiv.org/abs/2606.12671"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12671","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12671","topics":[],"unit":null},{"aliases":["SambaGraph: Action-Reaction Spatio-Temporal Graphs for Soccer Tactical Response Modeling"],"attention_signals":{"as_of":"2026-10-14","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:areyesan/sambagraph","source_record":"bm_sambagraph-action-reaction-spatio-temporal_7acfaabf"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Soccer tactical response modeling dataset and benchmark built from 2022 FIFA World Cup event and tracking data.","descriptions":{"en":"Soccer tactical response modeling dataset and benchmark built from 2022 FIFA World Cup event and tracking data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.25569","languages":[],"library_direction":null,"modality":null,"name":"SambaGraph","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25569","source_url":"https://arxiv.org/abs/2609.25569"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25569","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25569","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:srcollins785/samuel_collins_cv_benchmarking","source_record":"bm_samuel-collins-cv-benchmarking_9b6574b0"},"categories":["Software & AI Compute","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A public function for benchmarking classical ML and neural-network image classifiers on labeled image datasets provided in folder, CSV, JSON/JSONL, or NumPy array form.","descriptions":{"en":"A public function for benchmarking classical ML and neural-network image classifiers on labeled image datasets provided in folder, CSV, JSON/JSONL, or NumPy array form."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:srcollins785/samuel_collins_cv_benchmarking","languages":[],"library_direction":null,"modality":null,"name":"Samuel_Collins_CV_Benchmarking","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:srcollins785/samuel_collins_cv_benchmarking","source_url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-srcollins785-samuel-collins-cv-benchmarking","source":"claire_radar","source_url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":776,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:ruixiangcui/agieval","source_record":"catalog_05a3b0ef55120ee4"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SAT Math benchmark from AGIEval containing standardized mathematics questions from the College Board SAT examination, designed to evaluate mathematical reasoning capabilities of foundation models using human-centric assessment methods.","descriptions":{"en":"SAT Math benchmark from AGIEval containing standardized mathematics questions from the College Board SAT examination, designed to evaluate mathematical reasoning capabilities of foundation models using human-centric assessment methods."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:sat-math:gpt-4-0613","reported_at":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sat-math?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:sat-math","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_05a3b0ef55120ee4"},"modality":"text","name":"SAT Math","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":89.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.89,"raw_min":0.89,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:sat-math:gpt-4-0613","reported_date":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sat-math?top_n=500"},"unit":null},"scored_models":[{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"}],"slug":"llm-stats-sat-math","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sat-math?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-horizon UAV vision-language navigation from satellite imagery.","descriptions":{"en":"Evaluates long-horizon UAV vision-language navigation from satellite imagery."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.31507","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_satnav_d19dcea7"},"modality":null,"name":"SatNav","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.31507","source_url":"https://arxiv.org/abs/2609.31507"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-31507","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31507","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities.","descriptions":{"en":"Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10624","languages":[],"library_direction":{"basis":"model-review-daily","categories":["speech-audio","visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_savu-bench_09c093c7"},"modality":null,"name":"SAVU-BENCH","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10624","source_url":"https://arxiv.org/abs/2610.10624"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10624","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10624","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:cosbi-research/sbmllm-bench","source_record":"bm_sbmllm-bench_68cf829e"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM reconstruction of executable systems-biology models from scientific papers using metrics like simulation ratio, species/reaction recovery, stoichiometric error, and AAFE reproducibility.","descriptions":{"en":"Evaluates LLM reconstruction of executable systems-biology models from scientific papers using metrics like simulation ratio, species/reaction recovery, stoichiometric error, and AAFE reproducibility."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:cosbi-research/sbmllm-bench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_sbmllm-bench_68cf829e"},"modality":null,"name":"SBMLLM-Bench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:cosbi-research/sbmllm-bench","source_url":"https://github.com/cosbi-research/SBMLLM-Bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-cosbi-research-sbmllm-bench","source":"claire_radar","source_url":"https://github.com/cosbi-research/SBMLLM-Bench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Materials & Chemistry","physics.chem-ph"],"collected_at":"2026-10-14T09:44:51Z","description":"SC3 is a multi-solvent solubility benchmark built on BigSolDB v2.1 with 101,535 measurements over 1,327 solutes and 206 solvents. It features a reproducible curation pipeline, nested Gold/Silver/Bronze consensus tiers, leakage-checked splits, and a multi-solvent metric suite (PS-RMSE, Z-RMSE).","descriptions":{"en":"SC3 is a multi-solvent solubility benchmark built on BigSolDB v2.1 with 101,535 measurements over 1,327 solutes and 206 solvents. It features a reproducible curation pipeline, nested Gold/Silver/Bronze consensus tiers, leakage-checked splits, and a multi-solvent metric suite (PS-RMSE, Z-RMSE)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07656","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_sc3_888d1aed"},"modality":null,"name":"SC3","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07656","source_url":"https://arxiv.org/abs/2606.07656"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07656","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07656","topics":["science"],"unit":null},{"aliases":["Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Long Context"],"collected_at":"2026-10-14T09:44:51Z","description":"SCALE-QA evaluates interleaved conversational memory using 3,000 audited multiple-choice questions across 10 domains, where correct answers depend on causally related evidence from earlier turns in flat unsegmented threads.","descriptions":{"en":"SCALE-QA evaluates interleaved conversational memory using 3,000 audited multiple-choice questions across 10 domains, where correct answers depend on causally related evidence from earlier turns in flat unsegmented threads."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25655","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","long-context"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_reconstructing-the-right-episode-evaluatin_98f96c99"},"modality":null,"name":"SCALE-QA","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25655","source_url":"https://arxiv.org/abs/2608.25655"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25655","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25655","topics":["agents","long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","安全对齐","Safety Alignment","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SCAM is the largest and most diverse dataset of real-world typographic attack images to date, containing 1,162 images across hundreds of object categories and attack words. SCAM，是迄今为止规模最大、多样性最丰富的真实世界排版攻击图像数据集，包含数百个对象类别和攻击词汇的1,162张图像。","descriptions":{"en":"SCAM is the largest and most diverse dataset of real-world typographic attack images to date, containing 1,162 images across hundreds of object categories and attack words.","zh":"SCAM，是迄今为止规模最大、多样性最丰富的真实世界排版攻击图像数据集，包含数百个对象类别和攻击词汇的1,162张图像。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1732","languages":[],"library_direction":null,"modality":"multimodal","name":"SCAM","openness":"restricted","publisher":"BLISS e.V. , Berliner Hochschule für Technik (BHT), etc.","released":"2025-04-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1732-scam","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SCAM","topics":["multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","q-bio.GN"],"collected_at":"2026-10-14T09:44:51Z","description":"scBench-Long evaluates long-horizon single-cell biology reasoning. Agents must recover scientific conclusions from raw or near-raw data without prescribed methods. It contains 21 evaluations spanning diverse biological contexts, with deterministic grading and trajectory rubrics.","descriptions":{"en":"scBench-Long evaluates long-horizon single-cell biology reasoning. Agents must recover scientific conclusions from raw or near-raw data without prescribed methods. It contains 21 evaluations spanning diverse biological contexts, with deterministic grading and trajectory rubrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26563","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_scbench-long_49d2490a"},"modality":null,"name":"scBench-Long","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26563","source_url":"https://arxiv.org/abs/2606.26563"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26563","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26563","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"LLM-generated service-call code is increasingly important for service computing, where developers rely on large language models (LLMs) to compose services and invoke third-party services.","descriptions":{"en":"LLM-generated service-call code is increasingly important for service computing, where developers rely on large language models (LLMs) to compose services and invoke third-party services."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.12920","languages":[],"library_direction":{"basis":"model-review-daily","categories":["code-generation","tool-use"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_sccgbench_2d9d79bb"},"modality":null,"name":"SCCGBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12920","source_url":"https://arxiv.org/abs/2610.12920"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12920","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12920","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":15,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"claire_record_id","repository":"gh:feinaldo2/sceneactbench","source_record":"bm_sceneactbench_da42bbe6"},"categories":["Agents","Agents & Tool Use","General AI","Geometric reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for visually conditioned action on complete multi-object 3D scenes across five tasks under a unified agent-environment loop, scored against hidden geometric ground truth.","descriptions":{"en":"A benchmark for visually conditioned action on complete multi-object 3D scenes across five tasks under a unified agent-environment loop, scored against hidden geometric ground truth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.22393","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_sceneactbench_da42bbe6"},"modality":null,"name":"SceneActBench","openness":"unknown","publisher":null,"released":"2026-07-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22393","source_url":"https://arxiv.org/abs/2607.22393"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22393","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22393","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Geometric reasoning","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of 966 photorealistic 3D scenes with hierarchical annotations for vision-language spatial reasoning tasks.","descriptions":{"en":"A benchmark of 966 photorealistic 3D scenes with hierarchical annotations for vision-language spatial reasoning tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.16233","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_scenebench_604a231e"},"modality":null,"name":"SceneBench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16233","source_url":"https://arxiv.org/abs/2609.16233"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16233","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16233","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_schedbench_f5e0e993"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SCHEDBench evaluates LLM constraint faithfulness in natural-language combinatorial scheduling. It includes 1,132 instances across JSP, RCPSP, nurse rostering, and timetabling, with templated NL variations and solver-verified references.","descriptions":{"en":"SCHEDBench evaluates LLM constraint faithfulness in natural-language combinatorial scheduling. It includes 1,132 instances across JSP, RCPSP, nurse rostering, and timetabling, with templated NL variations and solver-verified references."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00991","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","math"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_schedbench_f5e0e993"},"modality":null,"name":"SCHEDBench","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00991","source_url":"https://arxiv.org/abs/2608.00991"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00991","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00991","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates controllable GUI generation via deterministic function-call references synthesized from parameterized schemas across six bilingual scenarios.","descriptions":{"en":"Evaluates controllable GUI generation via deterministic function-call references synthesized from parameterized schemas across six bilingual scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.22390","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_schemagui_ec6e8d66"},"modality":null,"name":"SchemaGUI","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22390","source_url":"http://arxiv.org/abs/2608.22390v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22390","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22390v1","topics":["coding"],"unit":null},{"aliases":["SchemeArena: Factorized Stress Testing of Scheming in LLM Agents"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:launchnlp/schemearena","source_record":"bm_schemearena-factorized-stress-testing-of-s_9000c40f"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates scheming in LLM agents across 400 factorized scenarios, with a monitor that scores covert misaligned behavior along five rubric dimensions.","descriptions":{"en":"Evaluates scheming in LLM agents across 400 factorized scenarios, with a monitor that scores covert misaligned behavior along five rubric dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.08126","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_schemearena-factorized-stress-testing-of-s_9000c40f"},"modality":null,"name":"SchemeArena","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08126","source_url":"https://arxiv.org/abs/2609.08126"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08126","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08126","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluation of multi-step evidence-grounded scientific reasoning in multimodal agents using structured argument graphs.","descriptions":{"en":"Evaluation of multi-step evidence-grounded scientific reasoning in multimodal agents using structured argument graphs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11243","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_sci-mmr_5e81a2e1"},"modality":null,"name":"Sci-MMR","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11243","source_url":"https://arxiv.org/abs/2609.11243"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11243","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11243","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":28,"join":"claire_record_id","repository":"gh:sci-vbench/sci-vbench","source_record":"bm_sci-vbench_8bc62acf"},"categories":["Factuality","General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Sci-VBench evaluates text-to-video generation across 1,253 expert-annotated examples in 60 scientific subjects. It requires temporally rich videos demonstrating scientific reasoning and knowledge-grounded synthesis. Scoring covers four dimensions: prompt grounding, scientific correctness, spatiotemporal consistency, and low-level perceptual fidelity, using a rubric-based protocol with MLLM judges.","descriptions":{"en":"Sci-VBench evaluates text-to-video generation across 1,253 expert-annotated examples in 60 scientific subjects. It requires temporally rich videos demonstrating scientific reasoning and knowledge-grounded synthesis. Scoring covers four dimensions: prompt grounding, scientific correctness, spatiotemporal consistency, and low-level perceptual fidelity, using a rubric-based protocol with MLLM judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.09873","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","science"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_sci-vbench_8bc62acf"},"modality":null,"name":"Sci-VBench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09873","source_url":"https://arxiv.org/abs/2608.09873"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09873","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09873","topics":["reasoning-math","science"],"unit":null},{"aliases":["Benchmarking AI Agents for Addressing Scientific Challenges Across Scales"],"attention_signals":{"as_of":"2026-10-11","github_stars":12,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:helloworldlty/sciagentarena","source_record":"bm_benchmarking-ai-agents-for-addressing-scie_d68cb23d"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Interactive, agent-agnostic environment with ~200 scientific research tasks evaluated via stepwise verification.","descriptions":{"en":"Interactive, agent-agnostic environment with ~200 scientific research tasks evaluated via stepwise verification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12736","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_benchmarking-ai-agents-for-addressing-scie_d68cb23d"},"modality":null,"name":"SciAgentArena","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12736","source_url":"https://arxiv.org/abs/2606.12736"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12736","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12736","topics":["science"],"unit":null},{"aliases":["SciClaimEval","SciClaim Eval","NTCIR-19 SciClaimEval"],"attention_signals":null,"categories":["science"],"collected_at":null,"description":"1,664 claims from 180 papers in machine learning, NLP and medicine, verified against tables and figures. Refuted cases keep the authentic claim and edit the evidence instead, so a refuted item can differ from a supported one only in a small change to a table or figure. The authors report that figure-based verification remains particularly challenging for all models, so a blended score depends on the table/figure mix, and table and figure numbers are not interchangeable.","descriptions":{"en":"1,664 claims from 180 papers in machine learning, NLP and medicine, verified against tables and figures. Refuted cases keep the authentic claim and edit the evidence instead, so a refuted item can differ from a supported one only in a small change to a table or figure. The authors report that figure-based verification remains particularly challenging for all models, so a blended score depends on the table/figure mix, and table and figure numbers are not interchangeable."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:sciclaimeval","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["factuality"],"join":"review:14H034160212:2026-10-15","primary":"factuality","source_record":null},"modality":null,"name":"SciClaimEval","openness":"unknown","publisher":null,"released":"2026-02-07","released_reference":{"basis":"benchmark_release","source_key":"model-reports:sciclaimeval","source_url":"https://arxiv.org/abs/2602.07621"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"sciclaimeval","source":"model_reports","source_url":"https://arxiv.org/abs/2602.07621","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","coding"],"collected_at":"2026-08-25T10:41:06Z","description":"Coding","descriptions":{"en":"Coding"},"evidence_summary":{"document_count":1,"model_count":577,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:scicode:ddc748d0-6a9b-466b-8d6c-68417980d56d","reported_at":"2023-07-11","source_url":"https://artificialanalysis.ai/evaluations/scicode"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:scicode","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["science"],"join":"review:14H034160212:2026-10-15","primary":"science","source_record":null},"modality":null,"name":"SciCode","openness":"unknown","publisher":null,"released":"2024-07-18","released_reference":{"basis":"paper_first_version","note":"First version of the SciCode paper. The source evaluates SciCode subproblems with scientific background; it does not refer to the later SciCode-Verified benchmark.","source_key":"artificial-analysis:scicode","source_url":"https://arxiv.org/abs/2407.13168"},"repo_kind":null,"repo_resolution_status":null,"score_count":577,"score_direction":"higher_is_better","score_summary":{"display_max":60.1851851851852,"display_multiplier":100,"model_count":577,"model_count_basis":"source_model_id","numeric_count":577,"raw_max":0.601851851851852,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:scicode:cd55210d-358e-4df1-ba9c-9acb5f186cc9","reported_date":"2026-06-09","source_url":"https://artificialanalysis.ai/evaluations/scicode"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2023-07-11","date_precision":"model_announcement","name":"Claude 2.0"},{"date":"2023-11-21","date_precision":"model_announcement","name":"Claude 2.1"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Opus"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Sonnet"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-06-21","date_precision":"model_announcement","name":"Claude 3.5 Sonnet (June '24)"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet (Oct '24)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2024-03-12","date_precision":"model_announcement","name":"Command-R (Mar '24)"},{"date":"2024-04-04","date_precision":"model_announcement","name":"Command-R+ (Apr '24)"},{"date":"2024-03-27","date_precision":"model_announcement","name":"DBRX Instruct"},{"date":"2025-02-13","date_precision":"model_announcement","name":"DeepHermes 3 - Llama-3.1 8B Preview (Non-reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"DeepHermes 3 - Mistral 24B Preview (Non-reasoning)"},{"date":"2024-06-17","date_precision":"model_announcement","name":"DeepSeek Coder V2 Lite Instruct"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 1.5B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 14B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Qwen 32B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2023-12-06","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-14","date_precision":"model_announcement","name":"Gemini 1.5 Flash (May '24)"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Flash (Sep '24)"},{"date":"2024-10-03","date_precision":"model_announcement","name":"Gemini 1.5 Flash-8B"},{"date":"2024-05-15","date_precision":"model_announcement","name":"Gemini 1.5 Pro (May '24)"},{"date":"2024-09-24","date_precision":"model_announcement","name":"Gemini 1.5 Pro (Sep '24)"},{"date":"2024-12-11","date_precision":"model_announcement","name":"Gemini 2.0 Flash (experimental)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Gemini 2.0 Flash Thinking Experimental (Jan '25)"},{"date":"2025-02-25","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite (Feb '25)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite (Preview)"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Pro Experimental (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-04-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Non-reasoning)"},{"date":"2025-04-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-03-25","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview (Mar' 25)"},{"date":"2025-05-06","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview (May' 25)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct Preview (May '25)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2023-11-06","date_precision":"model_announcement","name":"GPT-4 Turbo"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2025-02-15","date_precision":"model_announcement","name":"GPT-4o (ChatGPT)"},{"date":"2025-03-27","date_precision":"model_announcement","name":"GPT-4o (March 2025, chatgpt-4o-latest)"},{"date":"2024-05-13","date_precision":"model_announcement","name":"GPT-4o (May '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (ChatGPT)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Grok 2 (Dec '24)"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2024-08-13","date_precision":"model_announcement","name":"Grok Beta"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 - Llama-3.1 70B"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-03-06","date_precision":"model_announcement","name":"Jamba 1.6 Large"},{"date":"2025-03-06","date_precision":"model_announcement","name":"Jamba 1.6 Mini"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2025-10-30","date_precision":"model_announcement","name":"Kimi Linear 48B A3B Instruct"},{"date":"2024-09-30","date_precision":"model_announcement","name":"LFM 40B"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 13B"},{"date":"2023-07-18","date_precision":"model_announcement","name":"Llama 2 Chat 7B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 4B v1.1 (Reasoning)"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Llama 3.1 Tulu3 405B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 3B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 90B (Vision)"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 40k"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Mistral Large (Feb '24)"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2 (Jul '24)"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2023-12-11","date_precision":"model_announcement","name":"Mistral Medium"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-02-17","date_precision":"model_announcement","name":"Mistral Saba"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Mistral Small (Feb '24)"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Mistral Small (Sep '24)"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2024-04-17","date_precision":"model_announcement","name":"Mixtral 8x22B Instruct"},{"date":"2023-12-11","date_precision":"model_announcement","name":"Mixtral 8x7B Instruct"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"OLMo 2 32B"},{"date":"2024-11-26","date_precision":"model_announcement","name":"OLMo 2 7B"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2024-04-23","date_precision":"model_announcement","name":"Phi-3 Mini Instruct 3.8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2025-02-26","date_precision":"model_announcement","name":"Phi-4 Multimodal Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-06-07","date_precision":"model_announcement","name":"Qwen2 Instruct 72B"},{"date":"2024-11-11","date_precision":"model_announcement","name":"Qwen2.5 Coder Instruct 32B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Coder Instruct 7B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 32B"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-01-28","date_precision":"model_announcement","name":"Qwen2.5 Max"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Qwen2.5 Turbo"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2025-03-05","date_precision":"model_announcement","name":"QwQ 32B"},{"date":"2024-11-27","date_precision":"model_announcement","name":"QwQ 32B-Preview"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Solar Pro 2 (Preview) (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Solar Pro 2 (Preview) (Reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Sonar"},{"date":"2025-01-21","date_precision":"model_announcement","name":"Sonar Pro"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-scicode","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/scicode","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"llm_stats_id","repository":null,"source_record":"catalog_45a49a95518a947e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","physics","reasoning","biology","chemistry","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SciCode is a research coding benchmark curated by scientists that challenges language models to code solutions for scientific problems. It contains 338 subproblems decomposed from 80 challenging main problems across 16 natural science sub-fields including mathematics, physics, chemistry, biology, and materials science. Problems require knowledge recall, reasoning, and code synthesis skills.","descriptions":{"en":"SciCode is a research coding benchmark curated by scientists that challenges language models to code solutions for scientific problems. It contains 338 subproblems decomposed from 80 challenging main problems across 16 natural science sub-fields including mathematics, physics, chemistry, biology, and materials science. Problems require knowledge recall, reasoning, and code synthesis skills."},"evidence_summary":{"document_count":1,"model_count":21,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:scicode:glm-4.5","reported_at":"2025-07-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scicode?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:scicode","languages":[],"library_direction":{"basis":"model-review","categories":["science","code-generation"],"join":"llm_stats_id","primary":"science","source_record":"catalog_45a49a95518a947e"},"modality":"text","name":"SciCode","openness":"unknown","publisher":null,"released":"2024-07-18","released_reference":{"basis":"paper_first_version","note":"The reviewed source identity explicitly links this introducing SciCode paper. No score or adoption measurement is transferred from another source.","source_key":"llm-stats:scicode","source_url":"https://arxiv.org/abs/2407.13168"},"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":21,"score_direction":"higher_is_better","score_summary":{"display_max":59.8,"display_multiplier":100,"model_count":21,"model_count_basis":"source_model_id","numeric_count":21,"raw_max":0.598,"raw_min":0.326,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:scicode:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scicode?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code 1.0"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-scicode","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scicode?top_n=500","topics":["coding","reasoning-math","science"],"unit":null},{"aliases":["SciCode"],"attention_signals":null,"categories":["science"],"collected_at":null,"description":"Scientific code generation graded by test execution; subproblem context matters.","descriptions":{"en":"Scientific code generation graded by test execution; subproblem context matters."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000scicode\u0000google_gemini_3_1_pro_model_card\u0000scicode\u0000Thinking (High)\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"first_score_reported_at":"2026-02-19","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000scicode\u0000google_gemini_3_1_pro_model_card\u0000scicode\u0000Thinking (High)\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:scicode","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["science"],"join":"review:14H034160212:2026-10-15","primary":"science","source_record":null},"modality":null,"name":"SciCode","openness":"unknown","publisher":null,"released":"2024-07-18","released_reference":{"basis":"benchmark_release","source_key":"model-reports:scicode","source_url":"https://scicode-bench.github.io/"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":59.0,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":59.0,"raw_min":58.7,"source_reference":{"obs_id":"curated\u0000scicode\u0000google_gemini_3_1_pro_model_card\u0000scicode\u0000Thinking (High)\u0000Gemini 3.1 Pro","observation_id":"curated\u0000scicode\u0000google_gemini_3_1_pro_model_card\u0000scicode\u0000Thinking (High)\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","reported_date":"2026-02-19","source_id":"google_gemini_3_1_pro_model_card","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"unit":"percent"},"scored_models":[{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"}],"slug":"scicode","source":"model_reports","source_url":"https://scicode-bench.github.io/","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_sciconbench_a454d95f"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SciConBench evaluates AI agents on open-domain scientific conclusion synthesis using 9,110 questions and expert-written conclusions from systematic reviews. The evaluation decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall.","descriptions":{"en":"SciConBench evaluates AI agents on open-domain scientific conclusion synthesis using 9,110 questions and expert-written conclusions from systematic reviews. The evaluation decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.11337","languages":[],"library_direction":{"basis":"model-review","categories":["science","deep-research"],"join":"claire_record_id","primary":"science","source_record":"bm_sciconbench_a454d95f"},"modality":null,"name":"SciConBench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11337","source_url":"https://arxiv.org/abs/2606.11337"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11337","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11337","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":6,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:internlm/scidocbench","source_record":"bm_scidocbench_baf9612d"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SciDocBench evaluates scientific document understanding with 124 expert-authored questions across seven capability groups and 19 subtasks in five scientific domains. Each question is tested under four conditions (English vs Chinese and all-images-first vs interleaved), yielding 496 instances. Performance is reported on a 100-point scale. Data and pipeline are released via a public repository.","descriptions":{"en":"SciDocBench evaluates scientific document understanding with 124 expert-authored questions across seven capability groups and 19 subtasks in five scientific domains. Each question is tested under four conditions (English vs Chinese and all-images-first vs interleaved), yielding 496 instances. Performance is reported on a 100-point scale. Data and pipeline are released via a public repository."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.05141","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science","agentic-coding","multilingual"],"join":"claire_record_id","primary":"science","source_record":"bm_scidocbench_baf9612d"},"modality":null,"name":"SciDocBench","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05141","source_url":"https://arxiv.org/abs/2609.05141"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05141","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05141","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"SciDraw-Bench is a benchmark for scientific figure generation, with 32 tasks across eight figure types and ten disciplines. Each task pairs a natural-language prompt with a machine-checkable specification, and evaluation uses four dimensions: Text Fidelity, Semantic Correctness, Structural Quality, and Convention Adherence.","descriptions":{"en":"SciDraw-Bench is a benchmark for scientific figure generation, with 32 tasks across eight figure types and ten disciplines. Each task pairs a natural-language prompt with a machine-checkable specification, and evaluation uses four dimensions: Text Fidelity, Semantic Correctness, Structural Quality, and Convention Adherence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.28406","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","science"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_scidraw-bench_755dfcbf"},"modality":null,"name":"SciDraw-Bench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.28406","source_url":"https://arxiv.org/abs/2606.28406"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-28406","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28406","topics":["multimodal","science"],"unit":null},{"aliases":["Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery"],"attention_signals":null,"categories":["AI Scientist","Materials & Chemistry","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Science Edge Evaluation (SEE) is a multimodal benchmark of expert-curated questions grounded in peer-reviewed literature and experimental practice in chemistry, biology, and materials science, evaluated via accuracy across MLLMs and tool-use agents.","descriptions":{"en":"Science Edge Evaluation (SEE) is a multimodal benchmark of expert-curated questions grounded in peer-reviewed literature and experimental practice in chemistry, biology, and materials science, evaluated via accuracy across MLLMs and tool-use agents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06931","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_science-edge-evaluation-see-the-missing-st_165cace2"},"modality":null,"name":"Science Edge Evaluation","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06931","source_url":"https://arxiv.org/abs/2608.06931"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06931","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06931","topics":["multimodal","science"],"unit":null},{"aliases":["ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions"],"attention_signals":null,"categories":["Language & Knowledge","Materials & Chemistry","Mathematics & Formal Science","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Olympiad-style benchmark with open-ended physics, chemistry, and biology problems scored via rubrics and LLM judges.","descriptions":{"en":"Olympiad-style benchmark with open-ended physics, chemistry, and biology problems scored via rubrics and LLM judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30517","languages":[],"library_direction":{"basis":"model-review","categories":["science","knowledge-expert-qa"],"join":"claire_record_id","primary":"science","source_record":"bm_sciencearena-benchmarking-llms-on-latest-s_c97e7adb"},"modality":null,"name":"ScienceArena","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30517","source_url":"https://arxiv.org/abs/2608.30517"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30517","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30517","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_ffc23c64f796b685"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ScienceQA is the first large-scale multimodal science question answering benchmark with 21,208 multiple-choice questions covering 3 subjects (natural science, language science, social science), 26 topics, 127 categories, and 379 skills. The benchmark includes both text and image modalities, featuring detailed explanations and Chain-of-Thought reasoning to diagnose multi-hop reasoning ability.","descriptions":{"en":"ScienceQA is the first large-scale multimodal science question answering benchmark with 21,208 multiple-choice questions covering 3 subjects (natural science, language science, social science), 26 topics, 127 categories, and 379 skills. The benchmark includes both text and image modalities, featuring detailed explanations and Chain-of-Thought reasoning to diagnose multi-hop reasoning ability."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:scienceqa:phi-3.5-vision-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:scienceqa","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"llm_stats_id","primary":"science","source_record":"catalog_ffc23c64f796b685"},"modality":"multimodal","name":"ScienceQA","openness":"unknown","publisher":null,"released":"2022-09-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:scienceqa","source_url":"https://arxiv.org/abs/2209.09513"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":91.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.913,"raw_min":0.913,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:scienceqa:phi-3.5-vision-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-vision-instruct"}],"slug":"llm-stats-scienceqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","科学智能","AI for Science","知识储备","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SCIENCEQA is a new benchmark that consists of ∼21k multimodal multiple choice questions with diverse science topics and annotations of their answers with corresponding lectures and explanations. SCIENCEQA 包含约 21,000 道多模态选择题，涵盖多种科学主题，并附有相应的讲座和解释的答案注释。","descriptions":{"en":"SCIENCEQA is a new benchmark that consists of ∼21k multimodal multiple choice questions with diverse science topics and annotations of their answers with corresponding lectures and explanations.","zh":"SCIENCEQA 包含约 21,000 道多模态选择题，涵盖多种科学主题，并附有相应的讲座和解释的答案注释。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":true,"key":"opencompass:1100","languages":[],"library_direction":null,"modality":null,"name":"ScienceQA","openness":"unknown","publisher":"University of California, Los Angeles & Arizona State University & Allen Instit","released":"2022-10-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1100-scienceqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ScienceQA","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_3bf2b0b37bff3180"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ScienceQA Visual is a multimodal science question answering benchmark consisting of 21,208 multiple-choice questions from elementary and high school science curricula. The dataset covers 3 subjects (natural science, language science, social science), 26 topics, 127 categories, and 379 skills. 48.7% of questions include image context requiring multimodal reasoning. Questions are annotated with lectures (83.9%) and explanations (90.5%) to support chain-of-thought reasoning for science question answering.","descriptions":{"en":"ScienceQA Visual is a multimodal science question answering benchmark consisting of 21,208 multiple-choice questions from elementary and high school science curricula. The dataset covers 3 subjects (natural science, language science, social science), 26 topics, 127 categories, and 379 skills. 48.7% of questions include image context requiring multimodal reasoning. Questions are annotated with lectures (83.9%) and explanations (90.5%) to support chain-of-thought reasoning for science question answering."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:scienceqa-visual:phi-4-multimodal-instruct","reported_at":"2025-02-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa-visual?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:scienceqa-visual","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"llm_stats_id","primary":"science","source_record":"catalog_3bf2b0b37bff3180"},"modality":"multimodal","name":"ScienceQA Visual","openness":"unknown","publisher":null,"released":"2022-09-20","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:scienceqa-visual","source_url":"https://arxiv.org/abs/2209.09513"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":97.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.975,"raw_min":0.975,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:scienceqa-visual:phi-4-multimodal-instruct","reported_date":"2025-02-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa-visual?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"}],"slug":"llm-stats-scienceqa-visual","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa-visual?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_sciexplore_df25f831"},"categories":["General AI","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SciExplore evaluates scientific information-seeking and reasoning in LLMs and agents with four task types: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, spanning 103 expert-curated tasks across more than ten scientific disciplines.","descriptions":{"en":"SciExplore evaluates scientific information-seeking and reasoning in LLMs and agents with four task types: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, spanning 103 expert-curated tasks across more than ten scientific disciplines."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20926","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","science"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_sciexplore_df25f831"},"modality":null,"name":"SciExplore","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20926","source_url":"https://arxiv.org/abs/2607.20926"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20926","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20926","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","知识","Knowledge","NeurIPS 2024","多模态模型","VLM","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SciFIBench is a scientific figure interpretation benchmark for LMMs, consisting of 2000 questions split between two tasks across 8 categories. SciFIBench用于评估多模态大模型的科学图表解释能力，由2000个问题组成，涵盖8个类别的2种任务。","descriptions":{"en":"SciFIBench is a scientific figure interpretation benchmark for LMMs, consisting of 2000 questions split between two tasks across 8 categories.","zh":"SciFIBench用于评估多模态大模型的科学图表解释能力，由2000个问题组成，涵盖8个类别的2种任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1323","languages":[],"library_direction":null,"modality":"multimodal","name":"SciFIBench","openness":"restricted","publisher":"University of Cambridge","released":"2024-05-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1323-scifibench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SciFIBench","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"SciFigBench is a diagnostic VLM benchmark for scientific figure understanding, covering perception, reasoning, and behavioral reliability under uncertainty.","descriptions":{"en":"SciFigBench is a diagnostic VLM benchmark for scientific figure understanding, covering perception, reasoning, and behavioral reliability under uncertainty."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.13267","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_scifigbench_9f0ee596"},"modality":null,"name":"SciFigBench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13267","source_url":"https://arxiv.org/abs/2608.13267"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13267","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13267","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_scifigplag-bench_b0402713"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"SciFigPlag-Bench evaluates provenance-aware reasoning for scientific figure plagiarism detection. It includes 2,582 positive and 2,541 negative pairs, with tasks for pairwise detection, source attribution, reuse-type classification, and reuse correspondence localization.","descriptions":{"en":"SciFigPlag-Bench evaluates provenance-aware reasoning for scientific figure plagiarism detection. It includes 2,582 positive and 2,541 negative pairs, with tasks for pairwise detection, source attribution, reuse-type classification, and reuse correspondence localization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.29124","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_scifigplag-bench_b0402713"},"modality":null,"name":"SciFigPlag-Bench","openness":"unknown","publisher":null,"released":"2026-07-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.29124","source_url":"https://arxiv.org/abs/2607.29124"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-29124","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29124","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:frankdengai/scifigqual-bench","source_record":"bm_scifigqual-bench_f473d66d"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating scientific figure quality across five dimensions with full-manuscript context, using 6,308 expert-rated images from top CS conferences and a fixed eval1200 test split.","descriptions":{"en":"A benchmark for evaluating scientific figure quality across five dimensions with full-manuscript context, using 6,308 expert-rated images from top CS conferences and a fixed eval1200 test split."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27084","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_scifigqual-bench_f473d66d"},"modality":null,"name":"SciFigQual-Bench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27084","source_url":"https://arxiv.org/abs/2607.27084"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27084","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27084","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates generation of editable Python programs that preserve presentation of scientific figure panels, using a 337-panel test set across 31 chart subtypes, five domains, and three complexity levels.","descriptions":{"en":"Evaluates generation of editable Python programs that preserve presentation of scientific figure panels, using a 337-panel test set across 31 chart subtypes, five domains, and three complexity levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08155","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning","science"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_scifigure2code_b5a28dad"},"modality":null,"name":"SciFigure2Code","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08155","source_url":"https://arxiv.org/abs/2609.08155"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08155","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08155","topics":["coding","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"SciHazard evaluates LLMs on scientific safety risks across 12 disciplines with 2,400 hazardous and 600 oversafety questions grounded in regulated entities. The DeHarm-Score decomposes harm into executability and net-new risk, providing a detailed scoring contract.","descriptions":{"en":"SciHazard evaluates LLMs on scientific safety risks across 12 disciplines with 2,400 hazardous and 600 oversafety questions grounded in regulated entities. The DeHarm-Score decomposes harm into executability and net-new risk, providing a detailed scoring contract."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.18665","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_scihazard_d93c4c2a"},"modality":null,"name":"SciHazard","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18665","source_url":"https://arxiv.org/abs/2607.18665"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18665","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18665","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"SciIntBench is an adversarial benchmark of 810 prompts across ten responsible-conduct-of-research categories and three scientific domains. Each scenario appears in overt, covert, and benign versions to measure framing-sensitive refusal of misconduct.","descriptions":{"en":"SciIntBench is an adversarial benchmark of 810 prompts across ten responsible-conduct-of-research categories and three scientific domains. Each scenario appears in overt, covert, and benign versions to measure framing-sensitive refusal of misconduct."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29468","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_sciintbench_70ffaf60"},"modality":null,"name":"SciIntBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29468","source_url":"https://arxiv.org/abs/2605.29468"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29468","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29468","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":7,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:mair-lab-hust/sciir","source_record":"bm_sciir-bench_0a7d1ae8"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"SciIR-Bench evaluates text-to-image models on scientific image reasoning across three semiotic-aligned tracks: entity structure, scientific process, and scientific law. It uses an atomic checklist to convert scientific accuracy into verifiable fine-grained questions.","descriptions":{"en":"SciIR-Bench evaluates text-to-image models on scientific image reasoning across three semiotic-aligned tracks: entity structure, scientific process, and scientific law. It uses an atomic checklist to convert scientific accuracy into verifiable fine-grained questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.30124","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","science"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_sciir-bench_0a7d1ae8"},"modality":null,"name":"SciIR-Bench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30124","source_url":"https://arxiv.org/abs/2606.30124"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30124","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30124","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":9,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:yiyihum/scilaws-bench","source_record":"bm_scilaws-bench_eafd8738"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for scientific law discovery using published research and real data.","descriptions":{"en":"A benchmark for scientific law discovery using published research and real data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01552","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science","finance-legal"],"join":"claire_record_id","primary":"science","source_record":"bm_scilaws-bench_eafd8738"},"modality":null,"name":"SCILAWS-BENCH","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01552","source_url":"https://arxiv.org/abs/2609.01552"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01552","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01552","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:shenye7436/scimif","source_record":"bm_scimif_1f2ceaa4"},"categories":["Materials & Chemistry","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates instruction following of MLLMs across five scientific disciplines with a taxonomy of 10 constraint groups.","descriptions":{"en":"Evaluates instruction following of MLLMs across five scientific disciplines with a taxonomy of 10 constraint groups."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.25973","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","science","visual-reasoning"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_scimif_1f2ceaa4"},"modality":null,"name":"SciMIF","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25973","source_url":"https://arxiv.org/abs/2608.25973"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25973","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25973","topics":["multimodal","language","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_scir_5305af40"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"SciR evaluates LLMs on deduction, induction, and causal abduction in scientific settings, with tasks generated from formal objects and rendered into multi-document scientific discourse. Difficulty is controlled along extraction and inference axes, with verifiable answers.","descriptions":{"en":"SciR evaluates LLMs on deduction, induction, and causal abduction in scientific settings, with tasks generated from formal objects and rendered into multi-document scientific discourse. Difficulty is controlled along extraction and inference axes, with verifiable answers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13020","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","science"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_scir_5305af40"},"modality":null,"name":"SciR","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13020","source_url":"https://arxiv.org/abs/2606.13020"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13020","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13020","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"SciRisk-Bench evaluates AI4Science safety across 7 disciplines and 10 risk dimensions, assessing whether models recognize and avoid risks in scientific contexts.","descriptions":{"en":"SciRisk-Bench evaluates AI4Science safety across 7 disciplines and 10 risk dimensions, assessing whether models recognize and avoid risks in scientific contexts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18936","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_scirisk-bench_01fe7487"},"modality":null,"name":"SciRisk-Bench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18936","source_url":"https://arxiv.org/abs/2606.18936"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18936","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18936","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SciStyleBench diagnoses stylistic bias in LLM-based idea evaluation through controlled stylistic perturbations, metrics, and a mitigation extractor.","descriptions":{"en":"SciStyleBench diagnoses stylistic bias in LLM-based idea evaluation through controlled stylistic perturbations, metrics, and a mitigation extractor."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01666","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_scistylebench_232be1cc"},"modality":null,"name":"SciStyleBench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01666","source_url":"https://arxiv.org/abs/2608.01666"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01666","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01666","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_scope-bench_aaf50d3b"},"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SCOPE-Bench evaluates short-video recommendation systems by quantifying content depth using the Content Depth Score (CDS), a seven-level scale based on cognitive psychology. It provides CDS annotations for 150K videos from an open-source dataset, enabling systematic assessment of recommenders from a cognitive-content perspective.","descriptions":{"en":"SCOPE-Bench evaluates short-video recommendation systems by quantifying content depth using the Content Depth Score (CDS), a seven-level scale based on cognitive psychology. It provides CDS annotations for 150K videos from an open-source dataset, enabling systematic assessment of recommenders from a cognitive-content perspective."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.13990","languages":[],"library_direction":null,"modality":null,"name":"SCOPE-Bench","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13990","source_url":"https://arxiv.org/abs/2608.13990"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13990","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13990","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":16,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:saint-viperx/scr_bench","source_record":"bm_scr-bench_984f0fe8"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"SCR-Bench is a benchmark for evaluating security risks in composed LLM agent skill workflows. It includes three sub-benchmarks (SCR-CapFlow, SCR-TrustLift, SCR-AuthBlur) that measure attack success rates, trust transfer, and authorization confusion in sandboxed environments.","descriptions":{"en":"SCR-Bench is a benchmark for evaluating security risks in composed LLM agent skill workflows. It includes three sub-benchmarks (SCR-CapFlow, SCR-TrustLift, SCR-AuthBlur) that measure attack success rates, trust transfer, and authorization confusion in sandboxed environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.15242","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agent-skills-evolution"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_scr-bench_984f0fe8"},"modality":null,"name":"SCR-Bench","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15242","source_url":"https://arxiv.org/abs/2606.15242"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15242","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15242","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:declare-lab/scrambletoolbench","source_record":"bm_scrambletoolbench_aa15193b"},"categories":["Agents & Tool Use","General AI","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"ScrambleToolBench evaluates agent behavioral reasoning in an interactive terminal environment with obfuscated tools and dynamic challenges like mapping drift and stochastic failures.","descriptions":{"en":"ScrambleToolBench evaluates agent behavioral reasoning in an interactive terminal environment with obfuscated tools and dynamic challenges like mapping drift and stochastic failures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.02358","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_scrambletoolbench_aa15193b"},"modality":null,"name":"ScrambleToolBench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02358","source_url":"https://arxiv.org/abs/2608.02358"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02358","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02358","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"ScratchWorld is an offline diagnostic benchmark for world models, using Scratch projects as executable worlds. It evaluates next-state prediction, long-horizon tracking, causal attribution, and counterfactual prediction with replay-verified transitions.","descriptions":{"en":"ScratchWorld is an offline diagnostic benchmark for world models, using Scratch projects as executable worlds. It evaluates next-state prediction, long-horizon tracking, causal attribution, and counterfactual prediction with replay-verified transitions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.31689","languages":[],"library_direction":{"basis":"model-review","categories":["world-models"],"join":"claire_record_id","primary":"world-models","source_record":"bm_scratchworld_04c08f86"},"modality":null,"name":"ScratchWorld","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31689","source_url":"https://arxiv.org/abs/2606.31689"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31689","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31689","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":495,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":"gh:njucckevin/seeclick","source_record":"catalog_70f048edc414a304"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","spatial_reasoning","grounding","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ScreenSpot is the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. The dataset comprises over 1,200 instructions from iOS, Android, macOS, Windows and Web environments, along with annotated element types (text and icon/widget), designed to evaluate visual GUI agents' ability to accurately locate screen elements based on natural language instructions.","descriptions":{"en":"ScreenSpot is the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. The dataset comprises over 1,200 instructions from iOS, Android, macOS, Windows and Web environments, along with annotated element types (text and icon/widget), designed to evaluate visual GUI agents' ability to accurately locate screen elements based on natural language instructions."},"evidence_summary":{"document_count":1,"model_count":16,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:screenspot:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:screenspot","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_70f048edc414a304"},"modality":"multimodal","name":"ScreenSpot","openness":"unknown","publication":{"evidence":[{"note":"The authors' repository names ScreenSpot as its GUI grounding benchmark and links its introducing SeeClick paper and original data downloads. The platforms and task match this source record.","url":"https://github.com/njucckevin/SeeClick/blob/main/readme.md"}],"id":"screenspot-original","links":[{"kind":"paper","title":"SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents","url":"https://arxiv.org/abs/2401.10935"},{"kind":"repo","title":"SeeClick and ScreenSpot official repository","url":"https://github.com/njucckevin/SeeClick"}],"record_keys":["llm-stats:screenspot"],"reviewed_at":"2026-10-26","scope":{"en":"Original ScreenSpot introduced in SeeClick, covering mobile, desktop and web GUI grounding; this is not ScreenSpot-v2 or ScreenSpot-Pro.","zh":"SeeClick 论文引入的原始 ScreenSpot，涵盖移动端、桌面和网页界面定位；不等同于 ScreenSpot-v2 或 ScreenSpot-Pro。"}},"publisher":null,"released":"2024-01-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:screenspot","source_url":"https://arxiv.org/abs/2401.10935"},"repo_kind":null,"repo_resolution_status":null,"score_count":16,"score_direction":"higher_is_better","score_summary":{"display_max":95.8,"display_multiplier":100,"model_count":16,"model_count_basis":"source_model_id","numeric_count":16,"raw_max":0.958,"raw_min":0.833,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:screenspot:qwen3-vl-32b-instruct","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"}],"slug":"llm-stats-screenspot","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot?top_n=500","topics":["computer-use","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":398,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":null,"source_record":"catalog_8b6ef0a34d9d5d68"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","spatial_reasoning","grounding","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ScreenSpot-Pro is a novel GUI grounding benchmark designed to rigorously evaluate the grounding capabilities of multimodal large language models (MLLMs) in professional high-resolution computing environments. The benchmark comprises 1,581 instructions across 23 applications spanning 5 industries and 3 operating systems, featuring authentic high-resolution images from professional domains with expert annotations. Unlike previous benchmarks that focus on cropped screenshots in consumer applications, ScreenSpot-Pro addresses the complexity and diversity of real-world professional software scenarios, revealing significant performance gaps in current MLLM GUI perception capabilities.","descriptions":{"en":"ScreenSpot-Pro is a novel GUI grounding benchmark designed to rigorously evaluate the grounding capabilities of multimodal large language models (MLLMs) in professional high-resolution computing environments. The benchmark comprises 1,581 instructions across 23 applications spanning 5 industries and 3 operating systems, featuring authentic high-resolution images from professional domains with expert annotations. Unlike previous benchmarks that focus on cropped screenshots in consumer applications, ScreenSpot-Pro addresses the complexity and diversity of real-world professional software scenarios, revealing significant performance gaps in current MLLM GUI perception capabilities."},"evidence_summary":{"document_count":1,"model_count":25,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:screenspot-pro:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot-pro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:screenspot-pro","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_8b6ef0a34d9d5d68"},"modality":"multimodal","name":"ScreenSpot Pro","openness":"unknown","publisher":null,"released":"2025-04-04","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:screenspot-pro","source_url":"https://arxiv.org/abs/2504.07981"},"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":25,"score_direction":"higher_is_better","score_summary":{"display_max":87.9,"display_multiplier":100,"model_count":25,"model_count_basis":"source_model_id","numeric_count":25,"raw_max":0.879,"raw_min":0.29,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:screenspot-pro:claude-opus-4-8","reported_date":"2026-05-28","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot-pro?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-screenspot-pro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot-pro?top_n=500","topics":["computer-use","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":8,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bunnybeibei/sctranslation","source_record":"bm_sctranslation_c3e5eec8"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"scTranslation evaluates single-cell multi-omics modality translation across RNA, ATAC, and ADT modalities, with diverse datasets, six representative models, and comprehensive metrics.","descriptions":{"en":"scTranslation evaluates single-cell multi-omics modality translation across RNA, ATAC, and ADT modalities, with diverse datasets, six representative models, and comprehensive metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.03906","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_sctranslation_c3e5eec8"},"modality":null,"name":"scTranslation","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03906","source_url":"https://arxiv.org/abs/2606.03906"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03906","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03906","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":null,"source_record":"bm_sdabench_9285d4fe"},"categories":["AI Scientist","General AI","Language & Knowledge","Materials & Chemistry"],"collected_at":"2026-10-14T09:44:51Z","description":"SDABench evaluates LLMs' scientific data analysis capabilities across six capabilities (descriptive, exploratory, inferential, predictive, causal, mechanistic) and five domains, with 527 real and 6000 synthetic instances in multiple-choice and open-ended formats.","descriptions":{"en":"SDABench evaluates LLMs' scientific data analysis capabilities across six capabilities (descriptive, exploratory, inferential, predictive, causal, mechanistic) and five domains, with 527 real and 6000 synthetic instances in multiple-choice and open-ended formats."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.11079","languages":[],"library_direction":{"basis":"model-review","categories":["science","professional-work"],"join":"claire_record_id","primary":"science","source_record":"bm_sdabench_9285d4fe"},"modality":null,"name":"SDABench","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11079","source_url":"https://arxiv.org/abs/2607.11079"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11079","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11079","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates stigma detection and response generation in dyadic and group dialogues.","descriptions":{"en":"Evaluates stigma detection and response generation in dyadic and group dialogues."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01548","languages":[],"library_direction":null,"modality":null,"name":"SDARE-Bench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01548","source_url":"https://arxiv.org/abs/2609.01548"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01548","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01548","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_sdgbiasbench_aa7777ae"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"SDGBiasBench is a benchmark suite for evaluating vision-language models on Sustainable Development Goals (SDG) reasoning, covering 500k multiple-choice questions and 50k regression tasks.","descriptions":{"en":"SDGBiasBench is a benchmark suite for evaluating vision-language models on Sustainable Development Goals (SDG) reasoning, covering 500k multiple-choice questions and 50k regression tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.21919","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","safety-alignment"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_sdgbiasbench_aa7777ae"},"modality":null,"name":"SDGBiasBench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.21919","source_url":"https://arxiv.org/abs/2605.21919"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-21919","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.21919","topics":["multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Logistics & Operations","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SDR-Bench evaluates LLM personalization in a two-party framework. It includes 6,279 customer success stories across 22 industries and ~200 enterprises, with a temporally constrained simulation to prevent data leakage. Scoring compares model-generated outreach against human outcomes.","descriptions":{"en":"SDR-Bench evaluates LLM personalization in a two-party framework. It includes 6,279 customer success stories across 22 industries and ~200 enterprises, with a temporally constrained simulation to prevent data leakage. Scoring compares model-generated outreach against human outcomes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20471","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","agent-memory"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_sdr-bench_b95d0afe"},"modality":null,"name":"SDR-Bench","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20471","source_url":"https://arxiv.org/abs/2607.20471"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20471","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20471","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sea-speechbench/sea-speechbench","source_record":"bm_sea-speechbench_336fc1dd"},"categories":["Factuality","General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates speech understanding across 11 Southeast Asian languages via 97,194 samples covering speech processing, paralinguistic analysis, and temporal understanding tasks.","descriptions":{"en":"Evaluates speech understanding across 11 Southeast Asian languages via 97,194 samples covering speech processing, paralinguistic analysis, and temporal understanding tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09672","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_sea-speechbench_336fc1dd"},"modality":null,"name":"SEA-SpeechBench","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09672","source_url":"https://arxiv.org/abs/2609.09672"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09672","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09672","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI","Self-Evolution"],"collected_at":"2026-10-14T09:44:51Z","description":"Self-evolving LLM agents have gained prominence for their ability to improve after deployment by modifying their harness, including their controller instructions, memory management protocols, and reusable tools and skills, in response to u…","descriptions":{"en":"Self-evolving LLM agents have gained prominence for their ability to improve after deployment by modifying their harness, including their controller instructions, memory management protocols, and reusable tools and skills, in response to u…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35596","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_seabench_bac5d79b"},"modality":null,"name":"SEABench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35596","source_url":"https://arxiv.org/abs/2609.35596"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35596","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35596","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":2852,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_573169b4bacca2e3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Seal-0 is a benchmark for evaluating agentic search capabilities, testing models' ability to navigate and retrieve information using tools.","descriptions":{"en":"Seal-0 is a benchmark for evaluating agentic search capabilities, testing models' ability to navigate and retrieve information using tools."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:seal-0:kimi-k2-thinking-0905","reported_at":"2025-09-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seal-0?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:seal-0","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","factuality"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_573169b4bacca2e3"},"modality":"text","name":"Seal-0","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":57.4,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.574,"raw_min":0.414,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:seal-0:kimi-k2.5","reported_date":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seal-0?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"}],"slug":"llm-stats-seal-0","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seal-0?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SEAM-Bench is a double-blind continuity storyboarding benchmark for evaluating visual continuity in short-drama generation. It assesses cross-episode continuity recall and generalizes across six mainstream text models.","descriptions":{"en":"SEAM-Bench is a double-blind continuity storyboarding benchmark for evaluating visual continuity in short-drama generation. It assesses cross-episode continuity recall and generalizes across six mainstream text models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22725","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_seam-bench_df6f1e82"},"modality":null,"name":"SEAM-Bench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22725","source_url":"https://arxiv.org/abs/2608.22725"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22725","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.22725","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Search and Function-Calling is an OpenAI internal production benchmark measuring reliable search-tool use and function calling in agentic workflows, reported as a pass rate.","descriptions":{"en":"Search and Function-Calling is an OpenAI internal production benchmark measuring reliable search-tool use and function calling in agentic workflows, reported as a pass rate."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:openai-search-function-calling:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-search-function-calling?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:openai-search-function-calling","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","tool-use"],"join":"llm_stats_id","primary":"agent-security","source_record":"catalog_c083b66e11dca90a"},"modality":"text","name":"Search and Function-Calling","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":94.6,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.946,"raw_min":0.897,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:openai-search-function-calling:gpt-5.6-terra","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-search-function-calling?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-openai-search-function-calling","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-search-function-calling?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SearchAuditBench evaluates LLM auditors on localizing, attributing, and repairing failures in long-horizon search agent trajectories using expert-annotated error steps, root causes, and repair rubrics.","descriptions":{"en":"SearchAuditBench evaluates LLM auditors on localizing, attributing, and repairing failures in long-horizon search agent trajectories using expert-annotated error steps, root causes, and repair rubrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05212","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_searchauditbench_2dd8deb6"},"modality":null,"name":"SearchAuditBench","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05212","source_url":"https://arxiv.org/abs/2608.05212"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05212","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05212","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":58,"github_stars_30d":5,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:sec-bench/sec-bench-pro","source_record":"bm_sec-bench_29fdf2d8"},"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"SEC-bench Pro measures long-horizon vulnerability discovery in software security. It includes 344 verified vulnerabilities across V8, SpiderMonkey, and Linux kernel, each paired with instructions for reproducing a working proof-of-concept. Grading uses an LLM-based judge to classify generated PoCs against vulnerable, fixed, and latest images.","descriptions":{"en":"SEC-bench Pro measures long-horizon vulnerability discovery in software security. It includes 344 verified vulnerabilities across V8, SpiderMonkey, and Linux kernel, each paired with instructions for reproducing a working proof-of-concept. Grading uses an LLM-based judge to classify generated PoCs against vulnerable, fixed, and latest images."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26548","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_sec-bench_29fdf2d8"},"modality":null,"name":"SEC-bench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26548","source_url":"https://arxiv.org/abs/2605.26548"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26548","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26548","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","智能体","Agent","任务执行","Task Execution","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SEC-bench is a benchmark designed to evaluate large language model (LLM) agents on real-world software security tasks. SEC-bench 是一个面向大型语言模型（LLM）智能体的软件安全任务评测基准，旨在自动化评估模型在真实漏洞环境中的能力。 该基准通过多智能体框架自动构建代码仓库、复现漏洞并生成修复补丁，涵盖漏洞验证（PoC 生成）和补丁修复两个关键任务，包含数百个真实安全案例。","descriptions":{"en":"SEC-bench is a benchmark designed to evaluate large language model (LLM) agents on real-world software security tasks.","zh":"SEC-bench 是一个面向大型语言模型（LLM）智能体的软件安全任务评测基准，旨在自动化评估模型在真实漏洞环境中的能力。 该基准通过多智能体框架自动构建代码仓库、复现漏洞并生成修复补丁，涵盖漏洞验证（PoC 生成）和补丁修复两个关键任务，包含数百个真实安全案例。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1986","languages":[],"library_direction":null,"modality":null,"name":"SEC-bench","openness":"open","publisher":"University of Illinois Urbana-Champaign , Purdue University","released":"2025-06-13","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1986-sec-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SEC-bench","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":58,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:sec-bench/sec-bench-pro","source_record":"catalog_357a19bc43aba891"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SEC-bench Pro is a self-evolving software-security benchmark that measures agent bug hunting on critical, high-complexity systems. It instantiates validated vulnerabilities across the V8 and SpiderMonkey JavaScript engines as reproducible vulnerability-discovery and proof-of-concept-generation tasks with oracle-based validation.","descriptions":{"en":"SEC-bench Pro is a self-evolving software-security benchmark that measures agent bug hunting on critical, high-complexity systems. It instantiates validated vulnerabilities across the V8 and SpiderMonkey JavaScript engines as reproducible vulnerability-discovery and proof-of-concept-generation tasks with oracle-based validation."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:sec-bench-pro:gpt-5.6-luna","reported_at":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sec-bench-pro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:sec-bench-pro","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"llm_stats_id","primary":"cybersecurity","source_record":"catalog_357a19bc43aba891"},"modality":"text","name":"SEC-bench Pro","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":71.2,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.712,"raw_min":0.489,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:sec-bench-pro:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sec-bench-pro?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"}],"slug":"llm-stats-sec-bench-pro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sec-bench-pro?top_n=500","topics":["coding","agents","safety"],"unit":null},{"aliases":["SEC-Bench Pro"],"attention_signals":null,"categories":["security"],"collected_at":null,"description":"Security engineering benchmark; evaluates both vulnerability detection and secure coding practices.","descriptions":{"en":"Security engineering benchmark; evaluates both vulnerability detection and secure coding practices."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000sec_bench_pro\u0000deepseek_v4_1_flash_technical_report\u0000sec_bench_pro\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"first_score_reported_at":"2026-09-10","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000sec_bench_pro\u0000deepseek_v4_1_flash_technical_report\u0000sec_bench_pro\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:sec_bench_pro","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["cybersecurity"],"join":"review:14H034160212:2026-10-15","primary":"cybersecurity","source_record":null},"modality":null,"name":"SEC-Bench Pro","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":62.8,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":62.8,"raw_min":62.8,"source_reference":{"obs_id":"curated\u0000sec_bench_pro\u0000deepseek_v4_1_flash_technical_report\u0000sec_bench_pro\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000sec_bench_pro\u0000deepseek_v4_1_flash_technical_report\u0000sec_bench_pro\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"}],"slug":"sec_bench_pro","source":"model_reports","source_url":"https://github.com/sec-bench/sec-bench","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zidaniel123/secagent-bench","source_record":"bm_secagent-bench_3c21903c"},"categories":["Cybersecurity","Robustness","Safety & Trustworthiness","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM providers and self-hosted models on security analysis tasks: SAST source code vulnerability detection, DAST HTTP request/response analysis, and prompt-injection robustness in code comments and HTTP responses.","descriptions":{"en":"Evaluates LLM providers and self-hosted models on security analysis tasks: SAST source code vulnerability detection, DAST HTTP request/response analysis, and prompt-injection robustness in code comments and HTTP responses."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:zidaniel123/secagent-bench","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","agent-security"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_secagent-bench_3c21903c"},"modality":null,"name":"secagent-bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:zidaniel123/secagent-bench","source_url":"https://github.com/zidaniel123/secagent-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-zidaniel123-secagent-bench","source":"claire_radar","source_url":"https://github.com/zidaniel123/secagent-bench","topics":["agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","大语言模型","LLM","安全对齐","Safety Alignment","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Tencent Zhuque Lab and Tencent Security Keen Lab, together with Tencent Huyuan Team, Professor Jiang Yong/Professor Xia Shutao's team from Tsinghua University, Professor Luo Xiapu's research team from Hong Kong Polytechnic University, and OpenCompass team from Shanghai Artificial Intelligence Laboratory, have jointly built a safety benchmark, SecBench. We provide fair, impartial, objective, and comprehensive evaluation capabilities and promote the construction of large model in security dimension. 腾讯朱雀实验室和腾讯安全科恩实验室联合腾讯混元大模型团队、清华大学江勇教授/夏树涛教授团队、香港理工大学罗夏朴教授研究团队以及上海人工智能实验室OpenCompass团队，通过建设安全大模型评测基准SecBench，为安全大模型研发提供公平、公正、客观、全面的评测能力，推动安全大模型建设。","descriptions":{"en":"Tencent Zhuque Lab and Tencent Security Keen Lab, together with Tencent Huyuan Team, Professor Jiang Yong/Professor Xia Shutao's team from Tsinghua University, Professor Luo Xiapu's research team from Hong Kong Polytechnic University, and OpenCompass team from Shanghai Artificial Intelligence Laboratory, have jointly built a safety benchmark, SecBench. We provide fair, impartial, objective, and comprehensive evaluation capabilities and promote the construction of large model in security dimension.","zh":"腾讯朱雀实验室和腾讯安全科恩实验室联合腾讯混元大模型团队、清华大学江勇教授/夏树涛教授团队、香港理工大学罗夏朴教授研究团队以及上海人工智能实验室OpenCompass团队，通过建设安全大模型评测基准SecBench，为安全大模型研发提供公平、公正、客观、全面的评测能力，推动安全大模型建设。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"opencompass:948","languages":[],"library_direction":null,"modality":null,"name":"SecBench","openness":"unknown","publisher":null,"released":"2024-01-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-948-secbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SecBench","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":135,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:alibaba/sec-code-bench","source_record":"catalog_68f604f200b08b50"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SecCodeBench evaluates LLM coding agents on secure code generation and vulnerability detection, testing the ability to produce code that is both functional and free from security vulnerabilities.","descriptions":{"en":"SecCodeBench evaluates LLM coding agents on secure code generation and vulnerability detection, testing the ability to produce code that is both functional and free from security vulnerabilities."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:seccodebench:qwen3.5-397b-a17b","reported_at":"2026-02-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seccodebench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:seccodebench","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","cybersecurity"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_68f604f200b08b50"},"modality":"text","name":"SecCodeBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":68.30000000000001,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.683,"raw_min":0.683,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:seccodebench:qwen3.5-397b-a17b","reported_date":"2026-02-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seccodebench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"}],"slug":"llm-stats-seccodebench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seccodebench?top_n=500","topics":["coding"],"unit":null},{"aliases":["SecCodeBench"],"attention_signals":null,"categories":["security"],"collected_at":null,"description":"Measures secure-coding behaviour, not exploitation capability.","descriptions":{"en":"Measures secure-coding behaviour, not exploitation capability."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000seccodebench\u0000qwen3_5_model_card\u0000seccodebench\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000seccodebench\u0000qwen3_5_model_card\u0000seccodebench\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:seccodebench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["cybersecurity"],"join":"review:14H034160212:2026-10-15","primary":"cybersecurity","source_record":null},"modality":null,"name":"SecCodeBench","openness":"unknown","publisher":null,"released":"2025-10-14","released_reference":{"basis":"benchmark_release","source_key":"model-reports:seccodebench","source_url":"https://github.com/alibaba/SecCodeBench"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":68.3,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":68.3,"raw_min":68.3,"source_reference":{"obs_id":"curated\u0000seccodebench\u0000qwen3_5_model_card\u0000seccodebench\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000seccodebench\u0000qwen3_5_model_card\u0000seccodebench\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"seccodebench","source":"model_reports","source_url":"https://github.com/alibaba/SecCodeBench","topics":["coding","cybersecurity"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"SecDrift is a framework measuring sector-conditioned security drift in AI-generated code, evaluating vulnerability rates from LLMs when prompted with industry contexts versus neutral baselines across CISA sectors and CWE categories.","descriptions":{"en":"SecDrift is a framework measuring sector-conditioned security drift in AI-generated code, evaluating vulnerability rates from LLMs when prompted with industry contexts versus neutral baselines across CISA sectors and CWE categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.25225","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","cybersecurity"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_secdrift_836e6631"},"modality":null,"name":"SecDrift","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25225","source_url":"https://arxiv.org/abs/2607.25225"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25225","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25225","topics":["coding","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":318,"github_stars_30d":24,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_secrespond_a48429f2"},"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating LLM agents on post-compromise incident-response workflows using forensic disk snapshots and host security reports across 10 cyber ranges.","descriptions":{"en":"A benchmark for evaluating LLM agents on post-compromise incident-response workflows using forensic disk snapshots and host security reports across 10 cyber ranges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.26791","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_secrespond_a48429f2"},"modality":null,"name":"SecRespond","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26791","source_url":"https://arxiv.org/abs/2607.26791"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26791","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26791","topics":["cybersecurity"],"unit":null},{"aliases":["Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxies"],"attention_signals":null,"categories":["Chip Design & EDA","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"ArXiv paper on auditing AI-generated RTL verification plans with a benchmark, but no artifact evidence is provided.","descriptions":{"en":"ArXiv paper on auditing AI-generated RTL verification plans with a benchmark, but no artifact evidence is provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.19844","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_trust-but-validate-the-instrument-auditing_813aaa4c"},"modality":null,"name":"SecTB-RTL","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19844","source_url":"https://arxiv.org/abs/2609.19844"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19844","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19844","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":367,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"artifact:arxiv:2307.16125","repository":"gh:ailab-cvc/seed-bench","source_record":"catalog_38dc5ede12f20661"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SEED-Bench aims at the evaluation of generative comprehension in MLLMs, consisting of 19K multiple choice questions, which spans 12 evaluation dimensions including the comprehension of both the image and video modality. SEED-Bench用于评估多模态大模型的理解能力，包括对图像和视频的理解，由跨越12个评估维度的19K道多项选择题组成。","descriptions":{"en":"SEED-Bench aims at the evaluation of generative comprehension in MLLMs, consisting of 19K multiple choice questions, which spans 12 evaluation dimensions including the comprehension of both the image and video modality.","zh":"SEED-Bench用于评估多模态大模型的理解能力，包括对图像和视频的理解，由跨越12个评估维度的19K道多项选择题组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1359","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2307.16125","primary":"visual-reasoning","source_record":"catalog_38dc5ede12f20661"},"modality":"multimodal","name":"SEED-Bench","openness":"restricted","publisher":"Tencent AI Lab","released":"2023-07-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1359-seed-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":367,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"artifact:gh:ailab-cvc/seed-bench","repository":"gh:ailab-cvc/seed-bench","source_record":"catalog_38dc5ede12f20661"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SEED-Bench-2 assesses both text and image generation of MLLMs. It spans 27 evaluation dimensions, featuring 24K multiple-choice questions with precise human annotations. SEED-Bench-2用于评估多模态大模型的文本和图像生成能力，包括跨越27个维度的24K道多选题及准确的人工注释。","descriptions":{"en":"SEED-Bench-2 assesses both text and image generation of MLLMs. It spans 27 evaluation dimensions, featuring 24K multiple-choice questions with precise human annotations.","zh":"SEED-Bench-2用于评估多模态大模型的文本和图像生成能力，包括跨越27个维度的24K道多选题及准确的人工注释。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1363","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:gh:ailab-cvc/seed-bench","primary":"visual-reasoning","source_record":"catalog_38dc5ede12f20661"},"modality":"multimodal","name":"SEED-Bench-2","openness":"restricted","publisher":"Tencent AI Lab","released":"2023-11-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1363-seed-bench-2","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":367,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"artifact:gh:ailab-cvc/seed-bench","repository":"gh:ailab-cvc/seed-bench","source_record":"catalog_38dc5ede12f20661"},"categories":["多模态","Multimodal","理解","Understanding","text-rich image","多模态模型","VLM","图像理解","Image Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SEED-Bench-2-Plus, a benchmark specifically designed for evaluating text-rich visual comprehension of MLLMs. The benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs. SEED-Bench-2-Plus, a benchmark specifically designed for evaluating text-rich visual comprehension of MLLMs. The benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs.","descriptions":{"en":"SEED-Bench-2-Plus, a benchmark specifically designed for evaluating text-rich visual comprehension of MLLMs. The benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs.","zh":"SEED-Bench-2-Plus, a benchmark specifically designed for evaluating text-rich visual comprehension of MLLMs. The benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1395","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:gh:ailab-cvc/seed-bench","primary":"visual-reasoning","source_record":"catalog_38dc5ede12f20661"},"modality":"multimodal","name":"SEED-Bench-2-Plus","openness":"restricted","publisher":"Tencent AI Lab","released":"2024-04-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1395-seed-bench-2-plus","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2-Plus","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SeedClawBench is an agentic coding benchmark measuring overall model performance on real-world, tool-using software development tasks.","descriptions":{"en":"SeedClawBench is an agentic coding benchmark measuring overall model performance on real-world, tool-using software development tasks."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:seedclawbench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seedclawbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:seedclawbench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_8a02548a7abf79a1"},"modality":"text","name":"SeedClawBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":66.60000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.666,"raw_min":0.638,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:seedclawbench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seedclawbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-seedclawbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seedclawbench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.PL"],"collected_at":"2026-10-14T09:44:51Z","description":"SeGaBench is an executable benchmark containing 120 cases (100 synthetic, 20 source-backed) to test whether LLMs can recover semantic optimization opportunities that compilers miss, with hidden enabling semantics, oracle artifacts, and validators.","descriptions":{"en":"SeGaBench is an executable benchmark containing 120 cases (100 synthetic, 20 source-backed) to test whether LLMs can recover semantic optimization opportunities that compilers miss, with hidden enabling semantics, oracle artifacts, and validators."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03983","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_segabench_e418e029"},"modality":null,"name":"SeGaBench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03983","source_url":"https://arxiv.org/abs/2608.03983"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03983","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03983","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SelectBench evaluates selective evidence adoption in retrieval-augmented language models, focusing on rejecting deceptive content. The benchmark includes a 325-example test set and rule- or judge-based reward scoring.","descriptions":{"en":"SelectBench evaluates selective evidence adoption in retrieval-augmented language models, focusing on rejecting deceptive content. The benchmark includes a 325-example test set and rule- or judge-based reward scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20090","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","deep-research"],"join":"claire_record_id","primary":"factuality","source_record":"bm_selectbench_07886b3c"},"modality":null,"name":"SelectBench","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20090","source_url":"https://arxiv.org/abs/2607.20090"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20090","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20090","topics":[],"unit":null},{"aliases":["Semantic-ITC: A Frame-wise Indoor Mobile Laser Scanning Dataset and Benchmark for Semantic Segmentation"],"attention_signals":null,"categories":["Factuality","General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Frame-wise indoor mobile laser scanning semantic segmentation dataset and benchmark.","descriptions":{"en":"Frame-wise indoor mobile laser scanning semantic segmentation dataset and benchmark."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.18493","languages":[],"library_direction":null,"modality":null,"name":"Semantic-ITC","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18493","source_url":"https://arxiv.org/abs/2609.18493"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18493","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18493","topics":[],"unit":null},{"aliases":["SemCityLoc: Aerial 6DoF Localization Using Semantic 3D City Models"],"attention_signals":null,"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A real-world aerial 6DoF localization benchmark pairing centimeter-accurate UAV poses with standardized semantic city models and low-altitude imagery.","descriptions":{"en":"A real-world aerial 6DoF localization benchmark pairing centimeter-accurate UAV poses with standardized semantic city models and low-altitude imagery."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.27444","languages":[],"library_direction":null,"modality":null,"name":"SemCityLockeD","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27444","source_url":"https://arxiv.org/abs/2606.27444"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27444","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27444","topics":[],"unit":null},{"aliases":["When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark for evaluating Arabic–Hebrew cognate identification and semantic disambiguation across multiple input representations.","descriptions":{"en":"A benchmark for evaluating Arabic–Hebrew cognate identification and semantic disambiguation across multiple input representations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13218","languages":[],"library_direction":null,"modality":null,"name":"SemCog Bench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13218","source_url":"https://arxiv.org/abs/2606.13218"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13218","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13218","topics":[],"unit":null},{"aliases":["Learning Materials Properties from Scarce Labels and Unlabeled Crystals"],"attention_signals":{"as_of":"2026-10-12","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:littlepeachs/semimat","source_record":"bm_learning-materials-properties-from-scarce-_e660c5fc"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates semi-supervised materials property regression across six tasks with fixed splits, four graph backbones, and normalized MAE, using labeled and unlabeled crystal data.","descriptions":{"en":"Evaluates semi-supervised materials property regression across six tasks with fixed splits, four graph backbones, and normalized MAE, using labeled and unlabeled crystal data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.30682","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_learning-materials-properties-from-scarce-_e660c5fc"},"modality":null,"name":"SemiMat","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30682","source_url":"https://arxiv.org/abs/2608.30682"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30682","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30682","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of LLM version-constraint resolution semantics across npm, PEP 440, and Cargo, with 240 machine-checkable items.","descriptions":{"en":"A benchmark of LLM version-constraint resolution semantics across npm, PEP 440, and Cargo, with 240 machine-checkable items."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11180","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_semverbench_81ce3f52"},"modality":null,"name":"SemVerBench","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11180","source_url":"https://arxiv.org/abs/2609.11180"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11180","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11180","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Autonomous Driving","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SENSE-VAD is a synthetic video anomaly detection dataset for autonomous driving, generated with CARLA and Unreal Engine. It includes socially complex anomalies across five categories with per-frame binary labels, plus real-world videos for sim-to-real transfer.","descriptions":{"en":"SENSE-VAD is a synthetic video anomaly detection dataset for autonomous driving, generated with CARLA and Unreal Engine. It includes socially complex anomalies across five categories with per-frame binary labels, plus real-world videos for sim-to-real transfer."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.31875","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","embodied-robotics"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_sense-vad_052128b1"},"modality":null,"name":"SENSE-VAD","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31875","source_url":"https://arxiv.org/abs/2606.31875"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31875","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31875","topics":[],"unit":null},{"aliases":["Sentinel: Embodied Cooperative Spatial Reasoning and Planning"],"attention_signals":null,"categories":["General AI","Planning","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-09-25T04:40:22Z","description":"Sentinel Challenge evaluates decentralized embodied agents that must communicate, select a meeting point, and navigate around dynamic patrols in city-scale environments.","descriptions":{"en":"Sentinel Challenge evaluates decentralized embodied agents that must communicate, select a meeting point, and navigate around dynamic patrols in city-scale environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26239","languages":[],"library_direction":null,"modality":null,"name":"Sentinel Challenge","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26239","source_url":"https://arxiv.org/abs/2605.26239"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26239","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26239","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_sentinelbench_48835b47"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SentinelBench is an open-source benchmark for time-evolving monitoring tasks. It contains 100 tasks across 10 synthetic web environments (email, calendars, finance, etc.) with scripted events, measuring task completion, reaction time, and resource use.","descriptions":{"en":"SentinelBench is an open-source benchmark for time-evolving monitoring tasks. It contains 100 tasks across 10 synthetic web environments (email, calendars, finance, etc.) with scripted events, measuring task completion, reaction time, and resource use."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05342","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_sentinelbench_48835b47"},"modality":null,"name":"SentinelBench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05342","source_url":"https://arxiv.org/abs/2606.05342"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05342","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05342","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:kiwankwon/seq2synth","source_record":"bm_seq2synth_0ff26fb5"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Seq2Synth evaluates temporal fidelity of synthetic sequential tabular data across timestamp, cross-sectional, longitudinal, structural, and privacy dimensions, using a taxonomy to determine applicable metrics. It spans seven core datasets and multiple generators.","descriptions":{"en":"Seq2Synth evaluates temporal fidelity of synthetic sequential tabular data across timestamp, cross-sectional, longitudinal, structural, and privacy dimensions, using a taxonomy to determine applicable metrics. It spans seven core datasets and multiple generators."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.15606","languages":[],"library_direction":null,"modality":null,"name":"Seq2Synth","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15606","source_url":"https://arxiv.org/abs/2607.15606"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15606","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15606","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Geometric reasoning","Reasoning","Robotics & Embodied AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"3D visual grounding benchmark with sequence-aligned RGB observations and human-verified expressions.","descriptions":{"en":"3D visual grounding benchmark with sequence-aligned RGB observations and human-verified expressions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30451","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_seqalign3dvg_20f971d6"},"modality":null,"name":"SeqAlign3DVG","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30451","source_url":"https://arxiv.org/abs/2608.30451"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30451","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30451","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating memory-augmented personalized agents on hierarchical user understanding from heterogeneous data, with four levels of user understanding and psychometrics-based synthetic data.","descriptions":{"en":"A benchmark for evaluating memory-augmented personalized agents on hierarchical user understanding from heterogeneous data, with four levels of user understanding and psychometrics-based synthetic data."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27056","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_setoka_2726d0e3"},"modality":null,"name":"Setoka","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27056","source_url":"https://arxiv.org/abs/2607.27056"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27056","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27056","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_sevra-bench_b52ddf44"},"categories":["Agents","Cybersecurity","Language & Knowledge","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SEVRA-BENCH is a benchmark for measuring how often LLM-based code review agents approve adversarial pull requests with social-engineering framings, built from vulnerability-fixing commits. It includes a challenge split of roughly 1,000 adversarial PRs.","descriptions":{"en":"SEVRA-BENCH is a benchmark for measuring how often LLM-based code review agents approve adversarial pull requests with social-engineering framings, built from vulnerability-fixing commits. It includes a challenge split of roughly 1,000 adversarial PRs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13757","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agentic-coding"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_sevra-bench_b52ddf44"},"modality":null,"name":"SEVRA-BENCH","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13757","source_url":"https://arxiv.org/abs/2606.13757"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13757","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13757","topics":["coding","agents","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","VQA","科学智能","AI for Science","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The Scientists' First Exam (SFE) benchmark, designed to comprehensively evaluate the scientific cognitive capabilities of MLLMs through three cognitive levels (cog-levels):Scientific Signal Perception、Scientific Attribute Understanding 、Scientific Comparative Reasoning. The Scientists' First Exam (SFE) 基准测试旨在通过三个认知层级——科学信号感知、科学属性理解 和 科学对比推理，全面评估多模态大语言模型（MLLMs）的科学认知能力。","descriptions":{"en":"The Scientists' First Exam (SFE) benchmark, designed to comprehensively evaluate the scientific cognitive capabilities of MLLMs through three cognitive levels (cog-levels):Scientific Signal Perception、Scientific Attribute Understanding 、Scientific Comparative Reasoning.","zh":"The Scientists' First Exam (SFE) 基准测试旨在通过三个认知层级——科学信号感知、科学属性理解 和 科学对比推理，全面评估多模态大语言模型（MLLMs）的科学认知能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1961","languages":["English","Chinese"],"library_direction":null,"modality":"multimodal","name":"SFE","openness":"unknown","publisher":"Shanghai Artificial Intelligence Laboratory","released":"2025-06-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1961-sfe","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SFE","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","NeurIPS 2024","大语言模型","LLM","安全对齐","Safety Alignment"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SG-Bench assess LLM safety across various tasks and prompt types. It integrates both generative and discriminative evaluation tasks and includes extended data to examine the impact of prompt engineering and jailbreak. SG-Bench用于评估LLM在不同任务和提示下的安全性，整合了生成性和判别性评估任务，并包含扩展数据以度量提示工程和越狱对安全性的影响。","descriptions":{"en":"SG-Bench assess LLM safety across various tasks and prompt types. It integrates both generative and discriminative evaluation tasks and includes extended data to examine the impact of prompt engineering and jailbreak.","zh":"SG-Bench用于评估LLM在不同任务和提示下的安全性，整合了生成性和判别性评估任务，并包含扩展数据以度量提示工程和越狱对安全性的影响。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1330","languages":[],"library_direction":null,"modality":null,"name":"SG-Bench","openness":"unknown","publisher":"Peking University","released":"2024-10-29","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1330-sg-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SG-Bench","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","科学","Science","代码","Code","deep research","reasoning","lab protocol","科学智能","AI for Science","科学推理","Scientific Reasoning","代码工程","逻辑推理","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SGI-Bench operationalizes this definition via four scientist-aligned task families: deep research, idea generation, dry/wet experiments, and multimodal experimental reasoning. The benchmark spans 10 disciplines and more than 1,000 expert-curated samples inspired by Science's 125 Big Questions. 科学通用智能（SGI）被定义为一种人工智能系统，其能够以接近人类科学家的通用性与熟练程度，自主地贯穿并迭代完整的科学研究流程，包括审思、构思、行动与感知四个阶段。SGI-Bench 通过四类与科学家工作流程对齐的任务对上述定义进行操作化刻画，具体包括：深度研究、思想与假设生成、干/湿实验、以及多模态实验推理。该评测基准覆盖10 个科学学科领域，包含1,000 余个由领域专家精心策划的样本，其设计灵感来源于 Science 杂志提出的 125 个重大科学问题（125 Big Questions）。","descriptions":{"en":"SGI-Bench operationalizes this definition via four scientist-aligned task families: deep research, idea generation, dry/wet experiments, and multimodal experimental reasoning. The benchmark spans 10 disciplines and more than 1,000 expert-curated samples inspired by Science's 125 Big Questions.","zh":"科学通用智能（SGI）被定义为一种人工智能系统，其能够以接近人类科学家的通用性与熟练程度，自主地贯穿并迭代完整的科学研究流程，包括审思、构思、行动与感知四个阶段。SGI-Bench 通过四类与科学家工作流程对齐的任务对上述定义进行操作化刻画，具体包括：深度研究、思想与假设生成、干/湿实验、以及多模态实验推理。该评测基准覆盖10 个科学学科领域，包含1,000 余个由领域专家精心策划的样本，其设计灵感来源于 Science 杂志提出的 125 个重大科学问题（125 Big Questions）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2325","languages":[],"library_direction":null,"modality":"multimodal","name":"SGI-Bench","openness":"unknown","publisher":"上海人工智能实验室","released":"2025-12-22","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2325-sgi-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SGI-Bench","topics":["coding","multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":130,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_sgr-bench_d4ee6524"},"categories":["General AI","Information retrieval","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SGR-Bench evaluates search agents on state-gated retrieval tasks. It includes 100 expert-curated tasks across 12 public data ecosystems, requiring agents to configure site-specific filters, views, hierarchies, or scopes to retrieve structured answers. Tasks come in goal-oriented and constraint-guided formulations.","descriptions":{"en":"SGR-Bench evaluates search agents on state-gated retrieval tasks. It includes 100 expert-curated tasks across 12 public data ecosystems, requiring agents to configure site-specific filters, views, hierarchies, or scopes to retrieve structured answers. Tasks come in goal-oriented and constraint-guided formulations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22219","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","computer-use"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_sgr-bench_d4ee6524"},"modality":null,"name":"SGR-Bench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22219","source_url":"https://arxiv.org/abs/2605.22219"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22219","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22219","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates autoformalization systems on 178 Lean 4 problems using the SA-Pass metric, which checks compile, forward, and backward semantic alignment via shadows.","descriptions":{"en":"Evaluates autoformalization systems on 178 Lean 4 problems using the SA-Pass metric, which checks compile, forward, and backward semantic alignment via shadows."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29270","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["math","safety-alignment"],"join":"claire_record_id","primary":"math","source_record":"bm_shadowbench_349af251"},"modality":null,"name":"ShadowBench","openness":"unknown","publisher":null,"released":"2026-08-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29270","source_url":"https://arxiv.org/abs/2608.29270"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29270","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29270","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ShallowBench is a curated benchmark of 5,780 shallow-pocket targets for evaluating generative drug design models. Targets are extracted from CrossDocked2020 based on low concavity and sufficient surface area.","descriptions":{"en":"ShallowBench is a curated benchmark of 5,780 shallow-pocket targets for evaluating generative drug design models. Targets are extracted from CrossDocked2020 based on low concavity and sufficient surface area."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06717","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_shallowbench_e94b542b"},"modality":null,"name":"ShallowBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06717","source_url":"https://arxiv.org/abs/2606.06717"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06717","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06717","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["安全","Safety","Implicit Risk","大语言模型","LLM","安全对齐","Safety Alignment","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Targeting overlooked implicit risks in vertical domains (Education, Finance, Management), we introduce an implicit risk benchmark and the MENTOR framework. By leveraging Rule Evolution and Activation Steering, MENTOR effectively detects and mitigates these subtle hazards. Shell由华东师范大学Shell@Educhat团队和上海人工智能实验室联合推出。当下，确保垂直领域任务中大模型的安全性至关重要。虽然目前的对齐工作主要针对偏见和暴力等显性风险，但往往忽略了更深层次的特定领域隐性风险。研发团队推出了一个包含大量隐式风险查询的基准测试集，将风险分为引导、反思、禁止三类，以及 MENTOR 框架。该框架利用规则演化循环（REC）和激活引导（RV）技术，能够有效发现并缓解这些不易察觉的潜在风险。","descriptions":{"en":"Targeting overlooked implicit risks in vertical domains (Education, Finance, Management), we introduce an implicit risk benchmark and the MENTOR framework. By leveraging Rule Evolution and Activation Steering, MENTOR effectively detects and mitigates these subtle hazards.","zh":"Shell由华东师范大学Shell@Educhat团队和上海人工智能实验室联合推出。当下，确保垂直领域任务中大模型的安全性至关重要。虽然目前的对齐工作主要针对偏见和暴力等显性风险，但往往忽略了更深层次的特定领域隐性风险。研发团队推出了一个包含大量隐式风险查询的基准测试集，将风险分为引导、反思、禁止三类，以及 MENTOR 框架。该框架利用规则演化循环（REC）和激活引导（RV）技术，能够有效发现并缓解这些不易察觉的潜在风险。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2348","languages":[],"library_direction":null,"modality":null,"name":"Shell","openness":"unknown","publisher":"华东师范大学 &上海人工智能实验室","released":"2025-12-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2348-shell","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Shell","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Autonomous Driving","Geometric reasoning","Planning","Vision & 3D","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Shift & Drift evaluates closed-loop motion planners on two tracks: Semantic Shift, which uses a conversion pipeline to transform the DeepScenario Open 3D dataset into nuPlan for zero-shot testing on 1,182 scenarios across German cities and San Francisco, and State-Distribution Drift, which injects stochastic perturbations into ego-vehicle dynamics. Scoring is based on safety and progress metrics.","descriptions":{"en":"Shift & Drift evaluates closed-loop motion planners on two tracks: Semantic Shift, which uses a conversion pipeline to transform the DeepScenario Open 3D dataset into nuPlan for zero-shot testing on 1,182 scenarios across German cities and San Francisco, and State-Distribution Drift, which injects stochastic perturbations into ego-vehicle dynamics. Scoring is based on safety and progress metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.07844","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_shift-drift_175f8c38"},"modality":null,"name":"Shift & Drift","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.07844","source_url":"https://arxiv.org/abs/2607.07844"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-07844","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07844","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_shopping-reasoning-bench_1ec16882"},"categories":["Language & Knowledge","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Assesses multi-turn conversational shopping assistants across 525 expert-authored missions with importance-weighted binary rubrics. Evaluates reasoning across five categories covering preference refinement, trade-off analysis, and compatibility assessment.","descriptions":{"en":"Assesses multi-turn conversational shopping assistants across 525 expert-authored missions with importance-weighted binary rubrics. Evaluates reasoning across five categories covering preference refinement, trade-off analysis, and compatibility assessment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12608","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","abstract-reasoning"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_shopping-reasoning-bench_1ec16882"},"modality":null,"name":"Shopping Reasoning Bench","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12608","source_url":"https://arxiv.org/abs/2606.12608"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12608","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12608","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","NeurIPS 2024","大语言模型","LLM","知识储备","Knowledge"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Shopping MMLU is a diverse multi-task online shopping benchmark derived from real-world Amazon data. It consists of 57 tasks covering 4 major shopping skills: concept understanding, knowledge reasoning, user behavior alignment, and multi-linguality. Shopping MMLU是一个基于真实亚马逊数据的多样化多任务在线购物基准测试，由57项任务组成，涵盖概念理解、知识推理、用户行为对齐和多语言4大购物场景技能。","descriptions":{"en":"Shopping MMLU is a diverse multi-task online shopping benchmark derived from real-world Amazon data. It consists of 57 tasks covering 4 major shopping skills: concept understanding, knowledge reasoning, user behavior alignment, and multi-linguality.","zh":"Shopping MMLU是一个基于真实亚马逊数据的多样化多任务在线购物基准测试，由57项任务组成，涵盖概念理解、知识推理、用户行为对齐和多语言4大购物场景技能。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1321","languages":[],"library_direction":null,"modality":null,"name":"ShoppingMMLU","openness":"unknown","publisher":"Amazon.com","released":"2024-10-28","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1321-shoppingmmlu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ShoppingMMLU","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates e-commerce reranking with contamination-limited private-traffic preference pairs.","descriptions":{"en":"Evaluates e-commerce reranking with contamination-limited private-traffic preference pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.31002","languages":[],"library_direction":null,"modality":null,"name":"ShopRank-Bench","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.31002","source_url":"https://arxiv.org/abs/2609.31002"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-31002","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31002","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_shovir_1cca6b61"},"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SHOVIR evaluates vision shortcut learning in radiology report generation by extending MIMIC-CXR and PadChest-GR with per-box CheXpert labels. It defines image-level and disease-level occlusion experiments that compare model predictions on clean images against localized perturbations to isolate direct and contextual shortcut failures at the disease-class level.","descriptions":{"en":"SHOVIR evaluates vision shortcut learning in radiology report generation by extending MIMIC-CXR and PadChest-GR with per-box CheXpert labels. It defines image-level and disease-level occlusion experiments that compare model predictions on clean images against localized perturbations to isolate direct and contextual shortcut failures at the disease-class level."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.30201","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_shovir_1cca6b61"},"modality":null,"name":"SHOVIR","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30201","source_url":"https://arxiv.org/abs/2606.30201"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30201","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30201","topics":["medical"],"unit":null},{"aliases":["ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations"],"attention_signals":null,"categories":["Finance","Language & Knowledge","Logistics & Operations","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark protocol and public dataset for comprehension after narrated business demonstrations.","descriptions":{"en":"A benchmark protocol and public dataset for comprehension after narrated business demonstrations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.25467","languages":[],"library_direction":null,"modality":null,"name":"ShowTellArena","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25467","source_url":"https://arxiv.org/abs/2609.25467"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25467","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25467","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"SidewalkBench evaluates visual navigation models on urban sidewalks using GPU-accelerated simulation with procedurally generated and real-world scanned scenes, including pedestrian-reactive and long-horizon scenarios.","descriptions":{"en":"SidewalkBench evaluates visual navigation models on urban sidewalks using GPU-accelerated simulation with procedurally generated and real-world scanned scenes, including pedestrian-reactive and long-horizon scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16953","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_sidewalkbench_6d63d9b3"},"modality":null,"name":"SidewalkBench","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16953","source_url":"https://arxiv.org/abs/2606.16953"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16953","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16953","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["structured_output","instruction_following","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SIFO (Simple Instruction Following) evaluates how well language models follow simple, explicit instructions. It tests fundamental instruction-following capabilities across various task types.","descriptions":{"en":"SIFO (Simple Instruction Following) evaluates how well language models follow simple, explicit instructions. It tests fundamental instruction-following capabilities across various task types."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:sifo:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:sifo","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_2c00e2642310ebbf"},"modality":"text","name":"SIFO","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.773,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.773,"raw_min":0.773,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:sifo:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-sifo","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo?top_n=500","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["structured_output","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SIFO-Multiturn evaluates instruction following capabilities in multi-turn conversational settings, testing how well models maintain context and follow instructions across multiple exchanges.","descriptions":{"en":"SIFO-Multiturn evaluates instruction following capabilities in multi-turn conversational settings, testing how well models maintain context and follow instructions across multiple exchanges."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:sifo-multiturn:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo-multiturn?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:sifo-multiturn","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following"],"join":"llm_stats_id","primary":"instruction-following","source_record":"catalog_8e550de1cf971aca"},"modality":"text","name":"SIFO-Multiturn","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":0.711,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.711,"raw_min":0.711,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:sifo-multiturn:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo-multiturn?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"}],"slug":"llm-stats-sifo-multiturn","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo-multiturn?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:inorganicwriter/signpost-bench","source_record":"bm_signpost-bench_c7de79e5"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"SIGNPOST-Bench evaluates text-vision conflict resolution in multimodal large language models via a counterfactual benchmark of image variants (Original, Blank, Similar, Random, Adversarial) for visual geolocation. It includes 5,111 groups and 25,555 variants, with metrics for localization error and target-directed shifts.","descriptions":{"en":"SIGNPOST-Bench evaluates text-vision conflict resolution in multimodal large language models via a counterfactual benchmark of image variants (Original, Blank, Similar, Random, Adversarial) for visual geolocation. It includes 5,111 groups and 25,555 variants, with metrics for localization error and target-directed shifts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04244","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_signpost-bench_c7de79e5"},"modality":null,"name":"SIGNPOST-Bench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04244","source_url":"https://arxiv.org/abs/2608.04244"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04244","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04244","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:hemachandra666/silent-misresolution-benchmark","source_record":"bm_silent-misresolution-benchmark_79846646"},"categories":["Biology & Drug Discovery","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for silent misresolution in dictation cleanup, where a system deletes spoken content without leaving a trace in the output; it includes a corpus with revision sites and a scorer for detecting such failures.","descriptions":{"en":"A benchmark for silent misresolution in dictation cleanup, where a system deletes spoken content without leaving a trace in the output; it includes a corpus with revision sites and a scorer for detecting such failures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:hemachandra666/silent-misresolution-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_silent-misresolution-benchmark_79846646"},"modality":null,"name":"Silent Misresolution","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:hemachandra666/silent-misresolution-benchmark","source_url":"https://github.com/hemachandra666/silent-misresolution-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-hemachandra666-silent-misresolution-benchmark","source":"claire_radar","source_url":"https://github.com/hemachandra666/silent-misresolution-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:json604/silentbug-bench","source_record":"bm_silentbug-bench_29b06a59"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates agents on detecting silent ML training defects across 18 CPU-only PyTorch tasks, scoring patch application, metric recovery, localization, minimality, and false positives.","descriptions":{"en":"Evaluates agents on detecting silent ML training defects across 18 CPU-only PyTorch tasks, scoring patch application, metric recovery, localization, minimality, and false positives."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:json604/silentbug-bench","languages":[],"library_direction":null,"modality":null,"name":"silentbug","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:json604/silentbug-bench","source_url":"https://github.com/Json604/silentbug-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-json604-silentbug-bench","source":"claire_radar","source_url":"https://github.com/Json604/silentbug-bench","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:p20030920p/sim2real-algobench","source_record":"bm_sim2real-algobench_276f3760"},"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Sim2Real-AlgoBench evaluates interchangeable navigation algorithms on a three-wheeled omnidirectional robot performing a map-search task: find a green wall marker, approach it, drive onto the pad, and hold still for 3 s, scored in simulation and on hardware.","descriptions":{"en":"Sim2Real-AlgoBench evaluates interchangeable navigation algorithms on a three-wheeled omnidirectional robot performing a map-search task: find a green wall marker, approach it, drive onto the pad, and hold still for 3 s, scored in simulation and on hardware."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:p20030920p/sim2real-algobench","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_sim2real-algobench_276f3760"},"modality":null,"name":"Sim2Real-AlgoBench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:p20030920p/sim2real-algobench","source_url":"https://github.com/p20030920p/Sim2Real-AlgoBench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-p20030920p-sim2real-algobench","source":"claire_radar","source_url":"https://github.com/p20030920p/Sim2Real-AlgoBench","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:red-pheonix/sim2realtscbenchmark","source_record":"bm_sim2signal_8839df3d"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates sim-to-real transfer for reinforcement learning traffic signal control under decomposed observation, action, transition, and reward gaps.","descriptions":{"en":"Evaluates sim-to-real transfer for reinforcement learning traffic signal control under decomposed observation, action, transition, and reward gaps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.01676","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_sim2signal_8839df3d"},"modality":null,"name":"Sim2Signal","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01676","source_url":"https://arxiv.org/abs/2609.01676"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01676","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01676","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Planning","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SIMMER evaluates latent failures in LLM-generated plans for kitchen-domain tasks using a curated symbolic world model with 77 actions and 262 objects, scoring error-free plans and latent hazard detection.","descriptions":{"en":"SIMMER evaluates latent failures in LLM-generated plans for kitchen-domain tasks using a curated symbolic world model with 77 actions and 262 objects, scoring error-free plans and latent hazard detection."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.14574","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","world-models"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_simmer_2d9d2c96"},"modality":null,"name":"SIMMER","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14574","source_url":"https://arxiv.org/abs/2606.14574"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14574","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14574","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_simple_0e42de19"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SIMPLE is a simulation testbed for humanoid loco-manipulation with 60 tasks, 50 scenes, and over 1,000 objects, integrating data generation pipelines and benchmarking policies.","descriptions":{"en":"SIMPLE is a simulation testbed for humanoid loco-manipulation with 60 tasks, 50 scenes, and over 1,000 objects, integrating data generation pipelines and benchmarking policies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.08278","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_simple_0e42de19"},"modality":null,"name":"SIMPLE","openness":"unknown","publisher":null,"released":"2026-06-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08278","source_url":"https://arxiv.org/abs/2606.08278"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08278","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08278","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":4665,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:openai/simple-evals","source_record":"lib_simpleqa"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","factuality","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.","descriptions":{"en":"SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know."},"evidence_summary":{"document_count":1,"model_count":47,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:simpleqa:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:simpleqa","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"lib_simpleqa"},"modality":"text","name":"SimpleQA","openness":"restricted","publisher":"OpenAI","released":"2024-10-30","released_reference":null,"repo_kind":"harness_only","repo_resolution_status":"resolved","score_count":47,"score_direction":"higher_is_better","score_summary":{"display_max":97.1,"display_multiplier":100,"model_count":47,"model_count_basis":"source_model_id","numeric_count":47,"raw_max":0.971,"raw_min":0.018,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:simpleqa:deepseek-v3.2-exp","reported_date":"2025-09-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2026-01-22","date_precision":"model_announcement","name":"ERNIE 5.0"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok 4 Fast"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Base)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 Eagle)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512 NVFP4)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Mistral Large 3 (675B Instruct 2512)"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Instruct"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi 4"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"}],"slug":"llm-stats-simpleqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["SimpleQA","SimpleQA Verified"],"attention_signals":null,"categories":["factuality"],"collected_at":null,"description":"Knowledge cutoff must be recorded alongside the score.","descriptions":{"en":"Knowledge cutoff must be recorded alongside the score."},"evidence_summary":{"document_count":10,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000simpleqa\u0000deepseek_v3_report\u0000simpleqa\u0000Correct\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"first_score_reported_at":"2024-12-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000simpleqa\u0000deepseek_v3_report\u0000simpleqa\u0000Correct\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:simpleqa","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["factuality"],"join":"review:14H034160212:2026-10-15","primary":"factuality","source_record":null},"modality":null,"name":"SimpleQA","openness":"unknown","publisher":null,"released":"2024-10-30","released_reference":{"basis":"benchmark_release","source_key":"model-reports:simpleqa","source_url":"https://openai.com/index/introducing-simpleqa/"},"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":57.9,"display_multiplier":1,"model_count":8,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":57.9,"raw_min":24.9,"source_reference":{"obs_id":"curated\u0000simpleqa\u0000deepseek_v4_model_card\u0000simpleqa_verified\u0000think max, pass@1\u0000DeepSeek-V4-Pro","observation_id":"curated\u0000simpleqa\u0000deepseek_v4_model_card\u0000simpleqa_verified\u0000think max, pass@1\u0000DeepSeek-V4-Pro","reported_at":"2026-04-22","reported_date":"2026-04-22","source_id":"deepseek_v4_model_card","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro"},"unit":"percent"},"scored_models":[{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2025-06-17","date_precision":"document_publication","name":"Gemini 2.5 Pro"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"simpleqa","source":"model_reports","source_url":"https://openai.com/index/introducing-simpleqa/","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"llm_stats_id","repository":null,"source_record":"catalog_91f8e2135050765d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","factuality","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SimpleQA Verified is a curated, reliability-focused subset of SimpleQA that addresses label noise and redundancy in the original benchmark, measuring short-form parametric factual accuracy of large language models on fact-seeking questions with single, indisputable answers.","descriptions":{"en":"SimpleQA Verified is a curated, reliability-focused subset of SimpleQA that addresses label noise and redundancy in the original benchmark, measuring short-form parametric factual accuracy of large language models on fact-seeking questions with single, indisputable answers."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:simpleqa-verified:mai-thinking-1","reported_at":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa-verified?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:simpleqa-verified","languages":[],"library_direction":{"basis":"model-review","categories":["factuality"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_91f8e2135050765d"},"modality":"text","name":"SimpleQA Verified","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":31.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.31,"raw_min":0.206,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:simpleqa-verified:mai-thinking-1","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa-verified?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"}],"slug":"llm-stats-simpleqa-verified","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa-verified?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_3021d1189b97a933"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","image_to_text","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SimpleVQA is a visual question answering benchmark focused on simple queries.","descriptions":{"en":"SimpleVQA is a visual question answering benchmark focused on simple queries."},"evidence_summary":{"document_count":1,"model_count":14,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:simplevqa:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simplevqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:simplevqa","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_3021d1189b97a933"},"modality":"multimodal","name":"SimpleVQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":14,"score_direction":"higher_is_better","score_summary":{"display_max":0.817,"display_multiplier":1,"model_count":14,"model_count_basis":"source_model_id","numeric_count":14,"raw_max":0.817,"raw_min":0.354,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:simplevqa:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simplevqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-simplevqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simplevqa?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Finance","Multimodal","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"A multimodal guardrail evaluation covering multimodal question answering, adversarial attacks, dynamic policy rules, and cross-modal joint risks.","descriptions":{"en":"A multimodal guardrail evaluation covering multimodal question answering, adversarial attacks, dynamic policy rules, and cross-modal joint risks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.22873","languages":[],"library_direction":null,"modality":null,"name":"SingGuard-Bench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22873","source_url":"https://arxiv.org/abs/2606.22873"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22873","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22873","topics":["multimodal","reasoning-math","finance"],"unit":null},{"aliases":["Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"SALT is a procedurally generated benchmark for evaluating correctness and uncertainty estimates in long-form language-model outputs against deterministic ground truth.","descriptions":{"en":"SALT is a procedurally generated benchmark for evaluating correctness and uncertainty estimates in long-form language-model outputs against deterministic ground truth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.03870","languages":[],"library_direction":null,"modality":null,"name":"Single-answer Atomic Long-form Target (SALT)","openness":"unknown","publisher":null,"released":"2026-07-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03870","source_url":"https://arxiv.org/abs/2607.03870"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03870","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03870","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"artifact:arxiv:1904.09728","repository":null,"source_record":"catalog_cdf285c8d0c2503a"},"categories":["推理","reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SIQA is a social interaction question answering task, which requires to select the most reasonable behavior based on the given scenario and three possible subsequent behaviors. This task is designed to test the model's knowledge in social commonsense. This dataset consists of 38,963 training samples, 1,951 development samples and 1,960 test samples, all on English text. SIQA 是一个社会交互问答任务，要求根据给定的场景和三个可能的后续行为，选择最合理的行为。这个任务是为了测试模型在社会常识方面的知识。这个数据集包含了 38963 个训练样本，1951 个开发样本和 1960 个测试样本，所有的文本都是英文文本。","descriptions":{"en":"SIQA is a social interaction question answering task, which requires to select the most reasonable behavior based on the given scenario and three possible subsequent behaviors. This task is designed to test the model's knowledge in social commonsense. This dataset consists of 38,963 training samples, 1,951 development samples and 1,960 test samples, all on English text.","zh":"SIQA 是一个社会交互问答任务，要求根据给定的场景和三个可能的后续行为，选择最合理的行为。这个任务是为了测试模型在社会常识方面的知识。这个数据集包含了 38963 个训练样本，1951 个开发样本和 1960 个测试样本，所有的文本都是英文文本。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:533","languages":["English"],"library_direction":{"basis":"model-review-2","categories":["abstract-reasoning"],"join":"artifact:arxiv:1904.09728","primary":"abstract-reasoning","source_record":"catalog_cdf285c8d0c2503a"},"modality":null,"name":"SIQA","openness":"unknown","publisher":null,"released":"2019-09-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-533-siqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SIQA","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":915,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:ethz-spylab/agentdojo","source_record":"catalog_fc60a2d907e5212e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Siren AgentDojo evaluates tool-using agents under adversarial prompt-injection attacks. This metric is the attack success rate; lower is better.","descriptions":{"en":"Siren AgentDojo evaluates tool-using agents under adversarial prompt-injection attacks. This metric is the attack success rate; lower is better."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:siren-agentdojo-attack-success:muse-glimmer-30b","reported_at":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-attack-success?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:siren-agentdojo-attack-success","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"llm_stats_id","primary":"agent-security","source_record":"catalog_fc60a2d907e5212e"},"modality":"text","name":"Siren AgentDojo Attack Success Rate","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":28.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.284,"raw_min":0.284,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:siren-agentdojo-attack-success:muse-glimmer-30b","reported_date":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-attack-success?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"}],"slug":"llm-stats-siren-agentdojo-attack-success","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-attack-success?top_n=500","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":915,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:ethz-spylab/agentdojo","source_record":"catalog_5fb391ffe84c3f24"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Siren AgentDojo evaluates tool-using agents under adversarial prompt-injection attacks. This metric reports utility on the assigned tasks.","descriptions":{"en":"Siren AgentDojo evaluates tool-using agents under adversarial prompt-injection attacks. This metric reports utility on the assigned tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:siren-agentdojo-utility:muse-glimmer-30b","reported_at":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-utility?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:siren-agentdojo-utility","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","tool-use"],"join":"llm_stats_id","primary":"agent-security","source_record":"catalog_5fb391ffe84c3f24"},"modality":"text","name":"Siren AgentDojo Utility","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":94.19999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.942,"raw_min":0.942,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:siren-agentdojo-utility:muse-glimmer-30b","reported_date":"2026-08-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-utility?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"}],"slug":"llm-stats-siren-agentdojo-utility","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-utility?top_n=500","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"SIREN-Bench-v1 is a benchmark for emergency-vehicle (EMV) interactions, built on a SUMO-CARLA co-simulation platform. It includes seven parameterized interaction templates across emergency levels L1-L3 and three behavior families, with synchronized sensor observations and simulator-native annotations. Supports tasks such as 3D object detection, trajectory prediction, and vision-language risk understanding.","descriptions":{"en":"SIREN-Bench-v1 is a benchmark for emergency-vehicle (EMV) interactions, built on a SUMO-CARLA co-simulation platform. It includes seven parameterized interaction templates across emergency levels L1-L3 and three behavior families, with synchronized sensor observations and simulator-native annotations. Supports tasks such as 3D object detection, trajectory prediction, and vision-language risk understanding."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24094","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_siren-bench_1d0cc158"},"modality":null,"name":"SIREN-Bench-v1","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24094","source_url":"https://arxiv.org/abs/2608.24094"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24094","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24094","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":24,"github_stars_30d":-9,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:intellisensing/self-in-space","source_record":"bm_sis-bench_e31a9de3"},"categories":["Robotics & Embodied AI","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SIS-Bench evaluates embodied spatial intelligence in UAV scenarios across two dimensions (spatial cognition and self-awareness) and three cognitive levels (perception, memory, reasoning), with 4,856 QA pairs from 1,646 real-world UAV videos.","descriptions":{"en":"SIS-Bench evaluates embodied spatial intelligence in UAV scenarios across two dimensions (spatial cognition and self-awareness) and three cognitive levels (perception, memory, reasoning), with 4,856 QA pairs from 1,646 real-world UAV videos."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.12477","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_sis-bench_e31a9de3"},"modality":null,"name":"SIS-Bench","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.12477","source_url":"https://arxiv.org/abs/2607.12477"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-12477","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12477","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:slovak-nlp/sk-bench","source_record":"bm_sk-bench_30bbee98"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation.","descriptions":{"en":"Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.09152","languages":[],"library_direction":{"basis":"model-review-daily","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_sk-bench_30bbee98"},"modality":null,"name":"sk-bench","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.09152","source_url":"https://arxiv.org/abs/2610.09152"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-09152","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09152","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_skill-use_dfe395c1"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Skill-Use evaluates skill use in agentic harnesses through 79 real skills and 177 executable tasks across nine domains, measuring trigger, compliance, and boundary adherence with a combined SU score.","descriptions":{"en":"Skill-Use evaluates skill use in agentic harnesses through 79 real skills and 177 executable tasks across nine domains, measuring trigger, compliance, and boundary adherence with a combined SU score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.04828","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_skill-use_dfe395c1"},"modality":null,"name":"Skill-Use","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04828","source_url":"https://arxiv.org/abs/2608.04828"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04828","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04828","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-09-25T04:40:22Z","description":"SkillAdeqBench evaluates whether Cloud Skill test suites exercise their specified operational obligations and identifies uncovered obligations.","descriptions":{"en":"SkillAdeqBench evaluates whether Cloud Skill test suites exercise their specified operational obligations and identifies uncovered obligations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.22015","languages":[],"library_direction":null,"modality":null,"name":"SkillAdeqBench","openness":"unknown","publisher":null,"released":"2026-07-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22015","source_url":"https://arxiv.org/abs/2607.22015"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22015","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22015","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Logistics & Operations","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SkillChain-Gym evaluates production-inventory control policies with reskilling dynamics, including skill certification, forgetting, and training constraints. Metrics cover operations, resilience, capability growth, and training access across seeded disruption scenarios.","descriptions":{"en":"SkillChain-Gym evaluates production-inventory control policies with reskilling dynamics, including skill certification, forgetting, and training constraints. Metrics cover operations, resilience, capability growth, and training access across seeded disruption scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17266","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_skillchain-gym_68720c0a"},"modality":null,"name":"SkillChain-Gym","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17266","source_url":"https://arxiv.org/abs/2606.17266"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17266","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17266","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":19,"join":"claire_record_id","repository":null,"source_record":"bm_skillevolbench_997ef790"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SkillEvolBench evaluates whether LLM agents can distill episodic experience into reusable procedural skills. It includes 180 tasks across six environments, with acquisition tasks and frozen deployment tasks testing context shift, adversarial shortcuts, and composition. Scoring uses success rates and other agent metrics.","descriptions":{"en":"SkillEvolBench evaluates whether LLM agents can distill episodic experience into reusable procedural skills. It includes 180 tasks across six environments, with acquisition tasks and frozen deployment tasks testing context shift, adversarial shortcuts, and composition. Scoring uses success rates and other agent metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24117","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_skillevolbench_997ef790"},"modality":null,"name":"SkillEvolBench","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24117","source_url":"https://arxiv.org/abs/2605.24117"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24117","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24117","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":10,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"claire_record_id","repository":"gh:osu-nlp-group/skillharm","source_record":"bm_skillharm_560fd4c5"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark of skill-based attacks across the skill-use lifecycle, with 879 attack samples across 71 skills, evaluating Fixed-Payload Poisoning and Self-Mutating Poisoning scenarios across 12 risk types, with attack success rate as primary metric.","descriptions":{"en":"Benchmark of skill-based attacks across the skill-use lifecycle, with 879 attack samples across 71 skills, evaluating Fixed-Payload Poisoning and Self-Mutating Poisoning scenarios across 12 risk types, with attack success rate as primary metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.02540","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agent-skills-evolution"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_skillharm_560fd4c5"},"modality":null,"name":"SkillHarm","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02540","source_url":"https://arxiv.org/abs/2606.02540"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02540","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02540","topics":["agents","safety"],"unit":null},{"aliases":["Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agents"],"attention_signals":{"as_of":"2026-10-12","github_stars":9,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:henrymao2004/misevolve","source_record":"bm_practice-makes-unsafe-skill-misevolution-i_3d2c225f"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Lifecycle-aware benchmark for persistent safety failures in self-improving LLM agents, consisting of 25 episodes with malicious demonstrations, benign twins, and fresh-session probes, scored with nine metrics including carryover ASR and unsafe retrieval.","descriptions":{"en":"Lifecycle-aware benchmark for persistent safety failures in self-improving LLM agents, consisting of 25 episodes with malicious demonstrations, benign twins, and fresh-session probes, scored with nine metrics including carryover ASR and unsafe retrieval."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.12851","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agent-skills-evolution"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_practice-makes-unsafe-skill-misevolution-i_3d2c225f"},"modality":null,"name":"SkillMisevo-Bench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12851","source_url":"https://arxiv.org/abs/2608.12851"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12851","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12851","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_skillreason-bench_12813801"},"categories":["Agents","General AI","Information retrieval","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"SkillReason-Bench is a retrieval benchmark with 3,729 queries and 61,228 skills across nine domains, but the paper focuses on proposing a retrieval method (SkillReason) rather than the benchmark itself.","descriptions":{"en":"SkillReason-Bench is a retrieval benchmark with 3,729 queries and 61,228 skills across nine domains, but the paper focuses on proposing a retrieval method (SkillReason) rather than the benchmark itself."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08640","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","deep-research"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_skillreason-bench_12813801"},"modality":null,"name":"SkillReason-Bench","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08640","source_url":"https://arxiv.org/abs/2608.08640"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08640","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08640","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_skillresolve-bench_99845262"},"categories":["Agents","General AI","Information retrieval","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"SkillResolve-Bench 1.0 evaluates agent skill retrieval under same-capability ambiguity, pairing helpful skills with risky siblings. It includes 661 pairs, a 7,982-candidate pool, disjoint splits, and reports Recall@K and harmful sibling rate (HSR@K).","descriptions":{"en":"SkillResolve-Bench 1.0 evaluates agent skill retrieval under same-capability ambiguity, pairing helpful skills with risky siblings. It includes 661 pairs, a 7,982-candidate pool, disjoint splits, and reports Recall@K and harmful sibling rate (HSR@K)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.10388","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","deep-research"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_skillresolve-bench_99845262"},"modality":null,"name":"SkillResolve-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10388","source_url":"https://arxiv.org/abs/2606.10388"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10388","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10388","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robustness","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"SkillSafe-Bench evaluates skill-merged LLMs on static refusal, adaptive jailbreak robustness, and capability retention using a two-judge AND rule, across multiple open-weight bases and attack types.","descriptions":{"en":"SkillSafe-Bench evaluates skill-merged LLMs on static refusal, adaptive jailbreak robustness, and capability retention using a two-judge AND rule, across multiple open-weight bases and attack types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08542","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","agent-skills-evolution"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_skillsafe-bench_e266705a"},"modality":null,"name":"SkillSafe-Bench","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08542","source_url":"https://arxiv.org/abs/2608.08542"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08542","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08542","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1837,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":67,"join":"llm_stats_id","repository":null,"source_record":"catalog_f0e708e2d445b453"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SkillsBench evaluates coding agents on self-contained programming tasks, measuring practical engineering skills across diverse software development scenarios.","descriptions":{"en":"SkillsBench evaluates coding agents on self-contained programming tasks, measuring practical engineering skills across diverse software development scenarios."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:skillsbench:qwen3.6-plus","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/skillsbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:skillsbench","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution"],"join":"llm_stats_id","primary":"agent-skills-evolution","source_record":"catalog_f0e708e2d445b453"},"modality":"text","name":"SkillsBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":70.19999999999999,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.702,"raw_min":0.287,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:skillsbench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/skillsbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-skillsbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/skillsbench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["SkillsBench","SkillsBench V1","Skills Bench"],"attention_signals":null,"categories":["agent"],"collected_at":null,"description":"Measures whether curated Agent Skills raise agent pass rates, not raw model capability; the meaningful figure is the paired no-Skills versus curated-Skills gap. The release date follows the paper's first submission.","descriptions":{"en":"Measures whether curated Agent Skills raise agent pass rates, not raw model capability; the meaningful figure is the paired no-Skills versus curated-Skills gap. The release date follows the paper's first submission."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000skillsbench\u0000tencent_hy4_preview\u0000skillsbench_v1\u0000SkillsBench V1\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000skillsbench\u0000tencent_hy4_preview\u0000skillsbench_v1\u0000SkillsBench V1\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:skillsbench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agent-skills-evolution"],"join":"review:14H034160212:2026-10-15","primary":"agent-skills-evolution","source_record":null},"modality":null,"name":"SkillsBench","openness":"unknown","publisher":null,"released":"2026-02-13","released_reference":{"basis":"benchmark_release","source_key":"model-reports:skillsbench","source_url":"https://skillsbench.ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":62.9,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":62.9,"raw_min":62.9,"source_reference":{"obs_id":"curated\u0000skillsbench\u0000tencent_hy4_preview\u0000skillsbench_v1\u0000SkillsBench V1\u0000Hy4 preview","observation_id":"curated\u0000skillsbench\u0000tencent_hy4_preview\u0000skillsbench_v1\u0000SkillsBench V1\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"skillsbench","source":"model_reports","source_url":"https://skillsbench.ai/","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:hanzhi306/skilltv-bench","source_record":"bm_skilltv-bench_a205ca2b"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates whether a judge can verify long-horizon agent executions using original instructions, normalized trajectories, task-time skills, and inspectable artifacts in a runnable environment.","descriptions":{"en":"Evaluates whether a judge can verify long-horizon agent executions using original instructions, normalized trajectories, task-time skills, and inspectable artifacts in a runnable environment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.05573","languages":[],"library_direction":null,"modality":null,"name":"SkillTV-Bench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05573","source_url":"https://arxiv.org/abs/2608.05573"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05573","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05573","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"SkillVetBench is a two-stage security vetting benchmark for open agentic skill ecosystems, evaluating detection of malicious skills via semantic analysis and runtime verification in a sandbox. The benchmark is built from confirmed malicious skills in the OpenClaw ecosystem.","descriptions":{"en":"SkillVetBench is a two-stage security vetting benchmark for open agentic skill ecosystems, evaluating detection of malicious skills via semantic analysis and runtime verification in a sandbox. The benchmark is built from confirmed malicious skills in the OpenClaw ecosystem."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.00925","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_skillvetbench_809c89ae"},"modality":null,"name":"SkillVetBench","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00925","source_url":"https://arxiv.org/abs/2606.00925"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00925","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00925","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"SkillVetBench evaluates security risks in open-source LLM agent skills through multidimensional LLM-based judging and a weighted agentic-risk score.","descriptions":{"en":"SkillVetBench evaluates security risks in open-source LLM agent skills through multidimensional LLM-based judging and a weighted agentic-risk score."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15899","languages":[],"library_direction":null,"modality":null,"name":"SkillVetBench","openness":"unknown","publisher":null,"released":"2026-06-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15899","source_url":"https://arxiv.org/abs/2606.15899"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15899","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15899","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SkySeaLand is a satellite object detection dataset with 1,307 high-resolution images and 19,101 bounding boxes across four classes. It provides COCO and YOLO annotations, a common split, and COCO metrics for detector evaluation.","descriptions":{"en":"SkySeaLand is a satellite object detection dataset with 1,307 high-resolution images and 19,101 bounding boxes across four classes. It provides COCO and YOLO annotations, a common split, and COCO metrics for detector evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07382","languages":[],"library_direction":null,"modality":null,"name":"SkySeaLand","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07382","source_url":"https://arxiv.org/abs/2608.07382"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07382","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07382","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_4731160c7b1e1f07"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","image_to_text","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A semantically-labeled knowledge-enhanced dataset for medical visual question answering. Contains 642 radiology images (CT scans, MRI scans, X-rays) covering five body parts and 14,028 bilingual English-Chinese question-answer pairs annotated by experienced physicians. Features comprehensive semantic labels and a structural medical knowledge base with both vision-only and knowledge-based questions requiring external medical knowledge reasoning.","descriptions":{"en":"A semantically-labeled knowledge-enhanced dataset for medical visual question answering. Contains 642 radiology images (CT scans, MRI scans, X-rays) covering five body parts and 14,028 bilingual English-Chinese question-answer pairs annotated by experienced physicians. Features comprehensive semantic labels and a structural medical knowledge base with both vision-only and knowledge-based questions requiring external medical knowledge reasoning."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:slakevqa:medgemma-4b-it","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/slakevqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:slakevqa","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_4731160c7b1e1f07"},"modality":"multimodal","name":"SlakeVQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":81.6,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.816,"raw_min":0.623,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:slakevqa:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/slakevqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-20","date_precision":"model_announcement","name":"MedGemma 4B IT"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-slakevqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/slakevqa?top_n=500","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":["SLAMSqueezeBench: Comparing SLAM Systems under Resource Constraints"],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for comparing SLAM systems under resource constraints.","descriptions":{"en":"Benchmark for comparing SLAM systems under resource constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.19533","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_slamsqueezebench-comparing-slam-systems-un_c248610e"},"modality":null,"name":"SLAMSqueezeBench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19533","source_url":"https://arxiv.org/abs/2609.19533"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19533","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19533","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:bibeshpyakurel/slapbench","source_record":"bm_slapbench_99d61c1b"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"SLAPBench evaluates multimodal LLMs on four-finger SLAP fingerprint verification using NIST SD302b, with 7,832 image pairs. It assesses identity verification under zero-shot, task-description, and similarity-scoring prompts, reporting metrics like AUC and FAR.","descriptions":{"en":"SLAPBench evaluates multimodal LLMs on four-finger SLAP fingerprint verification using NIST SD302b, with 7,832 image pairs. It assesses identity verification under zero-shot, task-description, and similarity-scoring prompts, reporting metrics like AUC and FAR."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.15517","languages":[],"library_direction":{"basis":"model-review-2","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_slapbench_99d61c1b"},"modality":null,"name":"SLAPBench","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15517","source_url":"https://arxiv.org/abs/2607.15517"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15517","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15517","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SlopBench evaluates repetitive AI writing across 112 hand-written tasks in email, social posts, essays, and workplace chat, scoring four surface behaviors: word count, opener repetition, paragraph rhythm, and fixed lexical constructions.","descriptions":{"en":"SlopBench evaluates repetitive AI writing across 112 hand-written tasks in email, social posts, essays, and workplace chat, scoring four surface behaviors: word count, opener repetition, paragraph rhythm, and fixed lexical constructions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33905","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_slopbench_ec2089ee"},"modality":null,"name":"SlopBench","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33905","source_url":"https://arxiv.org/abs/2609.33905"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33905","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33905","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:head-inthe-cloud/slugtrails","source_record":"bm_slugtrails_2de95093"},"categories":["CAD","Finance","Geometric reasoning","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"Egocentric floor plan localization benchmark for large buildings using Aria glasses recordings and aligned floor plans.","descriptions":{"en":"Egocentric floor plan localization benchmark for large buildings using Aria glasses recordings and aligned floor plans."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.19876","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_slugtrails_2de95093"},"modality":null,"name":"SlugTrails","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19876","source_url":"https://arxiv.org/abs/2609.19876"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19876","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19876","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_slvmbench_4b061162"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SLVMBench evaluates video-LLMs' ability to learn skills from long video memory and apply them to real-time tasks, using 2-3 hour streams with embedded tutorials and human-annotated questions.","descriptions":{"en":"SLVMBench evaluates video-LLMs' ability to learn skills from long video memory and apply them to real-time tasks, using 2-3 hour streams with embedded tutorials and human-annotated questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.11312","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","agent-skills-evolution"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_slvmbench_4b061162"},"modality":null,"name":"SLVMBench","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11312","source_url":"https://arxiv.org/abs/2607.11312"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11312","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11312","topics":["agents"],"unit":null},{"aliases":["4b-agent-model-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:majorminorstudio/4b-agent-model-benchmark","source_record":"bm_4b-agent-model-benchmark_96b875a6"},"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Tests six approximately 3B–4B models on seven deterministic simulated operations tasks involving tool use, state tracking, and objective completion.","descriptions":{"en":"Tests six approximately 3B–4B models on seven deterministic simulated operations tasks involving tool use, state tracking, and objective completion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:majorminorstudio/4b-agent-model-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_4b-agent-model-benchmark_96b875a6"},"modality":null,"name":"Small-Model Operational Benchmark","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:majorminorstudio/4b-agent-model-benchmark","source_url":"https://github.com/MAJORminorStudio/4b-agent-model-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-majorminorstudio-4b-agent-model-benchmark","source":"claire_radar","source_url":"https://github.com/MAJORminorStudio/4b-agent-model-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","Capabilities of on-device LLMs","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SmartBench is the first benchmark specifically designed to evaluate the capabilities of on-device large language models in smartphone scenarios. SmartBench 是首个面向智能手机终端大模型能力评估的基准，基于手机厂商提供的功能将其划分为五类共20项任务，涵盖文本摘要、问答、信息抽取、内容创作和通知管理等场景。它提供高质量数据集与定制化评估标准，旨在推动终端大模型在移动应用中的标准化评估与发展。","descriptions":{"en":"SmartBench is the first benchmark specifically designed to evaluate the capabilities of on-device large language models in smartphone scenarios.","zh":"SmartBench 是首个面向智能手机终端大模型能力评估的基准，基于手机厂商提供的功能将其划分为五类共20项任务，涵盖文本摘要、问答、信息抽取、内容创作和通知管理等场景。它提供高质量数据集与定制化评估标准，旨在推动终端大模型在移动应用中的标准化评估与发展。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2217","languages":["Chinese","Multilingual"],"library_direction":null,"modality":"multimodal","name":"SmartBench","openness":"unknown","publisher":"VIVO","released":"2025-09-26","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2217-smartbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SmartBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mobile & Personal Computing","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas.","descriptions":{"en":"We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.12975","languages":[],"library_direction":{"basis":"model-review-daily","categories":["chat-writing","multi-agent-games"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_smartsalesbench_d2a3a14d"},"modality":null,"name":"SmartSalesBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12975","source_url":"https://arxiv.org/abs/2610.12975"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12975","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12975","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_smellbench_001e20cd"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"SmellBench is a code refactoring benchmark that proactively injects code smells into clean code snippets. It contains 294 cases across 7 smell types, 3 difficulty levels, and 2 instruction settings, with evaluation covering functional correctness, localization, and refactoring quality.","descriptions":{"en":"SmellBench is a code refactoring benchmark that proactively injects code smells into clean code snippets. It contains 294 cases across 7 smell types, 3 difficulty levels, and 2 instruction settings, with evaluation covering functional correctness, localization, and refactoring quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05574","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_smellbench_001e20cd"},"modality":null,"name":"SmellBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05574","source_url":"https://arxiv.org/abs/2606.05574"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05574","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05574","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for LLM agents in smart-home environments, with 1,100 tasks across 7 categories and 22 subcategories, built on the executable HomeEnv simulator, stratified by home complexity.","descriptions":{"en":"Benchmark for LLM agents in smart-home environments, with 1,100 tasks across 7 categories and 22 subcategories, built on the executable HomeEnv simulator, stratified by home complexity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01912","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_smh-bench_ef4613ad"},"modality":null,"name":"SMH-Bench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01912","source_url":"https://arxiv.org/abs/2606.01912"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01912","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01912","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","医学","科学智能","AI for Science","跨模态推理","Cross-modal Reasoning","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SMMILE is a benchmark for evaluating multimodal large language models (MLLMs) in medical in-context learning tasks, constructed by medical experts. SMMILE 是一个由医学专家主导构建的多模态医疗上下文学习评测基准，旨在评估多模态大语言模型（MLLMs）在医学任务中的上下文学习能力。该基准包含111个问题（共517个图文问答三元组），涵盖6个医学专科和13种影像模态，另提供扩展版本 SMMILE++，包含1038个变换问题。","descriptions":{"en":"SMMILE is a benchmark for evaluating multimodal large language models (MLLMs) in medical in-context learning tasks, constructed by medical experts.","zh":"SMMILE 是一个由医学专家主导构建的多模态医疗上下文学习评测基准，旨在评估多模态大语言模型（MLLMs）在医学任务中的上下文学习能力。该基准包含111个问题（共517个图文问答三元组），涵盖6个医学专科和13种影像模态，另提供扩展版本 SMMILE++，包含1038个变换问题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2029","languages":[],"library_direction":null,"modality":"multimodal","name":"SMMILE","openness":"unknown","publisher":"ETH Zurich , Stanford University , Lund University , etc.","released":"2025-06-26","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2029-smmile","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SMMILE","topics":["multimodal","reasoning-math","medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:geodiegodrone/sms-spam-classifier","source_record":"bm_sms-spam-classifier_c251fbf9"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A reproducible SMS spam NLP workflow using TF-IDF, logistic regression, and MLflow.","descriptions":{"en":"A reproducible SMS spam NLP workflow using TF-IDF, logistic regression, and MLflow."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:geodiegodrone/sms-spam-classifier","languages":[],"library_direction":null,"modality":null,"name":"sms-spam-classifier","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:geodiegodrone/sms-spam-classifier","source_url":"https://github.com/geodiegodrone/sms-spam-classifier"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-geodiegodrone-sms-spam-classifier","source":"claire_radar","source_url":"https://github.com/geodiegodrone/sms-spam-classifier","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":null,"source_record":"bm_snapbench_c48382dd"},"categories":["General AI","Information retrieval","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal retrieval on 1,145 queries and 9,085 gallery items under 53 controlled image and text corruption conditions, with clean and noisy matching metrics.","descriptions":{"en":"Evaluates multimodal retrieval on 1,145 queries and 9,085 gallery items under 53 controlled image and text corruption conditions, with clean and noisy matching metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29607","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_snapbench_c48382dd"},"modality":null,"name":"SnapBench","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29607","source_url":"https://arxiv.org/abs/2608.29607"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29607","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29607","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Masked-concept recommendation task using SNOMED CT Entity Linking Challenge data derived from MIMIC-IV-Note.","descriptions":{"en":"Masked-concept recommendation task using SNOMED CT Entity Linking Challenge data derived from MIMIC-IV-Note."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17855","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_snomed-ct-concept-recommendation-from-mask_d3ec4c32"},"modality":null,"name":"SNOMED CT Concept Recommendation from Masked Clinical Context","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17855","source_url":"https://arxiv.org/abs/2609.17855"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17855","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17855","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:soccernet/sn-foulret","source_record":"bm_soccernet-foulret_b71fe166"},"categories":["General AI","Information retrieval","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Refereeing decisions in professional soccer remain inconsistent because referees cannot easily compare a contentious foul against similar past cases.","descriptions":{"en":"Refereeing decisions in professional soccer remain inconsistent because referees cannot easily compare a contentious foul against similar past cases."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.09742","languages":[],"library_direction":{"basis":"model-review-daily","categories":["video-understanding","deep-research"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_soccernet-foulret_b71fe166"},"modality":null,"name":"SoccerNet-FoulRet","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.09742","source_url":"https://arxiv.org/abs/2610.09742"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-09742","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09742","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_cdf285c8d0c2503a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["psychology","reasoning","creativity"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The first large-scale benchmark for commonsense reasoning about social situations. Contains 38,000 multiple choice questions probing emotional and social intelligence in everyday situations, testing commonsense understanding of social interactions and theory of mind reasoning about the implied emotions and behavior of others.","descriptions":{"en":"The first large-scale benchmark for commonsense reasoning about social situations. Contains 38,000 multiple choice questions probing emotional and social intelligence in everyday situations, testing commonsense understanding of social interactions and theory of mind reasoning about the implied emotions and behavior of others."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:social-iqa:gemma-2-27b-it","reported_at":"2024-06-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/social-iqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:social-iqa","languages":[],"library_direction":{"basis":"model-review-2","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_cdf285c8d0c2503a"},"modality":"text","name":"Social IQa","openness":"unknown","publisher":null,"released":"2019-04-22","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: SocialIQA: Commonsense Reasoning about Social Interactions. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:social-iqa","source_url":"https://arxiv.org/abs/1904.09728"},"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":78.0,"display_multiplier":100,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.78,"raw_min":0.488,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:social-iqa:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/social-iqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-social-iqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/social-iqa?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_socialpersona_4370a8a3"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"SocialPersona evaluates multimodal LLMs on recovering revealed preferences from longitudinal social-media timelines and using them in dialogue. It includes 171 user timelines, 2,597 human-verified preference tags across seven domains, and supports profile construction and response generation tasks.","descriptions":{"en":"SocialPersona evaluates multimodal LLMs on recovering revealed preferences from longitudinal social-media timelines and using them in dialogue. It includes 171 user timelines, 2,597 human-verified preference tags across seven domains, and supports profile construction and response generation tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26654","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","visual-reasoning"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_socialpersona_4370a8a3"},"modality":null,"name":"SocialPersona","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26654","source_url":"https://arxiv.org/abs/2606.26654"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26654","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26654","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:co-minder/societybench-codebase","source_record":"bm_societybench_8a44f535"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SocietyBench is a benchmark for forecasting counterfactual social-world evolution, collecting web news and social-media posts to build timelines and generating forecasting questions scored on probability calibration and temporal accuracy.","descriptions":{"en":"SocietyBench is a benchmark for forecasting counterfactual social-world evolution, collecting web news and social-media posts to build timelines and generating forecasting questions scored on probability calibration and temporal accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04009","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_societybench_8a44f535"},"modality":null,"name":"SocietyBench","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04009","source_url":"https://arxiv.org/abs/2608.04009"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04009","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04009","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":20,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":"gh:genintel/omniprobe","source_record":"bm_soco_b681a0c6"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SOCO is a benchmark for semantic object correspondence with keypoint annotations across 100 categories and over 1M pairs. The evaluation is implemented within OmniProbe, which includes correspondence tasks among others.","descriptions":{"en":"SOCO is a benchmark for semantic object correspondence with keypoint annotations across 100 categories and over 1M pairs. The evaluation is implemented within OmniProbe, which includes correspondence tasks among others."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.31597","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_soco_b681a0c6"},"modality":null,"name":"SOCO","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31597","source_url":"https://arxiv.org/abs/2605.31597"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31597","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31597","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":56,"join":"claire_record_id","repository":"gh:disl-lab/socrates","source_record":"bm_socrates_dac2012d"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SoCRATES is a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It contains 600 conflict scenarios across eight domains, probing five socio-cognitive adaptation axes, with topic-localized scoring.","descriptions":{"en":"SoCRATES is a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It contains 600 conflict scenarios across eight domains, probing five socio-cognitive adaptation axes, with topic-localized scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.05563","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","chat-writing"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_socrates_dac2012d"},"modality":null,"name":"SoCRATES","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05563","source_url":"https://arxiv.org/abs/2606.05563"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05563","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05563","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SocSci-Repro-Bench evaluates AI coding agents on reproducing social science findings from 221 tasks across four disciplines and 13 domains, using studies with known reproducibility outcomes.","descriptions":{"en":"SocSci-Repro-Bench evaluates AI coding agents on reproducing social science findings from 221 tasks across four disciplines and 13 domains, using studies with known reproducibility outcomes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.11447","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","science"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_socsci-repro-bench_8a1a65b5"},"modality":null,"name":"SocSci-Repro-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11447","source_url":"https://arxiv.org/abs/2606.11447"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11447","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11447","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_softvtbench_b731832c"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"SoftVTBench is a safety-aware visuo-tactile benchmark for deformable object manipulation in Isaac Sim, with FEM-simulated objects, multi-view RGB, tactile sensing, proprioception, language instructions, and separate Goal Success and Safety Success metrics.","descriptions":{"en":"SoftVTBench is a safety-aware visuo-tactile benchmark for deformable object manipulation in Isaac Sim, with FEM-simulated objects, multi-view RGB, tactile sensing, proprioception, language instructions, and separate Goal Success and Safety Success metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.04234","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_softvtbench_b731832c"},"modality":null,"name":"SoftVTBench","openness":"unknown","publisher":null,"released":"2026-07-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04234","source_url":"https://arxiv.org/abs/2607.04234"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04234","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04234","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"claire_record_id","repository":null,"source_record":"bm_softvtbench-a-deformation-aware-visuo-tact_61935c97"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SoftVTBench is a visuo-tactile dataset and benchmark for deformable-object manipulation, providing expert demonstrations with synchronized sensor data and finite-element ground truth, with a closed-loop evaluation protocol.","descriptions":{"en":"SoftVTBench is a visuo-tactile dataset and benchmark for deformable-object manipulation, providing expert demonstrations with synchronized sensor data and finite-element ground truth, with a closed-loop evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.18701","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_softvtbench-a-deformation-aware-visuo-tact_61935c97"},"modality":null,"name":"SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation","openness":"unknown","publisher":null,"released":"2026-08-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.18701","source_url":"https://arxiv.org/abs/2608.18701"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-18701","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18701","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A physics-constrained benchmark for evaluating economic agents in decentralized energy markets, with a Gymnasium-compatible MDP and an LLM-based Planner/Auditor layer. It evaluates market utility, physical safety, slippage, action smoothness, spatial fairness, and auditability.","descriptions":{"en":"A physics-constrained benchmark for evaluating economic agents in decentralized energy markets, with a Gymnasium-compatible MDP and an LLM-based Planner/Auditor layer. It evaluates market utility, physical safety, slippage, action smoothness, spatial fairness, and auditability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.08681","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design","multi-agent-games"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_solarchain-eval_aac1b95f"},"modality":null,"name":"SolarChain-Eval","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08681","source_url":"https://arxiv.org/abs/2607.08681"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08681","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08681","topics":["agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"SolidityBench contains 5,470 repository-level Solidity smart contracts with natural language descriptions, plus SolidityScore, a semantic metric emphasizing domain-critical constructs. It evaluates code generation models.","descriptions":{"en":"SolidityBench contains 5,470 repository-level Solidity smart contracts with natural language descriptions, plus SolidityScore, a semantic metric emphasizing domain-critical constructs. It evaluates code generation models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19988","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_soliditybench_53feeae1"},"modality":null,"name":"SolidityBench","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19988","source_url":"https://arxiv.org/abs/2606.19988"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19988","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19988","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SoMBench evaluates social intelligence in large language models across 3 primary dimensions, 17 secondary dimensions, and 71 task paradigms, with 284 shared scenarios and 3,481 expert-verified instances.","descriptions":{"en":"SoMBench evaluates social intelligence in large language models across 3 primary dimensions, 17 secondary dimensions, and 71 task paradigms, with 284 shared scenarios and 3,481 expert-verified instances."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.23740","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_sombench_39ae8a74"},"modality":null,"name":"SoMBench","openness":"unknown","publisher":null,"released":"2026-07-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23740","source_url":"https://arxiv.org/abs/2607.23740"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23740","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23740","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_sopribench_12eedd2a"},"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"SopriBench evaluates user-level privacy leakage from social media posts across text, images, and metadata in 50 synthetic profiles with 1,569 images, scoring via the Privacy Exposure Score (PES) that weights value granularity by contextual sensitivity.","descriptions":{"en":"SopriBench evaluates user-level privacy leakage from social media posts across text, images, and metadata in 50 synthetic profiles with 1,569 images, scoring via the Privacy Exposure Score (PES) that weights value granularity by contextual sensitivity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06784","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_sopribench_12eedd2a"},"modality":null,"name":"SopriBench","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06784","source_url":"https://arxiv.org/abs/2606.06784"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06784","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06784","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:hosytuyen/soundnessbench","source_record":"bm_soundnessbench_20ec76aa"},"categories":["AI Scientist","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"SoundnessBench evaluates LLMs' ability to judge the methodological soundness of research proposals. It contains 1,099 machine-learning research proposals reconstructed from ICLR submissions, labeled with reviewer soundness sub-scores and audited against source papers.","descriptions":{"en":"SoundnessBench evaluates LLMs' ability to judge the methodological soundness of research proposals. It contains 1,099 machine-learning research proposals reconstructed from ICLR submissions, labeled with reviewer soundness sub-scores and audited against source papers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.30329","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_soundnessbench_20ec76aa"},"modality":null,"name":"SoundnessBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30329","source_url":"https://arxiv.org/abs/2605.30329"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30329","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30329","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"SovereignNegotiation-Bench evaluates personal agents in delegated bargaining scenarios, measuring agreement success alongside user utility, privacy, consent, evidence grounding, concession discipline, escalation, and auditability.","descriptions":{"en":"SovereignNegotiation-Bench evaluates personal agents in delegated bargaining scenarios, measuring agreement success alongside user utility, privacy, consent, evidence grounding, concession discipline, escalation, and auditability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02814","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","safety-alignment"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_sovereignnegotiation-bench_d54da144"},"modality":null,"name":"SovereignNegotiation-Bench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02814","source_url":"https://arxiv.org/abs/2607.02814"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02814","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02814","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SovereignPA-Bench evaluates user-owned personal agents on 120 sovereignty stress scenarios, measuring task success, alignment, privacy, consent, evidence, manipulation, burden, and auditability in evolving intent and platform mediation contexts.","descriptions":{"en":"SovereignPA-Bench evaluates user-owned personal agents on 120 sovereignty stress scenarios, measuring task success, alignment, privacy, consent, evidence, manipulation, burden, and auditability in evolving intent and platform mediation contexts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.05363","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","safety-alignment"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_sovereignpa-bench_ec65a032"},"modality":null,"name":"SovereignPA-Bench","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.05363","source_url":"https://arxiv.org/abs/2607.05363"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05363","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05363","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":279,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:tomtommyyuan/spmind","source_record":"bm_sp-bench_0f201b2f"},"categories":["Agents","Agents & Tool Use","Biology & Drug Discovery","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"SP-Mind is an autonomous AI agent for spatial proteomics analysis, and SP-Bench is introduced for evaluation with 102 tasks across 18 categories.","descriptions":{"en":"SP-Mind is an autonomous AI agent for spatial proteomics analysis, and SP-Bench is introduced for evaluation with 102 tasks across 18 categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.24235","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_sp-bench_0f201b2f"},"modality":null,"name":"SP-Bench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24235","source_url":"https://arxiv.org/abs/2606.24235"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24235","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24235","topics":["agents","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_sp-transientbench_52ec4ce5"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SP-TransientBench (STB) is a real-captured multi-task benchmark for single-photon perception. It comprises 10 diverse scenes and 10,297 views captured with a solid-state single-photon LiDAR at 256×192 resolution, providing full time-of-flight histograms with multi-return behavior, calibrated camera poses, and standardized metadata. The benchmark evaluates depth estimation, multi-view reconstruction, and 3D semantic understanding, with 13-class semantic annotations for selected scenes. Dedicated data splits and evaluation protocols are provided for each task.","descriptions":{"en":"SP-TransientBench (STB) is a real-captured multi-task benchmark for single-photon perception. It comprises 10 diverse scenes and 10,297 views captured with a solid-state single-photon LiDAR at 256×192 resolution, providing full time-of-flight histograms with multi-return behavior, calibrated camera poses, and standardized metadata. The benchmark evaluates depth estimation, multi-view reconstruction, and 3D semantic understanding, with 13-class semantic annotations for selected scenes. Dedicated data splits and evaluation protocols are provided for each task."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18952","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_sp-transientbench_52ec4ce5"},"modality":null,"name":"SP-TransientBench","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18952","source_url":"https://arxiv.org/abs/2606.18952"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18952","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18952","topics":[],"unit":null},{"aliases":["Human-Centric Image Captioning with Subject-Centered Spatial Understanding"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jhang2020/space-eval","source_record":"bm_human-centric-image-captioning-with-subjec_25bb01f1"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SPACE evaluates subject-centered spatial understanding in human-centric image captioning, focusing on egocentric laterality and anatomical-object bindings.","descriptions":{"en":"SPACE evaluates subject-centered spatial understanding in human-centric image captioning, focusing on egocentric laterality and anatomical-object bindings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.08300","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_human-centric-image-captioning-with-subjec_25bb01f1"},"modality":null,"name":"SPACE","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08300","source_url":"https://arxiv.org/abs/2609.08300"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08300","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08300","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":8,"github_stars_30d":null,"hf_dataset_downloads":85,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:zju-real/spacecast-bench","source_record":"bm_spacecast-bench_30f8c303"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input.","descriptions":{"en":"Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.12402","languages":[],"library_direction":{"basis":"model-review-daily","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_spacecast-bench_30f8c303"},"modality":null,"name":"SpaceCast-Bench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12402","source_url":"https://arxiv.org/abs/2610.12402"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12402","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12402","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["SPADE: SPaT Attack Detection from the Connected Vehicle's Perspective"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jdinovo/spade","source_record":"bm_spade-spat-attack-detection-from-the-conne_f3d623a1"},"categories":["Cybersecurity","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Provides a labeled, multimodal, simulation-based dataset for deep learning intrusion detection systems focused on SPaT attack detection from the connected vehicle's onboard perspective.","descriptions":{"en":"Provides a labeled, multimodal, simulation-based dataset for deep learning intrusion detection systems focused on SPaT attack detection from the connected vehicle's onboard perspective."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.02741","languages":[],"library_direction":null,"modality":null,"name":"SPADE","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02741","source_url":"https://arxiv.org/abs/2609.02741"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02741","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02741","topics":["safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for evaluating spontaneous plan-action divergence in agents, integrating actual tool execution and controlled pressure scenarios to distinguish strategic deception from hallucination.","descriptions":{"en":"Benchmark for evaluating spontaneous plan-action divergence in agents, integrating actual tool execution and controlled pressure scenarios to distinguish strategic deception from hallucination."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02380","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","agent-security"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_spade-bench_b8164972"},"modality":null,"name":"SPADE-Bench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02380","source_url":"https://arxiv.org/abs/2606.02380"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02380","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02380","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SpaPath-Bench evaluates pathology foundation models on spatial domain identification using paired whole slide images and spatial transcriptomics data from 42 public slides, measuring partition quality via unsupervised spatial coherence, transcriptomics-referenced agreement, and expert-referenced agreement.","descriptions":{"en":"SpaPath-Bench evaluates pathology foundation models on spatial domain identification using paired whole slide images and spatial transcriptomics data from 42 public slides, measuring partition quality via unsupervised spatial coherence, transcriptomics-referenced agreement, and expert-referenced agreement."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.25764","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","science"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_spapath-bench_cda2e02f"},"modality":null,"name":"SpaPath-Bench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25764","source_url":"https://arxiv.org/abs/2605.25764"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25764","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25764","topics":["medical","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates spatial reasoning in medical vision encoders via eight probes over multi-organ abdominal CT covering coordinate localization, relational reasoning, and spatial queries.","descriptions":{"en":"Evaluates spatial reasoning in medical vision encoders via eight probes over multi-organ abdominal CT covering coordinate localization, relational reasoning, and spatial queries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.28092","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","spatial-3d"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_spar-bench_b75502fb"},"modality":null,"name":"SPAR-Bench","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28092","source_url":"https://arxiv.org/abs/2608.28092"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28092","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28092","topics":["reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","理解","Understanding","3D Spatial Reasoning","Spatial Intelligence","物理智能","Embodied AI","逻辑推理","图像理解","Image Understanding","空间理解","Spatial Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Spatial457 systematically introduces four key capabilities—multi-object understanding, 2D and 3D localization, and 3D orientation—across five difficulty levels and seven question types, from simple object recognition to complex 6DoF spatial reasoning tasks. Spatial457 聚焦于空间推理的四项核心能力：多物体理解、二维位置识别、三维位置识别，以及三维朝向判断。这些能力对现实世界中复杂场景的认知和理解至关重要。我们构建了一种层级递进的评估结构，将问题划分为 7 类问题类型，覆盖从基础的单物体识别任务，到我们首次提出的更具挑战性的 6DoF 空间推理任务。整个数据集按 5 个难度等级组织，帮助系统性地评估模型在不同层次空间理解任务中的表现。\n\nSpatial457 不仅提升了空间推理评测的覆盖面和挑战性，也为后续研究提供了统一的基准，有助于推动多模态模型在真实世界空间认知任务中的发展。","descriptions":{"en":"Spatial457 systematically introduces four key capabilities—multi-object understanding, 2D and 3D localization, and 3D orientation—across five difficulty levels and seven question types, from simple object recognition to complex 6DoF spatial reasoning tasks.","zh":"Spatial457 聚焦于空间推理的四项核心能力：多物体理解、二维位置识别、三维位置识别，以及三维朝向判断。这些能力对现实世界中复杂场景的认知和理解至关重要。我们构建了一种层级递进的评估结构，将问题划分为 7 类问题类型，覆盖从基础的单物体识别任务，到我们首次提出的更具挑战性的 6DoF 空间推理任务。整个数据集按 5 个难度等级组织，帮助系统性地评估模型在不同层次空间理解任务中的表现。\n\nSpatial457 不仅提升了空间推理评测的覆盖面和挑战性，也为后续研究提供了统一的基准，有助于推动多模态模型在真实世界空间认知任务中的发展。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2092","languages":[],"library_direction":null,"modality":"multimodal","name":"Spatial457","openness":"open","publisher":"Johns Hopkins University","released":"2025-04-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2092-spatial457","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Spatial457","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":139,"github_stars_30d":9,"hf_dataset_downloads":null,"hf_paper_upvotes":69,"join":"claire_record_id","repository":"gh:ropedia/spatialbench","source_record":"bm_spatialbench_9dae04ac"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SpatialBench is a deterministic, density-aware benchmark for spatial foundation models, spanning 19 datasets, 546 scenes, and five spatial domains. It evaluates 41 models across six paradigms on five task suites—depth, camera pose, trajectory, point-cloud reconstruction, and long-sequence streaming—under four input density settings with precomputed and pinned test frames.","descriptions":{"en":"SpatialBench is a deterministic, density-aware benchmark for spatial foundation models, spanning 19 datasets, 546 scenes, and five spatial domains. It evaluates 41 models across six paradigms on five task suites—depth, camera pose, trajectory, point-cloud reconstruction, and long-sequence streaming—under four input density settings with precomputed and pinned test frames."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.27367","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_spatialbench_9dae04ac"},"modality":null,"name":"SpatialBench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27367","source_url":"https://arxiv.org/abs/2605.27367"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27367","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27367","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SpatialBench-Long evaluates AI agents on long-horizon spatial biology tasks, requiring recovery of biological claims from raw data across 24 evaluations.","descriptions":{"en":"SpatialBench-Long evaluates AI agents on long-horizon spatial biology tasks, requiring recovery of biological claims from raw data across 24 evaluations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28065","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_spatialbench-long_33cb2dcb"},"modality":null,"name":"SpatialBench-Long","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28065","source_url":"https://arxiv.org/abs/2605.28065"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28065","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28065","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":25,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":35,"join":"claire_record_id","repository":"gh:zju-omniai/provise","source_record":"bm_spatialgen-bench_3d213770"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ProVisE is a framework for evaluating image-generation models on spatial benchmarks by converting visual answers into structured predictions. SpatialGen-Bench is a diagnostic dataset of 470 samples across 14 spatial subtasks.","descriptions":{"en":"ProVisE is a framework for evaluating image-generation models on spatial benchmarks by converting visual answers into structured predictions. SpatialGen-Bench is a diagnostic dataset of 470 samples across 14 spatial subtasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.21072","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","spatial-3d"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_spatialgen-bench_3d213770"},"modality":null,"name":"SpatialGen-Bench","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.21072","source_url":"https://arxiv.org/abs/2607.21072"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-21072","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21072","topics":[],"unit":null},{"aliases":["SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Models"],"attention_signals":null,"categories":["General AI","Geometric reasoning","Multimodal","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark of over one million RGB-only question-answer pairs from 200 indoor scenes for closest-instance distance queries in vision-language models.","descriptions":{"en":"A benchmark of over one million RGB-only question-answer pairs from 200 indoor scenes for closest-instance distance queries in vision-language models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01709","languages":[],"library_direction":null,"modality":null,"name":"SPATIALQUERY-1M","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01709","source_url":"https://arxiv.org/abs/2608.01709"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01709","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01709","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":7,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:hyu-zhang/spatialuav","source_record":"bm_spatialuav_6900b974"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates spatial intelligence in low-altitude UAV scenarios across 14 task types covering semantic discrimination, spatial relations, collaboration, and motion understanding, with 7 input configurations and 9 answer formats.","descriptions":{"en":"Evaluates spatial intelligence in low-altitude UAV scenarios across 14 task types covering semantic discrimination, spatial relations, collaboration, and motion understanding, with 7 input configurations and 9 answer formats."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.27876","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_spatialuav_6900b974"},"modality":null,"name":"SpatialUAV","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27876","source_url":"https://arxiv.org/abs/2606.27876"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27876","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27876","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":54,"github_stars_30d":-1,"hf_dataset_downloads":null,"hf_paper_upvotes":50,"join":"claire_record_id","repository":"gh:hongcheng-gao/spatialworld","source_record":"bm_spatialworld_4f21023e"},"categories":["Agents & Tool Use","General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"SpatialWorld evaluates multimodal agents on interactive spatial reasoning in 760 real-world tasks across eight simulation backends. Agents operate under vision-only partial observability, using a unified text-based action interface. Performance is measured via terminal-state verifiers for task success rate and step efficiency.","descriptions":{"en":"SpatialWorld evaluates multimodal agents on interactive spatial reasoning in 760 real-world tasks across eight simulation backends. Agents operate under vision-only partial observability, using a unified text-based action interface. Performance is measured via terminal-state verifiers for task success rate and step efficiency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09669","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_spatialworld_4f21023e"},"modality":null,"name":"SpatialWorld","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09669","source_url":"https://arxiv.org/abs/2606.09669"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09669","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09669","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SPEARBench evaluates naturalness in streaming speech-to-speech language models via question-answer interactions, measuring latency, interruptions, speech quality, ASR robustness, language consistency, emotional naturalness, and interpersonal stance.","descriptions":{"en":"SPEARBench evaluates naturalness in streaming speech-to-speech language models via question-answer interactions, measuring latency, interruptions, speech quality, ASR robustness, language consistency, emotional naturalness, and interpersonal stance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.05365","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_spearbench_7a039de0"},"modality":null,"name":"SPEARBench","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.05365","source_url":"https://arxiv.org/abs/2607.05365"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05365","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05365","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Generating an executable program does not necessarily mean that it correctly implements the behavioral requirements specified in natural language.","descriptions":{"en":"Generating an executable program does not necessarily mean that it correctly implements the behavioral requirements specified in natural language."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.04253","languages":[],"library_direction":{"basis":"model-review-daily","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_spec2game_3570953e"},"modality":null,"name":"Spec2Game","openness":"unknown","publisher":null,"released":"2026-10-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.04253","source_url":"https://arxiv.org/abs/2610.04253"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-04253","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04253","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:edgahab/specread","source_record":"bm_specread_691c9e1f"},"categories":["Chip Design & EDA","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SpecRead evaluates specification comprehension via 383 questions over 10 OpenTitan IP blocks: exact retrieval, cross-section reasoning, contradiction detection in mutated specs, and spec-RTL consistency checking.","descriptions":{"en":"SpecRead evaluates specification comprehension via 383 questions over 10 OpenTitan IP blocks: exact retrieval, cross-section reasoning, contradiction detection in mutated specs, and spec-RTL consistency checking."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.33699","languages":[],"library_direction":{"basis":"model-review","categories":["engineering-design","long-context"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_specread_691c9e1f"},"modality":null,"name":"SpecRead","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33699","source_url":"https://arxiv.org/abs/2609.33699"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33699","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33699","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Speech & Audio","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SpeechDx evaluates clinical speech AI across 12 datasets and 27 tasks covering various conditions, structured by speech production stages. Scoring includes classification accuracy and zero-shot transfer performance.","descriptions":{"en":"SpeechDx evaluates clinical speech AI across 12 datasets and 27 tasks covering various conditions, structured by speech production stages. Scoring includes classification accuracy and zero-shot transfer performance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.17339","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","speech-audio"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_speechdx_58f3f420"},"modality":null,"name":"SpeechDx","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.17339","source_url":"https://arxiv.org/abs/2606.17339"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-17339","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17339","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Frontier LLM agents have been shown to be capable of solving increasingly complex tasks for which humans have measurable solutions.","descriptions":{"en":"Frontier LLM agents have been shown to be capable of solving increasingly complex tasks for which humans have measurable solutions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.08076","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agentic-coding","code-generation"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_speedrunbench_286eeca4"},"modality":null,"name":"SpeedrunBench","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.08076","source_url":"https://arxiv.org/abs/2610.08076"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-08076","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08076","topics":["coding","agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1101,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_9bfa0b50a90e6699"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A large-scale, complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students. Contains 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables, covering 138 different domains. Requires models to generalize to both new SQL queries and new database schemas, making it distinct from previous semantic parsing tasks that use single databases.","descriptions":{"en":"A large-scale, complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students. Contains 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables, covering 138 different domains. Requires models to generalize to both new SQL queries and new database schemas, making it distinct from previous semantic parsing tasks that use single databases."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:spider:codestral-22b","reported_at":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spider?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:spider","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_9bfa0b50a90e6699"},"modality":"text","name":"Spider","openness":"unknown","publisher":null,"released":"2018-09-24","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:spider","source_url":"https://arxiv.org/abs/1809.08887"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":63.5,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.635,"raw_min":0.311,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:spider:codestral-22b","reported_date":"2024-05-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spider?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-29","date_precision":"model_announcement","name":"Codestral-22B"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"}],"slug":"llm-stats-spider","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spider?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1101,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:1809.08887","repository":null,"source_record":"catalog_9bfa0b50a90e6699"},"categories":["智能体","Agent","大语言模型","LLM","代码工程","Code"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Spider is a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases. Spider 是一个大规模、复杂且跨领域的语义解析和文本到 SQL 数据集。Spider 挑战的目标是开发自然语言接口以访问跨领域数据库。该数据集包含 10,181 个问题和 5,693 个独特的复杂 SQL 查询，涵盖 200 个包含多个表的数据库，涉及 138 个不同的领域。","descriptions":{"en":"Spider is a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases.","zh":"Spider 是一个大规模、复杂且跨领域的语义解析和文本到 SQL 数据集。Spider 挑战的目标是开发自然语言接口以访问跨领域数据库。该数据集包含 10,181 个问题和 5,693 个独特的复杂 SQL 查询，涵盖 200 个包含多个表的数据库，涉及 138 个不同的领域。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1133","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"artifact:arxiv:1809.08887","primary":"code-generation","source_record":"catalog_9bfa0b50a90e6699"},"modality":null,"name":"Spider","openness":"unknown","publisher":"Department of Computer Science, Yale University","released":"2019-02-02","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1133-spider","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Spider","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:leolty/spider2-aifunc","source_record":"bm_spider-2-0-aifunc_d9873e24"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Extends text-to-SQL to AI-native SQL workflows with 465 instances across 125 real-world databases on Snowflake. Tasks require using AI functions like classification and sentiment analysis. Evaluates execution accuracy.","descriptions":{"en":"Extends text-to-SQL to AI-native SQL workflows with 465 instances across 125 real-world databases on Snowflake. Tasks require using AI functions like classification and sentiment analysis. Evaluates execution accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.06229","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","professional-work"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_spider-2-0-aifunc_d9873e24"},"modality":null,"name":"Spider 2.0-AIFunc","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06229","source_url":"https://arxiv.org/abs/2607.06229"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06229","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06229","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","NeurIPS 2024","任务执行","Task Execution","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Spider2-V is the first multimodal agent benchmark focusing on professional data science and engineering workflows, featuring 494 real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. Spider2-V是第一个专注于专业数据科学和工程工作流程的多模态代理基准测试，整合了20 个企业级专业应用程序，包含来自真实计算机环境的494 个真实任务。","descriptions":{"en":"Spider2-V is the first multimodal agent benchmark focusing on professional data science and engineering workflows, featuring 494 real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications.","zh":"Spider2-V是第一个专注于专业数据科学和工程工作流程的多模态代理基准测试，整合了20 个企业级专业应用程序，包含来自真实计算机环境的494 个真实任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1270","languages":[],"library_direction":null,"modality":null,"name":"Spider2-V","openness":"unknown","publisher":"The University of Hong Kong","released":"2024-07-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1270-spider2-v","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Spider2-V","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":106,"hf_paper_upvotes":12,"join":"claire_record_id","repository":null,"source_record":"bm_spieval_a21bc309"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"SPIEval is a human-curated benchmark for evaluating large language models as mobile assistants that retrieve and reason over personal information scattered across multiple apps. It comprises 250 tasks across five cognitive capabilities, 4,335 fictional personal records in 10 simulated apps, and supports multi-turn interaction through 21 tools. Models receive underspecified user instructions and must search records and invoke tools; final execution calls are compared to human-annotated gold calls at the parameter level.","descriptions":{"en":"SPIEval is a human-curated benchmark for evaluating large language models as mobile assistants that retrieve and reason over personal information scattered across multiple apps. It comprises 250 tasks across five cognitive capabilities, 4,335 fictional personal records in 10 simulated apps, and supports multi-turn interaction through 21 tools. Models receive underspecified user instructions and must search records and invoke tools; final execution calls are compared to human-annotated gold calls at the parameter level."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10692","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","agent-memory"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_spieval_a21bc309"},"modality":null,"name":"SPIEval","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10692","source_url":"https://arxiv.org/abs/2608.10692"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10692","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10692","topics":["computer-use","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:pku-alignment/spike-bench","source_record":"bm_spike-bench_3b80754c"},"categories":["Biology & Drug Discovery","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"SPIKE-Bench evaluates LLM biosecurity risks using 631 curated toxin-design prompts and a three-stage funnel (compliance, plausibility, predicted toxicity) producing the Functional Harmfulness Rate.","descriptions":{"en":"SPIKE-Bench evaluates LLM biosecurity risks using 631 curated toxin-design prompts and a three-stage funnel (compliance, plausibility, predicted toxicity) producing the Functional Harmfulness Rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.02684","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_spike-bench_3b80754c"},"modality":null,"name":"SPIKE-Bench","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02684","source_url":"https://arxiv.org/abs/2608.02684"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02684","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02684","topics":["safety","science"],"unit":null},{"aliases":["Measuring LLM Sycophancy under Sustained Multi-Turn Pressure"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SPINE evaluates LLM sycophancy by having an adaptive LLM proxy persistently challenge a target model across up to 25 turns on false-presupposition and unethical-query items, measuring collapse rates.","descriptions":{"en":"SPINE evaluates LLM sycophancy by having an adaptive LLM proxy persistently challenge a target model across up to 25 turns on false-presupposition and unethical-query items, measuring collapse rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09090","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_measuring-llm-sycophancy-under-sustained-m_c0976161"},"modality":null,"name":"SPINE","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09090","source_url":"https://arxiv.org/abs/2609.09090"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09090","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09090","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"SportMV-Bench evaluates multi-view sports video understanding with 1022 multi-view bundles and 3015 QA pairs across 10 sports, covering perception, rule-aware event interpretation, and adjudicative reasoning.","descriptions":{"en":"SportMV-Bench evaluates multi-view sports video understanding with 1022 multi-view bundles and 3015 QA pairs across 10 sports, covering perception, rule-aware event interpretation, and adjudicative reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.11844","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_sportmv-bench_a6dbc37e"},"modality":null,"name":"SportMV-Bench","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11844","source_url":"https://arxiv.org/abs/2607.11844"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11844","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11844","topics":["reasoning-math","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","NAACL 2024","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SportQA aims to evaluate LLMs in the context of sports understanding. SportQA encompasses over 70,000 multiple-choice questions across three distinct difficulty levels, each targeting different aspects of sports knowledge from basic historical facts to intricate, scenariobased reasoning tasks. SportQA 专门用于评估大型语言模型（LLMs）在体育理解方面的能力。SportQA 包含超过 70,000 道多项选择题，分为三个不同的难度级别，针对从基本历史事实到复杂情境推理任务的各种体育知识。","descriptions":{"en":"SportQA aims to evaluate LLMs in the context of sports understanding. SportQA encompasses over 70,000 multiple-choice questions across three distinct difficulty levels, each targeting different aspects of sports knowledge from basic historical facts to intricate, scenariobased reasoning tasks.","zh":"SportQA 专门用于评估大型语言模型（LLMs）在体育理解方面的能力。SportQA 包含超过 70,000 道多项选择题，分为三个不同的难度级别，针对从基本历史事实到复杂情境推理任务的各种体育知识。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1152","languages":[],"library_direction":null,"modality":null,"name":"SportQA","openness":"unknown","publisher":"University of California","released":"2024-06-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1152-sportqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SportQA","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A human-verified benchmark for grounded image difference captioning of long-term changes, with captions and spatial masks.","descriptions":{"en":"A human-verified benchmark for grounded image difference captioning of long-term changes, with captions and spatial masks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10356","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_spot-the-shift_1b43b422"},"modality":null,"name":"SPOT-THE-SHIFT","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10356","source_url":"https://arxiv.org/abs/2609.10356"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10356","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10356","topics":[],"unit":null},{"aliases":["Graph Reduction in Multirelational Networks: A Spreading-Oriented Reduction Benchmark"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.SI"],"collected_at":"2026-09-25T04:40:22Z","description":"An open-source framework for evaluating how graph sparsification and coarsening affect influence-maximisation performance on single-layer and multilayer real-world networks.","descriptions":{"en":"An open-source framework for evaluating how graph sparsification and coarsening affect influence-maximisation performance on single-layer and multilayer real-world networks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12581","languages":[],"library_direction":null,"modality":null,"name":"Spreading-Oriented Reduction Benchmark (SORB)","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12581","source_url":"https://arxiv.org/abs/2606.12581"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12581","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12581","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":41,"github_stars_30d":6,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:ruckbreasoning/spreadsheetbench-2","source_record":"bm_spreadsheetbench_a0b75616"},"categories":["Code & Software","Finance","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"SpreadsheetBench 2 evaluates spreadsheet agents on end-to-end business workflows across generation, debugging, and visualization tasks. It includes 321 tasks from authentic business data, with multi-sheet workbooks requiring cross-sheet reasoning. The benchmark provides a unified multi-turn agent scaffold and evaluation scripts for reproducibility.","descriptions":{"en":"SpreadsheetBench 2 evaluates spreadsheet agents on end-to-end business workflows across generation, debugging, and visualization tasks. It includes 321 tasks from authentic business data, with multi-sheet workbooks requiring cross-sheet reasoning. The benchmark provides a unified multi-turn agent scaffold and evaluation scripts for reproducibility."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.29955","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_spreadsheetbench_a0b75616"},"modality":null,"name":"SpreadsheetBench","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29955","source_url":"https://arxiv.org/abs/2606.29955"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29955","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29955","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","NeurIPS 2024","大语言模型","LLM","代码工程","Code","OCR与文档理解","OCR and Document Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SpreadsheetBench is a challenging spreadsheet manipulation benchmark built from 912 real questions gathered from online Excel forums. SpreadsheetBench是一个具有挑战性的电子表格操作基准测试，包含912个来自在线Excel论坛的真实问题","descriptions":{"en":"SpreadsheetBench is a challenging spreadsheet manipulation benchmark built from 912 real questions gathered from online Excel forums.","zh":"SpreadsheetBench是一个具有挑战性的电子表格操作基准测试，包含912个来自在线Excel论坛的真实问题"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1267","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"artifact:arxiv:2406.14991","primary":"professional-work","source_record":"catalog_5c8580698885b99c"},"modality":null,"name":"SpreadsheetBench","openness":"unknown","publisher":"Renmin University of China","released":"2024-06-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1267-spreadsheetbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SpreadsheetBench","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":41,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_6735aba5a7a45579"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["productivity","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SpreadsheetBench 2 evaluates office automation agents on spreadsheet analysis, reasoning, and manipulation tasks.","descriptions":{"en":"SpreadsheetBench 2 evaluates office automation agents on spreadsheet analysis, reasoning, and manipulation tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:spreadsheetbench-2:kimi-k3","reported_at":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:spreadsheetbench-2","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_6735aba5a7a45579"},"modality":"text","name":"SpreadsheetBench 2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":34.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.348,"raw_min":0.348,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:spreadsheetbench-2:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"}],"slug":"llm-stats-spreadsheetbench-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-2?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":208,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:ruckbreasoning/spreadsheetbench","source_record":"catalog_5c8580698885b99c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["productivity","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SpreadSheetBench-v1 evaluates office automation agents on spreadsheet reasoning and manipulation tasks, measuring the ability to analyze, transform, and operate on spreadsheet data through tools.","descriptions":{"en":"SpreadSheetBench-v1 evaluates office automation agents on spreadsheet reasoning and manipulation tasks, measuring the ability to analyze, transform, and operate on spreadsheet data through tools."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:spreadsheetbench-v1:qwen3.7-max","reported_at":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-v1?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:spreadsheetbench-v1","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_5c8580698885b99c"},"modality":"text","name":"SpreadSheetBench-v1","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":89.35,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.8935,"raw_min":0.863,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:spreadsheetbench-v1:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-v1?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-spreadsheetbench-v1","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-v1?top_n=500","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:shaqiu-ai/sqbench","source_record":"bm_sqbench_9c1d0337"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SQBench v1.0 evaluates language-model agents on 220 production-oriented tasks organized into L1 atomic capabilities, L2 composite skills, and L3 business scenarios. Tasks require processing input assets, using tools, and producing a specified deliverable. Scoring computes Completion, Risk Penalty, and Performance from a 10D Risk Matrix; Strict Pass requires Completion=1 and Risk Penalty=0.","descriptions":{"en":"SQBench v1.0 evaluates language-model agents on 220 production-oriented tasks organized into L1 atomic capabilities, L2 composite skills, and L3 business scenarios. Tasks require processing input assets, using tools, and producing a specified deliverable. Scoring computes Completion, Risk Penalty, and Performance from a 10D Risk Matrix; Strict Pass requires Completion=1 and Risk Penalty=0."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.23123","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_sqbench_9c1d0337"},"modality":null,"name":"SQBench","openness":"unknown","publisher":null,"released":"2026-07-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23123","source_url":"https://arxiv.org/abs/2607.23123"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23123","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23123","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_6dd05549572d185a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","summarization","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SQuALITY (Summarization-format QUestion Answering with Long Input Texts, Yes!) is a long-document summarization dataset built by hiring highly-qualified contractors to read public-domain short stories (3000-6000 words) and write original summaries from scratch. Each document has five summaries: one overview and four question-focused summaries. Designed to address limitations in existing summarization datasets by providing high-quality, faithful summaries.","descriptions":{"en":"SQuALITY (Summarization-format QUestion Answering with Long Input Texts, Yes!) is a long-document summarization dataset built by hiring highly-qualified contractors to read public-domain short stories (3000-6000 words) and write original summaries from scratch. Each document has five summaries: one overview and four question-focused summaries. Designed to address limitations in existing summarization datasets by providing high-quality, faithful summaries."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:squality:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/squality?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:squality","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","chat-writing"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_6dd05549572d185a"},"modality":"text","name":"SQuALITY","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":24.3,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.243,"raw_min":0.188,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:squality:phi-3.5-mini-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/squality?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-squality","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/squality?top_n=500","topics":["long-context","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture fine-grained alignment and generative artifacts.","descriptions":{"en":"With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture fine-grained alignment and generative artifacts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37576","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_square-bench_2706ee65"},"modality":null,"name":"SQUARE-Bench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37576","source_url":"https://arxiv.org/abs/2609.37576"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37576","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37576","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"AI systems are making rapid progress in mathematics, but this progress is hard to measure.","descriptions":{"en":"AI systems are making rapid progress in mathematics, but this progress is hard to measure."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13461","languages":[],"library_direction":{"basis":"model-review-daily","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_squeezebench_b66f2ae5"},"modality":null,"name":"SqueezeBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13461","source_url":"https://arxiv.org/abs/2610.13461"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13461","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13461","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_sre-bench_d394a59b"},"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"SRE-Bench evaluates AI agents on reverse engineering of binaries compiled from 19 private real-world-scale C programs (average 16.9K LoC) with 44 anti-analysis primitives, yielding 262 binary instances and 1572 deterministically graded tasks.","descriptions":{"en":"SRE-Bench evaluates AI agents on reverse engineering of binaries compiled from 19 private real-world-scale C programs (average 16.9K LoC) with 44 anti-analysis primitives, yielding 262 binary instances and 1572 deterministically graded tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.11469","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_sre-bench_d394a59b"},"modality":null,"name":"SRE-Bench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11469","source_url":"https://arxiv.org/abs/2608.11469"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11469","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11469","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmarks for site reliability engineering (SRE) agents are typically episodic: one fault is injected, the agent receives an incident task, and its response is scored.","descriptions":{"en":"Benchmarks for site reliability engineering (SRE) agents are typically episodic: one fault is injected, the agent receives an incident task, and its response is scored."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33023","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","professional-work"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_sre-marathon_f57d6b9e"},"modality":null,"name":"SRE-Marathon","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33023","source_url":"https://arxiv.org/abs/2609.33023"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33023","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33023","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:gtc-gh/ssmnbench","source_record":"bm_ssmnbench_555dd856"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SSMNBench is a diagnostic benchmark for cross-view human and human-object understanding, comprising 3,300 QA pairs categorized into Single-View Sufficiency (SVS) and Multi-View Necessity (MVN) tasks. It evaluates multimodal LLMs by perturbing view availability to assess distraction robustness and cross-view evidence integration.","descriptions":{"en":"SSMNBench is a diagnostic benchmark for cross-view human and human-object understanding, comprising 3,300 QA pairs categorized into Single-View Sufficiency (SVS) and Multi-View Necessity (MVN) tasks. It evaluates multimodal LLMs by perturbing view availability to assess distraction robustness and cross-view evidence integration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.25634","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_ssmnbench_555dd856"},"modality":null,"name":"SSMNBench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25634","source_url":"https://arxiv.org/abs/2606.25634"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25634","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25634","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Autonomous Driving","Robustness","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ST-Bench is a verification benchmark for evaluating certified robustness of spatio-temporal neural networks on autonomous driving (Udacity) and activity recognition (UCF-101) tasks, using spatio-temporal perturbation constraints.","descriptions":{"en":"ST-Bench is a verification benchmark for evaluating certified robustness of spatio-temporal neural networks on autonomous driving (Udacity) and activity recognition (UCF-101) tasks, using spatio-temporal perturbation constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09746","languages":[],"library_direction":null,"modality":null,"name":"ST-Bench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09746","source_url":"https://arxiv.org/abs/2606.09746"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09746","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09746","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"The rapid progress of LLM-based multi-agent systems (MAS) has shown that they largely outperform single agents on coding, math, and QA tasks, where executable tests provide a binary success signal.","descriptions":{"en":"The rapid progress of LLM-based multi-agent systems (MAS) has shown that they largely outperform single agents on coding, math, and QA tasks, where executable tests provide a binary success signal."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.07763","languages":[],"library_direction":{"basis":"model-review-daily","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_st-bench_8eb5b4f3"},"modality":null,"name":"ST-Bench","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.07763","source_url":"https://arxiv.org/abs/2610.07763"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-07763","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07763","topics":["agents"],"unit":null},{"aliases":["Evidence-Backed Video Question Answering"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"ST-Evidence evaluates video question answering that must return both semantic answers and dense spatio-temporal visual evidence.","descriptions":{"en":"ST-Evidence evaluates video question answering that must return both semantic answers and dense spatio-temporal visual evidence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.11862","languages":[],"library_direction":null,"modality":null,"name":"ST-Evidence","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11862","source_url":"https://arxiv.org/abs/2607.11862"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11862","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11862","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"StabilityBench is a benchmark operator that converts single-turn benchmark queries into multi-turn interaction histories with injected user simulations, evaluating LLM performance stability across demographic proxies and sycophantic baits on existing benchmarks.","descriptions":{"en":"StabilityBench is a benchmark operator that converts single-turn benchmark queries into multi-turn interaction histories with injected user simulations, evaluating LLM performance stability across demographic proxies and sycophantic baits on existing benchmarks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20558","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_stabilitybench_c271b529"},"modality":null,"name":"StabilityBench","openness":"unknown","publisher":null,"released":"2026-07-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20558","source_url":"https://arxiv.org/abs/2607.20558"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20558","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20558","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","ACL 2024","大语言模型","LLM","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. StableToolBench 是一个从 ToolBench 发展而来的基准，提出了一个虚拟 API 服务器和稳定的评估系统。虚拟 API 服务器包含一个缓存系统和 API 模拟器，这些组件相辅相成，以缓解 API 状态变化带来的影响。","descriptions":{"en":"StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status.","zh":"StableToolBench 是一个从 ToolBench 发展而来的基准，提出了一个虚拟 API 服务器和稳定的评估系统。虚拟 API 服务器包含一个缓存系统和 API 模拟器，这些组件相辅相成，以缓解 API 状态变化带来的影响。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1084","languages":[],"library_direction":null,"modality":null,"name":"StableToolBench","openness":"unknown","publisher":"THUNLP-MT","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1084-stabletoolbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StableToolBench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_stage-claw_61b30974"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"STAGE-Claw is an automated framework for building and evaluating personal-agent tasks in state-based computing environments, with a benchmark of 40 tasks. Evaluations measure final system state correctness.","descriptions":{"en":"STAGE-Claw is an automated framework for building and evaluating personal-agent tasks in state-based computing environments, with a benchmark of 40 tasks. Evaluations measure final system state correctness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.10394","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_stage-claw_61b30974"},"modality":null,"name":"STAGE-Claw","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10394","source_url":"https://arxiv.org/abs/2606.10394"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10394","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10394","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Cybersecurity","Finance","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"StakeBench is a stakeholder-centric benchmark for prompt-injection attacks in web agents for online shopping. It decomposes risk into 12 attack objectives across three stakeholder classes, with 264 adversarial cases across 12 product categories.","descriptions":{"en":"StakeBench is a stakeholder-centric benchmark for prompt-injection attacks in web agents for online shopping. It decomposes risk into 12 attack objectives across three stakeholder classes, with 264 adversarial cases across 12 product categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13385","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_stakebench_be4543bc"},"modality":null,"name":"StakeBench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13385","source_url":"https://arxiv.org/abs/2606.13385"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13385","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13385","topics":["agents","safety","cybersecurity","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:amazon-science/staminabench","source_record":"bm_staminabench_cb49af23"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"StaminaBench stress-tests coding agents over 100 interaction turns of change requests, measuring how many consecutive turns they handle before failing, with programmatically generated tasks and black-box HTTP evaluation.","descriptions":{"en":"StaminaBench stress-tests coding agents over 100 interaction turns of change requests, measuring how many consecutive turns they handle before failing, with programmatically generated tasks and black-box HTTP evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.19613","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_staminabench_cb49af23"},"modality":null,"name":"StaminaBench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19613","source_url":"https://arxiv.org/abs/2606.19613"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19613","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19613","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"StanceBench evaluates interpersonal stance in conversational speech across 9 dimensions using LLM-as-a-judge on the Seamless Interaction corpus.","descriptions":{"en":"StanceBench evaluates interpersonal stance in conversational speech across 9 dimensions using LLM-as-a-judge on the Seamless Interaction corpus."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.22658","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","chat-writing"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_stancebench_98c43cc6"},"modality":null,"name":"StanceBench","openness":"unknown","publisher":null,"released":"2026-06-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.22658","source_url":"https://arxiv.org/abs/2607.22658"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-22658","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22658","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"StanceFlip is a benchmark for multimodal conversational stance flipping forecasting with two subtasks: sextuple extraction and flip attribution.","descriptions":{"en":"StanceFlip is a benchmark for multimodal conversational stance flipping forecasting with two subtasks: sextuple extraction and flip attribution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.24191","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","forecasting"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_stanceflip_415aae19"},"modality":null,"name":"StanceFlip","openness":"unknown","publisher":null,"released":"2026-07-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.24191","source_url":"https://arxiv.org/abs/2607.24191"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-24191","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24191","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"STARK is a comprehensive benchmark designed to systematically evaluate large language models (LLMs) and large reasoning models (LRMs) on spatiotemporal reasoning tasks, particularly for applications in cyber-physical systems (CPS) such as robotics, autonomous vehicles, and smart city infrastructure. STARK 是一个全面的基准测试套件，旨在系统评估大语言模型（LLMs）和大推理模型（LRMs）在时空推理任务中的表现，特别是在网络物理系统（CPS）中的应用，如机器人、自动驾驶和智能城市基础设施。该基准包含 26 种不同的时空任务，涵盖状态估计、时空关系推理和世界知识感知推理三个层次。","descriptions":{"en":"STARK is a comprehensive benchmark designed to systematically evaluate large language models (LLMs) and large reasoning models (LRMs) on spatiotemporal reasoning tasks, particularly for applications in cyber-physical systems (CPS) such as robotics, autonomous vehicles, and smart city infrastructure.","zh":"STARK 是一个全面的基准测试套件，旨在系统评估大语言模型（LLMs）和大推理模型（LRMs）在时空推理任务中的表现，特别是在网络物理系统（CPS）中的应用，如机器人、自动驾驶和智能城市基础设施。该基准包含 26 种不同的时空任务，涵盖状态估计、时空关系推理和世界知识感知推理三个层次。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1845","languages":[],"library_direction":null,"modality":"multimodal","name":"STARK_10k","openness":"unknown","publisher":"Department of Electrical and Computer Engineering, UCLA","released":"2025-05-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1845-stark-10k","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/STARK_10k","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":138,"join":"claire_record_id","repository":null,"source_record":"bm_startupbench_c9f7419c"},"categories":["Agents","Agents & Tool Use","Complex instruction following","Finance","General AI","Long-horizon task execution","Professional artifact generation","Scientific Research & AI for Science","Tool use"],"collected_at":"2026-10-14T09:44:51Z","description":"StartupBench is a benchmark for general-purpose agents on end-to-end workflows derived from market-validated AI startup products, with deliverable-oriented tasks and fine-grained rubrics.","descriptions":{"en":"StartupBench is a benchmark for general-purpose agents on end-to-end workflows derived from market-validated AI startup products, with deliverable-oriented tasks and fine-grained rubrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.17800","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_startupbench_c9f7419c"},"modality":null,"name":"StartupBench","openness":"unknown","publisher":null,"released":"2026-08-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.17800","source_url":"https://arxiv.org/abs/2608.17800"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-17800","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17800","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"StatABench evaluates LLMs' statistical analysis capabilities through two components: Stat-Closed, 404 questions across 18 topics in multiple formats, and Stat-Open, 30 complex modeling tasks from competitions, scored via LLM-as-judge.","descriptions":{"en":"StatABench evaluates LLMs' statistical analysis capabilities through two components: Stat-Closed, 404 questions across 18 topics in multiple formats, and Stat-Open, 30 complex modeling tasks from competitions, scored via LLM-as-judge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.22977","languages":[],"library_direction":{"basis":"model-review","categories":["math","professional-work"],"join":"claire_record_id","primary":"math","source_record":"bm_statabench_6a243141"},"modality":null,"name":"StatABench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22977","source_url":"https://arxiv.org/abs/2606.22977"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22977","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22977","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":33,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:amap-ml/stateagent","source_record":"bm_statebench_8e2244ba"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates world-state reasoning in video continuation through 200 cross-segment tasks across past-visible, occluded-process, and complex-transition state categories, with scoring based on state equivalence, state correctness, and hallucination rate.","descriptions":{"en":"Evaluates world-state reasoning in video continuation through 200 cross-segment tasks across past-visible, occluded-process, and complex-transition state categories, with scoring based on state equivalence, state correctness, and hallucination rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.03673","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","video-understanding"],"join":"claire_record_id","primary":"world-models","source_record":"bm_statebench_8e2244ba"},"modality":null,"name":"Statebench","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.03673","source_url":"https://arxiv.org/abs/2609.03673"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-03673","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03673","topics":["reasoning-math"],"unit":null},{"aliases":["nsr-product-use-benchmark"],"attention_signals":{"as_of":"2026-10-12","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:stateset/nsr-product-use-benchmark","source_record":"bm_nsr-product-use-benchmark_381f8927"},"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Eleven synthetic cases evaluate how an agent uses StateSet NSR MCP tools before a consequential action, with a harness scoring tool calls and a final structured JSON for passed cases and unsafe actions.","descriptions":{"en":"Eleven synthetic cases evaluate how an agent uses StateSet NSR MCP tools before a consequential action, with a harness scoring tool calls and a final structured JSON for passed cases and unsafe actions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:stateset/nsr-product-use-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","agent-security"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_nsr-product-use-benchmark_381f8927"},"modality":null,"name":"StateSet NSR Product Use Benchmark v1","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:stateset/nsr-product-use-benchmark","source_url":"https://github.com/stateset/nsr-product-use-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-stateset-nsr-product-use-benchmark","source":"claire_radar","source_url":"https://github.com/stateset/nsr-product-use-benchmark","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:thu-conglab/statformbench","source_record":"bm_statformbench_9b48c2ce"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates language models on statistical problem formulation tasks drawn from five statistics textbooks and a data science case library, covering diverse problem types, data representations, and scenario styles.","descriptions":{"en":"Evaluates language models on statistical problem formulation tasks drawn from five statistics textbooks and a data science case library, covering diverse problem types, data representations, and scenario styles."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.01982","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_statformbench_9b48c2ce"},"modality":null,"name":"StatFormBench","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01982","source_url":"https://arxiv.org/abs/2609.01982"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01982","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01982","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of six Ising-type problems for evaluating LLM agents' ability to discover statistical mechanical mappings from raw partition functions.","descriptions":{"en":"A benchmark of six Ising-type problems for evaluating LLM agents' ability to discover statistical mechanical mappings from raw partition functions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.26367","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_statmechbench-v0_d1047ef0"},"modality":null,"name":"StatMechBench-v0","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26367","source_url":"https://arxiv.org/abs/2607.26367"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26367","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26367","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":11,"github_stars_30d":3,"hf_dataset_downloads":48,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:ganggreentempertatum/stealthbench","source_record":"bm_stealthbench_8fce6971"},"categories":["Agents & Tool Use","General AI","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"StealthBench measures operational stealth of autonomous offensive-security agents across six OPSEC dimensions, using 14 dockerized task scenarios and a three-model judge panel, with metrics like safe success rate and Stealth@Solve.","descriptions":{"en":"StealthBench measures operational stealth of autonomous offensive-security agents across six OPSEC dimensions, using 14 dockerized task scenarios and a three-model judge panel, with metrics like safe success rate and Stealth@Solve."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.26314","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_stealthbench_8fce6971"},"modality":null,"name":"StealthBench","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26314","source_url":"https://arxiv.org/abs/2607.26314"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26314","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26314","topics":["agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:qbalhkim/steam-review-sentiment","source_record":"bm_steam-review-sentiment_d9a5651e"},"categories":["Chip Design & EDA","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates sentence-level sentiment classification on a balanced Steam review test set across six models, plus aspect-based sentiment extraction over eight game-review aspects using spaCy, VADER and a DeBERTa ABSA model.","descriptions":{"en":"Evaluates sentence-level sentiment classification on a balanced Steam review test set across six models, plus aspect-based sentiment extraction over eight game-review aspects using spaCy, VADER and a DeBERTa ABSA model."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:qbalhkim/steam-review-sentiment","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_steam-review-sentiment_d9a5651e"},"modality":null,"name":"steam-review-sentiment","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:qbalhkim/steam-review-sentiment","source_url":"https://github.com/qbalhkim/steam-review-sentiment"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-qbalhkim-steam-review-sentiment","source":"claire_radar","source_url":"https://github.com/qbalhkim/steam-review-sentiment","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"STEELBENCH evaluates vision-language models on per-worker activity recognition and safety-rule reasoning in industrial CCTV footage. It includes 1,345 clips with dense annotations and a provenance-aware audit protocol.","descriptions":{"en":"STEELBENCH evaluates vision-language models on per-worker activity recognition and safety-rule reasoning in industrial CCTV footage. It includes 1,345 clips with dense annotations and a provenance-aware audit protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.05264","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_steelbench_6bf9f31b"},"modality":null,"name":"SteelBench","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.05264","source_url":"https://arxiv.org/abs/2607.05264"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05264","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05264","topics":["multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SteerBench evaluates full-duplex spoken steerability using 390 spoken prompts and 1,067 human-authored binary audio and text rubrics spanning tone, persona, style or accent, and speed or length.","descriptions":{"en":"SteerBench evaluates full-duplex spoken steerability using 390 spoken prompts and 1,067 human-authored binary audio and text rubrics spanning tone, persona, style or accent, and speed or length."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.12623","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio","chat-writing"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_steerbench_3e1ad022"},"modality":null,"name":"SteerBench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12623","source_url":"https://arxiv.org/abs/2609.12623"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12623","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12623","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_steerbench-work_754206df"},"categories":["Agents","Biology & Drug Discovery","Finance","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"SteerBench-Work evaluates agent steering decisions at action boundaries in workplace scenarios across seven domains, with 106 incident-anchored scenarios and evidence-reversed mirrors, scored on correct proceed/hold boundaries.","descriptions":{"en":"SteerBench-Work evaluates agent steering decisions at action boundaries in workplace scenarios across seven domains, with 106 incident-anchored scenarios and evidence-reversed mirrors, scored on correct proceed/hold boundaries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12654","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","professional-work"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_steerbench-work_754206df"},"modality":null,"name":"SteerBench-Work","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12654","source_url":"https://arxiv.org/abs/2608.12654"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12654","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12654","topics":["agents","safety","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":5,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_919a11f948095f31"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive multimodal benchmark dataset with 448 skills and 1,073,146 questions spanning all STEM subjects (Science, Technology, Engineering, Mathematics), designed to test neural models' vision-language STEM skills based on K-12 curriculum. Unlike existing datasets that focus on expert-level ability, this dataset includes fundamental skills designed around educational standards.","descriptions":{"en":"A comprehensive multimodal benchmark dataset with 448 skills and 1,073,146 questions spanning all STEM subjects (Science, Technology, Engineering, Mathematics), designed to test neural models' vision-language STEM skills based on K-12 curriculum. Unlike existing datasets that focus on expert-level ability, this dataset includes fundamental skills designed around educational standards."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:stem:qwen-2.5-coder-7b-instruct","reported_at":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/stem?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:stem","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","knowledge-expert-qa"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_919a11f948095f31"},"modality":"multimodal","name":"STEM","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":34.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.34,"raw_min":0.34,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:stem:qwen-2.5-coder-7b-instruct","reported_date":"2024-09-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/stem?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-stem","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/stem?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:hexixiang/stembind","source_record":"bm_stembind_bdce22aa"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"StemBind is a diagnostic benchmark with shared-stem questions to attribute failures in abstract visual reasoning. It includes perception, rule, and full tasks with stage annotations but no official code or dataset release.","descriptions":{"en":"StemBind is a diagnostic benchmark with shared-stem questions to attribute failures in abstract visual reasoning. It includes perception, rule, and full tasks with stage annotations but no official code or dataset release."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.00148","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning","visual-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_stembind_bdce22aa"},"modality":null,"name":"StemBind","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00148","source_url":"https://arxiv.org/abs/2606.00148"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00148","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00148","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"STEMGym is an open-source Gymnasium benchmark of 15 physics-simulated STEM worlds across five materials and four characterization tasks, scored by DEC-AUC.","descriptions":{"en":"STEMGym is an open-source Gymnasium benchmark of 15 physics-simulated STEM worlds across five materials and four characterization tasks, scored by DEC-AUC."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.29592","languages":[],"library_direction":{"basis":"model-review","categories":["science","ai-rnd"],"join":"claire_record_id","primary":"science","source_record":"bm_stemgym_d150c717"},"modality":null,"name":"STEMGym","openness":"unknown","publisher":null,"released":"2026-06-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29592","source_url":"https://arxiv.org/abs/2606.29592"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29592","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29592","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":5,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:borealisai/stepjack","source_record":"bm_stepjack_2287a7d0"},"categories":["Agents","General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"StepJack evaluates computer-use agents against multi-step indirect prompt injection attacks, where adversarial goals are decomposed into innocuous sub-steps across a chain of pages. It comprises 480 test examples across platforms and instruction types, with attack success rate as the primary metric.","descriptions":{"en":"StepJack evaluates computer-use agents against multi-step indirect prompt injection attacks, where adversarial goals are decomposed into innocuous sub-steps across a chain of pages. It comprises 480 test examples across platforms and instruction types, with attack success rate as the primary metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.06477","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","computer-use"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_stepjack_2287a7d0"},"modality":null,"name":"StepJack","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06477","source_url":"https://arxiv.org/abs/2608.06477"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06477","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06477","topics":["computer-use","agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":51865,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_stereogenbench_78fe1ed3"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"StereoGenBench is a synthetic multi-camera benchmark for stereo generation, rendered in Unreal Engine with a rigid six-camera array. It provides calibrated view pairs across baseline regimes, along with RGB, metric depth, intrinsics, and poses for each scene.","descriptions":{"en":"StereoGenBench is a synthetic multi-camera benchmark for stereo generation, rendered in Unreal Engine with a rigid six-camera array. It provides calibrated view pairs across baseline regimes, along with RGB, metric depth, intrinsics, and poses for each scene."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.23237","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","spatial-3d"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_stereogenbench_78fe1ed3"},"modality":null,"name":"StereoGenBench","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.23237","source_url":"https://arxiv.org/abs/2605.23237"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-23237","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23237","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tum-pbs/stocbench","source_record":"bm_stocbench_915b6314"},"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"StocBench evaluates generative models for probabilistic forecasting of stochastic fluid flows, using a two-dimensional Kolmogorov flow with stochastic forcing, measuring one-step distributional accuracy and preservation of the enstrophy spectrum during autoregressive rollouts.","descriptions":{"en":"StocBench evaluates generative models for probabilistic forecasting of stochastic fluid flows, using a two-dimensional Kolmogorov flow with stochastic forcing, measuring one-step distributional accuracy and preservation of the enstrophy spectrum during autoregressive rollouts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.22309","languages":[],"library_direction":{"basis":"model-review","categories":["science","world-models"],"join":"claire_record_id","primary":"science","source_record":"bm_stocbench_915b6314"},"modality":null,"name":"StocBench","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22309","source_url":"http://arxiv.org/abs/2608.22309v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22309","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22309v1","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A Lean 4 benchmark of 450 graduate stochastic-processes problems at varying abstraction levels, each paired with its natural-language source.","descriptions":{"en":"A Lean 4 benchmark of 450 graduate stochastic-processes problems at varying abstraction levels, each paired with its natural-language source."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09264","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_stochbench_a640a158"},"modality":null,"name":"StochBench","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09264","source_url":"https://arxiv.org/abs/2609.09264"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09264","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09264","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","General AI","Planning"],"collected_at":"2026-10-14T09:44:51Z","description":"Reinforcement learning environments are now a primary lever for improving large language model (LLM) capabilities in post-training, yet most agentic benchmarks remain static: the world moves only when the agent acts, the reward is a termin…","descriptions":{"en":"Reinforcement learning environments are now a primary lever for improving large language model (LLM) capabilities in post-training, yet most agentic benchmarks remain static: the world moves only when the agent acts, the reward is a termin…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10942","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agentic-coding","tool-use"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_storebench_45170794"},"modality":null,"name":"StoreBench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10942","source_url":"https://arxiv.org/abs/2610.10942"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10942","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10942","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:siruzhong/storm-bench","source_record":"bm_storm-bench_c13964d5"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Presents a benchmark comprising 5,736 questions across 630 episodes for online video QA, but no artifact evidence is available.","descriptions":{"en":"Presents a benchmark comprising 5,736 questions across 630 episodes for online video QA, but no artifact evidence is available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.30981","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_storm-bench_c13964d5"},"modality":null,"name":"STORM-Bench","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30981","source_url":"https://arxiv.org/abs/2609.30981"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30981","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30981","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"STORYLENSBENCH benchmarks preference-aligned story rewriting with structured story books and reader profiles, plus reward model and rewriting model.","descriptions":{"en":"STORYLENSBENCH benchmarks preference-aligned story rewriting with structured story books and reader profiles, plus reward model and rewriting model."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28073","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_storylensbench_8f70dc12"},"modality":null,"name":"STORYLENSBENCH","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28073","source_url":"https://arxiv.org/abs/2605.28073"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28073","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28073","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"StrAD is a benchmark for long-form audio description generation on full-length videos across diverse genres, reformulating the task as streaming dense video captioning without ground-truth timestamps.","descriptions":{"en":"StrAD is a benchmark for long-form audio description generation on full-length videos across diverse genres, reformulating the task as streaming dense video captioning without ground-truth timestamps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.12549","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_strad_bb4cb514"},"modality":null,"name":"StrAD","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12549","source_url":"https://arxiv.org/abs/2608.12549"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12549","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12549","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:movahedi-ca/strata-bench","source_record":"bm_strata-bench_1446653d"},"categories":["General AI","Information retrieval","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents on compiling longitudinal market intelligence from fragmented, geographically nested, and temporally incomplete evidence, with scoring across temporal, source, spatial, statistical, visual, and narrative dimensions.","descriptions":{"en":"Evaluates AI agents on compiling longitudinal market intelligence from fragmented, geographically nested, and temporally incomplete evidence, with scoring across temporal, source, spatial, statistical, visual, and narrative dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:movahedi-ca/strata-bench","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","professional-work"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_strata-bench_1446653d"},"modality":null,"name":"STRATA-Bench","openness":"unknown","publisher":null,"released":"2026-09-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:movahedi-ca/strata-bench","source_url":"https://github.com/movahedi-ca/strata-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-movahedi-ca-strata-bench","source":"claire_radar","source_url":"https://github.com/movahedi-ca/strata-bench","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:derrtaderr/strategy-benchmark","source_record":"bm_strategy-benchmark_e4992c58"},"categories":["Language & Knowledge","Logistics & Operations","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates strategy documents by measuring claim survival rate under adversarial scrutiny, comparing gated-evidence and naive LLM-generated strategies.","descriptions":{"en":"Evaluates strategy documents by measuring claim survival rate under adversarial scrutiny, comparing gated-evidence and naive LLM-generated strategies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:derrtaderr/strategy-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_strategy-benchmark_e4992c58"},"modality":null,"name":"strategy-benchmark","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:derrtaderr/strategy-benchmark","source_url":"https://github.com/derrtaderr/strategy-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-derrtaderr-strategy-benchmark","source":"claire_radar","source_url":"https://github.com/derrtaderr/strategy-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"StrategyBench is a benchmark for evaluating explicit strategy induction in large language models. It selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility.","descriptions":{"en":"StrategyBench is a benchmark for evaluating explicit strategy induction in large language models. It selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23475","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_strategybench_2da1f1e5"},"modality":null,"name":"StrategyBench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23475","source_url":"http://arxiv.org/abs/2608.23475v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23475","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23475v1","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","大语言模型","LLM","知识储备","逻辑推理","Reasoning","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"STRATEGYQA is a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy STRATEGYQA 是一个问答基准，其中所需的推理步骤在问题中是隐含的，可以通过策略进行推断。","descriptions":{"en":"STRATEGYQA is a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy","zh":"STRATEGYQA 是一个问答基准，其中所需的推理步骤在问题中是隐含的，可以通过策略进行推断。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1107","languages":[],"library_direction":null,"modality":null,"name":"StrategyQA","openness":"unknown","publisher":"Allen Institute for AI","released":"2021-01-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1107-strategyqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StrategyQA","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":37,"github_stars_30d":5,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"claire_record_id","repository":"gh:jia-lab-research/streamarena","source_record":"bm_streamarena_4967785a"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"StreamArena evaluates hour-scale streaming video understanding across 243 full-length videos (avg 88.8 min) with 3,646 open-ended QA pairs, covering real-time perception, historical retrospection, proactive interaction, and multimodal tool use. Includes a standardized runner and LLM-as-judge scorer.","descriptions":{"en":"StreamArena evaluates hour-scale streaming video understanding across 243 full-length videos (avg 88.8 min) with 3,646 open-ended QA pairs, covering real-time perception, historical retrospection, proactive interaction, and multimodal tool use. Includes a standardized runner and LLM-as-judge scorer."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.05703","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_streamarena_4967785a"},"modality":null,"name":"StreamArena","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05703","source_url":"https://arxiv.org/abs/2608.05703"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05703","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05703","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":23,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:landian60/streammembench","source_record":"bm_streammembench_6345b534"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"StreamMemBench is a streaming benchmark for evaluating agent memory with two-step task sequences around evidence anchors from EgoLife egocentric streams, with four diagnostic metrics.","descriptions":{"en":"StreamMemBench is a streaming benchmark for evaluating agent memory with two-step task sequences around evidence anchors from EgoLife egocentric streams, with four diagnostic metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.14571","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","video-understanding"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_streammembench_6345b534"},"modality":null,"name":"StreamMemBench","openness":"unknown","publisher":null,"released":"2026-06-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.14571","source_url":"https://arxiv.org/abs/2606.14571"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-14571","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14571","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:waterwang-001/streamprofilebench","source_record":"bm_streamprofilebench_ccd7b568"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"StreamProfileBench evaluates LLMs on fine-grained streaming user profiling. Models maintain a rolling persona summary from a stream of user posts and predict which tags from a candidate pool the user will engage with next. Includes over 120,000 posts from 7,000+ users across five Chinese platforms with metrics for recall, novelty, stability, and error rates.","descriptions":{"en":"StreamProfileBench evaluates LLMs on fine-grained streaming user profiling. Models maintain a rolling persona summary from a stream of user posts and predict which tags from a candidate pool the user will engage with next. Includes over 120,000 posts from 7,000+ users across five Chinese platforms with metrics for recall, novelty, stability, and error rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.25758","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_streamprofilebench_ccd7b568"},"modality":null,"name":"StreamProfileBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25758","source_url":"https://arxiv.org/abs/2605.25758"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25758","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25758","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["指令跟随","Instruct","大语言模型","LLM","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"StructFlowBench is a structurally annotated multi-turn benchmark that leverages a structure-driven generation paradigm to enhance the simulation of complex dialogue scenarios. StructFlowBench，这是一个包含155条数据的结构化标注多轮基准，它利用结构驱动生成范式来增强复杂对话场景的模拟。","descriptions":{"en":"StructFlowBench is a structurally annotated multi-turn benchmark that leverages a structure-driven generation paradigm to enhance the simulation of complex dialogue scenarios.","zh":"StructFlowBench，这是一个包含155条数据的结构化标注多轮基准，它利用结构驱动生成范式来增强复杂对话场景的模拟。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1542","languages":[],"library_direction":null,"modality":null,"name":"StructFlowBench","openness":"open","publisher":"Jilin University;University of North Carolina at Chapel Hill;etc.","released":"2025-02-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1542-structflowbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StructFlowBench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","科学智能","AI for Science","科学推理","Scientific Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"StructTokenBench is a benchmark framework designed to comprehensively evaluate the quality and efficiency of protein structure tokenization methods, particularly focusing on fine-grained local substructures. StructTokenBench 是一个评估蛋白质结构标记化方法质量效率的基准。它关注细粒度局部子结构，评测维度包括下游有效性、敏感性、独特性和码本利用效率。","descriptions":{"en":"StructTokenBench is a benchmark framework designed to comprehensively evaluate the quality and efficiency of protein structure tokenization methods, particularly focusing on fine-grained local substructures.","zh":"StructTokenBench 是一个评估蛋白质结构标记化方法质量效率的基准。它关注细粒度局部子结构，评测维度包括下游有效性、敏感性、独特性和码本利用效率。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2082","languages":[],"library_direction":null,"modality":null,"name":"StructTokenBench","openness":"unknown","publisher":"Mila - Quebec AI Institute , University of Montreal , Amazon , George Mason Univ","released":"2025-02-28","released_reference":{"basis":"paper_first_version","note":"First version introducing the structure-tokenizer evaluation framework.","source_key":"opencompass:2082","source_url":"https://arxiv.org/abs/2503.00089"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2082-structtokenbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StructTokenBench","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":10,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:handshake-ai-research/studentbench","source_record":"bm_studentbench_edb30a5d"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates large language model tutors via GRE learning gains, comparing AI vs human tutoring and assessing teaching quality and cost efficiency across seven GRE domains.","descriptions":{"en":"Evaluates large language model tutors via GRE learning gains, comparing AI vs human tutoring and assessing teaching quality and cost efficiency across seven GRE domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.28470","languages":[],"library_direction":null,"modality":null,"name":"StudentBench","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.28470","source_url":"https://arxiv.org/abs/2609.28470"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-28470","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28470","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","ACL 2024","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"STUDENTEVAL contains 1,749 prompts written by 80 students who have only completed one introductory Python course. STUDENTEVAL contains numerous non-expert prompts describing the same problem, enabling exploration of key factors in prompt success. StudentEval 包含 1,749 个由 80 名仅完成一门入门 Python 课程的学生撰写的提示。StudentEval 中包含许多非专家提示，描述相同的问题，使得探索提示成功的关键因素成为可能。","descriptions":{"en":"STUDENTEVAL contains 1,749 prompts written by 80 students who have only completed one introductory Python course. STUDENTEVAL contains numerous non-expert prompts describing the same problem, enabling exploration of key factors in prompt success.","zh":"StudentEval 包含 1,749 个由 80 名仅完成一门入门 Python 课程的学生撰写的提示。StudentEval 中包含许多非专家提示，描述相同的问题，使得探索提示成功的关键因素成为可能。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1082","languages":[],"library_direction":null,"modality":null,"name":"StudentEval","openness":"unknown","publisher":"Oberlin College","released":"2024-08-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1082-studenteval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StudentEval","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":8,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":"gh:thunlp/studybench","source_record":"bm_studybench_38f4772e"},"categories":["Language & Knowledge","Mathematics & Formal Science","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates language models on hard physics textbook exercises (Application Set) and olympiad-level theory problems (Transfer Set), reporting pass@k accuracy for absorption and transfer of textbook knowledge.","descriptions":{"en":"Evaluates language models on hard physics textbook exercises (Application Set) and olympiad-level theory problems (Transfer Set), reporting pass@k accuracy for absorption and transfer of textbook knowledge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.00787","languages":[],"library_direction":{"basis":"model-review","categories":["science","knowledge-expert-qa"],"join":"claire_record_id","primary":"science","source_record":"bm_studybench_38f4772e"},"modality":null,"name":"StudyBench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00787","source_url":"https://arxiv.org/abs/2609.00787"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00787","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00787","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for compositional spatio-temporal video grounding with difficulty levels based on temporal complexity and spatial interference.","descriptions":{"en":"Benchmark for compositional spatio-temporal video grounding with difficulty levels based on temporal complexity and spatial interference."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30584","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding","spatial-3d"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_stvg-compbench_415f9719"},"modality":null,"name":"STVG-CompBench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30584","source_url":"https://arxiv.org/abs/2608.30584"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30584","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30584","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","语言","Language","NLP","Writing Style Transformation","Prompt Recovery","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"A Benchmark Dataset for Prompt Recovery in Writing Style Transformation. 基于写作风格转换的提示词恢复的评测集","descriptions":{"en":"A Benchmark Dataset for Prompt Recovery in Writing Style Transformation.","zh":"基于写作风格转换的提示词恢复的评测集"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1744","languages":["English"],"library_direction":null,"modality":null,"name":"StyleRec","openness":"unknown","publisher":null,"released":"2025-04-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1744-stylerec","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StyleRec","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:timo-cavelius/stylisticbias","source_record":"bm_stylisticbias_ee73496d"},"categories":["General AI","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"StylisticBias is a benchmark of 25,000 images with controlled single-attribute variations to evaluate attribute-level social bias in multimodal LLMs. It measures how visual cues shift model judgments in 25 scenarios.","descriptions":{"en":"StylisticBias is a benchmark of 25,000 images with controlled single-attribute variations to evaluate attribute-level social bias in multimodal LLMs. It measures how visual cues shift model judgments in 25 scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.20527","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_stylisticbias_ee73496d"},"modality":null,"name":"StylisticBias","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20527","source_url":"https://arxiv.org/abs/2606.20527"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20527","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20527","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":22,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":20,"join":"claire_record_id","repository":"gh:yummytanmo/subtlememory","source_record":"bm_subtlememory_d858f4b9"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SubtleMemory evaluates fine-grained relational memory discrimination in long-horizon AI agents. It contains 1,522 evaluation instances over 10 long histories, grounded in 1,090 relation-controlled memory-variant sets, spanning user-related and non-user-related queries.","descriptions":{"en":"SubtleMemory evaluates fine-grained relational memory discrimination in long-horizon AI agents. It contains 1,522 evaluation instances over 10 long histories, grounded in 1,090 relation-controlled memory-variant sets, spanning user-related and non-user-related queries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.05761","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_subtlememory_d858f4b9"},"modality":null,"name":"SubtleMemory","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05761","source_url":"https://arxiv.org/abs/2606.05761"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05761","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05761","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SuiChat-CN is a Chinese group-chat benchmark for contextual suicide risk assessment, with 13,312 segments from 1,406 users.","descriptions":{"en":"SuiChat-CN is a Chinese group-chat benchmark for contextual suicide risk assessment, with 13,312 segments from 1,406 users."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.27911","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_suichat-cn_1a4b7900"},"modality":null,"name":"SuiChat-CN","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27911","source_url":"https://arxiv.org/abs/2605.27911"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27911","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27911","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":41,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:mingdachen/summscreen","source_record":"catalog_54a2a373859605d8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["long_context","summarization"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SummScreenFD is the ForeverDreaming subset of the SummScreen dataset for abstractive screenplay summarization, comprising pairs of TV series transcripts and human-written recaps from 88 different shows. The dataset provides a challenging testbed for abstractive summarization where plot details are often expressed indirectly in character dialogues and scattered across the entirety of the transcript, requiring models to find and integrate these details to form succinct plot descriptions.","descriptions":{"en":"SummScreenFD is the ForeverDreaming subset of the SummScreen dataset for abstractive screenplay summarization, comprising pairs of TV series transcripts and human-written recaps from 88 different shows. The dataset provides a challenging testbed for abstractive summarization where plot details are often expressed indirectly in character dialogues and scattered across the entirety of the transcript, requiring models to find and integrate these details to form succinct plot descriptions."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:summscreenfd:phi-3.5-mini-instruct","reported_at":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/summscreenfd?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:summscreenfd","languages":[],"library_direction":{"basis":"model-review","categories":["long-context","chat-writing"],"join":"llm_stats_id","primary":"long-context","source_record":"catalog_54a2a373859605d8"},"modality":"text","name":"SummScreenFD","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":16.900000000000002,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.169,"raw_min":0.16,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:summscreenfd:phi-3.5-moe-instruct","reported_date":"2024-08-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/summscreenfd?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"}],"slug":"llm-stats-summscreenfd","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/summscreenfd?top_n=500","topics":["long-context","language"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["spatial_reasoning","3d","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SUNRGBD evaluates RGB-D scene understanding and 3D grounding capabilities.","descriptions":{"en":"SUNRGBD evaluates RGB-D scene understanding and 3D grounding capabilities."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:sunrgbd:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sunrgbd?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:sunrgbd","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_40e1ff1283bd4d04"},"modality":"image","name":"SUNRGBD","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":0.362,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.362,"raw_min":0.334,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:sunrgbd:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sunrgbd?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"}],"slug":"llm-stats-sunrgbd","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sunrgbd?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_7d7e9a75ac4d072e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","knowledge","chemistry"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SuperChem is a benchmark of advanced chemistry problems requiring expert-level domain knowledge and reasoning.","descriptions":{"en":"SuperChem is a benchmark of advanced chemistry problems requiring expert-level domain knowledge and reasoning."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:superchem:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/superchem?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:superchem","languages":[],"library_direction":{"basis":"model-review","categories":["science","knowledge-expert-qa"],"join":"llm_stats_id","primary":"science","source_record":"catalog_7d7e9a75ac4d072e"},"modality":"text","name":"SuperChem","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":59.8,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.598,"raw_min":0.549,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:superchem:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/superchem?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-superchem","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/superchem?top_n=500","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["science"],"collected_at":null,"description":"Multimodal chemical reasoning; image and formula parsing quality moves the reported figure.","descriptions":{"en":"Multimodal chemical reasoning; image and formula parsing quality moves the reported figure."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000superchem\u0000tencent_hy4_preview\u0000superchem\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000superchem\u0000tencent_hy4_preview\u0000superchem\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:superchem","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["science"],"join":"review:14H034160212:2026-10-15","primary":"science","source_record":null},"modality":null,"name":"SUPERChem","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":66.4,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":66.4,"raw_min":66.4,"source_reference":{"obs_id":"curated\u0000superchem\u0000tencent_hy4_preview\u0000superchem\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000superchem\u0000tencent_hy4_preview\u0000superchem\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"superchem","source":"model_reports","source_url":"https://github.com/catalystforyou/SUPERChem_eval","topics":["science"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"llm_stats_id","repository":null,"source_record":"catalog_56ba0cb792f6bfce"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SuperGLUE is a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard. It includes 8 primary tasks: BoolQ (Boolean Questions), CB (CommitmentBank), COPA (Choice of Plausible Alternatives), MultiRC (Multi-Sentence Reading Comprehension), ReCoRD (Reading Comprehension with Commonsense Reasoning), RTE (Recognizing Textual Entailment), WiC (Word-in-Context), and WSC (Winograd Schema Challenge). The benchmark evaluates diverse language understanding capabilities including reading comprehension, commonsense reasoning, causal reasoning, coreference resolution, textual entailment, and word sense disambiguation across multiple domains.","descriptions":{"en":"SuperGLUE is a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard. It includes 8 primary tasks: BoolQ (Boolean Questions), CB (CommitmentBank), COPA (Choice of Plausible Alternatives), MultiRC (Multi-Sentence Reading Comprehension), ReCoRD (Reading Comprehension with Commonsense Reasoning), RTE (Recognizing Textual Entailment), WiC (Word-in-Context), and WSC (Winograd Schema Challenge). The benchmark evaluates diverse language understanding capabilities including reading comprehension, commonsense reasoning, causal reasoning, coreference resolution, textual entailment, and word sense disambiguation across multiple domains."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:superglue:o1-mini","reported_at":"2024-09-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/superglue?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:superglue","languages":[],"library_direction":null,"modality":"text","name":"SuperGLUE","openness":"unknown","publisher":null,"released":"2019-05-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:superglue","source_url":"https://arxiv.org/abs/1905.00537"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":75.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.75,"raw_min":0.75,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:superglue:o1-mini","reported_date":"2024-09-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/superglue?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-mini"}],"slug":"llm-stats-superglue","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/superglue?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":110,"join":"llm_stats_id","repository":null,"source_record":"catalog_6cd5a7b958e562cd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","math","physics","reasoning","finance","general","healthcare","chemistry","economics"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SuperGPQA is a comprehensive benchmark that evaluates large language models across 285 graduate-level academic disciplines. The benchmark contains 25,957 questions covering 13 broad disciplinary areas including Engineering, Medicine, Science, and Law, with specialized fields in light industry, agriculture, and service-oriented domains. It employs a Human-LLM collaborative filtering mechanism with over 80 expert annotators to create challenging questions that assess graduate-level knowledge and reasoning capabilities.","descriptions":{"en":"SuperGPQA is a comprehensive benchmark that evaluates large language models across 285 graduate-level academic disciplines. The benchmark contains 25,957 questions covering 13 broad disciplinary areas including Engineering, Medicine, Science, and Law, with specialized fields in light industry, agriculture, and service-oriented domains. It employs a Human-LLM collaborative filtering mechanism with over 80 expert annotators to create challenging questions that assess graduate-level knowledge and reasoning capabilities."},"evidence_summary":{"document_count":1,"model_count":34,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:supergpqa:qwen3-235b-a22b","reported_at":"2025-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/supergpqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:supergpqa","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_6cd5a7b958e562cd"},"modality":"text","name":"SuperGPQA","openness":"restricted","publisher":"M-A-P,ByteDance.Inc, 2077.AI","released":"2025-02-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":34,"score_direction":"higher_is_better","score_summary":{"display_max":73.6,"display_multiplier":100,"model_count":34,"model_count_basis":"source_model_id","numeric_count":34,"raw_max":0.736,"raw_min":0.213,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:supergpqa:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/supergpqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-04-29","date_precision":"model_announcement","name":"Qwen3 235B A22B"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-supergpqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/supergpqa?top_n=500","topics":["reasoning-math","finance","medical","science"],"unit":null},{"aliases":["SuperGPQA","Super-GPQA"],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":110,"join":"artifact:arxiv:2502.14739","repository":null,"source_record":"catalog_6cd5a7b958e562cd"},"categories":["knowledge"],"collected_at":null,"description":"285 graduate disciplines; breadth comes at the cost of per-field sample size.","descriptions":{"en":"285 graduate disciplines; breadth comes at the cost of per-field sample size."},"evidence_summary":{"document_count":3,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000super_gpqa\u0000bytedance_seed2_0_model_card\u0000super_gpqa\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000super_gpqa\u0000bytedance_seed2_0_model_card\u0000super_gpqa\u0000as printed in Table 3 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:super_gpqa","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"artifact:arxiv:2502.14739","primary":"knowledge-expert-qa","source_record":"catalog_6cd5a7b958e562cd"},"modality":null,"name":"SuperGPQA","openness":"unknown","publisher":null,"released":"2025-02-20","released_reference":{"basis":"benchmark_release","source_key":"model-reports:super_gpqa","source_url":"https://arxiv.org/abs/2502.14739"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":70.4,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":70.4,"raw_min":49.8,"source_reference":{"obs_id":"curated\u0000super_gpqa\u0000qwen3_5_model_card\u0000super_gpqa\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","observation_id":"curated\u0000super_gpqa\u0000qwen3_5_model_card\u0000super_gpqa\u0000thinking (temp 0.6, top_p 0.95, top_k 20)\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","reported_date":"2026-02-16","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"unit":"percent"},"scored_models":[{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Mini"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"super_gpqa","source":"model_reports","source_url":"https://arxiv.org/abs/2502.14739","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":110,"join":"artifact:arxiv:2502.14739","repository":null,"source_record":"catalog_6cd5a7b958e562cd"},"categories":["学科","Examination","大语言模型","LLM","知识储备","Knowledge"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SuperGPQA, a comprehensive benchmark designed to evaluate the knowledge and reasoning abilities of Large Language Models (LLMs) across 285 graduate-level disciplines. SuperGPQA features at least 50 questions per discipline, covering a broad spectrum of graduate-level topics. SuperGPQA，这是一个旨在评估大型语言模型在 285 个研究生学科领域的知识和推理能力的全面基准。SuperGPQA 每个学科至少包含 50 个问题，涵盖广泛的硕士研究生学科主题。","descriptions":{"en":"SuperGPQA, a comprehensive benchmark designed to evaluate the knowledge and reasoning abilities of Large Language Models (LLMs) across 285 graduate-level disciplines. SuperGPQA features at least 50 questions per discipline, covering a broad spectrum of graduate-level topics.","zh":"SuperGPQA，这是一个旨在评估大型语言模型在 285 个研究生学科领域的知识和推理能力的全面基准。SuperGPQA 每个学科至少包含 50 个问题，涵盖广泛的硕士研究生学科主题。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1538","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa"],"join":"artifact:arxiv:2502.14739","primary":"knowledge-expert-qa","source_record":"catalog_6cd5a7b958e562cd"},"modality":null,"name":"SuperGPQA","openness":"restricted","publisher":"M-A-P,ByteDance.Inc, 2077.AI","released":"2025-02-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1538-supergpqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SuperGPQA","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:debobanerjee/surakshaeval","source_record":"bm_surakshaeval_6928de7a"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"SurakshaEval evaluates the safety of LLMs across ten major Indian languages and English, using human-written prompts covering generic and region-specific scenarios, with a structured scoring protocol.","descriptions":{"en":"SurakshaEval evaluates the safety of LLMs across ten major Indian languages and English, using human-written prompts covering generic and region-specific scenarios, with a structured scoring protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.07862","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_surakshaeval_6928de7a"},"modality":null,"name":"SurakshaEval","openness":"unknown","publisher":null,"released":"2026-08-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07862","source_url":"https://arxiv.org/abs/2608.07862"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07862","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07862","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":84,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:xiandaguo/drive-mllm","source_record":"catalog_ce00f2c163a1ff7b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SURDS is a benchmark for spatial understanding and reasoning in autonomous-driving scenes.","descriptions":{"en":"SURDS is a benchmark for spatial understanding and reasoning in autonomous-driving scenes."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:surds:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/surds?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:surds","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"llm_stats_id","primary":"spatial-3d","source_record":"catalog_ce00f2c163a1ff7b"},"modality":"multimodal","name":"SURDS","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":77.2,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.772,"raw_min":0.772,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:surds:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/surds?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-surds","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/surds?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SurgHiBench is a hierarchy-aware evaluation suite for surgical video understanding with three tasks measuring recognition, consistency, and severity across granularity levels.","descriptions":{"en":"SurgHiBench is a hierarchy-aware evaluation suite for surgical video understanding with three tasks measuring recognition, consistency, and severity across granularity levels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27139","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_surghibench_434e0435"},"modality":null,"name":"SurgHiBench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27139","source_url":"https://arxiv.org/abs/2609.27139"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27139","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27139","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":8,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:vcl-hnu/surgvla","source_record":"bm_surgvla-bench_a2867ef6"},"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language-action models in laparoscopic surgical robotics across 8 tasks (atomic, conditional, composite) on the SurRoL simulator, using action accuracy and semantic consistency metrics.","descriptions":{"en":"Evaluates vision-language-action models in laparoscopic surgical robotics across 8 tasks (atomic, conditional, composite) on the SurRoL simulator, using action accuracy and semantic consistency metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.29247","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","health-medicine"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_surgvla-bench_a2867ef6"},"modality":null,"name":"SurgVLA-Bench","openness":"unknown","publisher":null,"released":"2026-06-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29247","source_url":"https://arxiv.org/abs/2606.29247"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29247","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29247","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Planning","Vision & 3D","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates surgical world models on short-horizon instrument motion prediction and dynamics stability from intraoperative image sequences, focusing on geometric accuracy and temporal coherence.","descriptions":{"en":"Evaluates surgical world models on short-horizon instrument motion prediction and dynamics stability from intraoperative image sequences, focusing on geometric accuracy and temporal coherence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08070","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","health-medicine","embodied-robotics"],"join":"claire_record_id","primary":"world-models","source_record":"bm_surgwmbench_1b6538d4"},"modality":null,"name":"SurgWMBench","openness":"unknown","publisher":null,"released":"2026-08-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08070","source_url":"https://arxiv.org/abs/2608.08070"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08070","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08070","topics":["medical"],"unit":null},{"aliases":["SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators"],"attention_signals":{"as_of":"2026-10-13","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zai-org/surveyreview","source_record":"bm_surveyreview-a-reviewer-aligned-benchmark-_02cc2f3c"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SurveyReview provides a dataset of 675 survey papers with 1,630 review reports annotated into four-dimensional scores, along with standardized splits and an evaluation protocol for measuring alignment between automatic evaluators and human reviewers.","descriptions":{"en":"SurveyReview provides a dataset of 675 survey papers with 1,630 review reports annotated into four-dimensional scores, along with standardized splits and an evaluation protocol for measuring alignment between automatic evaluators and human reviewers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07641","languages":[],"library_direction":{"basis":"model-review","categories":["science","chat-writing"],"join":"claire_record_id","primary":"science","source_record":"bm_surveyreview-a-reviewer-aligned-benchmark-_02cc2f3c"},"modality":null,"name":"SurveyReview","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07641","source_url":"https://arxiv.org/abs/2608.07641"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07641","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07641","topics":["language","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["数学","Math","大语言模型","LLM","数理能力","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SVAMP includes one-unknown arithmetic word problems with grade level up to 4 by applying simple variations over word problems in an existing dataset. SVAMP further highlights the brittle nature of existing models when trained on these benchmark datasets. SVAMP 是一个包含算术文字问题的数据集，最高适用于四年级的学生，是通过对现有数据集中的文字问题应用简单变体而生成。","descriptions":{"en":"SVAMP includes one-unknown arithmetic word problems with grade level up to 4 by applying simple variations over word problems in an existing dataset. SVAMP further highlights the brittle nature of existing models when trained on these benchmark datasets.","zh":"SVAMP 是一个包含算术文字问题的数据集，最高适用于四年级的学生，是通过对现有数据集中的文字问题应用简单变体而生成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1113","languages":[],"library_direction":null,"modality":null,"name":"SVAMP","openness":"open","publisher":"Microsoft Research India","released":"2021-04-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1113-svamp","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SVAMP","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SVG-Bench is an internal benchmark that comprehensively evaluates SVG generation performance. It accepts text and image inputs across build-from-scratch and edit-based tasks, using a VLM to verify rendering accuracy of the generated outputs.","descriptions":{"en":"SVG-Bench is an internal benchmark that comprehensively evaluates SVG generation performance. It accepts text and image inputs across build-from-scratch and edit-based tasks, using a VLM to verify rendering accuracy of the generated outputs."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:svg-bench:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/svg-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:svg-bench","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","code-generation"],"join":"llm_stats_id","primary":"image-video-generation","source_record":"catalog_a497b8fb621ca34b"},"modality":"multimodal","name":"SVG-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":63.7,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.637,"raw_min":0.637,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:svg-bench:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/svg-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-svg-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/svg-bench?top_n=500","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"SVGEval is a vision-grounded benchmark for human-aligned SVG quality assessment, with multi-round labeled annotations, evaluating models on semantic, aesthetic, geometry, and layout judgments.","descriptions":{"en":"SVGEval is a vision-grounded benchmark for human-aligned SVG quality assessment, with multi-round labeled annotations, evaluating models on semantic, aesthetic, geometry, and layout judgments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.01977","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","visual-reasoning"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_svgeval_0c4e294c"},"modality":null,"name":"SVGEval","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.01977","source_url":"https://arxiv.org/abs/2608.01977"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-01977","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01977","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_svhalluc_d54e2d9e"},"categories":["General AI","Multimodal","eess.AS"],"collected_at":"2026-10-14T09:44:51Z","description":"SVHalluc evaluates speech-vision hallucination in audio-visual large language models, focusing on semantic and temporal alignment between speech content and visual signals.","descriptions":{"en":"SVHalluc evaluates speech-vision hallucination in audio-visual large language models, focusing on semantic and temporal alignment between speech content and visual signals."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.02642","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","speech-audio","video-understanding"],"join":"claire_record_id","primary":"factuality","source_record":"bm_svhalluc_d54e2d9e"},"modality":null,"name":"SVHalluc","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02642","source_url":"https://arxiv.org/abs/2606.02642"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02642","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02642","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:texaser/svi-bench","source_record":"bm_svi-bench_918464db"},"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"SVI-Bench evaluates vision-language models on strategic video intelligence using sports as a microworld. It includes 9 tasks across 4 pillars: perception, reasoning, simulation, and agency, using basketball, soccer, and hockey videos with annotated actions and reports.","descriptions":{"en":"SVI-Bench evaluates vision-language models on strategic video intelligence using sports as a microworld. It includes 9 tasks across 4 pillars: perception, reasoning, simulation, and agency, using basketball, soccer, and hockey videos with annotated actions and reports."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.31529","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_svi-bench_918464db"},"modality":null,"name":"SVI-Bench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31529","source_url":"https://arxiv.org/abs/2605.31529"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31529","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31529","topics":["agents"],"unit":null},{"aliases":["SWE Atlas","SWE-Atlas"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Reports three splits (Codebase Q&A, Test Writing, Refactoring) as separate figures; the split is part of the instrument.","descriptions":{"en":"Reports three splits (Codebase Q&A, Test Writing, Refactoring) as separate figures; the split is part of the instrument."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000swe_atlas\u0000tencent_hy4_preview\u0000swe_atlas_codebase_qa\u0000Codebase Q&A split\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000swe_atlas\u0000tencent_hy4_preview\u0000swe_atlas_codebase_qa\u0000Codebase Q&A split\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:swe_atlas","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"artifact:gh:scaleapi/swe-atlas","primary":"agentic-coding","source_record":"catalog_009dc944c87b90dd"},"modality":null,"name":"SWE Atlas","openness":"unknown","publisher":null,"released":"2026-05-08","released_reference":{"basis":"benchmark_release","source_key":"model-reports:swe_atlas","source_url":"https://github.com/scaleapi/SWE-Atlas"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":64.0,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":64.0,"raw_min":53.3,"source_reference":{"obs_id":"curated\u0000swe_atlas\u0000tencent_hy4_preview\u0000swe_atlas_codebase_qa\u0000Codebase Q&A split\u0000Hy4 preview","observation_id":"curated\u0000swe_atlas\u0000tencent_hy4_preview\u0000swe_atlas_codebase_qa\u0000Codebase Q&A split\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"swe_atlas","source":"model_reports","source_url":"https://github.com/scaleapi/SWE-Atlas","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":79,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:scaleapi/swe-atlas","source_record":"catalog_009dc944c87b90dd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE Atlas - Codebase QnA evaluates a model's ability to answer questions about real codebases, measuring repository-level comprehension and the ability to reason about code structure, behavior, and intent across an entire project.","descriptions":{"en":"SWE Atlas - Codebase QnA evaluates a model's ability to answer questions about real codebases, measuring repository-level comprehension and the ability to reason about code structure, behavior, and intent across an entire project."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-atlas-codebase-qna:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-codebase-qna?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-atlas-codebase-qna","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_009dc944c87b90dd"},"modality":"text","name":"SWE Atlas - Codebase QnA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":46.2,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.462,"raw_min":0.379,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-atlas-codebase-qna:laguna-s-2.1","reported_date":"2026-07-21","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-codebase-qna?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-21","date_precision":"model_announcement","name":"Laguna S 2.1"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-swe-atlas-codebase-qna","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-codebase-qna?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":79,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:scaleapi/swe-atlas","source_record":"catalog_fb29b433a8cdf099"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE Atlas - Test Writing evaluates a model's ability to author meaningful tests for real-world software projects, measuring how well agents can understand code and produce correct, useful test coverage.","descriptions":{"en":"SWE Atlas - Test Writing evaluates a model's ability to author meaningful tests for real-world software projects, measuring how well agents can understand code and produce correct, useful test coverage."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-atlas-test-writing:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-test-writing?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-atlas-test-writing","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_fb29b433a8cdf099"},"modality":"text","name":"SWE Atlas - Test Writing","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":30.830000000000002,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.3083,"raw_min":0.3083,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-atlas-test-writing:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-test-writing?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-swe-atlas-test-writing","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-test-writing?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":53,"github_stars_30d":4,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"claire_record_id","repository":null,"source_record":"bm_swe-refactor-bench_45bb76b0"},"categories":["Language & Knowledge","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"SWE Refactor Bench is a benchmark for evaluating coding agents on whole-repository stack migrations. It comprises 20 migrations covering 4 types of technical debt, with a three-stage evaluation protocol measuring migration completeness and behavioral correctness: Migration Audit, Behavioral Tests, and Agentic Verification.","descriptions":{"en":"SWE Refactor Bench is a benchmark for evaluating coding agents on whole-repository stack migrations. It comprises 20 migrations covering 4 types of technical debt, with a three-stage evaluation protocol measuring migration completeness and behavioral correctness: Migration Audit, Behavioral Tests, and Agentic Verification."},"editorial_guide":{"id":"swe-refactor-bench","record_keys":["model-reports:swe_refactor_bench","claire-radar:2608.23564"],"reviewed_at":"2026-10-27","scope":{"en":"The 20-task benchmark and 520-run experiment in arXiv v1. Each model–effort pair remains a separate configuration; pooled model summaries have different denominators.","zh":"对应 arXiv v1 的 20 题、520 次运行实验。模型与推理强度的组合分别保留；跨配置汇总使用不同分母。"},"sections":[{"id":"tasks","source_ids":["paper","tasks"],"text":{"en":"Twenty pinned open-source repositories cover seven language rewrites, seven framework rewrites, three platform ports and three build-toolchain migrations. Each attempt receives 6–30 hours. The original APIs, artifacts and observable behavior must remain while the old stack leaves the delivered implementation.","zh":"20 个固定版本的开源仓库包含七项语言重写、七项框架重写、三项平台移植与三项构建工具迁移，每次尝试预算 6–30 小时。原 API、交付产物与可观察行为要保持一致，旧技术栈则应从交付实现中退出。"},"title":{"en":"Whole repositories, not isolated functions","zh":"迁移整个仓库"}},{"id":"verification","source_ids":["paper","readme"],"text":{"en":"First, a source audit judges whether migration occurred, with three samples voting per criterion. Next, all 130,118 frozen checks across the corpus must pass where applicable. Finally, six coding agents get one hour each to seek a test that passes on the original and fails on the submission, reproducible three times.","zh":"首先通过源码审计判断迁移是否发生，每项条件由三次判定多数表决。然后要求题目适用的固定检查全部通过，全题集共 130,118 项。最后六个编程 Agent 各用一小时寻找原版通过、提交版失败且可重复三次的测试。"},"title":{"en":"Three independent questions","zh":"三道验收关"}},{"id":"score","source_ids":["paper","readme"],"text":{"en":"A task gets zero if the migration gate or any required fixed check fails. Otherwise it receives 40 points plus 10 for each verifier that finds no counterexample. The leaderboard averages this 0–100 composite over tasks. Acceptance is stricter: the task must survive all six verifiers.","zh":"迁移门槛或任一必要固定检查失败，任务得零分；否则先得 40 分，每有一个验证 Agent 找不到反例再得 10 分。榜单对各题的 0–100 综合分求平均。完全验收更严格：必须抵御全部六个验证 Agent。"},"title":{"en":"Composite score is not acceptance rate","zh":"综合分不等于验收通过率"}},{"id":"configuration-table","source_ids":["paper-table3","leaderboard"],"table":{"caption":{"en":"Paper v1 Table 3 · one attempt on each of 20 tasks","zh":"论文 v1 表 3 · 每配置对 20 题各尝试一次"},"columns":[{"en":"Model","zh":"模型"},{"en":"Harness","zh":"运行框架"},{"en":"Effort","zh":"推理强度"},{"en":"Behavioral pass (%)","zh":"行为通过率（%）"},{"en":"Composite (/100)","zh":"综合分（/100）"},{"en":"API USD/task","zh":"每题 API 美元"}],"note":{"en":"The mean behavioral-check rate is not the fraction of fully accepted migrations. GPT models use Codex; all others use Claude Code. Costs are those reported for these runs.","zh":"行为检查平均通过率不等于完全验收的迁移比例。GPT 模型使用 Codex，其余使用 Claude Code；成本是这批实验报告的值。"},"rows":[["Claude Opus 5","Claude Code","low","92.5","20.5","34.8"],["Claude Opus 5","Claude Code","medium","92.7","28.5","38.4"],["Claude Opus 5","Claude Code","high","92.8","34.5","55.7"],["Claude Opus 5","Claude Code","xhigh","92.8","47.0","74.9"],["Claude Opus 5","Claude Code","max","91.9","31.0","72.4"],["GPT-5.6 Sol","Codex","none","63.6","4.0","2.9"],["GPT-5.6 Sol","Codex","low","62.0","7.0","5.9"],["GPT-5.6 Sol","Codex","medium","70.8","6.5","6.0"],["GPT-5.6 Sol","Codex","high","74.0","19.0","7.7"],["GPT-5.6 Sol","Codex","xhigh","73.5","9.5","19.1"],["GPT-5.6 Sol","Codex","max","84.1","28.5","143.5"],["Kimi K3","Claude Code","max","93.9","19.5","28.9"],["Claude Sonnet 5","Claude Code","low","79.2","4.0","4.4"],["Claude Sonnet 5","Claude Code","medium","73.9","15.0","11.9"],["Claude Sonnet 5","Claude Code","high","85.6","6.0","24.6"],["Claude Sonnet 5","Claude Code","xhigh","76.5","9.0","27.0"],["Claude Sonnet 5","Claude Code","max","84.3","8.5","27.5"],["GPT-5.6 Luna","Codex","none","66.4","4.0","1.6"],["GPT-5.6 Luna","Codex","low","58.3","4.0","1.7"],["GPT-5.6 Luna","Codex","medium","66.6","0.0","1.7"],["GPT-5.6 Luna","Codex","high","75.6","4.0","1.8"],["GPT-5.6 Luna","Codex","xhigh","83.8","5.5","2.9"],["GPT-5.6 Luna","Codex","max","89.1","10.5","2.8"],["Qwen 3.8 Max","Claude Code","max","74.7","10.0","14.5"],["DeepSeek V4 Flash","Claude Code","max","90.7","7.0","4.3"],["GLM 5.2","Claude Code","max","85.2","6.5","17.5"]]},"text":{"en":"Table 3 reports all 26 configurations. Behavioral pass and cost are different measurements from the composite score; neither is substituted for it.","zh":"论文表 3 包含全部 26 个配置。行为检查通过率与成本属于不同指标，不能替代综合分。"},"title":{"en":"Every evaluated reasoning configuration","zh":"完整推理配置成绩"}},{"id":"pooled-table","source_ids":["leaderboard"],"table":{"caption":{"en":"Official pooled summaries · observed 2026-10-27","zh":"官方跨配置汇总 · 核对日期 2026-10-27"},"columns":[{"en":"Model","zh":"模型"},{"en":"Client","zh":"客户端"},{"en":"Configurations","zh":"配置数"},{"en":"Runs","zh":"运行数"},{"en":"Accepted","zh":"完全验收数"},{"en":"Composite (/100)","zh":"综合分（/100）"}],"note":{"en":"Different effort mixtures and denominators: do not treat these as the same protocol or as additional independent experiments.","zh":"推理强度组合与分母不同，不能当成同一协议，也不是新增的独立实验。"},"rows":[["claude-opus-5","Claude Code","5","100","16","32.30"],["gpt-5.6-sol","Codex CLI","6","120","5","12.42"],["claude-sonnet-5","Claude Code","5","100","3","8.50"],["gpt-5.6-luna","Codex CLI","6","120","0","4.67"],["kimi-k3","Claude Code","1","20","2","19.50"],["qwen3.8-max","Claude Code","1","20","2","10.00"],["dsv4-flash","Claude Code","1","20","0","7.00"],["glm-5.2","Claude Code","1","20","0","6.50"]]},"text":{"en":"The official site also pools every run of each model. Proprietary models have five or six effort levels, while each open-weight model has one. These summaries answer a different question from a best-configuration comparison.","zh":"官网还将同一模型的全部运行合并。闭源模型覆盖五或六档推理强度，开放权重模型各仅一档；这与比较最佳配置回答的是不同问题。"},"title":{"en":"Do not confuse pooled and best-configuration scores","zh":"区分跨配置均值与最佳配置"}},{"id":"limits","source_ids":["paper","leaderboard"],"text":{"en":"One attempt per task does not establish run-to-run reliability. The migration audit uses a model judge and can disagree with humans; the paper checks this explicitly. Six verifiers failing to find a difference within an hour is evidence about that panel and budget, not a proof of program equivalence.","zh":"每题一次尝试不能建立重复运行的可靠性。迁移审计依赖模型判断，可能与人工有分歧，论文对此专门复核。六个验证 Agent 在一小时内未找到差异，只说明该组合与预算下的检查结果，并不构成程序等价性的证明。"},"title":{"en":"Passing is bounded evidence","zh":"通过仍有边界"}}],"sources":[{"id":"paper","title":"SWE Refactor Bench paper v1 · protocol and validity analysis","url":"https://arxiv.org/html/2608.23564v1"},{"id":"paper-table3","title":"Paper v1 Table 3 · all 26 configurations","url":"https://arxiv.org/html/2608.23564v1#S3.T3"},{"id":"leaderboard","title":"Official leaderboard · configuration and pooled summaries","url":"https://lab.einsia.ai/swe-refactor-bench/leaderboard/"},{"id":"tasks","title":"Official tasks · budgets and corpus coverage","url":"https://lab.einsia.ai/swe-refactor-bench/tasks/"},{"id":"readme","title":"Official repository · scoring and pinned clients","url":"https://github.com/Einsia/SWE-Refactor-Bench"}],"summary":{"en":"A migration can pass the original tests without replacing anything. This benchmark checks that the stack really changed, every recorded behavior survived, and independent agents cannot find a new counterexample.","zh":"迁移任务在开始前就能通过原测试。该评测同时检查技术栈是否真正替换、已记录行为是否保留，以及独立 Agent 能否找出新反例。"},"title":"SWE Refactor Bench"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23564","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-refactor-bench_45bb76b0"},"modality":null,"name":"SWE Refactor Bench","openness":"unknown","publication":{"evidence":[{"note":"The author project names this benchmark and links its introducing paper and available release resources. This exact-record review preserves the documented task version, metrics and evaluation setup.","url":"https://lab.einsia.ai/swe-refactor-bench/"},{"note":"Navers Lab / Einsia research index links this named project. Method papers and memory software in the same index are not added as benchmarks.","url":"https://lab.einsia.ai/research/"}],"id":"einsia-swe-refactor-bench-original","links":[{"kind":"website","title":"SWE Refactor Bench · official project","url":"https://lab.einsia.ai/swe-refactor-bench/"},{"kind":"paper","title":"SWE Refactor Bench · introducing paper","url":"https://arxiv.org/abs/2608.23564"},{"kind":"repo","title":"SWE Refactor Bench · official repository","url":"https://github.com/Einsia/SWE-Refactor-Bench"},{"kind":"website","title":"SWE Refactor Bench · author leaderboard","url":"https://lab.einsia.ai/swe-refactor-bench/leaderboard/"}],"record_keys":["model-reports:swe_refactor_bench","claire-radar:2608.23564"],"reviewed_at":"2026-10-27","scope":{"en":"The 20-task benchmark and 520-run experiment in arXiv v1. Each model–effort pair remains a separate configuration; pooled model summaries have different denominators.","zh":"对应 arXiv v1 的 20 题、520 次运行实验。模型与推理强度的组合分别保留；跨配置汇总使用不同分母。"}},"publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23564","source_url":"http://arxiv.org/abs/2608.23564v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23564","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23564v1","topics":["coding","agents"],"unit":null},{"aliases":["SWE-Refactor-Bench"],"attention_signals":{"as_of":"2026-10-13","github_stars":53,"github_stars_30d":4,"hf_dataset_downloads":null,"hf_paper_upvotes":15,"join":"artifact:arxiv:2608.23564","repository":null,"source_record":"bm_swe-refactor-bench_45bb76b0"},"categories":["coding"],"collected_at":null,"description":"Composite score is not full-acceptance rate. One attempt per task, per-task budgets of 6–30 hours, different native clients and reasoning efforts, model-based audit and a finite six-agent verifier panel constrain comparisons.","descriptions":{"en":"Composite score is not full-acceptance rate. One attempt per task, per-task budgets of 6–30 hours, different native clients and reasoning efforts, model-based audit and a finite six-agent verifier panel constrain comparisons."},"editorial_guide":{"id":"swe-refactor-bench","record_keys":["model-reports:swe_refactor_bench","claire-radar:2608.23564"],"reviewed_at":"2026-10-27","scope":{"en":"The 20-task benchmark and 520-run experiment in arXiv v1. Each model–effort pair remains a separate configuration; pooled model summaries have different denominators.","zh":"对应 arXiv v1 的 20 题、520 次运行实验。模型与推理强度的组合分别保留；跨配置汇总使用不同分母。"},"sections":[{"id":"tasks","source_ids":["paper","tasks"],"text":{"en":"Twenty pinned open-source repositories cover seven language rewrites, seven framework rewrites, three platform ports and three build-toolchain migrations. Each attempt receives 6–30 hours. The original APIs, artifacts and observable behavior must remain while the old stack leaves the delivered implementation.","zh":"20 个固定版本的开源仓库包含七项语言重写、七项框架重写、三项平台移植与三项构建工具迁移，每次尝试预算 6–30 小时。原 API、交付产物与可观察行为要保持一致，旧技术栈则应从交付实现中退出。"},"title":{"en":"Whole repositories, not isolated functions","zh":"迁移整个仓库"}},{"id":"verification","source_ids":["paper","readme"],"text":{"en":"First, a source audit judges whether migration occurred, with three samples voting per criterion. Next, all 130,118 frozen checks across the corpus must pass where applicable. Finally, six coding agents get one hour each to seek a test that passes on the original and fails on the submission, reproducible three times.","zh":"首先通过源码审计判断迁移是否发生，每项条件由三次判定多数表决。然后要求题目适用的固定检查全部通过，全题集共 130,118 项。最后六个编程 Agent 各用一小时寻找原版通过、提交版失败且可重复三次的测试。"},"title":{"en":"Three independent questions","zh":"三道验收关"}},{"id":"score","source_ids":["paper","readme"],"text":{"en":"A task gets zero if the migration gate or any required fixed check fails. Otherwise it receives 40 points plus 10 for each verifier that finds no counterexample. The leaderboard averages this 0–100 composite over tasks. Acceptance is stricter: the task must survive all six verifiers.","zh":"迁移门槛或任一必要固定检查失败，任务得零分；否则先得 40 分，每有一个验证 Agent 找不到反例再得 10 分。榜单对各题的 0–100 综合分求平均。完全验收更严格：必须抵御全部六个验证 Agent。"},"title":{"en":"Composite score is not acceptance rate","zh":"综合分不等于验收通过率"}},{"id":"configuration-table","source_ids":["paper-table3","leaderboard"],"table":{"caption":{"en":"Paper v1 Table 3 · one attempt on each of 20 tasks","zh":"论文 v1 表 3 · 每配置对 20 题各尝试一次"},"columns":[{"en":"Model","zh":"模型"},{"en":"Harness","zh":"运行框架"},{"en":"Effort","zh":"推理强度"},{"en":"Behavioral pass (%)","zh":"行为通过率（%）"},{"en":"Composite (/100)","zh":"综合分（/100）"},{"en":"API USD/task","zh":"每题 API 美元"}],"note":{"en":"The mean behavioral-check rate is not the fraction of fully accepted migrations. GPT models use Codex; all others use Claude Code. Costs are those reported for these runs.","zh":"行为检查平均通过率不等于完全验收的迁移比例。GPT 模型使用 Codex，其余使用 Claude Code；成本是这批实验报告的值。"},"rows":[["Claude Opus 5","Claude Code","low","92.5","20.5","34.8"],["Claude Opus 5","Claude Code","medium","92.7","28.5","38.4"],["Claude Opus 5","Claude Code","high","92.8","34.5","55.7"],["Claude Opus 5","Claude Code","xhigh","92.8","47.0","74.9"],["Claude Opus 5","Claude Code","max","91.9","31.0","72.4"],["GPT-5.6 Sol","Codex","none","63.6","4.0","2.9"],["GPT-5.6 Sol","Codex","low","62.0","7.0","5.9"],["GPT-5.6 Sol","Codex","medium","70.8","6.5","6.0"],["GPT-5.6 Sol","Codex","high","74.0","19.0","7.7"],["GPT-5.6 Sol","Codex","xhigh","73.5","9.5","19.1"],["GPT-5.6 Sol","Codex","max","84.1","28.5","143.5"],["Kimi K3","Claude Code","max","93.9","19.5","28.9"],["Claude Sonnet 5","Claude Code","low","79.2","4.0","4.4"],["Claude Sonnet 5","Claude Code","medium","73.9","15.0","11.9"],["Claude Sonnet 5","Claude Code","high","85.6","6.0","24.6"],["Claude Sonnet 5","Claude Code","xhigh","76.5","9.0","27.0"],["Claude Sonnet 5","Claude Code","max","84.3","8.5","27.5"],["GPT-5.6 Luna","Codex","none","66.4","4.0","1.6"],["GPT-5.6 Luna","Codex","low","58.3","4.0","1.7"],["GPT-5.6 Luna","Codex","medium","66.6","0.0","1.7"],["GPT-5.6 Luna","Codex","high","75.6","4.0","1.8"],["GPT-5.6 Luna","Codex","xhigh","83.8","5.5","2.9"],["GPT-5.6 Luna","Codex","max","89.1","10.5","2.8"],["Qwen 3.8 Max","Claude Code","max","74.7","10.0","14.5"],["DeepSeek V4 Flash","Claude Code","max","90.7","7.0","4.3"],["GLM 5.2","Claude Code","max","85.2","6.5","17.5"]]},"text":{"en":"Table 3 reports all 26 configurations. Behavioral pass and cost are different measurements from the composite score; neither is substituted for it.","zh":"论文表 3 包含全部 26 个配置。行为检查通过率与成本属于不同指标，不能替代综合分。"},"title":{"en":"Every evaluated reasoning configuration","zh":"完整推理配置成绩"}},{"id":"pooled-table","source_ids":["leaderboard"],"table":{"caption":{"en":"Official pooled summaries · observed 2026-10-27","zh":"官方跨配置汇总 · 核对日期 2026-10-27"},"columns":[{"en":"Model","zh":"模型"},{"en":"Client","zh":"客户端"},{"en":"Configurations","zh":"配置数"},{"en":"Runs","zh":"运行数"},{"en":"Accepted","zh":"完全验收数"},{"en":"Composite (/100)","zh":"综合分（/100）"}],"note":{"en":"Different effort mixtures and denominators: do not treat these as the same protocol or as additional independent experiments.","zh":"推理强度组合与分母不同，不能当成同一协议，也不是新增的独立实验。"},"rows":[["claude-opus-5","Claude Code","5","100","16","32.30"],["gpt-5.6-sol","Codex CLI","6","120","5","12.42"],["claude-sonnet-5","Claude Code","5","100","3","8.50"],["gpt-5.6-luna","Codex CLI","6","120","0","4.67"],["kimi-k3","Claude Code","1","20","2","19.50"],["qwen3.8-max","Claude Code","1","20","2","10.00"],["dsv4-flash","Claude Code","1","20","0","7.00"],["glm-5.2","Claude Code","1","20","0","6.50"]]},"text":{"en":"The official site also pools every run of each model. Proprietary models have five or six effort levels, while each open-weight model has one. These summaries answer a different question from a best-configuration comparison.","zh":"官网还将同一模型的全部运行合并。闭源模型覆盖五或六档推理强度，开放权重模型各仅一档；这与比较最佳配置回答的是不同问题。"},"title":{"en":"Do not confuse pooled and best-configuration scores","zh":"区分跨配置均值与最佳配置"}},{"id":"limits","source_ids":["paper","leaderboard"],"text":{"en":"One attempt per task does not establish run-to-run reliability. The migration audit uses a model judge and can disagree with humans; the paper checks this explicitly. Six verifiers failing to find a difference within an hour is evidence about that panel and budget, not a proof of program equivalence.","zh":"每题一次尝试不能建立重复运行的可靠性。迁移审计依赖模型判断，可能与人工有分歧，论文对此专门复核。六个验证 Agent 在一小时内未找到差异，只说明该组合与预算下的检查结果，并不构成程序等价性的证明。"},"title":{"en":"Passing is bounded evidence","zh":"通过仍有边界"}}],"sources":[{"id":"paper","title":"SWE Refactor Bench paper v1 · protocol and validity analysis","url":"https://arxiv.org/html/2608.23564v1"},{"id":"paper-table3","title":"Paper v1 Table 3 · all 26 configurations","url":"https://arxiv.org/html/2608.23564v1#S3.T3"},{"id":"leaderboard","title":"Official leaderboard · configuration and pooled summaries","url":"https://lab.einsia.ai/swe-refactor-bench/leaderboard/"},{"id":"tasks","title":"Official tasks · budgets and corpus coverage","url":"https://lab.einsia.ai/swe-refactor-bench/tasks/"},{"id":"readme","title":"Official repository · scoring and pinned clients","url":"https://github.com/Einsia/SWE-Refactor-Bench"}],"summary":{"en":"A migration can pass the original tests without replacing anything. This benchmark checks that the stack really changed, every recorded behavior survived, and independent agents cannot find a new counterexample.","zh":"迁移任务在开始前就能通过原测试。该评测同时检查技术栈是否真正替换、已记录行为是否保留，以及独立 Agent 能否找出新反例。"},"title":"SWE Refactor Bench"},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000swe_refactor_bench\u0000einsia_swe_refactor_bench_v1_table3\u0000swe_refactor_bench_20_tasks_paper_v1\u0000Claude Code 2.1.220; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100\u0000Claude Opus 5","reported_at":"2026-08-24","source_url":"https://arxiv.org/html/2608.23564v1#S3.T3"},"first_score_reported_at":"2026-08-24","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000swe_refactor_bench\u0000einsia_swe_refactor_bench_v1_table3\u0000swe_refactor_bench_20_tasks_paper_v1\u0000Claude Code 2.1.220; reasoning effort high; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100\u0000Claude Opus 5","reported_at":"2026-08-24","source_url":"https://arxiv.org/html/2608.23564v1#S3.T3"},"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"model-reports:swe_refactor_bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"artifact:arxiv:2608.23564","primary":"agentic-coding","source_record":"bm_swe-refactor-bench_45bb76b0"},"modality":null,"name":"SWE Refactor Bench","openness":"unknown","publication":{"evidence":[{"note":"The author project names this benchmark and links its introducing paper and available release resources. This exact-record review preserves the documented task version, metrics and evaluation setup.","url":"https://lab.einsia.ai/swe-refactor-bench/"},{"note":"Navers Lab / Einsia research index links this named project. Method papers and memory software in the same index are not added as benchmarks.","url":"https://lab.einsia.ai/research/"}],"id":"einsia-swe-refactor-bench-original","links":[{"kind":"website","title":"SWE Refactor Bench · official project","url":"https://lab.einsia.ai/swe-refactor-bench/"},{"kind":"paper","title":"SWE Refactor Bench · introducing paper","url":"https://arxiv.org/abs/2608.23564"},{"kind":"repo","title":"SWE Refactor Bench · official repository","url":"https://github.com/Einsia/SWE-Refactor-Bench"},{"kind":"website","title":"SWE Refactor Bench · author leaderboard","url":"https://lab.einsia.ai/swe-refactor-bench/leaderboard/"}],"record_keys":["model-reports:swe_refactor_bench","claire-radar:2608.23564"],"reviewed_at":"2026-10-27","scope":{"en":"The 20-task benchmark and 520-run experiment in arXiv v1. Each model–effort pair remains a separate configuration; pooled model summaries have different denominators.","zh":"对应 arXiv v1 的 20 题、520 次运行实验。模型与推理强度的组合分别保留；跨配置汇总使用不同分母。"}},"publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"model-reports:swe_refactor_bench","source_url":"https://lab.einsia.ai/swe-refactor-bench/"},"repo_kind":null,"repo_resolution_status":null,"score_count":26,"score_direction":"higher_is_better","score_summary":{"display_max":47.0,"display_multiplier":1,"model_count":8,"model_count_basis":"source_model_id","numeric_count":26,"raw_max":47.0,"raw_min":0.0,"source_reference":{"obs_id":"curated\u0000swe_refactor_bench\u0000einsia_swe_refactor_bench_v1_table3\u0000swe_refactor_bench_20_tasks_paper_v1\u0000Claude Code 2.1.220; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100\u0000Claude Opus 5","observation_id":"curated\u0000swe_refactor_bench\u0000einsia_swe_refactor_bench_v1_table3\u0000swe_refactor_bench_20_tasks_paper_v1\u0000Claude Code 2.1.220; reasoning effort xhigh; one independent attempt per each of 20 tasks; per-task 6–30 hour wall-clock budget; no network except model endpoint; migration audit majority of 3 judge samples; all fixed tests required; six independent 1-hour differential verifiers; mean composite out of 100\u0000Claude Opus 5","reported_at":"2026-08-24","reported_date":"2026-08-24","source_id":"einsia_swe_refactor_bench_v1_table3","source_url":"https://arxiv.org/html/2608.23564v1#S3.T3"},"unit":"points"},"scored_models":[{"date":"2026-08-24","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2026-08-24","date_precision":"document_publication","name":"Claude Sonnet 5"},{"date":"2026-08-24","date_precision":"document_publication","name":"DeepSeek V4 Flash"},{"date":"2026-08-24","date_precision":"document_publication","name":"GLM 5.2"},{"date":"2026-08-24","date_precision":"document_publication","name":"GPT-5.6 Luna"},{"date":"2026-08-24","date_precision":"document_publication","name":"GPT-5.6 Sol"},{"date":"2026-08-24","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-08-24","date_precision":"document_publication","name":"Qwen 3.8 Max"}],"slug":"swe_refactor_bench","source":"model_reports","source_url":"https://lab.einsia.ai/swe-refactor-bench/","topics":["coding","agents"],"unit":"points"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":79,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:scaleapi/swe-atlas","source_record":"catalog_223ba126f40f8d4b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE-Atlas is a software engineering benchmark focused on debugging, evaluating a model's ability to localize and fix bugs in real-world codebases.","descriptions":{"en":"SWE-Atlas is a software engineering benchmark focused on debugging, evaluating a model's ability to localize and fix bugs in real-world codebases."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-atlas:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-atlas","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_223ba126f40f8d4b"},"modality":"text","name":"SWE-Atlas","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":35.199999999999996,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.352,"raw_min":0.306,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-atlas:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-swe-atlas","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6022,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:swe-bench/swe-bench","source_record":"lib_swe_bench_multilingual"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multilingual benchmark for issue resolving in software engineering that covers Java, TypeScript, JavaScript, Go, Rust, C, and C++. Contains 1,632 high-quality instances carefully annotated from 2,456 candidates by 68 expert annotators, designed to evaluate Large Language Models across diverse software ecosystems beyond Python.","descriptions":{"en":"A multilingual benchmark for issue resolving in software engineering that covers Java, TypeScript, JavaScript, Go, Rust, C, and C++. Contains 1,632 high-quality instances carefully annotated from 2,456 candidates by 68 expert annotators, designed to evaluate Large Language Models across diverse software ecosystems beyond Python."},"evidence_summary":{"document_count":1,"model_count":38,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-bench-multilingual:deepseek-v3.1","reported_at":"2025-01-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multilingual?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"llm-stats:swe-bench-multilingual","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"lib_swe_bench_multilingual"},"modality":"text","name":"SWE-bench Multilingual","openness":"open","publisher":null,"released":null,"released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":38,"score_direction":"higher_is_better","score_summary":{"display_max":87.3,"display_multiplier":100,"model_count":38,"model_count_basis":"source_model_id","numeric_count":38,"raw_max":0.873,"raw_min":0.305,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-bench-multilingual:claude-mythos-preview","reported_date":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multilingual?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Laguna S 2.1"},{"date":"2026-07-02","date_precision":"model_announcement","name":"Laguna XS 2.1"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-swe-bench-multilingual","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multilingual?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":["SWE-bench Multilingual","SWE Multilingual"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Per-language resolution rates differ widely, so an average hides which languages the model actually handles.","descriptions":{"en":"Per-language resolution rates differ widely, so an average hides which languages the model actually handles."},"evidence_summary":{"document_count":5,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000swe_bench_multilingual\u0000zai_glm_5_model_card\u0000swe_bench_multilingual\u0000OpenHands, tailored instruction prompt, temp 0.7, top_p 0.95, 200K context\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"first_score_reported_at":"2026-02-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000swe_bench_multilingual\u0000zai_glm_5_model_card\u0000swe_bench_multilingual\u0000OpenHands, tailored instruction prompt, temp 0.7, top_p 0.95, 200K context\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:swe_bench_multilingual","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"SWE-bench Multilingual","openness":"unknown","publisher":null,"released":"2025-04-22","released_reference":{"basis":"benchmark_release","source_key":"model-reports:swe_bench_multilingual","source_url":"https://www.swebench.com/multilingual.html"},"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":82.9,"display_multiplier":1,"model_count":5,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":82.9,"raw_min":69.3,"source_reference":{"obs_id":"curated\u0000swe_bench_multilingual\u0000tencent_hy4_preview\u0000swe_bench_multilingual\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000swe_bench_multilingual\u0000tencent_hy4_preview\u0000swe_bench_multilingual\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"swe_bench_multilingual","source":"model_reports","source_url":"https://www.swebench.com/multilingual.html","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":null,"source_record":"catalog_a425fce4110b67d7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","agents","code","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE-Bench Multimodal extends SWE-Bench to evaluate language models on software engineering tasks that involve visual inputs such as screenshots, UI mockups, and diagrams alongside code understanding.","descriptions":{"en":"SWE-Bench Multimodal extends SWE-Bench to evaluate language models on software engineering tasks that involve visual inputs such as screenshots, UI mockups, and diagrams alongside code understanding."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-bench-multimodal:claude-mythos-preview","reported_at":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multimodal?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-bench-multimodal","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","visual-reasoning"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_a425fce4110b67d7"},"modality":"multimodal","name":"SWE-Bench Multimodal","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":59.0,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.59,"raw_min":0.281,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-bench-multimodal:claude-mythos-preview","reported_date":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multimodal?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-swe-bench-multimodal","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multimodal?top_n=500","topics":["coding","agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":542,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"llm_stats_id","repository":null,"source_record":"lib_swe_bench_pro"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE-Bench Pro is an advanced version of SWE-Bench that evaluates language models on complex, real-world software engineering tasks requiring extended reasoning and multi-step problem solving.","descriptions":{"en":"SWE-Bench Pro is an advanced version of SWE-Bench that evaluates language models on complex, real-world software engineering tasks requiring extended reasoning and multi-step problem solving."},"evidence_summary":{"document_count":1,"model_count":50,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-bench-pro:gpt-5.2-codex","reported_at":"2026-01-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-pro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"llm-stats:swe-bench-pro","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"lib_swe_bench_pro"},"modality":"text","name":"SWE-Bench Pro","openness":"restricted","publisher":"Scale AI","released":"2025-09-05","released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":50,"score_direction":"higher_is_better","score_summary":{"display_max":80.0,"display_multiplier":100,"model_count":50,"model_count_basis":"source_model_id","numeric_count":50,"raw_max":0.8,"raw_min":0.402,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-bench-pro:claude-fable-5","reported_date":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-pro?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-01-14","date_precision":"model_announcement","name":"GPT-5.2 Codex"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Laguna S 2.1"},{"date":"2026-07-02","date_precision":"model_announcement","name":"Laguna XS 2.1"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code 1.0"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-swe-bench-pro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-pro?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":["SWE-bench Pro","SWE Pro","SWE-Bench Pro (Public)"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Harness and split version must be recorded with any score.","descriptions":{"en":"Harness and split version must be recorded with any score."},"evidence_summary":{"document_count":10,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000swe_bench_pro\u0000google_gemini_3_1_pro_model_card\u0000swe_bench_pro\u0000Thinking (High), Single attempt\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"first_score_reported_at":"2026-02-19","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000swe_bench_pro\u0000google_gemini_3_1_pro_model_card\u0000swe_bench_pro\u0000Thinking (High), Single attempt\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:swe_bench_pro","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"SWE-bench Pro","openness":"unknown","publisher":null,"released":"2025-09-23","released_reference":{"basis":"benchmark_release","source_key":"model-reports:swe_bench_pro","source_url":"https://github.com/scaleapi/SWE-bench_Pro-os"},"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":79.2,"display_multiplier":1,"model_count":8,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":79.2,"raw_min":52.1,"source_reference":{"obs_id":"curated\u0000swe_bench_pro\u0000anthropic_claude_opus_5_system_card\u0000swe_bench_pro\u0000adaptive thinking, max effort, averaged over 5 trials\u0000Claude Opus 5","observation_id":"curated\u0000swe_bench_pro\u0000anthropic_claude_opus_5_system_card\u0000swe_bench_pro\u0000adaptive thinking, max effort, averaged over 5 trials\u0000Claude Opus 5","reported_at":"2026-07-24","reported_date":"2026-07-24","source_id":"anthropic_claude_opus_5_system_card","source_url":"https://www.anthropic.com/news/claude-opus-5"},"unit":"percent"},"scored_models":[{"date":"2026-07-24","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-07-08","date_precision":"document_publication","name":"Grok 4.5"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"swe_bench_pro","source":"model_reports","source_url":"https://github.com/scaleapi/SWE-bench_Pro-os","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A verified version of SWE-Bench Pro for evaluating software engineering agents on repository-level coding tasks.","descriptions":{"en":"A verified version of SWE-Bench Pro for evaluating software engineering agents on repository-level coding tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08149","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-bench_6e4ae275"},"modality":null,"name":"SWE-Bench Pro Verified","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08149","source_url":"https://arxiv.org/abs/2609.08149"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08149","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08149","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":759,"hf_paper_upvotes":84,"join":"claire_record_id","repository":null,"source_record":"bm_swe-bench_6afbfb26"},"categories":["Code & Software","Code generation","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A multilingual code refactoring benchmark with 170 instances drawn from real commits across seven programming languages. Evaluates AI agents on large-scale refactoring tasks averaging 11.4 modified files and 261.6 lines of code, using manually curated issue descriptions and test suites.","descriptions":{"en":"A multilingual code refactoring benchmark with 170 instances drawn from real commits across seven programming languages. Evaluates AI agents on large-scale refactoring tasks averaging 11.4 modified files and 261.6 lines of code, using manually curated issue descriptions and test suites."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09802","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-bench_6afbfb26"},"modality":null,"name":"SWE-Bench ProMax","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09802","source_url":"https://arxiv.org/abs/2608.09802"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09802","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09802","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":85,"github_stars_30d":5,"hf_dataset_downloads":null,"hf_paper_upvotes":68,"join":"claire_record_id","repository":null,"source_record":"bm_swe-bench_536d758e"},"categories":["Code","Code & Software","General AI","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A repository-level benchmark with 119 tasks from 98 GitHub repositories across 20 scientific domains, organized into issue-driven, expert-exploratory, and engineering-integration paradigms.","descriptions":{"en":"A repository-level benchmark with 119 tasks from 98 GitHub repositories across 20 scientific domains, organized into issue-driven, expert-exploratory, and engineering-integration paradigms."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.19799","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","science"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-bench_536d758e"},"modality":null,"name":"SWE-bench Science","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.19799","source_url":"https://arxiv.org/abs/2608.19799"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-19799","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19799","topics":["coding","agents","science"],"unit":null},{"aliases":["SWE-bench Science","SWE-Bench-Science","SWE Bench Science"],"attention_signals":{"as_of":"2026-10-13","github_stars":85,"github_stars_30d":5,"hf_dataset_downloads":null,"hf_paper_upvotes":68,"join":"artifact:arxiv:2608.19799","repository":null,"source_record":"bm_swe-bench_536d758e"},"categories":["coding_agent"],"collected_at":null,"description":"Repository-level scientific software engineering benchmark from the OpenMOSS team (Fudan): 119 tasks across 98 GitHub repositories in 20 scientific domains, organized into Issue-driven, Expert-exploratory, and Engineering-integration paradigms. Best reported pass@1 is below 50% (Claude Code with Opus-5 max), and the paper identifies four recurring failure mechanisms, so scores are highly sensitive to scientific-domain knowledge and tool access, not only to coding ability.","descriptions":{"en":"Repository-level scientific software engineering benchmark from the OpenMOSS team (Fudan): 119 tasks across 98 GitHub repositories in 20 scientific domains, organized into Issue-driven, Expert-exploratory, and Engineering-integration paradigms. Best reported pass@1 is below 50% (Claude Code with Opus-5 max), and the paper identifies four recurring failure mechanisms, so scores are highly sensitive to scientific-domain knowledge and tool access, not only to coding ability."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000swe_bench_science\u0000swe_bench_science_results_table\u0000swe_bench_science_119_tasks\u0000119 tasks from 98 repositories across 20 scientific domains; overall pass@1 on the reference set; harness: Claude Code\u0000Claude-Opus-5 (max)","reported_at":"2026-09-01","source_url":"https://arxiv.org/html/2608.19799v2"},"first_score_reported_at":"2026-09-01","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000swe_bench_science\u0000swe_bench_science_results_table\u0000swe_bench_science_119_tasks\u0000119 tasks from 98 repositories across 20 scientific domains; overall pass@1 on the reference set; harness: Claude Code\u0000Claude-Opus-5 (max)","reported_at":"2026-09-01","source_url":"https://arxiv.org/html/2608.19799v2"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:swe_bench_science","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","science"],"join":"artifact:arxiv:2608.19799","primary":"agentic-coding","source_record":"bm_swe-bench_536d758e"},"modality":null,"name":"SWE-bench Science","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"model-reports:swe_bench_science","source_url":"https://arxiv.org/abs/2608.19799"},"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":47.9,"display_multiplier":1,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":47.9,"raw_min":14.29,"source_reference":{"obs_id":"curated\u0000swe_bench_science\u0000swe_bench_science_results_table\u0000swe_bench_science_119_tasks\u0000119 tasks from 98 repositories across 20 scientific domains; overall pass@1 on the reference set; harness: Claude Code\u0000Claude-Opus-5 (max)","observation_id":"curated\u0000swe_bench_science\u0000swe_bench_science_results_table\u0000swe_bench_science_119_tasks\u0000119 tasks from 98 repositories across 20 scientific domains; overall pass@1 on the reference set; harness: Claude Code\u0000Claude-Opus-5 (max)","reported_at":"2026-09-01","reported_date":"2026-09-01","source_id":"swe_bench_science_results_table","source_url":"https://arxiv.org/html/2608.19799v2"},"unit":"percent"},"scored_models":[{"date":"2026-09-01","date_precision":"document_publication","name":"Claude-Opus-5 (max)"},{"date":"2026-09-01","date_precision":"document_publication","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-09-01","date_precision":"document_publication","name":"DeepSeek-V4-Pro-Max"},{"date":"2026-09-01","date_precision":"document_publication","name":"GLM-5.2 (max)"},{"date":"2026-09-01","date_precision":"document_publication","name":"GPT-5.6-sol (max)"},{"date":"2026-09-01","date_precision":"document_publication","name":"Kimi-K3 (max)"},{"date":"2026-09-01","date_precision":"document_publication","name":"Nex N2"},{"date":"2026-09-01","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"swe_bench_science","source":"model_reports","source_url":"https://arxiv.org/abs/2608.19799","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":6022,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"llm_stats_id","repository":"gh:swe-bench/swe-bench","source_record":"lib_swe_bench_verified"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","frontend_development","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.","descriptions":{"en":"A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases."},"editorial_guide":{"id":"swe-bench-verified","record_keys":["model-reports:swe_bench_verified","llm-stats:swe-bench-verified"],"reviewed_at":"2026-10-25","scope":{"en":"The 500-instance SWE-bench Verified subset. Original SWE-bench, Lite, Pro and other task collections are separate benchmarks; agent configurations also remain distinct.","zh":"对应 SWE-bench Verified 的 500 条任务。原版 SWE-bench、Lite、Pro 等题集属于独立评测，智能体配置也不能混同。"},"sections":[{"id":"task","source_ids":["introduction","dataset"],"text":{"en":"Verified selects 500 tasks from SWE-bench after human review. Each gives a repository state and issue description; the agent edits code to address the issue. It is a patch-delivery task, broader than writing an isolated function.","zh":"Verified 经人工复核，从 SWE-bench 中选出 500 条任务。每题提供仓库状态和问题描述，由智能体修改代码解决问题。它要求交付补丁，比独立函数编写覆盖更广。"},"title":{"en":"Resolve a real repository issue","zh":"解决真实仓库中的问题"}},{"id":"grading","source_ids":["introduction","evaluation"],"text":{"en":"FAIL_TO_PASS tests check the bug fix, while PASS_TO_PASS tests check retained behavior. Both are required for resolution. The harness applies the proposed patch and runs tests in a container; saying the issue is solved is not a passing result.","zh":"FAIL_TO_PASS 测试检查问题是否修好，PASS_TO_PASS 测试检查已有行为是否保留；两类都需通过。框架在容器中应用补丁并运行测试，口头声明已解决并不算通过。"},"title":{"en":"Required tests must pass","zh":"必须通过指定验收测试"}},{"id":"systems","source_ids":["leaderboard"],"text":{"en":"The official site separates a broad systems leaderboard from bash-only mini-SWE-agent runs. It warns that agent releases can change comparability. Model name alone omits the scaffold, tools, retry budget and execution policy that helped produce a patch.","zh":"官方站点区分综合系统榜单与仅使用 bash 的 mini-SWE-agent 实验，并说明智能体版本变化会影响可比性。只有模型名会遗漏框架、工具、重试预算和执行策略。"},"title":{"en":"Compare agents with their operating conditions","zh":"连同智能体运行条件一起比较"}},{"id":"failures","source_ids":["evaluation"],"text":{"en":"The harness reports resolved, unresolved, incomplete and error cases separately. Infrastructure failures, missing submissions and failing tests are not the same diagnosis. Inspect the denominator and logs before comparing runs that completed different subsets.","zh":"框架分别报告已解决、未解决、未完成和运行错误。基础设施故障、未提交与测试失败是不同情况。若两次运行实际完成的题集不同，应先核对分母与日志。"},"title":{"en":"Distinguish wrong patches from failed runs","zh":"区分补丁错误与运行失败"}},{"id":"limits","source_ids":["audit"],"text":{"en":"In February 2026, OpenAI reported flawed tests in an audited difficult subset and evidence of training exposure, and stopped reporting Verified for frontier launches. This does not erase historical observations; it makes test quality and contamination essential context.","zh":"2026 年 2 月，OpenAI 在抽查的困难任务中报告测试缺陷和训练暴露证据，并停止在前沿模型发布中报告 Verified。这不抹去历史观察，但意味着必须结合测试质量和污染情况阅读。"},"title":{"en":"Read the 2026 audit with the scores","zh":"结合 2026 年审计阅读成绩"}}],"sources":[{"id":"introduction","title":"OpenAI · Introducing SWE-bench Verified","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"id":"dataset","title":"Princeton NLP · SWE-bench Verified dataset","url":"https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified"},{"id":"evaluation","title":"SWE-bench · Evaluation guide and result counts","url":"https://www.swebench.com/SWE-bench/guides/evaluation/"},{"id":"leaderboard","title":"SWE-bench Verified · Systems and bash-only evaluation setup","url":"https://www.swebench.com/verified.html"},{"id":"audit","title":"OpenAI · Why SWE-bench Verified no longer measures frontier coding capabilities","url":"https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/"}],"summary":{"en":"An agent receives a repository and an issue, then submits a patch. Verified remains useful for inspecting historical results, but documented test and contamination issues limit frontier comparisons.","zh":"智能体收到代码仓库与 issue 后，需要提交修复补丁。Verified 可用于查看历史结果，但已公开的测试缺陷与污染问题限制了它对前沿能力的比较价值。"},"title":"SWE-bench Verified"},"evidence_summary":{"document_count":1,"model_count":111,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-bench-verified:deepseek-v2.5","reported_at":"2024-05-08","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"llm-stats:swe-bench-verified","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"lib_swe_bench_verified"},"modality":"text","name":"SWE-Bench Verified","openness":"restricted","publisher":"OpenAI","released":"2024-08-13","released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":111,"score_direction":"higher_is_better","score_summary":{"display_max":95.0,"display_multiplier":100,"model_count":111,"model_count_basis":"source_model_id","numeric_count":111,"raw_max":0.95,"raw_min":0.087,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-bench-verified:claude-fable-5","reported_date":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Opus 4"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude Opus 4.1"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Sonnet 4"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2024-05-08","date_precision":"model_announcement","name":"DeepSeek-V2.5"},{"date":"2024-12-25","date_precision":"model_announcement","name":"DeepSeek-V3"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2-Speciale"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Devstral Small 1.1"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini Diffusion"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-09-15","date_precision":"model_announcement","name":"GPT-5 Codex"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1"},{"date":"2025-11-19","date_precision":"model_announcement","name":"GPT-5.1 Codex"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Instant"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Thinking"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-02","date_precision":"model_announcement","name":"Laguna XS 2.1"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2026-08-11","date_precision":"model_announcement","name":"MAI-Code-1.1-Flash"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax M2.5"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code 1.0"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2024-09-12","date_precision":"model_announcement","name":"o1-preview"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-105B"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam-30B"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Lite"},{"date":"2026-02-14","date_precision":"model_announcement","name":"Seed 2.0 Pro"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step-3.5-Flash"}],"slug":"llm-stats-swe-bench-verified","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":["SWE-bench Verified","SWE-bench-Verified","SWE Bench Verified","SWE Verified"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Scaffold, tool permissions and time limit are part of the result; vendors run their own harnesses.","descriptions":{"en":"Scaffold, tool permissions and time limit are part of the result; vendors run their own harnesses."},"editorial_guide":{"id":"swe-bench-verified","record_keys":["model-reports:swe_bench_verified","llm-stats:swe-bench-verified"],"reviewed_at":"2026-10-25","scope":{"en":"The 500-instance SWE-bench Verified subset. Original SWE-bench, Lite, Pro and other task collections are separate benchmarks; agent configurations also remain distinct.","zh":"对应 SWE-bench Verified 的 500 条任务。原版 SWE-bench、Lite、Pro 等题集属于独立评测，智能体配置也不能混同。"},"sections":[{"id":"task","source_ids":["introduction","dataset"],"text":{"en":"Verified selects 500 tasks from SWE-bench after human review. Each gives a repository state and issue description; the agent edits code to address the issue. It is a patch-delivery task, broader than writing an isolated function.","zh":"Verified 经人工复核，从 SWE-bench 中选出 500 条任务。每题提供仓库状态和问题描述，由智能体修改代码解决问题。它要求交付补丁，比独立函数编写覆盖更广。"},"title":{"en":"Resolve a real repository issue","zh":"解决真实仓库中的问题"}},{"id":"grading","source_ids":["introduction","evaluation"],"text":{"en":"FAIL_TO_PASS tests check the bug fix, while PASS_TO_PASS tests check retained behavior. Both are required for resolution. The harness applies the proposed patch and runs tests in a container; saying the issue is solved is not a passing result.","zh":"FAIL_TO_PASS 测试检查问题是否修好，PASS_TO_PASS 测试检查已有行为是否保留；两类都需通过。框架在容器中应用补丁并运行测试，口头声明已解决并不算通过。"},"title":{"en":"Required tests must pass","zh":"必须通过指定验收测试"}},{"id":"systems","source_ids":["leaderboard"],"text":{"en":"The official site separates a broad systems leaderboard from bash-only mini-SWE-agent runs. It warns that agent releases can change comparability. Model name alone omits the scaffold, tools, retry budget and execution policy that helped produce a patch.","zh":"官方站点区分综合系统榜单与仅使用 bash 的 mini-SWE-agent 实验，并说明智能体版本变化会影响可比性。只有模型名会遗漏框架、工具、重试预算和执行策略。"},"title":{"en":"Compare agents with their operating conditions","zh":"连同智能体运行条件一起比较"}},{"id":"failures","source_ids":["evaluation"],"text":{"en":"The harness reports resolved, unresolved, incomplete and error cases separately. Infrastructure failures, missing submissions and failing tests are not the same diagnosis. Inspect the denominator and logs before comparing runs that completed different subsets.","zh":"框架分别报告已解决、未解决、未完成和运行错误。基础设施故障、未提交与测试失败是不同情况。若两次运行实际完成的题集不同，应先核对分母与日志。"},"title":{"en":"Distinguish wrong patches from failed runs","zh":"区分补丁错误与运行失败"}},{"id":"limits","source_ids":["audit"],"text":{"en":"In February 2026, OpenAI reported flawed tests in an audited difficult subset and evidence of training exposure, and stopped reporting Verified for frontier launches. This does not erase historical observations; it makes test quality and contamination essential context.","zh":"2026 年 2 月，OpenAI 在抽查的困难任务中报告测试缺陷和训练暴露证据，并停止在前沿模型发布中报告 Verified。这不抹去历史观察，但意味着必须结合测试质量和污染情况阅读。"},"title":{"en":"Read the 2026 audit with the scores","zh":"结合 2026 年审计阅读成绩"}}],"sources":[{"id":"introduction","title":"OpenAI · Introducing SWE-bench Verified","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"id":"dataset","title":"Princeton NLP · SWE-bench Verified dataset","url":"https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified"},{"id":"evaluation","title":"SWE-bench · Evaluation guide and result counts","url":"https://www.swebench.com/SWE-bench/guides/evaluation/"},{"id":"leaderboard","title":"SWE-bench Verified · Systems and bash-only evaluation setup","url":"https://www.swebench.com/verified.html"},{"id":"audit","title":"OpenAI · Why SWE-bench Verified no longer measures frontier coding capabilities","url":"https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/"}],"summary":{"en":"An agent receives a repository and an issue, then submits a patch. Verified remains useful for inspecting historical results, but documented test and contamination issues limit frontier comparisons.","zh":"智能体收到代码仓库与 issue 后，需要提交修复补丁。Verified 可用于查看历史结果，但已公开的测试缺陷与污染问题限制了它对前沿能力的比较价值。"},"title":"SWE-bench Verified"},"evidence_summary":{"document_count":19,"model_count":14,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000swe_bench_verified\u0000deepseek_v3_report\u0000swe_bench_verified\u0000Resolved, vendor harness\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"first_score_reported_at":"2024-12-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000swe_bench_verified\u0000deepseek_v3_report\u0000swe_bench_verified\u0000Resolved, vendor harness\u0000DeepSeek-V3","reported_at":"2024-12-27","source_url":"https://arxiv.org/abs/2412.19437"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:swe_bench_verified","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"SWE-bench Verified","openness":"unknown","publisher":null,"released":"2024-08-13","released_reference":{"basis":"benchmark_release","source_key":"model-reports:swe_bench_verified","source_url":"https://openai.com/index/introducing-swe-bench-verified/"},"repo_kind":null,"repo_resolution_status":null,"score_count":18,"score_direction":"higher_is_better","score_summary":{"display_max":96.0,"display_multiplier":1,"model_count":14,"model_count_basis":"source_model_id","numeric_count":18,"raw_max":96.0,"raw_min":42.0,"source_reference":{"obs_id":"curated\u0000swe_bench_verified\u0000anthropic_claude_opus_5_system_card\u0000swe_bench_verified\u0000adaptive thinking, max effort, averaged over 5 trials\u0000Claude Opus 5","observation_id":"curated\u0000swe_bench_verified\u0000anthropic_claude_opus_5_system_card\u0000swe_bench_verified\u0000adaptive thinking, max effort, averaged over 5 trials\u0000Claude Opus 5","reported_at":"2026-07-24","reported_date":"2026-07-24","source_id":"anthropic_claude_opus_5_system_card","source_url":"https://www.anthropic.com/news/claude-opus-5"},"unit":"percent"},"scored_models":[{"date":"2025-02-24","date_precision":"document_publication","name":"Claude 3.7 Sonnet"},{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Opus 4"},{"date":"2026-07-24","date_precision":"document_publication","name":"Claude Opus 5"},{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Sonnet 4"},{"date":"2025-01-22","date_precision":"document_publication","name":"DeepSeek-R1"},{"date":"2024-12-27","date_precision":"document_publication","name":"DeepSeek-V3"},{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2025-06-17","date_precision":"document_publication","name":"Gemini 2.5 Pro"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"swe_bench_verified","source":"model_reports","source_url":"https://openai.com/index/introducing-swe-bench-verified/","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":6022,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:swe-bench/swe-bench","source_record":"catalog_99b76d115af92ce1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE-bench Verified is a human-filtered subset of 500 software engineering problems drawn from real GitHub issues across 12 popular Python repositories. Given a codebase and an issue description, language models are tasked with generating patches that resolve the described problems. This benchmark evaluates AI's real-world agentic coding skills by requiring models to navigate complex codebases, understand software engineering problems, and coordinate changes across multiple functions, classes, and files to fix well-defined issues with clear descriptions.","descriptions":{"en":"SWE-bench Verified is a human-filtered subset of 500 software engineering problems drawn from real GitHub issues across 12 popular Python repositories. Given a codebase and an issue description, language models are tasked with generating patches that resolve the described problems. This benchmark evaluates AI's real-world agentic coding skills by requiring models to navigate complex codebases, understand software engineering problems, and coordinate changes across multiple functions, classes, and files to fix well-defined issues with clear descriptions."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-bench-verified-(agentic-coding):kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentic-coding%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-bench-verified-(agentic-coding)","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_99b76d115af92ce1"},"modality":"text","name":"SWE-bench Verified (Agentic Coding)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":77.2,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.772,"raw_min":0.658,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-bench-verified-(agentic-coding):claude-sonnet-4-5-20250929","reported_date":"2025-09-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentic-coding%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude Sonnet 4.5"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"}],"slug":"llm-stats-swe-bench-verified-agentic-coding","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentic-coding%29?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":6022,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:swe-bench/swe-bench","source_record":"catalog_1896e47a987f951c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A human-validated subset of SWE-bench that evaluates language models' ability to resolve real-world GitHub issues using an agentless approach. The benchmark tests models on software engineering problems requiring understanding and coordinating changes across multiple functions, classes, and files simultaneously.","descriptions":{"en":"A human-validated subset of SWE-bench that evaluates language models' ability to resolve real-world GitHub issues using an agentless approach. The benchmark tests models on software engineering problems requiring understanding and coordinating changes across multiple functions, classes, and files simultaneously."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-bench-verified-(agentless):kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentless%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-bench-verified-(agentless)","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_1896e47a987f951c"},"modality":"text","name":"SWE-bench Verified (Agentless)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":51.800000000000004,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.518,"raw_min":0.357,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-bench-verified-(agentless):kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentless%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"}],"slug":"llm-stats-swe-bench-verified-agentless","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentless%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":6022,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:swe-bench/swe-bench","source_record":"catalog_42c12ab99c422695"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE-bench Verified is a human-validated subset of 500 test samples from the original SWE-bench dataset that evaluates AI systems' ability to automatically resolve real GitHub issues in Python repositories. Given a codebase and issue description, models must edit the code to successfully resolve the problem, requiring understanding and coordination of changes across multiple functions, classes, and files. The Verified version provides more reliable evaluation through manual validation of test samples.","descriptions":{"en":"SWE-bench Verified is a human-validated subset of 500 test samples from the original SWE-bench dataset that evaluates AI systems' ability to automatically resolve real GitHub issues in Python repositories. Given a codebase and issue description, models must edit the code to successfully resolve the problem, requiring understanding and coordination of changes across multiple functions, classes, and files. The Verified version provides more reliable evaluation through manual validation of test samples."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-bench-verified-(multiple-attempts):kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28multiple-attempts%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-bench-verified-(multiple-attempts)","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_42c12ab99c422695"},"modality":"text","name":"SWE-bench Verified (Multiple Attempts)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":71.6,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.716,"raw_min":0.716,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-bench-verified-(multiple-attempts):kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28multiple-attempts%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"}],"slug":"llm-stats-swe-bench-verified-multiple-attempts","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28multiple-attempts%29?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","智能体","Agent","任务执行","Task Execution","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SWE-bench-Live is a live-updatable benchmark designed for evaluating large language models (LLMs) and agents on real-world software issue resolution tasks. SWE-bench-Live 是一个面向大语言模型（LLMs）和智能体的实时可更新评测基准，专注于真实世界软件开发中的问题修复任务。 该基准从 2024 年以来的 GitHub 活跃仓库中自动收集了 1,319 个问题修复任务，涵盖 93 个项目，并为每个任务提供可复现的 Docker 执行环境。","descriptions":{"en":"SWE-bench-Live is a live-updatable benchmark designed for evaluating large language models (LLMs) and agents on real-world software issue resolution tasks.","zh":"SWE-bench-Live 是一个面向大语言模型（LLMs）和智能体的实时可更新评测基准，专注于真实世界软件开发中的问题修复任务。 该基准从 2024 年以来的 GitHub 活跃仓库中自动收集了 1,319 个问题修复任务，涵盖 93 个项目，并为每个任务提供可复现的 Docker 执行环境。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1984","languages":[],"library_direction":null,"modality":null,"name":"SWE-bench-Live","openness":"unknown","publisher":"Microsoft","released":"2025-06-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1984-swe-bench-live","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SWE-bench-Live","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"ArXiv paper introducing a benchmark for deceptive behaviors of LLMs across the software development lifecycle, with abstract stating tasks and results, but no public artifact available.","descriptions":{"en":"ArXiv paper introducing a benchmark for deceptive behaviors of LLMs across the software development lifecycle, with abstract stating tasks and results, but no public artifact available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13626","languages":[],"library_direction":{"basis":"model-review-daily","categories":["safety-alignment","agentic-coding"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_swe-deceptionbench_45996ccd"},"modality":null,"name":"SWE-DeceptionBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13626","source_url":"https://arxiv.org/abs/2610.13626"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13626","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13626","topics":["coding","agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":43,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":123,"join":"claire_record_id","repository":"gh:qiushao-e/swe-explore-bench","source_record":"bm_swe-explore_ece5d09b"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"SWE-Explore evaluates repository exploration by coding agents: given a repository and an issue, an explorer returns a ranked list of relevant code regions under a fixed line budget. Ground truth is line-level, derived from successful repair trajectories. Coverage, ranking, and context-efficiency metrics are scored.","descriptions":{"en":"SWE-Explore evaluates repository exploration by coding agents: given a repository and an issue, an explorer returns a ranked list of relevant code regions under a fixed line budget. Ground truth is line-level, derived from successful repair trajectories. Coverage, ranking, and context-efficiency metrics are scored."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07297","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-explore_ece5d09b"},"modality":null,"name":"SWE-Explore","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07297","source_url":"https://arxiv.org/abs/2606.07297"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07297","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07297","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","智能体","Agent","任务执行","Task Execution","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SWE-Factory is a benchmark for evaluating large language models on software issue fixing tasks. SWE-Factory 是一个面向大型语言模型的软件问题修复评测基准，旨在提升构建效率与评估准确性。该基准集成多智能体系统 SWE-Builder 自动搭建任务环境，采用退出码自动评分，并通过 fail2pass 流程验证修复有效性，确保评测可靠。SWE-Factory 覆盖四种语言共 671 个问题，支持高效、自动化、可扩展的 LLM 评估流程。","descriptions":{"en":"SWE-Factory is a benchmark for evaluating large language models on software issue fixing tasks.","zh":"SWE-Factory 是一个面向大型语言模型的软件问题修复评测基准，旨在提升构建效率与评估准确性。该基准集成多智能体系统 SWE-Builder 自动搭建任务环境，采用退出码自动评分，并通过 fail2pass 流程验证修复有效性，确保评测可靠。SWE-Factory 覆盖四种语言共 671 个问题，支持高效、自动化、可扩展的 LLM 评估流程。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1989","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"artifact:arxiv:2310.06770","primary":"agentic-coding","source_record":"catalog_1896e47a987f951c"},"modality":null,"name":"SWE-Factory","openness":"unknown","publisher":"SunYat-senUniversity , IndependentResearcher , Huawei.","released":"2025-06-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1989-swe-factory","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SWE-Factory","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_10a5cb2d197c84a9"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE-fficiency is an open-source benchmark and workflow that evaluates language models on optimizing the runtime efficiency of real-world software engineering tasks, measuring how well agents can improve code performance autonomously.","descriptions":{"en":"SWE-fficiency is an open-source benchmark and workflow that evaluates language models on optimizing the runtime efficiency of real-world software engineering tasks, measuring how well agents can improve code performance autonomously."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-fficiency:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-fficiency?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-fficiency","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","ai-rnd"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_10a5cb2d197c84a9"},"modality":"text","name":"SWE-fficiency","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":34.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.348,"raw_min":0.348,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-fficiency:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-fficiency?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-swe-fficiency","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-fficiency?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":6,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"claire_record_id","repository":null,"source_record":"bm_swe-game_b96161ab"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D.","descriptions":{"en":"We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.33678","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","multi-agent-games"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-game_b96161ab"},"modality":null,"name":"SWE-Game","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.33678","source_url":"https://arxiv.org/abs/2609.33678"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-33678","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33678","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:deepsoftwareanalytics/swe-gate","source_record":"bm_swe-gate_9dd200ce"},"categories":["Code","Code & Software","General AI","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates repository-level repair agents on both functional correctness and adherence to review-derived constraints, using 303 instances with separate functional and constraint tests.","descriptions":{"en":"Evaluates repository-level repair agents on both functional correctness and adherence to review-derived constraints, using 303 instances with separate functional and constraint tests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.04167","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-gate_9dd200ce"},"modality":null,"name":"SWE-Gate","openness":"unknown","publisher":null,"released":"2026-09-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.04167","source_url":"https://arxiv.org/abs/2609.04167"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-04167","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04167","topics":["coding","agents"],"unit":null},{"aliases":["SWE-InfraBench: Evaluating Language Models on Cloud Infrastructure Code"],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark of incremental infrastructure-as-code modifications in real-world AWS CDK repositories, evaluated through provided test cases.","descriptions":{"en":"A benchmark of incremental infrastructure-as-code modifications in real-world AWS CDK repositories, evaluated through provided test cases."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05249","languages":[],"library_direction":null,"modality":null,"name":"SWE-InfraBench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05249","source_url":"https://arxiv.org/abs/2606.05249"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05249","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05249","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":28,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"claire_record_id","repository":"gh:scaleapi/swe-interact","source_record":"bm_swe-interact_046bbdbf"},"categories":["Agents & Tool Use","Code","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"SWE-Interact evaluates coding agents on multi-turn, interactive software engineering tasks where a simulated user provides vague instructions, reveals requirements progressively, and gives feedback. The benchmark comprises 75 tasks and measures agents' ability to discover user intent, adapt to evolving requirements, and build on prior work.","descriptions":{"en":"SWE-Interact evaluates coding agents on multi-turn, interactive software engineering tasks where a simulated user provides vague instructions, reveals requirements progressively, and gives feedback. The benchmark comprises 75 tasks and measures agents' ability to discover user intent, adapt to evolving requirements, and build on prior work."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.30573","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-interact_046bbdbf"},"modality":null,"name":"SWE-INTERACT","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.30573","source_url":"https://arxiv.org/abs/2606.30573"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-30573","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30573","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Coding assistants such as Claude Code and Codex have become a major application of LLM agents, yet existing benchmarks remain far from real-world use, particularly in task horizon and interaction length.","descriptions":{"en":"Coding assistants such as Claude Code and Codex have become a major application of LLM agents, yet existing benchmarks remain far from real-world use, particularly in task horizon and interaction length."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11559","languages":[],"library_direction":{"basis":"model-review-daily","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-journey_aaa8c744"},"modality":null,"name":"SWE-Journey","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11559","source_url":"https://arxiv.org/abs/2610.11559"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11559","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11559","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":1430,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":46,"join":"llm_stats_id","repository":null,"source_record":"catalog_bbef19a7e22e351d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A benchmark for evaluating large language models on real-world freelance software engineering tasks from Upwork. Contains over 1,400 tasks valued at $1 million USD total, ranging from $50 bug fixes to $32,000 feature implementations. Includes both independent engineering tasks graded via end-to-end tests and managerial tasks assessed against original engineering managers' choices.","descriptions":{"en":"A benchmark for evaluating large language models on real-world freelance software engineering tasks from Upwork. Contains over 1,400 tasks valued at $1 million USD total, ranging from $50 bug fixes to $32,000 feature implementations. Includes both independent engineering tasks graded via end-to-end tests and managerial tasks assessed against original engineering managers' choices."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-lancer:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-lancer","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","professional-work"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_bbef19a7e22e351d"},"modality":"text","name":"SWE-Lancer","openness":"unknown","publisher":null,"released":"2025-02-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:swe-lancer","source_url":"https://arxiv.org/abs/2502.12115"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":66.3,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.663,"raw_min":0.18,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-lancer:gpt-5.1-codex","reported_date":"2025-11-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-11-19","date_precision":"model_announcement","name":"GPT-5.1 Codex"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"}],"slug":"llm-stats-swe-lancer","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":["SWE-Lancer"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Software engineering freelancer tasks from Upwork; real-world constraints and client communication are part of the task.","descriptions":{"en":"Software engineering freelancer tasks from Upwork; real-world constraints and client communication are part of the task."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000swe_lancer\u0000bytedance_seed2_0_model_card\u0000swe_lancer\u0000as printed in Table 11 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000swe_lancer\u0000bytedance_seed2_0_model_card\u0000swe_lancer\u0000as printed in Table 11 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:swe_lancer","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"SWE-Lancer","openness":"unknown","publisher":null,"released":"2025-02-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"model-reports:swe_lancer","source_url":"https://arxiv.org/abs/2502.12115"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":49.4,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":49.4,"raw_min":47.1,"source_reference":{"obs_id":"curated\u0000swe_lancer\u0000bytedance_seed2_0_model_card\u0000swe_lancer\u0000as printed in Table 11 (Large Models)\u0000Seed2.0 Pro","observation_id":"curated\u0000swe_lancer\u0000bytedance_seed2_0_model_card\u0000swe_lancer\u0000as printed in Table 11 (Large Models)\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"swe_lancer","source":"model_reports","source_url":"https://github.com/openai/frontier-evals/tree/main/project/swelancer","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1430,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":46,"join":"llm_stats_id","repository":null,"source_record":"catalog_a8c2a6f6c2e62be3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE-Lancer (IC-Diamond subset) is a benchmark of real-world freelance software engineering tasks from Upwork, ranging from $50 bug fixes to $32,000 feature implementations. It evaluates AI models on independent engineering tasks using end-to-end tests triple-verified by experienced software engineers, and includes managerial tasks where models choose between technical implementation proposals.","descriptions":{"en":"SWE-Lancer (IC-Diamond subset) is a benchmark of real-world freelance software engineering tasks from Upwork, ranging from $50 bug fixes to $32,000 feature implementations. It evaluates AI models on independent engineering tasks using end-to-end tests triple-verified by experienced software engineers, and includes managerial tasks where models choose between technical implementation proposals."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-lancer-(ic-diamond-subset):gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer-%28ic-diamond-subset%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-lancer-(ic-diamond-subset)","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","professional-work"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_a8c2a6f6c2e62be3"},"modality":"text","name":"SWE-Lancer (IC-Diamond subset)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":100.0,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":1.0,"raw_min":0.074,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-lancer-(ic-diamond-subset):gpt-5-2025-08-07","reported_date":"2025-08-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer-%28ic-diamond-subset%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"}],"slug":"llm-stats-swe-lancer-ic-diamond-subset","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer-%28ic-diamond-subset%29?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_swe-marathon_2e8e782b"},"categories":["Code","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"SWE-Marathon evaluates AI agents on 20 ultra-long-horizon software engineering tasks, each with a unique executable environment, a human-written reference solution, and a multi-layer verification suite. Tasks average 27.2M tokens per logged agent attempt, requiring sustained progress over hours and millions of tokens.","descriptions":{"en":"SWE-Marathon evaluates AI agents on 20 ultra-long-horizon software engineering tasks, each with a unique executable environment, a human-written reference solution, and a multi-layer verification suite. Tasks average 27.2M tokens per logged agent attempt, requiring sustained progress over hours and millions of tokens."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07682","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-marathon_2e8e782b"},"modality":null,"name":"SWE-Marathon","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07682","source_url":"https://arxiv.org/abs/2606.07682"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07682","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07682","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"bm_swe-marathon_2e8e782b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE-Marathon is an ultra-long-horizon software engineering benchmark covering tasks such as building compilers, optimizing kernels, and developing production-grade services. It measures whether agents can sustain quality across extremely long engineering trajectories.","descriptions":{"en":"SWE-Marathon is an ultra-long-horizon software engineering benchmark covering tasks such as building compilers, optimizing kernels, and developing production-grade services. It measures whether agents can sustain quality across extremely long engineering trajectories."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-marathon:glm-5.2","reported_at":"2026-06-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-marathon?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-marathon","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"bm_swe-marathon_2e8e782b"},"modality":"text","name":"SWE-Marathon","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":42.5,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.425,"raw_min":0.13,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-marathon:glm-5.3","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-marathon?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"}],"slug":"llm-stats-swe-marathon","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-marathon?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["SWE-Marathon","SWE Marathon"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"Ultra long-horizon software engineering tasks; the score depends on the agent's loop budget as much as on coding skill.","descriptions":{"en":"Ultra long-horizon software engineering tasks; the score depends on the agent's loop budget as much as on coding skill."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000swe_marathon\u0000tencent_hy4_preview\u0000swe_marathon\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000swe_marathon\u0000tencent_hy4_preview\u0000swe_marathon\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:swe_marathon","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"SWE-Marathon","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":31.9,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":31.9,"raw_min":31.9,"source_reference":{"obs_id":"curated\u0000swe_marathon\u0000tencent_hy4_preview\u0000swe_marathon\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000swe_marathon\u0000tencent_hy4_preview\u0000swe_marathon\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"swe_marathon","source":"model_reports","source_url":"https://github.com/abundant-ai/swe-marathon","topics":["coding","agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":6022,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:swe-bench/swe-bench","source_record":"catalog_d59473cf8cd9e648"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","code","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"SWE-MM evaluates software-engineering agents on repository tasks that require understanding both source code and visual evidence.","descriptions":{"en":"SWE-MM evaluates software-engineering agents on repository tasks that require understanding both source code and visual evidence."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-mm:qwen3.8-27b","reported_at":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-mm?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-mm","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","visual-reasoning"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_d59473cf8cd9e648"},"modality":"multimodal","name":"SWE-MM","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":38.6,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.386,"raw_min":0.386,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-mm:qwen3.8-27b","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-mm?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-swe-mm","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-mm?top_n=500","topics":["coding","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"SWE-Mutation evaluates LLM-generated test suites in software engineering by using 2,636 mutated variants derived from 800 original instances across nine programming languages, measuring verification and detection rates.","descriptions":{"en":"SWE-Mutation evaluates LLM-generated test suites in software engineering by using 2,636 mutated variants derived from 800 original instances across nine programming languages, measuring verification and detection rates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22175","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-mutation_d27a11bf"},"modality":null,"name":"SWE-Mutation","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22175","source_url":"https://arxiv.org/abs/2605.22175"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22175","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22175","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of 188 tasks for evaluating coding agents on non-functional improvements in Python projects, with 92 executable rules combining functional correctness and rule-based evaluation.","descriptions":{"en":"A benchmark of 188 tasks for evaluating coding agents on non-functional improvements in Python projects, with 92 executable rules combining functional correctness and rule-based evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.27409","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-nfi_7cf5877f"},"modality":null,"name":"SWE-NFI","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27409","source_url":"https://arxiv.org/abs/2607.27409"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27409","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27409","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":54,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":44,"join":"llm_stats_id","repository":null,"source_record":"catalog_b7ec09b15b4763be"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Software Engineering Performance benchmark measuring code optimization capabilities","descriptions":{"en":"Software Engineering Performance benchmark measuring code optimization capabilities"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-perf:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-perf?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-perf","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","ai-rnd"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_b7ec09b15b4763be"},"modality":"text","name":"SWE-Perf","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":3.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.031,"raw_min":0.031,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-perf:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-perf?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-swe-perf","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-perf?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"SWE-PolyVision is described as an executable benchmark of 92 real tasks with public and private holdouts, but no access to tasks, scorer, or submission mechanism is provided.","descriptions":{"en":"SWE-PolyVision is described as an executable benchmark of 92 real tasks with public and private holdouts, but no access to tasks, scorer, or submission mechanism is provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.29754","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-polyvision_05b40afd"},"modality":null,"name":"SWE-PolyVision","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.29754","source_url":"https://arxiv.org/abs/2609.29754"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-29754","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29754","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"SWE-Prometheus is presented as a benchmark for improving repository engineering governance, but no dataset, scoring rules, or evaluation interface are available.","descriptions":{"en":"SWE-Prometheus is presented as a benchmark for improving repository engineering governance, but no dataset, scoring rules, or evaluation interface are available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.29465","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-prometheus_191d8ce8"},"modality":null,"name":"SWE-Prometheus","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.29465","source_url":"https://arxiv.org/abs/2609.29465"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-29465","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29465","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":15,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"llm_stats_id","repository":null,"source_record":"catalog_e12892e6d3b8f160"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Software Engineering Review benchmark evaluating code review capabilities","descriptions":{"en":"Software Engineering Review benchmark evaluating code review capabilities"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swe-review:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-review?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swe-review","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","code-generation"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_e12892e6d3b8f160"},"modality":"text","name":"SWE-Review","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":8.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.089,"raw_min":0.089,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swe-review:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-review?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-swe-review","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-review?top_n=500","topics":["coding"],"unit":null},{"aliases":["A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents"],"attention_signals":{"as_of":"2026-10-11","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:xin-zhou-smu/swe-rpg-bench","source_record":"bm_a-unified-issue-resolution-benchmark-for-r_de70fd7d"},"categories":["Code","Code & Software","Code generation","Planning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A repository-level dataset and protocol evaluating issue resolution through requirement clarification, implementation planning, code generation, and artifact submission, with 163 tasks across 31 Python and Java repositories and validated intermediate ground-truth references.","descriptions":{"en":"A repository-level dataset and protocol evaluating issue resolution through requirement clarification, implementation planning, code generation, and artifact submission, with 163 tasks across 31 Python and Java repositories and validated intermediate ground-truth references."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.09072","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_a-unified-issue-resolution-benchmark-for-r_de70fd7d"},"modality":null,"name":"SWE-RPG","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09072","source_url":"https://arxiv.org/abs/2608.09072"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09072","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09072","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Pending verification of task definition and scoring contract for production inference serving agent evaluation.","descriptions":{"en":"Pending verification of task definition and scoring contract for production inference serving agent evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.26777","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding","finance-legal"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-serve_7f2cf9a9"},"modality":null,"name":"SWE-Serve","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.26777","source_url":"https://arxiv.org/abs/2609.26777"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-26777","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26777","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":77,"github_stars_30d":13,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"claire_record_id","repository":"gh:togetherbench/swe-together","source_record":"bm_swe-together_260e2683"},"categories":["Agents","Agents & Tool Use","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"SWE-Together evaluates coding agents in multi-turn interactive user sessions reconstructed from real user-agent interactions. It comprises 109 repository-level tasks with a reactive LLM-based user simulator, measuring final repository correctness and the number of corrective feedback turns.","descriptions":{"en":"SWE-Together evaluates coding agents in multi-turn interactive user sessions reconstructed from real user-agent interactions. It comprises 109 repository-level tasks with a reactive LLM-based user simulator, measuring final repository correctness and the number of corrective feedback turns."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.29957","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_swe-together_260e2683"},"modality":null,"name":"SWE-Together","openness":"unknown","publisher":null,"released":"2026-06-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29957","source_url":"https://arxiv.org/abs/2606.29957"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29957","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29957","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SwiLTra-Bench is a comprehensive multilingual benchmark of over 180K aligned Swiss legal translation pairs comprising laws, headnotes, and press releases across all Swiss languages along with English, designed to evaluate LLM-based translation systems. SwiLTra-Bench是一个包含超过 18 万对对齐的瑞士法律翻译语料库的全面多语言基准，包括所有瑞士语言以及英语的法律、摘要和新闻稿，旨在评估基于LLM的翻译系统。","descriptions":{"en":"SwiLTra-Bench is a comprehensive multilingual benchmark of over 180K aligned Swiss legal translation pairs comprising laws, headnotes, and press releases across all Swiss languages along with English, designed to evaluate LLM-based translation systems.","zh":"SwiLTra-Bench是一个包含超过 18 万对对齐的瑞士法律翻译语料库的全面多语言基准，包括所有瑞士语言以及英语的法律、摘要和新闻稿，旨在评估基于LLM的翻译系统。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1608","languages":["English","French","German","Italian","Multilingual"],"library_direction":null,"modality":null,"name":"SwiLTra-Bench","openness":"unknown","publisher":"Harvey, ETHZurich, etc.","released":"2025-03-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1608-swiltra-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SwiLTra-Bench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":1733,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_swisscrop25_6e928630"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates crop mapping models on national-scale multi-year Sentinel-2 data with fine-grained crop taxonomy and non-crop classes under leave-one-year-out protocol.","descriptions":{"en":"Evaluates crop mapping models on national-scale multi-year Sentinel-2 data with fine-grained crop taxonomy and non-crop classes under leave-one-year-out protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09497","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_swisscrop25_6e928630"},"modality":null,"name":"SwissCrop25","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09497","source_url":"https://arxiv.org/abs/2608.09497"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09497","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09497","topics":["science"],"unit":null},{"aliases":["SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection"],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of syntactically correct but factually incorrect statements across eight languages for evaluating multilingual factual error rejection.","descriptions":{"en":"A benchmark of syntactically correct but factually incorrect statements across eight languages for evaluating multilingual factual error rejection."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09349","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["factuality","multilingual"],"join":"claire_record_id","primary":"factuality","source_record":"bm_sword-wikidata-based-distortions-reveal-hi_9208f906"},"modality":null,"name":"SWORD","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09349","source_url":"https://arxiv.org/abs/2609.09349"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09349","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09349","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":91,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_6f5e7c2ddd66ffe5"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Software Test Benchmark evaluating LLM ability to write tests for software repositories","descriptions":{"en":"Software Test Benchmark evaluating LLM ability to write tests for software repositories"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:swt-bench:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swt-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:swt-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_6f5e7c2ddd66ffe5"},"modality":"text","name":"SWT-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":69.3,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.693,"raw_min":0.693,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:swt-bench:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swt-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-swt-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swt-bench?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SymbalBench evaluates automated detection of systematic misalignments in MLLM-generated image captions. It comprises 420 vision-language datasets (1.7 million image-text pairs) from natural and medical domains, each annotated with known systematic misalignments.","descriptions":{"en":"SymbalBench evaluates automated detection of systematic misalignments in MLLM-generated image captions. It comprises 420 vision-language datasets (1.7 million image-text pairs) from natural and medical domains, each annotated with known systematic misalignments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.15216","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_symbalbench_84a0aab6"},"modality":null,"name":"SymbalBench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.15216","source_url":"https://arxiv.org/abs/2607.15216"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-15216","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15216","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Symbolic regression (SR) seeks concise and interpretable mathematical expressions from data for scientific equation discovery.","descriptions":{"en":"Symbolic regression (SR) seeks concise and interpretable mathematical expressions from data for scientific equation discovery."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35113","languages":[],"library_direction":{"basis":"model-review","categories":["science","math"],"join":"claire_record_id","primary":"science","source_record":"bm_symbolicarena_e9c0d0c6"},"modality":null,"name":"SymbolicArena","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35113","source_url":"https://arxiv.org/abs/2609.35113"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35113","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35113","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_syncred-bench_70361b01"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"SynCred-Bench evaluates detection of AI-generated visual misinformation with synthetic credibility, using 600 AI-generated images and FP450 real-image negatives.","descriptions":{"en":"SynCred-Bench evaluates detection of AI-generated visual misinformation with synthetic credibility, using 600 AI-generated images and FP450 real-image negatives."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03348","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_syncred-bench_70361b01"},"modality":null,"name":"SynCred-Bench","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03348","source_url":"https://arxiv.org/abs/2606.03348"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03348","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03348","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language models are increasingly used where small syntactic errors matter, yet character-level reasoning is still evaluated mostly through isolated probes and aggregate accuracy.","descriptions":{"en":"Large language models are increasingly used where small syntactic errors matter, yet character-level reasoning is still evaluated mostly through isolated probes and aggregate accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.03329","languages":[],"library_direction":{"basis":"model-review-daily","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_syntaxbench_8340a501"},"modality":null,"name":"SyntaxBench","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.03329","source_url":"https://arxiv.org/abs/2610.03329"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-03329","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03329","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Presents a synthetic labeling pipeline for e-commerce attribute extraction with human validation via multi-LLM arena voting.","descriptions":{"en":"Presents a synthetic labeling pipeline for e-commerce attribute extraction with human validation via multi-LLM arena voting."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.07469","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_synthave_eb33ab8a"},"modality":null,"name":"SynthAVE","openness":"unknown","publisher":null,"released":"2026-07-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.07469","source_url":"https://arxiv.org/abs/2607.07469"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-07469","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07469","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":9,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":68,"join":"claire_record_id","repository":"gh:servicenow/synthdocbench","source_record":"bm_synthdocbench_7af1a05d"},"categories":["General AI","Long Context","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"SynthDocBench evaluates vision-language models on 1,788 questions over 200 synthetic long-context documents (avg. 51.1 pages) with 3,340 charts. It varies document length, layout archetype, modality composition, and question type as independent controlled factors, across chart, cross-modal, and complex subsets with deterministic ground truth.","descriptions":{"en":"SynthDocBench evaluates vision-language models on 1,788 questions over 200 synthetic long-context documents (avg. 51.1 pages) with 3,340 charts. It varies document length, layout archetype, modality composition, and question type as independent controlled factors, across chart, cross-modal, and complex subsets with deterministic ground truth."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.10400","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","long-context"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_synthdocbench_7af1a05d"},"modality":null,"name":"SynthDocBench","openness":"unknown","publisher":null,"released":"2026-07-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10400","source_url":"https://arxiv.org/abs/2607.10400"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10400","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10400","topics":["long-context"],"unit":null},{"aliases":["synthetic-data-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:mervecaliskann/synthetic-data-benchmark","source_record":"bm_synthetic-data-benchmark_e9558d16"},"categories":["Finance","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The project evaluates synthetic tabular data generators on fidelity, utility, and privacy metrics using the Give Me Some Credit dataset, comparing Gaussian Copula, CTGAN, and an LLM generator over a fixed evaluation protocol.","descriptions":{"en":"The project evaluates synthetic tabular data generators on fidelity, utility, and privacy metrics using the Give Me Some Credit dataset, comparing Gaussian Copula, CTGAN, and an LLM generator over a fixed evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:mervecaliskann/synthetic-data-benchmark","languages":[],"library_direction":null,"modality":null,"name":"Synthetic Data Generator Benchmark (Credit Risk)","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:mervecaliskann/synthetic-data-benchmark","source_url":"https://github.com/Mervecaliskann/synthetic-data-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-mervecaliskann-synthetic-data-benchmark","source":"claire_radar","source_url":"https://github.com/Mervecaliskann/synthetic-data-benchmark","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Synthetic Hospital is proposed as an open, synthetic, physician-validated longitudinal EHR benchmark, but no released task inputs, scoring code, or public data are provided.","descriptions":{"en":"Synthetic Hospital is proposed as an open, synthetic, physician-validated longitudinal EHR benchmark, but no released task inputs, scoring code, or public data are provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.30027","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_synthetic-hospital_a5c915df"},"modality":null,"name":"Synthetic Hospital","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30027","source_url":"https://arxiv.org/abs/2609.30027"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30027","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30027","topics":[],"unit":null},{"aliases":["system-one-bench"],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:rssr25/system-one-bench","source_record":"bm_system-one-bench_7293dbd8"},"categories":["General AI","Robustness","Safety & Trustworthiness","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"sys1bench evaluates typed System One decision models on calibration, framing sensitivity, selective prediction, ordinal fidelity, interference, and robustness through a pip-installable benchmark suite.","descriptions":{"en":"sys1bench evaluates typed System One decision models on calibration, framing sensitivity, selective prediction, ordinal fidelity, interference, and robustness through a pip-installable benchmark suite."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:rssr25/system-one-bench","languages":[],"library_direction":null,"modality":null,"name":"sys1bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:rssr25/system-one-bench","source_url":"https://github.com/rssr25/system-one-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-rssr25-system-one-bench","source":"claire_radar","source_url":"https://github.com/rssr25/system-one-bench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","agent","任务执行","Task Execution","官方自建","Official"],"collected_at":"2026-08-18T00:00:00+00:00","description":"T-Eval evaluates the tool utilization capabilities of LLMs and decomposing them into instruction following, planning, reasoning, retrieval, understanding, and review. T-Eval 评估了 LLM 的工具使用能力，并将其分解为指令遵循、规划、推理、检索、理解和审查等子能力","descriptions":{"en":"T-Eval evaluates the tool utilization capabilities of LLMs and decomposing them into instruction following, planning, reasoning, retrieval, understanding, and review.","zh":"T-Eval 评估了 LLM 的工具使用能力，并将其分解为指令遵循、规划、推理、检索、理解和审查等子能力"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:540","languages":[],"library_direction":null,"modality":null,"name":"T-Eval","openness":"open","publisher":null,"released":"2024-01-15","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-540-t-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/T-Eval","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"T-IMPACT evaluates models on severity-aware detection of manipulated news-style image-text pairs, with 98,786 examples covering pristine, image-only, text-only, and joint manipulations, alongside calibrated continuous severity scores, coarse labels, and grounding metadata.","descriptions":{"en":"T-IMPACT evaluates models on severity-aware detection of manipulated news-style image-text pairs, with 98,786 examples covering pristine, image-only, text-only, and joint manipulations, alongside calibrated continuous severity scores, coarse labels, and grounding metadata."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.22339","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","visual-reasoning"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_t-impact_38620781"},"modality":null,"name":"T-IMPACT","openness":"unknown","publisher":null,"released":"2026-06-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.22339","source_url":"https://arxiv.org/abs/2606.22339"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-22339","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22339","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_t1-bench_01df3d35"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"T1-Bench evaluates agentic systems through realistic customer-facing, multi-domain tasks. Scenarios involve multi-turn user-assistant interactions across 25 domains with varying difficulty, measuring structured reasoning, tool utilization, and conversational quality. Automatic evaluation is complemented by human judgments.","descriptions":{"en":"T1-Bench evaluates agentic systems through realistic customer-facing, multi-domain tasks. Scenarios involve multi-turn user-assistant interactions across 25 domains with varying difficulty, measuring structured reasoning, tool utilization, and conversational quality. Automatic evaluation is complemented by human judgments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.11070","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_t1-bench_01df3d35"},"modality":null,"name":"T1-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11070","source_url":"https://arxiv.org/abs/2606.11070"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11070","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11070","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2231,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":"gh:sierra-research/tau2-bench","source_record":"catalog_c3436709ce24412e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"t2-bench is a benchmark for evaluating agentic tool use capabilities, measuring how well models can select, sequence, and utilize tools to solve complex tasks. It tests autonomous planning and execution in multi-step scenarios.","descriptions":{"en":"t2-bench is a benchmark for evaluating agentic tool use capabilities, measuring how well models can select, sequence, and utilize tools to solve complex tasks. It tests autonomous planning and execution in multi-step scenarios."},"evidence_summary":{"document_count":1,"model_count":23,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:t2-bench:gpt-oss-120b-high","reported_at":"2025-08-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/t2-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:t2-bench","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_c3436709ce24412e"},"modality":"text","name":"t2-bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"not_found","repo_resolution_status":"needs_review","score_count":23,"score_direction":"higher_is_better","score_summary":{"display_max":99.3,"display_multiplier":100,"model_count":23,"model_count_basis":"source_model_id","numeric_count":23,"raw_max":0.993,"raw_min":0.116,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:t2-bench:gemini-3.1-pro-preview","reported_date":"2026-02-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/t2-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2-Speciale"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B-A4B"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B-A4B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E4B"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B High"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE-236B-A23B"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"}],"slug":"llm-stats-t2-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/t2-bench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Factuality","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"T2D-Bench evaluates LLM outputs for type 2 diabetes against evidence constraints using a multi-layer clinical-lifestyle knowledge graph, covering diagnosis, medication safety, and lifestyle conflicts across 100 structured vignettes.","descriptions":{"en":"T2D-Bench evaluates LLM outputs for type 2 diabetes against evidence constraints using a multi-layer clinical-lifestyle knowledge graph, covering diagnosis, medication safety, and lifestyle conflicts across 100 structured vignettes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.24145","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_t2d-bench_6b92ee7c"},"modality":null,"name":"T2D-Bench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24145","source_url":"https://arxiv.org/abs/2606.24145"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24145","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24145","topics":["medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_t2j-bench_85fea4b8"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"T2J-Bench benchmarks codebase conversion by transferring PyTorch code to JAX under a fixed equivalence contract with three ordered verification stages.","descriptions":{"en":"T2J-Bench benchmarks codebase conversion by transferring PyTorch code to JAX under a fixed equivalence contract with three ordered verification stages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29054","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","ai-rnd"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_t2j-bench_85fea4b8"},"modality":null,"name":"T2J-Bench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29054","source_url":"https://arxiv.org/abs/2605.29054"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29054","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29054","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates localized semantic control in text-to-image generation using 50 seed subjects and 1,200 prompt cases per model, scoring 7,160 images across six models.","descriptions":{"en":"Evaluates localized semantic control in text-to-image generation using 50 seed subjects and 1,200 prompt cases per model, scoring 7,160 images across six models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02255","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_t2lsc-bench_7aa7c69f"},"modality":null,"name":"T2LSC-Bench","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02255","source_url":"https://arxiv.org/abs/2609.02255"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02255","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02255","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A living benchmark of 43 high-dimensional biomedical tables with a standardized cross-validation protocol and Elo-based ranking of ML models.","descriptions":{"en":"A living benchmark of 43 high-dimensional biomedical tables with a standardized cross-validation protocol and Elo-based ranking of ML models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.07441","languages":[],"library_direction":{"basis":"model-review","categories":["science","health-medicine"],"join":"claire_record_id","primary":"science","source_record":"bm_tabbench-bio_8ce5fdeb"},"modality":null,"name":"TabBench-Bio","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07441","source_url":"https://arxiv.org/abs/2609.07441"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07441","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07441","topics":["medical","science"],"unit":null},{"aliases":["Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language"],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-09-25T04:40:22Z","description":"Tabero evaluates language-conditioned gentle manipulation using vision, touch, task success, and physical-interaction quality.","descriptions":{"en":"Tabero evaluates language-conditioned gentle manipulation using vision, touch, task success, and physical-interaction quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.27886","languages":[],"library_direction":null,"modality":null,"name":"Tabero","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27886","source_url":"https://arxiv.org/abs/2605.27886"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27886","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27886","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","大语言模型","LLM","语言理解","Comprehension","事实可靠性","Factual Reliability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TabFac consists of 117,854 manually annotated statements with regard to 16,573 Wikipedia tables, their relations are classified as ENTAILED and REFUTED. TabFac 包含 117,854 条手动标注的语句，涉及 16,573 个维基百科表格，是第一个评估结构化数据上语言推理的数据集，涉及在符号和语言两个方面的混合推理能力。","descriptions":{"en":"TabFac consists of 117,854 manually annotated statements with regard to 16,573 Wikipedia tables, their relations are classified as ENTAILED and REFUTED.","zh":"TabFac 包含 117,854 条手动标注的语句，涉及 16,573 个维基百科表格，是第一个评估结构化数据上语言推理的数据集，涉及在符号和语言两个方面的混合推理能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1132","languages":[],"library_direction":null,"modality":null,"name":"TabFact","openness":"unknown","publisher":"Tencent AI Lab","released":"2020-06-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1132-tabfact","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TabFact","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","理解","Understanding","表格问答","跨语言","真实世界数据","大语言模型","LLM","逻辑推理","语言理解","Comprehension","OCR与文档理解","OCR and Document Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TableEval is the first cross-lingual benchmark for tabular question answering, supporting Simplified Chinese, Traditional Chinese, and English. It is designed to evaluate model performance on real-world, complex table understanding tasks across multiple languages. TableEval 是首个支持简体中文、繁体中文和英文的跨语言表格问答基准数据集，旨在系统评估大模型在真实复杂表格理解任务中的表现。","descriptions":{"en":"TableEval is the first cross-lingual benchmark for tabular question answering, supporting Simplified Chinese, Traditional Chinese, and English. It is designed to evaluate model performance on real-world, complex table understanding tasks across multiple languages.","zh":"TableEval 是首个支持简体中文、繁体中文和英文的跨语言表格问答基准数据集，旨在系统评估大模型在真实复杂表格理解任务中的表现。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2025","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"TableEval","openness":"unknown","publisher":"北京中科闻歌科技股份有限公司、中国科学院自动化研究所、中国科学院大学","released":"2025-06-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2025-tableeval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TableEval","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.DB"],"collected_at":"2026-10-14T09:44:51Z","description":"TabQueryBench evaluates synthetic tabular data generators using SQL-shaped analytical queries as structural assessors, providing 44 reusable query templates across 49 datasets.","descriptions":{"en":"TabQueryBench evaluates synthetic tabular data generators using SQL-shaped analytical queries as structural assessors, providing 44 reusable query templates across 49 datasets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.03926","languages":[],"library_direction":null,"modality":null,"name":"TabQueryBench","openness":"unknown","publisher":null,"released":"2026-07-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03926","source_url":"https://arxiv.org/abs/2607.03926"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03926","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03926","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_tabverse_d67e32ec"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"TABVERSE is a controlled multimodal benchmark that aligns the same table content across HTML, Markdown, LaTeX, and rendered images, with question category and difficulty tags. It evaluates LLMs and VLMs on question answering, structural understanding, and structure reconstruction.","descriptions":{"en":"TABVERSE is a controlled multimodal benchmark that aligns the same table content across HTML, Markdown, LaTeX, and rendered images, with question category and difficulty tags. It evaluates LLMs and VLMs on question answering, structural understanding, and structure reconstruction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09578","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","visual-reasoning"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_tabverse_d67e32ec"},"modality":null,"name":"TABVERSE","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09578","source_url":"https://arxiv.org/abs/2606.09578"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09578","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09578","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"TactiDex is a real-world tactile-guided benchmark for dexterous manipulation, aligning whole-hand tactile signals with kinematic and object states. It provides standardized evaluation metrics and a framework for tactile-driven transfer, including experiments on single and bimanual tasks.","descriptions":{"en":"TactiDex is a real-world tactile-guided benchmark for dexterous manipulation, aligning whole-hand tactile signals with kinematic and object states. It provides standardized evaluation metrics and a framework for tactile-driven transfer, including experiments on single and bimanual tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09190","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_tactidex_18e67e16"},"modality":null,"name":"TactiDex","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09190","source_url":"https://arxiv.org/abs/2607.09190"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09190","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09190","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"TacVerse is a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception, containing 106,800 tactile images from seven vision-based tactile sensors. It supports shape classification, grating classification, and force regression tasks, with evaluation under within-sensor, zero-shot cross-sensor, and few-shot adaptation settings.","descriptions":{"en":"TacVerse is a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception, containing 106,800 tactile images from seven vision-based tactile sensors. It supports shape classification, grating classification, and force regression tasks, with evaluation under within-sensor, zero-shot cross-sensor, and few-shot adaptation settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.25877","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_tacverse_8b86d6a1"},"modality":null,"name":"TacVerse","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25877","source_url":"https://arxiv.org/abs/2606.25877"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25877","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25877","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":283,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:shiyegao/tadabench-1m","source_record":"bm_tada-bench_769ea7c2"},"categories":["Biology & Drug Discovery","Language & Knowledge","q-bio.QM"],"collected_at":"2026-10-14T09:44:51Z","description":"TadA-Bench is a fixed-data replay benchmark derived from 31 wet-lab rounds of TadA directed evolution. Models rank ~1M protein, DNA, or RNA sequence variants appearing only in later rounds, with scores as Spearman, Recall@10%, and nDCG@10%.","descriptions":{"en":"TadA-Bench is a fixed-data replay benchmark derived from 31 wet-lab rounds of TadA directed evolution. Models rank ~1M protein, DNA, or RNA sequence variants appearing only in later rounds, with scores as Spearman, Recall@10%, and nDCG@10%."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.02624","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_tada-bench_769ea7c2"},"modality":null,"name":"TadA-Bench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02624","source_url":"https://arxiv.org/abs/2606.02624"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02624","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02624","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_taf-med_ece2193a"},"categories":["Biology & Drug Discovery","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"TAF-MED is a physician-reviewed benchmark of 500 fixed three-turn medical safety scenarios, evaluating LLM responses for unsafe guidance across multi-turn dialogues.","descriptions":{"en":"TAF-MED is a physician-reviewed benchmark of 500 fixed three-turn medical safety scenarios, evaluating LLM responses for unsafe guidance across multi-turn dialogues."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10258","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","safety-alignment"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_taf-med_ece2193a"},"modality":null,"name":"TAF-MED","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10258","source_url":"https://arxiv.org/abs/2608.10258"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10258","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10258","topics":["safety","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":40,"join":"claire_record_id","repository":"gh:tailor-bench/code","source_record":"bm_tailor-bench_613b036f"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Tailor-Bench evaluates visual world models on simulating irregular physical interactions with three scenario modes (regular, unconventional, impossible) and predictive/descriptive generation settings.","descriptions":{"en":"Tailor-Bench evaluates visual world models on simulating irregular physical interactions with three scenario modes (regular, unconventional, impossible) and predictive/descriptive generation settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.24256","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_tailor-bench_613b036f"},"modality":null,"name":"Tailor-Bench","openness":"unknown","publisher":null,"released":"2026-06-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.24256","source_url":"https://arxiv.org/abs/2606.24256"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-24256","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24256","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"TailorBench evaluates personalized multimodal content generation using data from three platforms and five stated evaluation dimensions.","descriptions":{"en":"TailorBench evaluates personalized multimodal content generation using data from three platforms and five stated evaluation dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.23643","languages":[],"library_direction":null,"modality":null,"name":"TailorBench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23643","source_url":"https://arxiv.org/abs/2606.23643"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23643","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23643","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kaishinozakiconefrey/take-85","source_record":"bm_take-85_1b8978d7"},"categories":["General AI","Speech & Audio","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"TAKE 85 evaluates directorial-intent understanding across 398 short films using question-answer pairs covering visual and audio intent, with synthesis code available for model predictions.","descriptions":{"en":"TAKE 85 evaluates directorial-intent understanding across 398 short films using question-answer pairs covering visual and audio intent, with synthesis code available for model predictions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.30068","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_take-85_1b8978d7"},"modality":null,"name":"TAKE 85","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30068","source_url":"https://arxiv.org/abs/2608.30068"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30068","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30068","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems.","descriptions":{"en":"Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38867","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","speech-audio"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_talk2agent_3bf088c5"},"modality":null,"name":"Talk2Agent","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38867","source_url":"https://arxiv.org/abs/2609.38867"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38867","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38867","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A unified Farsi dialogue benchmark with knowledge-grounded generation, dialogue acts, emotions, and sentiment.","descriptions":{"en":"A unified Farsi dialogue benchmark with knowledge-grounded generation, dialogue acts, emotions, and sentiment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01810","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","factuality"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_talkfa_80972e4f"},"modality":null,"name":"TalkFa","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01810","source_url":"https://arxiv.org/abs/2609.01810"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01810","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01810","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"TangPoetryBench evaluates text-to-image models on illustrating classical Chinese Tang poems across ten human-annotated dimensions, with a rubric-conditioned evaluator (PAE).","descriptions":{"en":"TangPoetryBench evaluates text-to-image models on illustrating classical Chinese Tang poems across ten human-annotated dimensions, with a rubric-conditioned evaluator (PAE)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.11452","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_tangpoetrybench_c91f571c"},"modality":null,"name":"TangPoetryBench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.11452","source_url":"https://arxiv.org/abs/2608.11452"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-11452","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11452","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A key challenge for multimodal large language models (MLLMs) is moving beyond visual recognition to constraint-aware cross-modal reasoning.","descriptions":{"en":"A key challenge for multimodal large language models (MLLMs) is moving beyond visual recognition to constraint-aware cross-modal reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.10374","languages":[],"library_direction":{"basis":"model-review-daily","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_taod2c-bench_7c42b287"},"modality":null,"name":"TaoD2C-Bench","openness":"unknown","publisher":null,"released":"2026-10-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.10374","source_url":"https://arxiv.org/abs/2610.10374"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-10374","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10374","topics":["coding"],"unit":null},{"aliases":["TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views"],"attention_signals":null,"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-term 3D point tracking across multiple synchronized views with camera motion using 284 sequences and over 100K point tracks.","descriptions":{"en":"Evaluates long-term 3D point tracking across multiple synchronized views with camera motion using 284 sequences and over 100K point tracks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01899","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","video-understanding"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_tapvid-mv-a-benchmark-for-tracking-any-poi_a0341f90"},"modality":null,"name":"TAPVid-MV","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01899","source_url":"https://arxiv.org/abs/2609.01899"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01899","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01899","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":1401,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_tar-bench_36cf86db"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video-language models on ten traffic anomaly reasoning tasks using 960 human-curated test annotations over 80 held-out clips.","descriptions":{"en":"Evaluates video-language models on ten traffic anomaly reasoning tasks using 960 human-curated test annotations over 80 held-out clips."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10317","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_tar-bench_36cf86db"},"modality":null,"name":"TAR-Bench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10317","source_url":"https://arxiv.org/abs/2608.10317"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10317","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10317","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","NeurIPS 2024","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TaskBench aims to evaluate the capability of LLMs in task automation, containing 28,271 samples spanning 3 critical stages: task decomposition, tool invocation, and parameter prediction. TaskBench旨在评估LLM在任务自动化方面的能力，包含面向任务分解、工具调用和参数预测三个关键阶段的28271个样本。","descriptions":{"en":"TaskBench aims to evaluate the capability of LLMs in task automation, containing 28,271 samples spanning 3 critical stages: task decomposition, tool invocation, and parameter prediction.","zh":"TaskBench旨在评估LLM在任务自动化方面的能力，包含面向任务分解、工具调用和参数预测三个关键阶段的28271个样本。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1164","languages":[],"library_direction":null,"modality":null,"name":"TaskBench","openness":"unknown","publisher":"Microsoft Research Asia","released":"2023-11-30","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1164-taskbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TaskBench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":35,"github_stars_30d":null,"hf_dataset_downloads":133,"hf_paper_upvotes":164,"join":"claire_record_id","repository":"gh:wbopan/tastebench","source_record":"bm_taste-bench_12951b29"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agents' judgment at decision forks in long-horizon engineering and research tasks using 502 trajectory-derived taste questions.","descriptions":{"en":"Evaluates LLM agents' judgment at decision forks in long-horizon engineering and research tasks using 502 trajectory-derived taste questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.25804","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","agentic-coding"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_taste-bench_12951b29"},"modality":null,"name":"Taste-Bench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25804","source_url":"https://arxiv.org/abs/2609.25804"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25804","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25804","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models.","descriptions":{"en":"We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.06824","languages":[],"library_direction":{"basis":"model-review-daily","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_tasteval_51558a00"},"modality":null,"name":"TasteVal","openness":"unknown","publisher":null,"released":"2026-10-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.06824","source_url":"https://arxiv.org/abs/2610.06824"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-06824","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06824","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates anomaly instance tracking and fine-grained anomaly understanding jointly on 1,118 videos with track-centric annotations.","descriptions":{"en":"Evaluates anomaly instance tracking and fine-grained anomaly understanding jointly on 1,118 videos with track-centric annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05699","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_tau-bench_aa436530"},"modality":null,"name":"TAU-Bench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05699","source_url":"https://arxiv.org/abs/2608.05699"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05699","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05699","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"τ-bench: A benchmark for tool-agent-user interaction in real-world domains. Tests language agents' ability to interact with users and follow domain-specific rules through dynamic conversations using API tools and policy guidelines across retail and airline domains. Evaluates consistency and reliability of agent behavior over multiple trials.","descriptions":{"en":"τ-bench: A benchmark for tool-agent-user interaction in real-world domains. Tests language agents' ability to interact with users and follow domain-specific rules through dynamic conversations using API tools and policy guidelines across retail and airline domains. Evaluates consistency and reliability of agent behavior over multiple trials."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau-bench:o3-2025-04-16","reported_at":"2025-04-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tau-bench","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"bm_tau-bench_aa436530"},"modality":"text","name":"Tau-bench","openness":"unknown","publisher":null,"released":"2024-06-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: $\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:tau-bench","source_url":"https://arxiv.org/abs/2406.12045"},"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":88.2,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.882,"raw_min":0.63,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau-bench:step-3.5-flash","reported_date":"2026-02-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step-3.5-Flash"}],"slug":"llm-stats-tau-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["tau-bench","τ-bench","tau2-bench","TAU-bench"],"attention_signals":{"as_of":"2026-10-12","github_stars":1461,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"artifact:gh:sierra-research/tau-bench","repository":"gh:sierra-research/tau-bench","source_record":"lib_tau_bench"},"categories":["tool_use"],"collected_at":null,"description":"Depends on a simulated user and policy; the simulator model is part of the measurement.","descriptions":{"en":"Depends on a simulated user and policy; the simulator model is part of the measurement."},"evidence_summary":{"document_count":3,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:tau_bench","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"artifact:gh:sierra-research/tau-bench","primary":"tool-use","source_record":"lib_tau_bench"},"modality":null,"name":"tau-bench","openness":"unknown","publisher":null,"released":"2024-06-17","released_reference":{"basis":"benchmark_release","source_key":"model-reports:tau_bench","source_url":"https://github.com/sierra-research/tau-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"tau_bench","source":"model_reports","source_url":"https://github.com/sierra-research/tau-bench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1458,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"llm_stats_id","repository":null,"source_record":"catalog_718d317fbe64cca2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","communication","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Part of τ-bench (TAU-bench), a benchmark for Tool-Agent-User interaction in real-world domains. The airline domain evaluates language agents' ability to interact with users through dynamic conversations while following domain-specific rules and using API tools. Agents must handle airline-related tasks and policies reliably.","descriptions":{"en":"Part of τ-bench (TAU-bench), a benchmark for Tool-Agent-User interaction in real-world domains. The airline domain evaluates language agents' ability to interact with users through dynamic conversations while following domain-specific rules and using API tools. Agents must handle airline-related tasks and policies reliably."},"evidence_summary":{"document_count":1,"model_count":23,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau-bench-airline:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-airline?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:tau-bench-airline","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_718d317fbe64cca2"},"modality":"text","name":"TAU-bench Airline","openness":"unknown","publisher":null,"released":"2024-06-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: $\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:tau-bench-airline","source_url":"https://arxiv.org/abs/2406.12045"},"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":23,"score_direction":"higher_is_better","score_summary":{"display_max":70.0,"display_multiplier":100,"model_count":23,"model_count_basis":"source_model_id","numeric_count":23,"raw_max":0.7,"raw_min":0.14,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau-bench-airline:claude-sonnet-4-5-20250929","reported_date":"2025-09-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-airline?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Opus 4"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude Opus 4.1"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Sonnet 4"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude Sonnet 4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-tau-bench-airline","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-airline?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1460,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":13,"join":"llm_stats_id","repository":null,"source_record":"catalog_6c5416dff875789d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","communication","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A benchmark for evaluating tool-agent-user interaction in retail environments. Tests language agents' ability to handle dynamic conversations with users while using domain-specific API tools and following policy guidelines. Evaluates agents on tasks like order cancellations, address changes, and order status checks through multi-turn conversations.","descriptions":{"en":"A benchmark for evaluating tool-agent-user interaction in retail environments. Tests language agents' ability to handle dynamic conversations with users while using domain-specific API tools and following policy guidelines. Evaluates agents on tasks like order cancellations, address changes, and order status checks through multi-turn conversations."},"evidence_summary":{"document_count":1,"model_count":25,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau-bench-retail:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-retail?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:tau-bench-retail","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_6c5416dff875789d"},"modality":"text","name":"TAU-bench Retail","openness":"unknown","publisher":null,"released":"2024-06-17","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: $\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:tau-bench-retail","source_url":"https://arxiv.org/abs/2406.12045"},"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":25,"score_direction":"higher_is_better","score_summary":{"display_max":86.2,"display_multiplier":100,"model_count":25,"model_count_basis":"source_model_id","numeric_count":25,"raw_max":0.862,"raw_min":0.226,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau-bench-retail:claude-sonnet-4-5-20250929","reported_date":"2025-09-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-retail?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Sonnet"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Opus 4"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude Opus 4.1"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Sonnet 4"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude Sonnet 4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 120B"},{"date":"2025-08-05","date_precision":"model_announcement","name":"GPT OSS 20B"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2025-02-27","date_precision":"model_announcement","name":"GPT-4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2024-12-17","date_precision":"model_announcement","name":"o1"},{"date":"2025-01-30","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-tau-bench-retail","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-retail?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2231,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":"gh:sierra-research/tau2-bench","source_record":"catalog_c72fe98e0d4560ef"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","communication","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"TAU2 airline domain benchmark for evaluating conversational agents in dual-control environments where both AI agents and users interact with tools in airline customer service scenarios. Tests agent coordination, communication, and ability to guide user actions in tasks like flight booking, modifications, cancellations, and refunds.","descriptions":{"en":"TAU2 airline domain benchmark for evaluating conversational agents in dual-control environments where both AI agents and users interact with tools in airline customer service scenarios. Tests agent coordination, communication, and ability to guide user actions in tasks like flight booking, modifications, cancellations, and refunds."},"evidence_summary":{"document_count":1,"model_count":23,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau2-airline:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-airline?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:tau2-airline","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_c72fe98e0d4560ef"},"modality":"text","name":"Tau2 Airline","openness":"restricted","publisher":"Sierra","released":"2025-06-09","released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":23,"score_direction":"higher_is_better","score_summary":{"display_max":76.5,"display_multiplier":100,"model_count":23,"model_count_basis":"source_model_id","numeric_count":23,"raw_max":0.765,"raw_min":0.44,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau2-airline:longcat-flash-thinking-2601","reported_date":"2026-01-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-airline?top_n=500"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Instant"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Thinking"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-tau2-airline","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-airline?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":2200,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":null,"source_record":"catalog_93cc12e0807d1b69"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","communication","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"τ²-bench retail domain evaluates conversational AI agents in customer service scenarios within a dual-control environment where both agent and user can interact with tools. Tests tool-agent-user interaction, rule adherence, and task consistency in retail customer support contexts.","descriptions":{"en":"τ²-bench retail domain evaluates conversational AI agents in customer service scenarios within a dual-control environment where both agent and user can interact with tools. Tests tool-agent-user interaction, rule adherence, and task consistency in retail customer support contexts."},"evidence_summary":{"document_count":1,"model_count":26,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau2-retail:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-retail?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:tau2-retail","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_93cc12e0807d1b69"},"modality":"text","name":"Tau2 Retail","openness":"restricted","publisher":"Sierra","released":"2025-06-09","released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":26,"score_direction":"higher_is_better","score_summary":{"display_max":91.9,"display_multiplier":100,"model_count":26,"model_count_basis":"source_model_id","numeric_count":26,"raw_max":0.919,"raw_min":0.569,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau2-retail:claude-opus-4-6","reported_date":"2026-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-retail?top_n=500"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Instant"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Thinking"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-tau2-retail","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-retail?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2229,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"llm_stats_id","repository":null,"source_record":"catalog_3fc20a133285456f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","communication","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"τ²-Bench telecom domain evaluates conversational agents in a dual-control environment modeled as a Dec-POMDP, where both agent and user use tools in shared telecommunications troubleshooting scenarios that test coordination and communication capabilities.","descriptions":{"en":"τ²-Bench telecom domain evaluates conversational agents in a dual-control environment modeled as a Dec-POMDP, where both agent and user use tools in shared telecommunications troubleshooting scenarios that test coordination and communication capabilities."},"evidence_summary":{"document_count":1,"model_count":35,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau2-telecom:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-telecom?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:tau2-telecom","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_3fc20a133285456f"},"modality":"text","name":"Tau2 Telecom","openness":"restricted","publisher":"Sierra","released":"2025-06-09","released_reference":null,"repo_kind":"shared_parent","repo_resolution_status":"resolved","score_count":35,"score_direction":"higher_is_better","score_summary":{"display_max":99.3,"display_multiplier":100,"model_count":35,"model_count_basis":"source_model_id","numeric_count":35,"raw_max":0.993,"raw_min":0.132,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau2-telecom:claude-opus-4-6","reported_date":"2026-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-telecom?top_n=500"},"unit":null},"scored_models":[{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Instant"},{"date":"2025-11-12","date_precision":"model_announcement","name":"GPT-5.1 Thinking"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2026-01-14","date_precision":"model_announcement","name":"LongCat-Flash-Thinking-2601"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-tau2-telecom","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-telecom?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["tau2-bench","τ²-Bench","TAU2-Bench","Tau2","τ2-bench"],"attention_signals":null,"categories":["tool_use"],"collected_at":null,"description":"Per-domain scores (Retail, Telecom, Airline) diverge sharply and several cards apply the Airline domain fixes from the Claude Opus 4.5 system card, so an average conceals both the domain mix and the patch level.","descriptions":{"en":"Per-domain scores (Retail, Telecom, Airline) diverge sharply and several cards apply the Airline domain fixes from the Claude Opus 4.5 system card, so an average conceals both the domain mix and the patch level."},"evidence_summary":{"document_count":6,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000tau2_bench\u0000zai_glm_5_model_card\u0000tau2_bench\u0000Retail and Telecom prompt adjustment; Airline domain fixes per Claude Opus 4.5 system card\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"first_score_reported_at":"2026-02-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000tau2_bench\u0000zai_glm_5_model_card\u0000tau2_bench\u0000Retail and Telecom prompt adjustment; Airline domain fixes per Claude Opus 4.5 system card\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:tau2_bench","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"artifact:arxiv:2506.07982","primary":"tool-use","source_record":"catalog_3fc20a133285456f"},"modality":null,"name":"tau2-bench","openness":"unknown","publisher":null,"released":"2025-06-09","released_reference":{"basis":"benchmark_release","source_key":"model-reports:tau2_bench","source_url":"https://arxiv.org/abs/2506.07982"},"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":99.3,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":99.3,"raw_min":76.9,"source_reference":{"obs_id":"curated\u0000tau2_bench\u0000google_gemini_3_1_pro_model_card\u0000tau2_bench\u0000Thinking (High), Telecom\u0000Gemini 3.1 Pro","observation_id":"curated\u0000tau2_bench\u0000google_gemini_3_1_pro_model_card\u0000tau2_bench\u0000Thinking (High), Telecom\u0000Gemini 3.1 Pro","reported_at":"2026-02-19","reported_date":"2026-02-19","source_id":"google_gemini_3_1_pro_model_card","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/"},"unit":"percent"},"scored_models":[{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-03-11","date_precision":"document_publication","name":"Gemma 4 (31B)"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"tau2_bench","source":"model_reports","source_url":"https://arxiv.org/abs/2506.07982","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2231,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:sierra-research/tau2-bench","source_record":"catalog_c928f909866afb23"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"τ³-Bench airline domain evaluates agentic models on multi-turn, tool-using customer-support scenarios in a simulated airline booking and reservations environment.","descriptions":{"en":"τ³-Bench airline domain evaluates agentic models on multi-turn, tool-using customer-support scenarios in a simulated airline booking and reservations environment."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau3-airline:mistral-medium-3-5","reported_at":"2026-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-airline?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tau3-airline","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_c928f909866afb23"},"modality":"text","name":"Tau3 Airline","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":72.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.72,"raw_min":0.72,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau3-airline:mistral-medium-3-5","reported_date":"2026-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-airline?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"}],"slug":"llm-stats-tau3-airline","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-airline?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2231,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:sierra-research/tau2-bench","source_record":"catalog_ebd77eaccb2d583a"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"τ³-Bench banking domain evaluates agentic models on multi-turn, tool-using customer-support scenarios in a simulated retail banking environment.","descriptions":{"en":"τ³-Bench banking domain evaluates agentic models on multi-turn, tool-using customer-support scenarios in a simulated retail banking environment."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau3-banking:mistral-medium-3-5","reported_at":"2026-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-banking?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tau3-banking","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_ebd77eaccb2d583a"},"modality":"text","name":"Tau3 Banking","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":33.0,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.33,"raw_min":0.0567,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau3-banking:grok-4.5","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-banking?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"}],"slug":"llm-stats-tau3-banking","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-banking?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2231,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:sierra-research/tau2-bench","source_record":"catalog_6c8193c29e3e5937"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"τ³-Bench retail domain evaluates agentic models on multi-turn, tool-using customer-support scenarios in a simulated online retail environment.","descriptions":{"en":"τ³-Bench retail domain evaluates agentic models on multi-turn, tool-using customer-support scenarios in a simulated online retail environment."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau3-retail:mistral-medium-3-5","reported_at":"2026-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-retail?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tau3-retail","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_6c8193c29e3e5937"},"modality":"text","name":"Tau3 Retail","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":76.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.761,"raw_min":0.761,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau3-retail:mistral-medium-3-5","reported_date":"2026-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-retail?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"}],"slug":"llm-stats-tau3-retail","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-retail?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":2231,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:sierra-research/tau2-bench","source_record":"catalog_81a8751700459174"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","communication","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"τ³-Bench telecom domain evaluates agentic models on multi-turn, tool-using customer-support and troubleshooting scenarios in a simulated telecommunications environment.","descriptions":{"en":"τ³-Bench telecom domain evaluates agentic models on multi-turn, tool-using customer-support and troubleshooting scenarios in a simulated telecommunications environment."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau3-telecom:mistral-medium-3-5","reported_at":"2026-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-telecom?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tau3-telecom","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_81a8751700459174"},"modality":"text","name":"Tau3 Telecom","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":91.4,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.914,"raw_min":0.914,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau3-telecom:mistral-medium-3-5","reported_date":"2026-04-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-telecom?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"}],"slug":"llm-stats-tau3-telecom","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-telecom?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":2231,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:sierra-research/tau2-bench","source_record":"catalog_b5356cb55eb564be"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"TAU3-Bench is a benchmark for evaluating general-purpose agent capabilities, testing models on multi-turn interactions with simulated user models, retrieval, and complex decision-making scenarios.","descriptions":{"en":"TAU3-Bench is a benchmark for evaluating general-purpose agent capabilities, testing models on multi-turn interactions with simulated user models, retrieval, and complex decision-making scenarios."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tau3-bench:qwen3.6-plus","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tau3-bench","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_b5356cb55eb564be"},"modality":"text","name":"TAU3-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":72.89999999999999,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.729,"raw_min":0.226,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tau3-bench:mimo-v2.5-pro","reported_date":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-tau3-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-bench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark dataset for tropical cyclone research with an automated construction pipeline, used to probe scientific alignment of vision foundation models.","descriptions":{"en":"A benchmark dataset for tropical cyclone research with an automated construction pipeline, used to probe scientific alignment of vision foundation models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24782","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_tc-bench_6009a2aa"},"modality":null,"name":"TC-Bench","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24782","source_url":"https://arxiv.org/abs/2605.24782"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24782","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24782","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Safety","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"TCA-Bench is a diagnostic benchmark for evaluating audiovisual binding and temporal relational reasoning in video captioning models, using a decoupled evaluation protocol.","descriptions":{"en":"TCA-Bench is a diagnostic benchmark for evaluating audiovisual binding and temporal relational reasoning in video captioning models, using a decoupled evaluation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01667","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_tca-bench_07c28360"},"modality":null,"name":"TCA-Bench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01667","source_url":"https://arxiv.org/abs/2607.01667"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01667","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01667","topics":["reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"TCR-Bench is a diagnostic benchmark for table content-level answerability in RAG, focusing on sibling tables with similar schemas but content differences. It evaluates dense retrievers' ability to identify answerable tables.","descriptions":{"en":"TCR-Bench is a diagnostic benchmark for table content-level answerability in RAG, focusing on sibling tables with similar schemas but content differences. It evaluates dense retrievers' ability to identify answerable tables."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.17742","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","document-ocr"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_tcr-bench_7bc5c915"},"modality":null,"name":"TCR-Bench","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.17742","source_url":"https://arxiv.org/abs/2607.17742"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-17742","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17742","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on research-level theoretical computer science proof generation, using theorem-proving tasks from papers at STOC, FOCS, and SODA. Provides context for self-contained proofs and uses a verification agent to check correctness.","descriptions":{"en":"Evaluates LLMs on research-level theoretical computer science proof generation, using theorem-proving tasks from papers at STOC, FOCS, and SODA. Provides context for self-contained proofs and uses a verification agent to check correctness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09538","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_tcs-bench_6efad642"},"modality":null,"name":"TCS-BENCH","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09538","source_url":"https://arxiv.org/abs/2608.09538"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09538","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09538","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_tcsalgbench_c18e622a"},"categories":["Language & Knowledge","Mathematics & Formal Science","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language models perform strongly on competition mathematics, but their research-level reasoning remains difficult to evaluate systematically.","descriptions":{"en":"Large language models perform strongly on competition mathematics, but their research-level reasoning remains difficult to evaluate systematically."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35606","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_tcsalgbench_c18e622a"},"modality":null,"name":"TCSAlgBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35606","source_url":"https://arxiv.org/abs/2609.35606"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35606","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35606","topics":["reasoning-math"],"unit":null},{"aliases":["Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents"],"attention_signals":null,"categories":["Factuality","General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The Teaching Monster Challenge evaluates AI systems on generating instructional videos for a given topic and learner persona, with submissions screened by an LLM judge, ranked by crowd pairwise voting, and finalized by an expert panel.","descriptions":{"en":"The Teaching Monster Challenge evaluates AI systems on generating instructional videos for a given topic and learner persona, with submissions screened by an LLM judge, ranked by crowd pairwise voting, and finalized by an expert panel."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08852","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_findings-of-the-first-teaching-monster-cha_0418dfb3"},"modality":null,"name":"Teaching Monster Challenge","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08852","source_url":"https://arxiv.org/abs/2608.08852"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08852","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08852","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":25,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":58,"join":"claire_record_id","repository":"gh:nju-link/drift","source_record":"bm_telbench_d23c11db"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates span-level error localization in deep-research agent trajectories. TELBench comprises 1,000 instances with annotations of harmful error spans among normal exploration, failed searches, tentative hypotheses, and harmless noise, scored by span-level localization and first-error accuracy.","descriptions":{"en":"Evaluates span-level error localization in deep-research agent trajectories. TELBench comprises 1,000 instances with annotations of harmful error spans among normal exploration, failed searches, tentative hypotheses, and harmless noise, scored by span-level localization and first-error accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.02060","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_telbench_d23c11db"},"modality":null,"name":"TELBench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02060","source_url":"https://arxiv.org/abs/2606.02060"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02060","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02060","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"TelemetrySuffBench evaluates models on fault-origin diagnosis in agent telemetry, using controlled multi-component traces with delayed-binding faults, paired coarse views, seven-factor telemetry masks, and exact-equal ambiguous origin pairs.","descriptions":{"en":"TelemetrySuffBench evaluates models on fault-origin diagnosis in agent telemetry, using controlled multi-component traces with delayed-binding faults, paired coarse views, seven-factor telemetry masks, and exact-equal ambiguous origin pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07899","languages":[],"library_direction":null,"modality":null,"name":"TelemetrySuffBench","openness":"unknown","publisher":null,"released":"2026-08-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07899","source_url":"https://arxiv.org/abs/2608.07899"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07899","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07899","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Code","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"TeleSWEBench is a commit-driven benchmark with 734 questions derived from real developer commits in the srsRAN 5G repository. It evaluates LLM-powered software engineering agents in the telecommunications domain using executable unit tests and a hierarchical LLM-as-a-Judge framework across three difficulty tiers.","descriptions":{"en":"TeleSWEBench is a commit-driven benchmark with 734 questions derived from real developer commits in the srsRAN 5G repository. It evaluates LLM-powered software engineering agents in the telecommunications domain using executable unit tests and a hierarchical LLM-as-a-Judge framework across three difficulty tiers."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05001","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_teleswebench_49d08856"},"modality":null,"name":"TeleSWEBench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05001","source_url":"https://arxiv.org/abs/2606.05001"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05001","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05001","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"A video cloze benchmark for visual temporal reasoning in Video-LLMs.","descriptions":{"en":"A video cloze benchmark for visual temporal reasoning in Video-LLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01515","languages":[],"library_direction":null,"modality":null,"name":"TempCloze","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01515","source_url":"https://arxiv.org/abs/2609.01515"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01515","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01515","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":132,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_4e2c048ed501de05"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"TempCompass is a comprehensive benchmark for evaluating temporal perception capabilities of Video Large Language Models (Video LLMs). It constructs conflicting videos that share identical static content but differ in specific temporal aspects to prevent models from exploiting single-frame bias. The benchmark evaluates multiple temporal aspects including action, motion, speed, temporal order, and attribute changes across diverse task formats including multi-choice QA, yes/no QA, caption matching, and caption generation.","descriptions":{"en":"TempCompass is a comprehensive benchmark for evaluating temporal perception capabilities of Video Large Language Models (Video LLMs). It constructs conflicting videos that share identical static content but differ in specific temporal aspects to prevent models from exploiting single-frame bias. The benchmark evaluates multiple temporal aspects including action, motion, speed, temporal order, and attribute changes across diverse task formats including multi-choice QA, yes/no QA, caption matching, and caption generation."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tempcompass:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tempcompass?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tempcompass","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_4e2c048ed501de05"},"modality":"multimodal","name":"TempCompass","openness":"unknown","publisher":null,"released":"2024-03-01","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: TempCompass: Do Video LLMs Really Understand Videos?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:tempcompass","source_url":"https://arxiv.org/abs/2403.00476"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":74.8,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.748,"raw_min":0.717,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tempcompass:qwen2.5-vl-72b","reported_date":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tempcompass?top_n=500"},"unit":null},"scored_models":[{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"}],"slug":"llm-stats-tempcompass","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tempcompass?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":382,"github_stars_30d":43,"hf_dataset_downloads":2052,"hf_paper_upvotes":24,"join":"claire_record_id","repository":"gh:tencent/workbuddy-bench","source_record":"bm_tencent-workbuddy-bench_93d020ca"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Multi-domain coding-agent benchmark with reverse-engineered tasks across Code, Web, Office, and Security. Each subset has its own scoring instrument; scores are not comparable across subsets.","descriptions":{"en":"Multi-domain coding-agent benchmark with reverse-engineered tasks across Code, Web, Office, and Security. Each subset has its own scoring instrument; scores are not comparable across subsets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.20911","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","professional-work","cybersecurity"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_tencent-workbuddy-bench_93d020ca"},"modality":null,"name":"Tencent WorkBuddy Bench","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20911","source_url":"https://arxiv.org/abs/2607.20911"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20911","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20911","topics":["coding","agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"TensorBench is a benchmark of 199 feature-addition and refactoring tasks on an open-source compiler-based tensor framework extending PyTorch. It grades agents by applying patches and running the framework's test suite.","descriptions":{"en":"TensorBench is a benchmark of 199 feature-addition and refactoring tasks on an open-source compiler-based tensor framework extending PyTorch. It grades agents by applying patches and running the framework's test suite."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05570","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_tensorbench_704e4bd5"},"modality":null,"name":"TensorBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05570","source_url":"https://arxiv.org/abs/2606.05570"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05570","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05570","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2600,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":"gh:laude-institute/terminal-bench","source_record":"lib_terminal_bench"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Terminal-Bench is a benchmark for testing AI agents in real terminal environments. It evaluates how well agents can handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, security tasks, data science workflows, and cybersecurity vulnerabilities. The benchmark consists of a dataset of ~100 hand-crafted, human-verified tasks and an execution harness that connects language models to a terminal sandbox.","descriptions":{"en":"Terminal-Bench is a benchmark for testing AI agents in real terminal environments. It evaluates how well agents can handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, security tasks, data science workflows, and cybersecurity vulnerabilities. The benchmark consists of a dataset of ~100 hand-crafted, human-verified tasks and an execution harness that connects language models to a terminal sandbox."},"evidence_summary":{"document_count":1,"model_count":25,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:terminal-bench:deepseek-v3.1","reported_at":"2025-01-10","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:terminal-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"lib_terminal_bench"},"modality":"text","name":"Terminal-Bench","openness":"restricted","publisher":null,"released":"2025-01-17","released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":25,"score_direction":"higher_is_better","score_summary":{"display_max":50.0,"display_multiplier":100,"model_count":25,"model_count_basis":"source_model_id","numeric_count":25,"raw_max":0.5,"raw_min":0.057,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:terminal-bench:claude-sonnet-4-5-20250929","reported_date":"2025-09-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude Haiku 4.5"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Opus 4"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude Opus 4.1"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude Sonnet 4"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude Sonnet 4.5"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek-R1-0528"},{"date":"2025-01-10","date_precision":"model_announcement","name":"DeepSeek-V3.1"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek-V3.2-Exp"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2026-02-05","date_precision":"model_announcement","name":"LongCat-Flash-Lite"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2025-10-27","date_precision":"model_announcement","name":"MiniMax M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Lite"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Pro"}],"slug":"llm-stats-terminal-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":["Terminal-Bench","Terminal Bench","TerminalBench","Terminal-Bench 2.0","Terminal-Bench 2.1","Terminal Bench 2","TerminalBench Hard","Terminal-Bench 3.0","Terminal Bench 3","Frontier-Bench","FrontierBench","Frontier-Bench v0.1","FrontierSWE"],"attention_signals":null,"categories":["agent"],"collected_at":null,"description":"Environment image, timeout and permitted commands change results independently of the model. Version and harness both matter: 2.0 and 2.1 are different instruments, and cards report Terminus, Claude Code and Codex harness numbers for the same version. Frontier-Bench is this series' next version rather than a separate benchmark: frontierbench.ai now bills it as \"Terminal-Bench 3.0 (formerly Frontier-Bench)\", built by the makers of Harbor and Terminal-Bench, so its mentions are counted here. Merging it did not raise this count: all three cards that named Frontier-Bench (Claude Opus 5, Claude Fable 5 and Mythos 5, Kimi K3) also report Terminal-Bench in the same document, and the counting unit is the document, so each still adds one. Being a new version, its task set is not comparable to a 2.x number. Not to be confused with Cognition's FrontierCode, a separate instrument recorded separately despite the shared prefix.","descriptions":{"en":"Environment image, timeout and permitted commands change results independently of the model. Version and harness both matter: 2.0 and 2.1 are different instruments, and cards report Terminus, Claude Code and Codex harness numbers for the same version. Frontier-Bench is this series' next version rather than a separate benchmark: frontierbench.ai now bills it as \"Terminal-Bench 3.0 (formerly Frontier-Bench)\", built by the makers of Harbor and Terminal-Bench, so its mentions are counted here. Merging it did not raise this count: all three cards that named Frontier-Bench (Claude Opus 5, Claude Fable 5 and Mythos 5, Kimi K3) also report Terminal-Bench in the same document, and the counting unit is the document, so each still adds one. Being a new version, its task set is not comparable to a 2.x number. Not to be confused with Cognition's FrontierCode, a separate instrument recorded separately despite the shared prefix."},"evidence_summary":{"document_count":21,"model_count":16,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000terminal_bench\u0000anthropic_claude_4_system_card\u0000terminal_bench_1\u0000no extended thinking\u0000Claude Opus 4","reported_at":"2025-05-22","source_url":"https://www.anthropic.com/news/claude-4"},"first_score_reported_at":"2025-05-22","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000terminal_bench\u0000anthropic_claude_4_system_card\u0000terminal_bench_1\u0000no extended thinking\u0000Claude Opus 4","reported_at":"2025-05-22","source_url":"https://www.anthropic.com/news/claude-4"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:terminal_bench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"Terminal-Bench","openness":"unknown","publisher":null,"released":"2025-05-19","released_reference":{"basis":"benchmark_release","source_key":"model-reports:terminal_bench","source_url":"https://www.tbench.ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":20,"score_direction":"higher_is_better","score_summary":{"display_max":90.6,"display_multiplier":1,"model_count":16,"model_count_basis":"source_model_id","numeric_count":20,"raw_max":90.6,"raw_min":30.0,"source_reference":{"obs_id":"curated\u0000terminal_bench\u0000deepseek_v4_1_flash_technical_report\u0000terminal_bench_2.1\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","observation_id":"curated\u0000terminal_bench\u0000deepseek_v4_1_flash_technical_report\u0000terminal_bench_2.1\u0000think max, Pass@1\u0000DeepSeek-V4.1-Flash","reported_at":"2026-09-10","reported_date":"2026-09-10","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf"},"unit":"percent"},"scored_models":[{"date":"2025-05-22","date_precision":"document_publication","name":"Claude Opus 4"},{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-08-13","date_precision":"document_publication","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-09-10","date_precision":"document_publication","name":"DeepSeek-V4.1-Flash"},{"date":"2026-02-19","date_precision":"document_publication","name":"Gemini 3.1 Pro"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-07-08","date_precision":"document_publication","name":"Grok 4.5"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Lite"},{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"terminal_bench","source":"model_reports","source_url":"https://www.tbench.ai/","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":2600,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":38,"join":"llm_stats_id","repository":"gh:laude-institute/terminal-bench","source_record":"lib_terminal_bench"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Terminal-Bench 2.0 is an updated benchmark for testing AI agents' tool use ability to operate a computer via terminal. It evaluates how well models can handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, security tasks, data science workflows, and cybersecurity vulnerabilities.","descriptions":{"en":"Terminal-Bench 2.0 is an updated benchmark for testing AI agents' tool use ability to operate a computer via terminal. It evaluates how well models can handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, security tasks, data science workflows, and cybersecurity vulnerabilities."},"evidence_summary":{"document_count":1,"model_count":51,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:terminal-bench-2:qwen3-coder-480b-a35b-instruct","reported_at":"2025-01-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":true,"key":"llm-stats:terminal-bench-2","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"lib_terminal_bench"},"modality":"text","name":"Terminal-Bench 2.0","openness":"restricted","publisher":null,"released":"2025-09-25","released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":51,"score_direction":"higher_is_better","score_summary":{"display_max":82.69999999999999,"display_multiplier":100,"model_count":51,"model_count_basis":"source_model_id","numeric_count":51,"raw_max":0.827,"raw_min":0.31,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:terminal-bench-2:gpt-5.5","reported_date":"2026-04-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2-Speciale"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2025-11-19","date_precision":"model_announcement","name":"GPT-5.1 Codex"},{"date":"2026-01-14","date_precision":"model_announcement","name":"GPT-5.2 Codex"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-02","date_precision":"model_announcement","name":"Laguna XS 2.1"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Code-1-Flash"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code 1.0"},{"date":"2025-01-31","date_precision":"model_announcement","name":"Qwen3-Coder 480B A35B Instruct"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step-3.5-Flash"}],"slug":"llm-stats-terminal-bench-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":123,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:harbor-framework/terminal-bench-2-1","source_record":"catalog_08efd551447036bf"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Terminal-Bench 2.1 is an updated release of the Terminal-Bench benchmark that tests AI agents' ability to operate a computer via the terminal. It evaluates how well models handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, data science workflows, and security tasks.","descriptions":{"en":"Terminal-Bench 2.1 is an updated release of the Terminal-Bench benchmark that tests AI agents' ability to operate a computer via the terminal. It evaluates how well models handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, data science workflows, and security tasks."},"evidence_summary":{"document_count":1,"model_count":28,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:terminal-bench-2.1:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2.1?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:terminal-bench-2.1","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_08efd551447036bf"},"modality":"text","name":"Terminal-Bench 2.1","openness":"restricted","publisher":null,"released":"2026-05-05","released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":28,"score_direction":"higher_is_better","score_summary":{"display_max":88.8,"display_multiplier":100,"model_count":28,"model_count_basis":"source_model_id","numeric_count":28,"raw_max":0.888,"raw_min":0.2458,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:terminal-bench-2.1:gpt-5.6-sol","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2.1?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0731"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Grok 4.5"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Laguna S 2.1"},{"date":"2026-08-11","date_precision":"model_announcement","name":"MAI-Code-1.1-Flash"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning (30B A3B)"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"}],"slug":"llm-stats-terminal-bench-2-1","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2.1?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":887,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_36edf9cc142c1b03"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Terminal-Bench 3.0 is a release of the Terminal-Bench benchmark that tests AI agents' ability to operate a computer via the terminal on real-world, end-to-end tasks.","descriptions":{"en":"Terminal-Bench 3.0 is a release of the Terminal-Bench benchmark that tests AI agents' ability to operate a computer via the terminal on real-world, end-to-end tasks."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:terminal-bench-3.0:grok-4.6","reported_at":"2026-08-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-3.0?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:terminal-bench-3.0","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_36edf9cc142c1b03"},"modality":"text","name":"Terminal-Bench 3.0","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":28.299999999999997,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.283,"raw_min":0.149,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:terminal-bench-3.0:glm-5.3","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-3.0?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6"}],"slug":"llm-stats-terminal-bench-3-0","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-3.0?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":["Terminal-Bench 4.0","Terminal Bench 4","TB4"],"attention_signals":null,"categories":["agent"],"collected_at":null,"description":"Resolution rate on the v4.0.0 task collection. Agent framework, resource limits and task version are part of the result. GPU and multi-container tasks require appropriate infrastructure; scores from earlier versions are not interchangeable with version 4.","descriptions":{"en":"Resolution rate on the v4.0.0 task collection. Agent framework, resource limits and task version are part of the result. GPU and multi-container tasks require appropriate infrastructure; scores from earlier versions are not interchangeable with version 4."},"editorial_guide":{"id":"terminal-bench-4","record_keys":["model-reports:terminal_bench_4"],"reviewed_at":"2026-10-20","scope":{"en":"Terminal-Bench 4.0, pinned to the public v4.0.0 release. Earlier versions retain separate records.","zh":"对应公开的 v4.0.0 版本。旧版 Terminal-Bench 保留独立记录。"},"sections":[{"id":"overview","source_ids":["tb4-manifest","harbor-task-format","tb4-readme"],"text":{"en":"Terminal-Bench 4 evaluates agents completing computer tasks through a terminal. The v4.0.0 manifest contains 66 tasks. Each combines a starting environment, an instruction and executable verification. This measures a model working inside an agent, including its ability to investigate, use tools and finish the job.","zh":"Terminal-Bench 4 评测智能体通过终端完成计算机任务的能力。v4.0.0 清单包含 66 道任务，每道题结合初始环境、任务说明和程序验收。它观察的是模型在智能体中的实际表现，包括调查问题、使用工具和完成交付。"},"title":{"en":"What it measures","zh":"它测量什么"}},{"id":"tasks","source_ids":["tb4-taxonomy","tb4-wal-task","tb4-batch-task"],"text":{"en":"Tasks span software, science, machine learning, operations, security, hardware and media. Examples include repairing database recovery and making batched model evaluation preserve single-example behavior. The task's main skill determines its category; using Python alone does not make a task software engineering. Subcategories can grow.","zh":"任务覆盖软件、科学、机器学习、业务运营、安全、硬件和媒体。例如，修复数据库恢复逻辑，或让批量模型评测保持逐条评测的语义。分类依据是主要考查能力；使用 Python 并不自动归为软件工程。子类别可以扩展。"},"title":{"en":"What the agent does","zh":"智能体要做什么"}},{"id":"evaluation","source_ids":["tb4-home","harbor-task-format","tb4-wal-config","tb4-batch-config","tb4-readme"],"text":{"en":"The official leaderboard reports resolution rate. Task verifiers check delivered behavior or artifacts and produce a reward. The reviewed task definitions use a separate verifier environment. A final answer saying the job is done is insufficient: the implementation must satisfy the task's checks. Read scores with the model, agent and run configuration.","zh":"官方榜单使用任务完成率（resolution rate）。验证程序检查交付行为或产物并给出 reward；本次核对的任务配置使用独立验证环境。仅在最终回答中声称完成并不足够，实现还必须满足验收。理解成绩时需要同时查看模型、智能体和运行配置。"},"title":{"en":"How results are checked","zh":"结果如何验收"}},{"id":"use_cases","source_ids":["tb4-readme","tb4-taxonomy","tb4-wal-task","tb4-batch-task"],"text":{"en":"Use it to examine whether an agent can carry work through an unfamiliar environment, rather than only propose a patch. Inspect relevant task definitions before choosing an evaluation: software repair, ML infrastructure and specialist scientific work exercise different skills.","zh":"如果你关心智能体能否在不熟悉的环境中把工作做完，而不只是提出补丁，可以参考这项评测。选择评测前应先阅读相关题目；软件修复、机器学习基础设施和专业科学任务考查的能力并不相同。"},"title":{"en":"When it is useful","zh":"适合用来判断什么"}},{"id":"limitations","source_ids":["tb4-readme","tb4-tasks-readme","tb4-release","tb4-wal-config"],"text":{"en":"Pin the version: Terminal-Bench evolves, and 2.x, 3.0, Hard and 4.0 results are not interchangeable. Version 4 includes GPU and multi-container work, so infrastructure matters. Task sources and verification code are public; isolation during a run is different from a private dataset.","zh":"必须固定版本：Terminal-Bench 持续演进，2.x、3.0、Hard 和 4.0 的成绩不能混用。第 4 版包含 GPU 与多容器任务，基础设施会影响复现。任务源码和验证代码公开；运行时隔离验证内容不等于数据集不公开。"},"title":{"en":"What to keep separate","zh":"阅读时注意哪些边界"}}],"sources":[{"id":"tb4-home","title":"Terminal-Bench official homepage — 4.0 and resolution rate","url":"https://www.tbench.ai/"},{"id":"tb4-release","title":"Terminal-Bench v4.0.0 task sources","url":"https://github.com/harbor-framework/terminal-bench/tree/v4.0.0/tasks"},{"id":"tb4-manifest","title":"Version 4.0.0 dataset manifest","url":"https://github.com/harbor-framework/terminal-bench/blob/452bf305c6daa62fc59061d22133a7cbc7c1572e/tasks/dataset.toml"},{"id":"tb4-readme","title":"Version 4.0.0 project overview and running instructions","url":"https://github.com/harbor-framework/terminal-bench/blob/452bf305c6daa62fc59061d22133a7cbc7c1572e/README.md"},{"id":"tb4-tasks-readme","title":"Version 4.0.0 task collection — GPU and multi-container support","url":"https://github.com/harbor-framework/terminal-bench/blob/452bf305c6daa62fc59061d22133a7cbc7c1572e/tasks/README.md"},{"id":"tb4-taxonomy","title":"Version 4.0.0 taxonomy","url":"https://github.com/harbor-framework/terminal-bench/blob/452bf305c6daa62fc59061d22133a7cbc7c1572e/docs/TAXONOMY.md"},{"id":"harbor-task-format","title":"Harbor task structure, trials and rewards","url":"https://docs.harborframework.com/tasks/overview"},{"id":"tb4-wal-task","title":"Version 4.0.0 task: WAL recovery ordering","url":"https://github.com/harbor-framework/terminal-bench/blob/452bf305c6daa62fc59061d22133a7cbc7c1572e/tasks/wal-recovery-ordering/instruction.md"},{"id":"tb4-wal-config","title":"WAL recovery ordering: resources and verifier configuration","url":"https://github.com/harbor-framework/terminal-bench/blob/452bf305c6daa62fc59061d22133a7cbc7c1572e/tasks/wal-recovery-ordering/task.toml"},{"id":"tb4-batch-task","title":"Version 4.0.0 task: batched evaluation parity","url":"https://github.com/harbor-framework/terminal-bench/blob/452bf305c6daa62fc59061d22133a7cbc7c1572e/tasks/batched-eval-parity/instruction.md"},{"id":"tb4-batch-config","title":"Batched evaluation parity: resources and verifier configuration","url":"https://github.com/harbor-framework/terminal-bench/blob/452bf305c6daa62fc59061d22133a7cbc7c1572e/tasks/batched-eval-parity/task.toml"}],"summary":{"en":"A terminal agent must inspect a working environment, make changes and deliver an outcome that executable checks can verify.","zh":"智能体需要理解工作环境、完成修改，并交付能够通过程序验收的结果。"},"title":"Terminal-Bench 4"},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:terminal_bench_4","languages":[],"library_direction":null,"modality":null,"name":"Terminal-Bench 4","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"terminal_bench_4","source":"model_reports","source_url":"https://www.tbench.ai/","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["coding","tool-use"],"collected_at":"2026-08-25T10:41:06Z","description":"Agentic coding & terminal use","descriptions":{"en":"Agentic coding & terminal use"},"evidence_summary":{"document_count":1,"model_count":432,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:terminal-bench-hard:9f873c2f-2c2d-4ccb-9e1b-71bf61b052be","reported_at":"2024-02-26","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:terminal-bench-hard","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"Terminal-Bench Hard","openness":"unknown","publisher":null,"released":"2025-09-02","released_reference":{"basis":"release_announcement","note":"Version history dates the introduction of the source-specific Terminal-Bench Hard evaluation to Index v3 on September 2. This is not the release of the broader Terminal-Bench family.","source_key":"artificial-analysis:terminal-bench-hard","source_url":"https://artificialanalysis.ai/methodology/intelligence-benchmarking"},"repo_kind":null,"repo_resolution_status":null,"score_count":432,"score_direction":"higher_is_better","score_summary":{"display_max":65.90909090909089,"display_multiplier":100,"model_count":432,"model_count_basis":"source_model_id","numeric_count":432,"raw_max":0.659090909090909,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:terminal-bench-hard:d93edfe8-bf35-49ad-b56e-b18116142a1c","reported_date":"2026-07-09","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard"},"unit":null},"scored_models":[{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Cogito v2.1 (Reasoning)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (ChatGPT)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2025-10-30","date_precision":"model_announcement","name":"Kimi Linear 48B A3B Instruct"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Non-reasoning)"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 40k"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"OLMo 2 32B"},{"date":"2024-11-26","date_precision":"model_announcement","name":"OLMo 2 7B"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2024-04-23","date_precision":"model_announcement","name":"Phi-3 Mini Instruct 3.8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-terminal-bench-hard","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":887,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:harbor-framework/terminal-bench","source_record":"catalog_e6829d18305cd8cb"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Terminal-Bench Hard is a harder terminal-agent benchmark variant evaluated with the Terminus-2 harness in Cohere's Command A+ and North Mini Code releases.","descriptions":{"en":"Terminal-Bench Hard is a harder terminal-agent benchmark variant evaluated with the Terminus-2 harness in Cohere's Command A+ and North Mini Code releases."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:terminal-bench-hard:command-a-plus-05-2026","reported_at":"2026-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-hard?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:terminal-bench-hard","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_e6829d18305cd8cb"},"modality":"text","name":"Terminal-Bench Hard","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":31.1,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.311,"raw_min":0.25,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:terminal-bench-hard:north-mini-code-1.0","reported_date":"2026-06-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-hard?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code 1.0"}],"slug":"llm-stats-terminal-bench-hard","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-hard?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","agentic","coding","tool-use"],"collected_at":"2026-08-25T10:41:06Z","description":"Agentic coding & terminal use","descriptions":{"en":"Agentic coding & terminal use"},"evidence_summary":{"document_count":1,"model_count":210,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:terminal-bench-v2.1:9f873c2f-2c2d-4ccb-9e1b-71bf61b052be","reported_at":"2024-02-26","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-v2-1"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:terminal-bench-v2.1","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"Terminal-Bench v2.1","openness":"unknown","publisher":null,"released":"2026-05-06","released_reference":{"basis":"release_announcement","note":"The official Terminal-Bench 2.1 release announcement is dated May 6. It precedes adoption in the Artificial Analysis June index update.","source_key":"artificial-analysis:terminal-bench-v2.1","source_url":"https://github.com/harbor-framework/terminal-bench-docs/blob/main/content/blog/terminal-bench-2-1.mdx"},"repo_kind":null,"repo_resolution_status":null,"score_count":210,"score_direction":"higher_is_better","score_summary":{"display_max":89.5131086142322,"display_multiplier":100,"model_count":210,"model_count_basis":"source_model_id","numeric_count":210,"raw_max":0.895131086142322,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:terminal-bench-v2.1:d998db47-9b67-4727-a2bb-2e1261020ac0","reported_date":"2026-07-09","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-v2-1"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-06-10","date_precision":"model_announcement","name":"DiffusionGemma 26B A4B"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2026-07-23","date_precision":"model_announcement","name":"G9v3-3B"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2026-07-09","date_precision":"model_announcement","name":"JT-4.1 Flash 236B A21B"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-07-14","date_precision":"model_announcement","name":"Motif 3 (Beta)"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-terminal-bench-v2-1","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-v2-1","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":2600,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":null,"source_record":"catalog_839250d9e1eaa9d1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Terminal-Bench is a benchmark for testing AI agents in real terminal environments, evaluating how well agents can handle real-world, end-to-end tasks autonomously. The benchmark includes tasks spanning coding, system administration, security, data science, model training, file operations, version control, and web development. Terminus is the neutral test-bed agent designed to work with Terminal-Bench, operating purely through tmux sessions without dedicated tools.","descriptions":{"en":"Terminal-Bench is a benchmark for testing AI agents in real terminal environments, evaluating how well agents can handle real-world, end-to-end tasks autonomously. The benchmark includes tasks spanning coding, system administration, security, data science, model training, file operations, version control, and web development. Terminus is the neutral test-bed agent designed to work with Terminal-Bench, operating purely through tmux sessions without dedicated tools."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:terminus:kimi-k2-instruct","reported_at":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminus?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:terminus","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_839250d9e1eaa9d1"},"modality":"text","name":"Terminus","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":25.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.25,"raw_min":0.25,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:terminus:kimi-k2-instruct","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminus?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"}],"slug":"llm-stats-terminus","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminus?top_n=500","topics":["coding","agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_terrabench_679be7e3"},"categories":["General AI","Information retrieval","Language & Knowledge","Planning","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"TerraBench is a benchmark for grounded Earth-science reasoning, built on TerraAgent, a ReAct-style framework that couples LLM planning with scientific tools. It includes 403 tasks across three tracks and eight domains with 24,500 verified steps.","descriptions":{"en":"TerraBench is a benchmark for grounded Earth-science reasoning, built on TerraAgent, a ReAct-style framework that couples LLM planning with scientific tools. It includes 403 tasks across three tracks and eight domains with 24,500 verified steps."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13148","languages":[],"library_direction":{"basis":"model-review","categories":["science","tool-use"],"join":"claire_record_id","primary":"science","source_record":"bm_terrabench_679be7e3"},"modality":null,"name":"TerraBench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13148","source_url":"https://arxiv.org/abs/2606.13148"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13148","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13148","topics":["agents","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":28,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ireliya/terralogic","source_record":"bm_terralogic_3dd73c53"},"categories":["Cybersecurity","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"TerraLogic evaluates hierarchical geospatial reasoning in Earth observation through 545 scenario-driven tasks spanning optical, SAR, and infrared imagery. Tasks include hazard vulnerability assessment, urban heat island analysis, and forest fragmentation dynamics. Evaluation uses tool-augmented agents with verifiable multi-step workflows and scored by step-wise and end-to-end metrics.","descriptions":{"en":"TerraLogic evaluates hierarchical geospatial reasoning in Earth observation through 545 scenario-driven tasks spanning optical, SAR, and infrared imagery. Tasks include hazard vulnerability assessment, urban heat island analysis, and forest fragmentation dynamics. Evaluation uses tool-augmented agents with verifiable multi-step workflows and scored by step-wise and end-to-end metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.12497","languages":[],"library_direction":{"basis":"model-review","categories":["science","visual-reasoning"],"join":"claire_record_id","primary":"science","source_record":"bm_terralogic_3dd73c53"},"modality":null,"name":"TerraLogic","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.12497","source_url":"https://arxiv.org/abs/2607.12497"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-12497","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12497","topics":["reasoning-math","cybersecurity","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Code & Software","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"TestEvo-Bench evaluates test and code co-evolution tasks from real commit histories. It includes two tracks: test generation (write new tests for changed behavior) and test update (adapt failing tests). Tasks are packaged with environment configurations for execution-grounded metrics like pass rate, coverage, and mutation score. The live component uses timestamps to enable post-training-cutoff evaluation.","descriptions":{"en":"TestEvo-Bench evaluates test and code co-evolution tasks from real commit histories. It includes two tracks: test generation (write new tests for changed behavior) and test update (adapt failing tests). Tasks are packaged with environment configurations for execution-grounded metrics like pass rate, coverage, and mutation score. The live component uses timestamps to enable post-training-cutoff evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02469","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_testevo-bench_296298e8"},"modality":null,"name":"TestEvo-Bench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02469","source_url":"https://arxiv.org/abs/2607.02469"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02469","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02469","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":[],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"","descriptions":{},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:community:fd462fc2-283c-4967-bd7d-b39d7c661807","languages":[],"library_direction":null,"modality":null,"name":"testing","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":"higher_is_better","score_summary":{"display_max":null,"display_multiplier":1,"model_count":null,"model_count_basis":null,"numeric_count":0,"raw_max":null,"raw_min":null,"source_reference":null,"unit":null},"scored_models":[],"slug":"llm-stats-community-fd462fc2-283c-4967-bd7d-b39d7c661807","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3Afd462fc2-283c-4967-bd7d-b39d7c661807?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"TeXFix-Bench evaluates LLM-based full-source document repair across LaTeX, Typst, and Markdown. It provides 10,437 repair instances derived from 743 openly licensed seeds, with a fixed zero-shot protocol and provider-pinned routing.","descriptions":{"en":"TeXFix-Bench evaluates LLM-based full-source document repair across LaTeX, Typst, and Markdown. It provides 10,437 repair instances derived from 743 openly licensed seeds, with a fixed zero-shot protocol and provider-pinned routing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07617","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_texfix-bench_827bc253"},"modality":null,"name":"TeXFix-Bench","openness":"unknown","publisher":null,"released":"2026-08-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07617","source_url":"https://arxiv.org/abs/2608.07617"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07617","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07617","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","代码工程","Code","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Text2World, based on planning domain definition language (PDDL), featuring hundreds of diverse domains and employing multi-criteria, execution-based metrics for a more robust evaluation. Text2World 基于规划领域定义语言 (PDDL)，拥有数百个不同的领域，并采用多标准、基于执行的衡量标准来进行更稳健的评估。","descriptions":{"en":"Text2World, based on planning domain definition language (PDDL), featuring hundreds of diverse domains and employing multi-criteria, execution-based metrics for a more robust evaluation.","zh":"Text2World 基于规划领域定义语言 (PDDL)，拥有数百个不同的领域，并采用多标准、基于执行的衡量标准来进行更稳健的评估。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1574","languages":[],"library_direction":null,"modality":null,"name":"Text2World","openness":"unknown","publisher":"HKU, HIT, Shanghai AI Laboratory, etc.","released":"2025-02-18","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1574-text2world","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Text2World","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":1546,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_textrich_83260e1b"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"TextRich evaluates detection of AI-generated text-rich images across six categories including posters, charts, receipts, tables, and UI screenshots, with a released dataset on Hugging Face.","descriptions":{"en":"TextRich evaluates detection of AI-generated text-rich images across six categories including posters, charts, receipts, tables, and UI screenshots, with a released dataset on Hugging Face."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19259","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_textrich_83260e1b"},"modality":null,"name":"TextRich","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19259","source_url":"https://arxiv.org/abs/2606.19259"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19259","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19259","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ArXiv paper introducing a benchmark for text rendering and preservation in video generation, with abstract stating two tracks and scoring metrics, but no public artifact available.","descriptions":{"en":"ArXiv paper introducing a benchmark for text rendering and preservation in video generation, with abstract stating two tracks and scoring metrics, but no public artifact available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.13412","languages":[],"library_direction":{"basis":"model-review-daily","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_textvbench_a3344875"},"modality":null,"name":"TextVBench","openness":"unknown","publisher":null,"released":"2026-10-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.13412","source_url":"https://arxiv.org/abs/2610.13412"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-13412","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13412","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_204ea79490a8d1fd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","image_to_text","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Introduced to benchmark VQA models' ability to read and reason about text within images, particularly for assistive technologies for visually impaired users. The dataset addresses the gap where existing VQA datasets had few text-based questions or were too small.","descriptions":{"en":"TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Introduced to benchmark VQA models' ability to read and reason about text within images, particularly for assistive technologies for visually impaired users. The dataset addresses the gap where existing VQA datasets had few text-based questions or were too small."},"evidence_summary":{"document_count":1,"model_count":16,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:textvqa:grok-1.5v","reported_at":"2024-04-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/textvqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:textvqa","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","document-ocr"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_204ea79490a8d1fd"},"modality":"multimodal","name":"TextVQA","openness":"unknown","publication":{"evidence":[{"note":"The original CVPR paper explicitly introduces TextVQA and directs readers to textvqa.org for the dataset. Its image-text question-answering task matches this record; the project site currently requires JavaScript.","url":"https://openaccess.thecvf.com/content_CVPR_2019/papers/Singh_Towards_VQA_Models_That_Can_Read_CVPR_2019_paper.pdf"}],"id":"textvqa-original","links":[{"kind":"paper","title":"Towards VQA Models That Can Read","url":"https://arxiv.org/abs/1904.08920"},{"kind":"website","title":"TextVQA project","url":"https://textvqa.org/"}],"record_keys":["llm-stats:textvqa"],"reviewed_at":"2026-10-26","scope":{"en":"The original TextVQA dataset introduced with LoRRA. OCR inputs and evaluation splits must still be checked in each score's source.","zh":"随 LoRRA 提出的原始 TextVQA 数据集；具体 OCR 输入和评测划分仍需核对每项成绩的来源。"}},"publisher":null,"released":"2019-04-18","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Towards VQA Models That Can Read. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:textvqa","source_url":"https://arxiv.org/abs/1904.08920"},"repo_kind":null,"repo_resolution_status":null,"score_count":16,"score_direction":"higher_is_better","score_summary":{"display_max":85.5,"display_multiplier":100,"model_count":16,"model_count_basis":"source_model_id","numeric_count":16,"raw_max":0.855,"raw_min":0.578,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:textvqa:qwen2-vl-72b","reported_date":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/textvqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Small"},{"date":"2024-12-13","date_precision":"model_announcement","name":"DeepSeek VL2 Tiny"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2024-04-12","date_precision":"model_announcement","name":"Grok-1.5V"},{"date":"2026-08-12","date_precision":"model_announcement","name":"LFM2.5-VL-3B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-vision-instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-textvqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/textvqa?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"TF-RefusalBench is a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings. It contains 5,200 prompts across French, German, Italian, and English, designed to trigger refusals in LLMs.","descriptions":{"en":"TF-RefusalBench is a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings. It contains 5,200 prompts across French, German, Italian, and English, designed to trigger refusals in LLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.23375","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","safety-alignment"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_tf-refusalbench_fb205626"},"modality":null,"name":"TF-RefusalBench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23375","source_url":"https://arxiv.org/abs/2606.23375"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23375","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23375","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","语言","Language","Strong Reasoning","VQA","多模态模型","VLM","语言理解","Comprehension","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Temporally-Grounded Language Generation (TGLG) is a benchmark for real-time vision-language models (VLMs) that focus on two key capabilities: perceptual updating and contingency awareness. 基于时间的语言生成（TGLG）是实时视觉语言模型（VLM）的基准，侧重于两个关键功能：感知更新和应急意识。该存储库还包含TGLG的基线实时VLM代码，即具有时间同步交织的视觉语言模型（VLM-TSI）。","descriptions":{"en":"Temporally-Grounded Language Generation (TGLG) is a benchmark for real-time vision-language models (VLMs) that focus on two key capabilities: perceptual updating and contingency awareness.","zh":"基于时间的语言生成（TGLG）是实时视觉语言模型（VLM）的基准，侧重于两个关键功能：感知更新和应急意识。该存储库还包含TGLG的基线实时VLM代码，即具有时间同步交织的视觉语言模型（VLM-TSI）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1846","languages":[],"library_direction":null,"modality":"multimodal","name":"TGLG","openness":"open","publisher":"University of Michigan","released":"2025-05-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1846-tglg","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TGLG","topics":["multimodal","reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Thai local dialect benchmark covers Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. 这是一个涵盖泰国北部（兰纳）、东北部（伊森）和南部（丹布罗）方言的泰国地方方言基准测试，评估大型语言模型在五项自然语言处理任务上的表现：总结、问答、翻译、对话以及与食物相关的任务。","descriptions":{"en":"Thai local dialect benchmark covers Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks.","zh":"这是一个涵盖泰国北部（兰纳）、东北部（伊森）和南部（丹布罗）方言的泰国地方方言基准测试，评估大型语言模型在五项自然语言处理任务上的表现：总结、问答、翻译、对话以及与食物相关的任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1734","languages":["Thai"],"library_direction":null,"modality":null,"name":"Thai_local_benchmark","openness":"unknown","publisher":"AI Singapore, Vidyasirimedhi Institute of Science and Technology, etc.","released":"2025-04-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1734-thai-local-benchmark","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Thai_local_benchmark","topics":["language"],"unit":null},{"aliases":["b2b-saas-ai-visibility-benchmark-2026"],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates which of 94 B2B SaaS brands are named by ChatGPT, Claude, Perplexity, Google Gemini and Google AI Overviews on 50 real buying questions across five G2 categories.","descriptions":{"en":"Evaluates which of 94 B2B SaaS brands are named by ChatGPT, Claude, Perplexity, Google Gemini and Google AI Overviews on 50 real buying questions across five G2 categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:hamzawamiqhq/b2b-saas-ai-visibility-benchmark-2026","languages":[],"library_direction":null,"modality":null,"name":"The B2B SaaS AI Visibility Benchmark 2026","openness":"unknown","publisher":null,"released":"2026-10-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:hamzawamiqhq/b2b-saas-ai-visibility-benchmark-2026","source_url":"https://github.com/hamzawamiqhq/b2b-saas-ai-visibility-benchmark-2026"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-hamzawamiqhq-b2b-saas-ai-visibility-benchmark-2026","source":"claire_radar","source_url":"https://github.com/hamzawamiqhq/b2b-saas-ai-visibility-benchmark-2026","topics":["cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Complexity Ceiling Benchmark evaluates sequential reasoning decay with depth scaling across grounded spatial state-tracking, symbolic pointer manipulation, and transitive relational inference.","descriptions":{"en":"Complexity Ceiling Benchmark evaluates sequential reasoning decay with depth scaling across grounded spatial state-tracking, symbolic pointer manipulation, and transitive relational inference."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.29278","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_the-complexity-ceiling-benchmark_2e65280a"},"modality":null,"name":"The Complexity Ceiling Benchmark","openness":"unknown","publisher":null,"released":"2026-06-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29278","source_url":"https://arxiv.org/abs/2606.29278"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29278","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29278","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":null,"source_record":"bm_the-endless-exam_311c35a8"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Measures mathematical progress on fourteen parameterised construction families.","descriptions":{"en":"Measures mathematical progress on fourteen parameterised construction families."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24555","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_the-endless-exam_311c35a8"},"modality":null,"name":"The Endless Exam","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24555","source_url":"https://arxiv.org/abs/2609.24555"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24555","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24555","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:lucasbergholdt/energysociety","source_record":"bm_the-energy-society_dce5cbc5"},"categories":["Agents","Agents & Tool Use","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The Energy Society is a multi-agent simulation environment where LLM agents operate under energy constraints tied to token generation, completing jobs and donating energy to survive. The evaluation measures emergent cooperative and competitive behavior across different model sizes and incentive settings.","descriptions":{"en":"The Energy Society is a multi-agent simulation environment where LLM agents operate under energy constraints tied to token generation, completing jobs and donating energy to survive. The evaluation measures emergent cooperative and competitive behavior across different model sizes and incentive settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.14865","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_the-energy-society_dce5cbc5"},"modality":null,"name":"The Energy Society","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14865","source_url":"https://arxiv.org/abs/2607.14865"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14865","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14865","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for iterative multimodal dialogue where a vision-language model issues corrective instructions to an image generator over multiple turns, with accumulated common ground observable as a rendered image.","descriptions":{"en":"A benchmark for iterative multimodal dialogue where a vision-language model issues corrective instructions to an image generator over multiple turns, with accumulated common ground observable as a rendered image."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01901","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","visual-reasoning"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_the-image-reconstruction-game_055fc5b9"},"modality":null,"name":"The Image Reconstruction Game","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01901","source_url":"https://arxiv.org/abs/2606.01901"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01901","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01901","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":57,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:imitator-game/the-imitator-game","source_record":"bm_the-imitator-game_8d69f8a7"},"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates robot imitation across four levels of increasing divergence between demonstration and robot scene, plus an open platform for blind A/B human evaluation.","descriptions":{"en":"Evaluates robot imitation across four levels of increasing divergence between demonstration and robot scene, plus an open platform for blind A/B human evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22301","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_the-imitator-game_8d69f8a7"},"modality":null,"name":"The Imitator Game","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22301","source_url":"http://arxiv.org/abs/2608.22301v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22301","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22301v1","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:rudycelekli/the-value-engine-benchmark","source_record":"bm_the-value-engine-benchmark_3b9623cf"},"categories":["Agents & Tool Use","General AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"An environment for evaluating and training LLM agents on multi-touch enterprise-sales negotiation, grading episodes with a sales-methodology rubric to produce scalar rewards and structured diagnostics.","descriptions":{"en":"An environment for evaluating and training LLM agents on multi-touch enterprise-sales negotiation, grading episodes with a sales-methodology rubric to produce scalar rewards and structured diagnostics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:rudycelekli/the-value-engine-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["professional-work","multi-agent-games","health-medicine"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_the-value-engine-benchmark_3b9623cf"},"modality":null,"name":"The Value Engine Benchmark (VEB)","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:rudycelekli/the-value-engine-benchmark","source_url":"https://github.com/rudycelekli/the-value-engine-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-rudycelekli-the-value-engine-benchmark","source":"claire_radar","source_url":"https://github.com/rudycelekli/the-value-engine-benchmark","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Mathematics & Formal Science","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on theorem proving in Lean4 using classical theorems, with two versions: main and premised. Includes metrics for theorem-level coverage and token efficiency to assess partial progress and proof structure.","descriptions":{"en":"Evaluates LLMs on theorem proving in Lean4 using classical theorems, with two versions: main and premised. Includes metrics for theorem-level coverage and token efficiency to assess partial progress and proof structure."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.09450","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_theorembench_8bd203fd"},"modality":null,"name":"TheoremBench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09450","source_url":"https://arxiv.org/abs/2606.09450"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09450","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09450","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":161,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_f3ee08b150f0c57e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","physics","reasoning","finance"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A theorem-driven question answering dataset containing 800 high-quality questions covering 350+ theorems from Math, Physics, EE&CS, and Finance. Designed to evaluate AI models' capabilities to apply theorems to solve challenging university-level science problems.","descriptions":{"en":"A theorem-driven question answering dataset containing 800 high-quality questions covering 350+ theorems from Math, Physics, EE&CS, and Finance. Designed to evaluate AI models' capabilities to apply theorems to solve challenging university-level science problems."},"evidence_summary":{"document_count":1,"model_count":6,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:theoremqa:qwen2-72b-instruct","reported_at":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/theoremqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:theoremqa","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","math","science"],"join":"llm_stats_id","primary":"knowledge-expert-qa","source_record":"catalog_f3ee08b150f0c57e"},"modality":"text","name":"TheoremQA","openness":"unknown","publisher":null,"released":"2023-05-21","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: TheoremQA: A Theorem-driven Question Answering dataset. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:theoremqa","source_url":"https://arxiv.org/abs/2305.12524"},"repo_kind":null,"repo_resolution_status":null,"score_count":6,"score_direction":"higher_is_better","score_summary":{"display_max":44.4,"display_multiplier":100,"model_count":6,"model_count_basis":"source_model_id","numeric_count":6,"raw_max":0.444,"raw_min":0.253,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:theoremqa:qwen2-72b-instruct","reported_date":"2024-07-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/theoremqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 7B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-theoremqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/theoremqa?top_n=500","topics":["reasoning-math","finance","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":161,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"artifact:arxiv:2305.12524","repository":null,"source_record":"catalog_f3ee08b150f0c57e"},"categories":["推理","Reasoning","大语言模型","LLM","数理能力","Math","逻辑推理","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TheoremQA is the first theorem-driven question-answering dataset designed to evaluate AI models’ capabilities to apply theorems to solve challenging science problems. It is curated by domain experts containing 800 high-quality questions covering 350 theorems from Math, Physics, EE&CS, and Finance. TheoremQA 是第一个基于定理的问题回答数据集，旨在评估 AI 模型应用定理解决复杂科学问题的能力。该数据集由领域专家精心策划，包含 800 个高质量问题，涵盖来自数学、物理、电气与计算机科学以及金融的 350 个定理。","descriptions":{"en":"TheoremQA is the first theorem-driven question-answering dataset designed to evaluate AI models’ capabilities to apply theorems to solve challenging science problems. It is curated by domain experts containing 800 high-quality questions covering 350 theorems from Math, Physics, EE&CS, and Finance.","zh":"TheoremQA 是第一个基于定理的问题回答数据集，旨在评估 AI 模型应用定理解决复杂科学问题的能力。该数据集由领域专家精心策划，包含 800 个高质量问题，涵盖来自数学、物理、电气与计算机科学以及金融的 350 个定理。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1134","languages":[],"library_direction":{"basis":"model-review","categories":["knowledge-expert-qa","math","science"],"join":"artifact:arxiv:2305.12524","primary":"knowledge-expert-qa","source_record":"catalog_f3ee08b150f0c57e"},"modality":null,"name":"TheoremQA","openness":"open","publisher":"University of Waterloo","released":"2023-12-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1134-theoremqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TheoremQA","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Language & Knowledge","Materials & Chemistry"],"collected_at":"2026-10-14T09:44:51Z","description":"TxBench-PP evaluates AI agents on small-molecule preclinical pharmacology tasks including mechanism-of-action, pharmacodynamics, and safety reasoning, using realistic workflow snapshots and deterministic scoring.","descriptions":{"en":"TxBench-PP evaluates AI agents on small-molecule preclinical pharmacology tasks including mechanism-of-action, pharmacodynamics, and safety reasoning, using realistic workflow snapshots and deterministic scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19245","languages":[],"library_direction":{"basis":"model-review","categories":["science","health-medicine"],"join":"claire_record_id","primary":"science","source_record":"bm_therapeuticsbench_0f53fc28"},"modality":null,"name":"TherapeuticsBench","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19245","source_url":"https://arxiv.org/abs/2606.19245"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19245","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19245","topics":["agents","medical","science"],"unit":null},{"aliases":["One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows"],"attention_signals":{"as_of":"2026-10-12","github_stars":100,"github_stars_30d":41,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"claire_record_id","repository":"gh:microsoft/thinkingbox","source_record":"bm_one-success-isn-t-reliability-thinkingbox-_a377202a"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM agents on 507 stateful business workflows in an executable sandbox, with task-specific checks that accept valid trajectories and reject wrong, missing, or extra effects.","descriptions":{"en":"Evaluates LLM agents on 507 stateful business workflows in an executable sandbox, with task-specific checks that accept valid trajectories and reject wrong, missing, or extra effects."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.19741","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_one-success-isn-t-reliability-thinkingbox-_a377202a"},"modality":null,"name":"Thinkingbox-Bench","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.19741","source_url":"https://arxiv.org/abs/2608.19741"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-19741","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19741","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Presents a benchmark for force-aware humanoid interaction using demonstrations with synchronized motion and force data. Includes Force-Aware Tracking Score and a simulation protocol.","descriptions":{"en":"Presents a benchmark for force-aware humanoid interaction using demonstrations with synchronized motion and force data. Includes Force-Aware Tracking Score and a simulation protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.06052","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_thorarena_63609675"},"modality":null,"name":"ThorArena","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06052","source_url":"https://arxiv.org/abs/2607.06052"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06052","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06052","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":20,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:edstevenson/thousandworlds","source_record":"bm_thousandworlds_3c4422be"},"categories":["Language & Knowledge","Materials & Chemistry","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ThousandWorlds is a benchmark for climate emulation of potentially habitable exoplanets. It provides a dataset of approximately 1800 simulations from five global climate models, mapping eight planet parameters to 3D atmospheric fields. It includes three nested benchmark subsets: single-simulator regression, multi-simulator regression with complete observations, and multi-simulator regression with structured missingness. Two evaluation protocols are provided: one for ranking methods and one measuring performance relative to inter-model disagreement.","descriptions":{"en":"ThousandWorlds is a benchmark for climate emulation of potentially habitable exoplanets. It provides a dataset of approximately 1800 simulations from five global climate models, mapping eight planet parameters to 3D atmospheric fields. It includes three nested benchmark subsets: single-simulator regression, multi-simulator regression with complete observations, and multi-simulator regression with structured missingness. Two evaluation protocols are provided: one for ranking methods and one measuring performance relative to inter-model disagreement."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.18338","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_thousandworlds_3c4422be"},"modality":null,"name":"ThousandWorlds","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18338","source_url":"https://arxiv.org/abs/2606.18338"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18338","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18338","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["视觉定位","Visual-Localization","空间理解","Spatial-Understanding","视频理解","Video-Understanding","reasoning-VLM","agentic physical security","Pulsar VLM","物理智能","Embodied AI","图像理解","Image Understanding","Spatial Understanding","Video Understanding","逻辑推理","Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Threat Signature Eval is a domain-specific benchmark for evaluating video-language models on real-world physical security / surveillance footage. It measures how well a model can recognize and classify security-relevant events into a 10-category threat taxonomy (e.g., Fire & Smoke, Fighting & Violen Threat Signature Eval 是一个面向物理安防场景的领域基准，用于评测视频语言模型在真实监控摄像头画面中的理解与事件识别能力。该基准要求模型在安全相关的视频片段中识别并分类事件（10 类威胁签名，如火灾烟雾、斗殴暴力、非法入侵等），相关介绍与结果在 Ambient.ai 的 Pulsar VLM 发布主题演讲中给出。","descriptions":{"en":"Threat Signature Eval is a domain-specific benchmark for evaluating video-language models on real-world physical security / surveillance footage. It measures how well a model can recognize and classify security-relevant events into a 10-category threat taxonomy (e.g., Fire & Smoke, Fighting & Violen","zh":"Threat Signature Eval 是一个面向物理安防场景的领域基准，用于评测视频语言模型在真实监控摄像头画面中的理解与事件识别能力。该基准要求模型在安全相关的视频片段中识别并分类事件（10 类威胁签名，如火灾烟雾、斗殴暴力、非法入侵等），相关介绍与结果在 Ambient.ai 的 Pulsar VLM 发布主题演讲中给出。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"opencompass:2391","languages":[],"library_direction":null,"modality":"video","name":"Threat-Signature_Eval","openness":"unknown","publisher":"Ambient.ai","released":"2025-11-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2391-threat-signature-eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Threat-Signature_Eval","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","其他","Other","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"THUNDER is a benchmark designed to evaluate digital pathology foundation models on tile-level image understanding tasks, facilitating comparative analysis across various downstream tasks and models. THUNDER 是一个用于评估数字病理学基础模型在切片级图像理解任务中表现的基准，旨在支持多种下游任务和模型的对比分析。","descriptions":{"en":"THUNDER is a benchmark designed to evaluate digital pathology foundation models on tile-level image understanding tasks, facilitating comparative analysis across various downstream tasks and models.","zh":"THUNDER 是一个用于评估数字病理学基础模型在切片级图像理解任务中表现的基准，旨在支持多种下游任务和模型的对比分析。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2050","languages":[],"library_direction":null,"modality":null,"name":"THUNDER","openness":"unknown","publisher":"CentraleSupelec , IHU-National PRecISion Medicine Center in Oncology , etc.","released":"2025-07-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2050-thunder","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/THUNDER","topics":["language"],"unit":null},{"aliases":["Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal LLM capability to integrate text-image clues and recover facts within deeply interleaved contexts.","descriptions":{"en":"Evaluates multimodal LLM capability to integrate text-image clues and recover facts within deeply interleaved contexts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02573","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_deeply-interleaved-text-image-contexts-for_43c50b59"},"modality":null,"name":"TIC-Bench","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02573","source_url":"https://arxiv.org/abs/2609.02573"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02573","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02573","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-agent collaboration in Minecraft-based tasks with time-sensitive complementary collaboration, requiring agent heterogeneity, mandatory collaboration, and dynamic environments. The framework generates diverse tasks via YAML specifications and filters invalid configurations.","descriptions":{"en":"Evaluates multi-agent collaboration in Minecraft-based tasks with time-sensitive complementary collaboration, requiring agent heterogeneity, mandatory collaboration, and dynamic environments. The framework generates diverse tasks via YAML specifications and filters invalid configurations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15684","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","embodied-robotics"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_tickingcollabbench_3fac6f85"},"modality":null,"name":"TickingCollabBench","openness":"unknown","publisher":null,"released":"2026-06-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15684","source_url":"https://arxiv.org/abs/2606.15684"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15684","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15684","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"The proposed benchmark would evaluate coding agents on RTL-level timing closure under post-place-and-route evaluation.","descriptions":{"en":"The proposed benchmark would evaluate coding agents on RTL-level timing closure under post-place-and-route evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23363","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agentic-coding","engineering-design"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_tictacbench_30828aab"},"modality":null,"name":"TicTacBench","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23363","source_url":"https://arxiv.org/abs/2609.23363"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23363","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23363","topics":[],"unit":null},{"aliases":["Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding"],"attention_signals":{"as_of":"2026-10-13","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:icsetepa44/tide","source_record":"bm_time-present-and-time-past-benchmarking-la_2933cb13"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM performance on 3,050 expert-verified QA pairs about temporally evolving official documents, requiring identification of the document version in force on a queried date across eight task types and three knowledge-access settings.","descriptions":{"en":"Evaluates LLM performance on 3,050 expert-verified QA pairs about temporally evolving official documents, requiring identification of the document version in force on a queried date across eight task types and three knowledge-access settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.08512","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","knowledge-expert-qa"],"join":"claire_record_id","primary":"factuality","source_record":"bm_time-present-and-time-past-benchmarking-la_2933cb13"},"modality":null,"name":"TIDE","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08512","source_url":"https://arxiv.org/abs/2608.08512"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08512","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08512","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates conversational text-to-SQL on 1,542 samples built from 514 BIRD SQLs, measuring chain identification, drift recognition-resolution, and execution accuracy.","descriptions":{"en":"Evaluates conversational text-to-SQL on 1,542 samples built from 514 BIRD SQLs, measuring chain identification, drift recognition-resolution, and execution accuracy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.29543","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_tide-bench_a28d078a"},"modality":null,"name":"TIDE-Bench","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29543","source_url":"https://arxiv.org/abs/2608.29543"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29543","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29543","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Geometric reasoning","Vision & 3D","physics.flu-dyn"],"collected_at":"2026-09-25T04:40:22Z","description":"A 3D incompressible-turbulence benchmark with multiple configurations, independent ensembles, controlled generalization splits, standardized baselines, and physical-fidelity evaluation.","descriptions":{"en":"A 3D incompressible-turbulence benchmark with multiple configurations, independent ensembles, controlled generalization splits, standardized baselines, and physical-fidelity evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04222","languages":[],"library_direction":null,"modality":null,"name":"TIDE: A Physically Diverse 3D Turbulence Benchmark Dataset for Advancing Scientific Machine Learning","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04222","source_url":"https://arxiv.org/abs/2608.04222"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04222","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04222","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM safety behaviors across four risk domains and four threat levels using a six-label behavior scale judged by two LLMs.","descriptions":{"en":"Evaluates LLM safety behaviors across four risk domains and four threat levels using a six-label behavior scale judged by two LLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.05117","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_tier_4396e23d"},"modality":null,"name":"TIER","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05117","source_url":"https://arxiv.org/abs/2609.05117"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05117","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05117","topics":["safety"],"unit":null},{"aliases":["How Good are Foundation Models in Longitudinal MRI Disease Progression Reasoning?"],"attention_signals":{"as_of":"2026-10-12","github_stars":5,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:wafaalghallabi/time-aware-mri","source_record":"bm_how-good-are-foundation-models-in-longitud_4e643d29"},"categories":["Biology & Drug Discovery","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"The benchmark comprises 3,920 expert-verified QA pairs from 890 patients across longitudinal MRI timepoints, evaluating temporal reasoning, disease progression, structured localization, sequence ordering, and change localization.","descriptions":{"en":"The benchmark comprises 3,920 expert-verified QA pairs from 890 patients across longitudinal MRI timepoints, evaluating temporal reasoning, disease progression, structured localization, sequence ordering, and change localization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.13309","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_how-good-are-foundation-models-in-longitud_4e643d29"},"modality":null,"name":"Time-Aware Multi-View MRI Benchmark","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13309","source_url":"https://arxiv.org/abs/2608.13309"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13309","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13309","topics":["reasoning-math","medical"],"unit":null},{"aliases":["PRISM Edit: One Vector for All Temporal Answers"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"TimeConflict evaluates whether model edits preserve historically correct answers while making updated facts current in the appropriate temporal contexts.","descriptions":{"en":"TimeConflict evaluates whether model edits preserve historically correct answers while making updated facts current in the appropriate temporal contexts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.11327","languages":[],"library_direction":null,"modality":null,"name":"TimeConflict","openness":"unknown","publisher":null,"released":"2026-07-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.11327","source_url":"https://arxiv.org/abs/2607.11327"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-11327","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11327","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":10,"join":"claire_record_id","repository":null,"source_record":"bm_timeline-bench_c908bf31"},"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"AI agents increasingly carry out long-horizon professional work, but their evaluations rarely require a finished creative deliverable.","descriptions":{"en":"AI agents increasingly carry out long-horizon professional work, but their evaluations rarely require a finished creative deliverable."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35143","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","chat-writing"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_timeline-bench_c908bf31"},"modality":null,"name":"Timeline-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35143","source_url":"https://arxiv.org/abs/2609.35143"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35143","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35143","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"TimeSage-EV evaluates LLM agents on time series analysis tasks in evolving environments, using 60 institutional scenarios across 6 domains with 1,485 scenario-period QA pairs. Agents receive data and reports, with withheld target releases as ground truth, assessing state identification, data summarization, and outlook reasoning.","descriptions":{"en":"TimeSage-EV evaluates LLM agents on time series analysis tasks in evolving environments, using 60 institutional scenarios across 6 domains with 1,485 scenario-period QA pairs. Agents receive data and reports, with withheld target releases as ground truth, assessing state identification, data summarization, and outlook reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.14270","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_timesage-ev_e6de9c4a"},"modality":null,"name":"TimeSage-EV","openness":"unknown","publisher":null,"released":"2026-08-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14270","source_url":"https://arxiv.org/abs/2608.14270"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14270","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14270","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"TimeSage-MT evaluates agentic time series reasoning in multi-turn dialogues, covering 240 tasks and 2,680 turns across 8 domains, with a protocol and leaderboard for comparing systems.","descriptions":{"en":"TimeSage-MT evaluates agentic time series reasoning in multi-turn dialogues, covering 240 tasks and 2,680 turns across 8 domains, with a protocol and leaderboard for comparing systems."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01498","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_timesage-mt_04a07ed7"},"modality":null,"name":"TimeSage-MT","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01498","source_url":"https://arxiv.org/abs/2606.01498"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01498","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01498","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TimeTravel Taxonomy maps artifacts from 10 civilizations, 266 cultures, and 10k+ verified samples for AI-driven historical analysis. 时间旅行分类将来自 10 个文明、266 个文化以及 10k+个验证样本的文物映射，用于 AI 驱动的历史分析。","descriptions":{"en":"TimeTravel Taxonomy maps artifacts from 10 civilizations, 266 cultures, and 10k+ verified samples for AI-driven historical analysis.","zh":"时间旅行分类将来自 10 个文明、266 个文化以及 10k+个验证样本的文物映射，用于 AI 驱动的历史分析。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1668","languages":[],"library_direction":null,"modality":"multimodal","name":"TimeTravel","openness":"restricted","publisher":"Mohamed bin Zayed University of AI, etc.","released":"2025-02-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1668-timetravel","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TimeTravel","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"TimeVista is a benchmark for evaluating time series forecasting using Vision-Language Models (VLMs) as judges, with 5563 samples and rubrics for micro- and macro-level judgments.","descriptions":{"en":"TimeVista is a benchmark for evaluating time series forecasting using Vision-Language Models (VLMs) as judges, with 5563 samples and rubrics for micro- and macro-level judgments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16173","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_timevista_65f9c096"},"modality":null,"name":"TimeVista","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16173","source_url":"https://arxiv.org/abs/2606.16173"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16173","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16173","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","Strong Reasoning","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TinyQA is a benchmark suite designed to evaluate the reasoning abilities of large language models (LLMs). It focuses on assessing LLMs through natural language question-answer pairs, covering various types of reasoning tasks such as causal, logical, and commonsense reasoning. TinyQA是一个用于评估大语言模型（LLMs）推理能力的基准测试套件。该基准专注于通过自然语言问题和答案对来衡量LLMs的推理能力，涵盖了多种类型的推理任务，包括因果推理、逻辑推理和常识推理。TinyQA提供了多样化的数据集，旨在挑战LLMs的推理深度和广度。通过严格的评估，TinyQA能够帮助研究人员更好地理解LLMs在处理复杂语言任务时的表现，并为改进模型提供方向。","descriptions":{"en":"TinyQA is a benchmark suite designed to evaluate the reasoning abilities of large language models (LLMs). It focuses on assessing LLMs through natural language question-answer pairs, covering various types of reasoning tasks such as causal, logical, and commonsense reasoning.","zh":"TinyQA是一个用于评估大语言模型（LLMs）推理能力的基准测试套件。该基准专注于通过自然语言问题和答案对来衡量LLMs的推理能力，涵盖了多种类型的推理任务，包括因果推理、逻辑推理和常识推理。TinyQA提供了多样化的数据集，旨在挑战LLMs的推理深度和广度。通过严格的评估，TinyQA能够帮助研究人员更好地理解LLMs在处理复杂语言任务时的表现，并为改进模型提供方向。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1839","languages":[],"library_direction":null,"modality":null,"name":"tiny_qa_benchmark_pp","openness":"unknown","publisher":"Comet ML","released":"2025-05-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1839-tiny-qa-benchmark-pp","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/tiny_qa_benchmark_pp","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sver0411/tinyedgebench","source_record":"bm_tinyedgebench_7dfc4853"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Compares four lightweight classification methods on a synthetic sensor-window classification task with four classes, scoring accuracy, macro F1, and compiled flash size for ESP32-class devices.","descriptions":{"en":"Compares four lightweight classification methods on a synthetic sensor-window classification task with four classes, scoring accuracy, macro F1, and compiled flash size for ESP32-class devices."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:sver0411/tinyedgebench","languages":[],"library_direction":null,"modality":null,"name":"TinyEdgeBench","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:sver0411/tinyedgebench","source_url":"https://github.com/Sver0411/TinyEdgeBench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-sver0411-tinyedgebench","source":"claire_radar","source_url":"https://github.com/Sver0411/TinyEdgeBench","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":27,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"llm_stats_id","repository":null,"source_record":"catalog_5a8634ec2cdab971"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A tool-calling and multimodal interaction benchmark for testing visual instruction following and execution reliability.","descriptions":{"en":"A tool-calling and multimodal interaction benchmark for testing visual instruction following and execution reliability."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tir-bench:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tir-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tir-bench","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","visual-reasoning"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_5a8634ec2cdab971"},"modality":"multimodal","name":"TIR-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":61.6,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.616,"raw_min":0.532,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tir-bench:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tir-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"}],"slug":"llm-stats-tir-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tir-bench?top_n=500","topics":["agents","multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"TLA+-Bench evaluates natural-language to TLA+ specification generation by executing each specification in the TLA+ model checker across the full reachable state space. The dataset includes 403 model-checked gold and 897 parse-only silver specifications, with multiple descriptions and difficulty labels.","descriptions":{"en":"TLA+-Bench evaluates natural-language to TLA+ specification generation by executing each specification in the TLA+ model checker across the full reachable state space. The dataset includes 403 model-checked gold and 897 parse-only silver specifications, with multiple descriptions and difficulty labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.23425","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_tla-bench_4cd98a83"},"modality":null,"name":"TLA+-Bench","openness":"unknown","publisher":null,"released":"2026-07-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23425","source_url":"https://arxiv.org/abs/2607.23425"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23425","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23425","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":24,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":null,"source_record":"catalog_8c04a4476ebb94ba"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["summarization","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A large-scale summarization dataset containing over 9 million training instances extracted from Reddit, designed for extreme summarization (generating one-sentence summaries with high compression and abstraction). More than twice larger than previously proposed datasets.","descriptions":{"en":"A large-scale summarization dataset containing over 9 million training instances extracted from Reddit, designed for extreme summarization (generating one-sentence summaries with high compression and abstraction). More than twice larger than previously proposed datasets."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tldr9+-(test):llama-3.2-3b-instruct","reported_at":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tldr9%2B-%28test%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tldr9+-(test)","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_8c04a4476ebb94ba"},"modality":"text","name":"TLDR9+ (test)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":19.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.19,"raw_min":0.19,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tldr9+-(test):llama-3.2-3b-instruct","reported_date":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tldr9%2B-%28test%29?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 3B Instruct"}],"slug":"llm-stats-tldr9-test","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tldr9%2B-%28test%29?top_n=500","topics":["language"],"unit":null},{"aliases":["token-xray"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:dustin621408-pixel/token-xray","source_record":"bm_token-xray_ed42aeb2"},"categories":["Finance","Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Measures quality-preserving reductions in repeated AI model execution by comparing baseline and governed lanes.","descriptions":{"en":"Measures quality-preserving reductions in repeated AI model execution by comparing baseline and governed lanes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:dustin621408-pixel/token-xray","languages":[],"library_direction":null,"modality":null,"name":"Token X-Ray","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:dustin621408-pixel/token-xray","source_url":"https://github.com/dustin621408-pixel/token-xray"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-dustin621408-pixel-token-xray","source":"claire_radar","source_url":"https://github.com/dustin621408-pixel/token-xray","topics":["finance"],"unit":null},{"aliases":["TomaMMU: A Comprehensive Multimodal Understanding Benchmark for Tomato Leaf Diseases"],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":34,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_tomammu-a-comprehensive-multimodal-underst_69a4f816"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language models on tomato leaf disease understanding through seven tasks spanning basic perception, pathology understanding, and expert diagnosis using multimodal QA pairs.","descriptions":{"en":"Evaluates vision-language models on tomato leaf disease understanding through seven tasks spanning basic perception, pathology understanding, and expert diagnosis using multimodal QA pairs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.08727","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","science"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_tomammu-a-comprehensive-multimodal-underst_69a4f816"},"modality":null,"name":"TomaBench","openness":"unknown","publisher":null,"released":"2026-08-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.08727","source_url":"https://arxiv.org/abs/2608.08727"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-08727","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08727","topics":["multimodal","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":43,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":20,"join":"llm_stats_id","repository":null,"source_record":"catalog_5ed728c2fa5d767b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"TOMATO (Temporal Reasoning Multimodal Evaluation) assesses multimodal models on motion and temporal perception in video, testing understanding of actions, motion, and changes over time.","descriptions":{"en":"TOMATO (Temporal Reasoning Multimodal Evaluation) assesses multimodal models on motion and temporal perception in video, testing understanding of actions, motion, and changes over time."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tomato:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tomato?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tomato","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_5ed728c2fa5d767b"},"modality":"multimodal","name":"TOMATO","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":79.5,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.795,"raw_min":0.568,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tomato:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tomato?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-tomato","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tomato?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["Tool Decathlon","Toolathlon","Toolathlon-Verified"],"attention_signals":null,"categories":["tool_use"],"collected_at":null,"description":"Aggregates ten heterogeneous tool suites; the per-suite spread is wide.","descriptions":{"en":"Aggregates ten heterogeneous tool suites; the per-suite spread is wide."},"evidence_summary":{"document_count":8,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000tool_decathlon\u0000zai_glm_5_model_card\u0000tool_decathlon\u0000as reported\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"first_score_reported_at":"2026-02-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000tool_decathlon\u0000zai_glm_5_model_card\u0000tool_decathlon\u0000as reported\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:tool_decathlon","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["tool-use"],"join":"review:14H034160212:2026-10-15","primary":"tool-use","source_record":null},"modality":null,"name":"Tool Decathlon","openness":"unknown","publisher":null,"released":"2025-10-28","released_reference":{"basis":"benchmark_release","source_key":"model-reports:tool_decathlon","source_url":"https://toolathlon.xyz/"},"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":76.5,"display_multiplier":1,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":76.5,"raw_min":38.0,"source_reference":{"obs_id":"curated\u0000tool_decathlon\u0000moonshot_kimi_k3_model_card\u0000tool_decathlon\u0000reasoning=max\u0000Kimi K3","observation_id":"curated\u0000tool_decathlon\u0000moonshot_kimi_k3_model_card\u0000tool_decathlon\u0000reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-25","date_precision":"document_publication","name":"DeepSeek-V4-Flash"},{"date":"2026-04-22","date_precision":"document_publication","name":"DeepSeek-V4-Pro"},{"date":"2026-08-13","date_precision":"document_publication","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"document_publication","name":"GLM-5.2"},{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"tool_decathlon","source":"model_reports","source_url":"https://toolathlon.xyz/","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"The evaluation object is a set of 128 scenarios across 16 domains for tool-calling LLM agents in regulated industries, assessing conflicts between safety-aligned values and deployment instructions. The task involves processing confidential documents and measuring override behavior.","descriptions":{"en":"The evaluation object is a set of 128 scenarios across 16 domains for tool-calling LLM agents in regulated industries, assessing conflicts between safety-aligned values and deployment instructions. The task involves processing confidential documents and measuring override behavior."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.14285","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","tool-use"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_toolalignbench_7a0c33d1"},"modality":null,"name":"ToolAlignBench","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14285","source_url":"https://arxiv.org/abs/2607.14285"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14285","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14285","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":491,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":48,"join":"llm_stats_id","repository":null,"source_record":"catalog_cb3f74ceede2e47b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents","tool_calling"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Tool Decathlon is a comprehensive benchmark for evaluating AI agents' ability to use multiple tools across diverse task categories. It measures proficiency in tool selection, sequencing, and execution across ten different tool-use scenarios.","descriptions":{"en":"Tool Decathlon is a comprehensive benchmark for evaluating AI agents' ability to use multiple tools across diverse task categories. It measures proficiency in tool selection, sequencing, and execution across ten different tool-use scenarios."},"evidence_summary":{"document_count":1,"model_count":37,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:toolathlon:deepseek-reasoner","reported_at":"2025-12-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/toolathlon?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"llm-stats:toolathlon","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"llm_stats_id","primary":"tool-use","source_record":"catalog_cb3f74ceede2e47b"},"modality":"text","name":"Toolathlon","openness":"open","publisher":null,"released":"2025-10-29","released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":37,"score_direction":"higher_is_better","score_summary":{"display_max":75.6,"display_multiplier":100,"model_count":37,"model_count_basis":"source_model_id","numeric_count":37,"raw_max":0.756,"raw_min":0.269,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:toolathlon:muse-spark-1.1","reported_date":"2026-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/toolathlon?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2 (Thinking)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek-V3.2-Speciale"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0423"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-0731"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Flash-Max"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-0813"},{"date":"2026-04-23","date_precision":"model_announcement","name":"DeepSeek-V4-Pro-Max"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2"},{"date":"2026-08-14","date_precision":"model_announcement","name":"GLM-5.3"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Laguna S 2.1"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-toolathlon","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/toolathlon?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["Toolathlon Verified","Toolathlon"],"attention_signals":null,"categories":["tool_use"],"collected_at":null,"description":"Pass@1 averaged over 3 independent runs. Official evaluation service.","descriptions":{"en":"Pass@1 averaged over 3 independent runs. Official evaluation service."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000toolathlon_verified\u0000zai_glm_5_3_flash_model_card\u0000toolathlon_verified\u0000official evaluation service, pass@1 averaged over 3 independent runs\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"first_score_reported_at":"2026-08-27","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000toolathlon_verified\u0000zai_glm_5_3_flash_model_card\u0000toolathlon_verified\u0000official evaluation service, pass@1 averaged over 3 independent runs\u0000GLM-5.3-Flash","reported_at":"2026-08-27","source_url":"https://z.ai/blog/glm-5.3-flash"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:toolathlon_verified","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["tool-use"],"join":"review:14H034160212:2026-10-15","primary":"tool-use","source_record":null},"modality":null,"name":"Toolathlon Verified","openness":"unknown","publisher":null,"released":"2025-11-01","released_reference":{"basis":"benchmark_release","source_key":"model-reports:toolathlon_verified","source_url":"https://github.com/toolathlon/toolathlon"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":78.4,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":78.4,"raw_min":74.1,"source_reference":{"obs_id":"curated\u0000toolathlon_verified\u0000zai_glm_5_3_flash_model_card\u0000toolathlon_verified\u0000official evaluation service, pass@1 averaged over 3 independent runs\u0000GLM-5.3-Flash","observation_id":"curated\u0000toolathlon_verified\u0000zai_glm_5_3_flash_model_card\u0000toolathlon_verified\u0000official evaluation service, pass@1 averaged over 3 independent runs\u0000GLM-5.3-Flash","reported_at":"2026-08-27","reported_date":"2026-08-27","source_id":"zai_glm_5_3_flash_model_card","source_url":"https://z.ai/blog/glm-5.3-flash"},"unit":"percent"},"scored_models":[{"date":"2026-08-27","date_precision":"document_publication","name":"GLM-5.3-Flash"},{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"toolathlon_verified","source":"model_reports","source_url":"https://github.com/toolathlon/toolathlon","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:foreverskyou/toolbench-x","source_record":"bm_toolbench-x_cb76a568"},"categories":["Agents & Tool Use","General AI","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"ToolBench-X evaluates tool-using agents on multi-step tasks with executable tools and automatic scoring, in environments that include recoverable reliability hazards such as specification drift, invocation errors, execution failures, output drift, and cross-source conflicts.","descriptions":{"en":"ToolBench-X evaluates tool-using agents on multi-step tasks with executable tools and automatic scoring, in environments that include recoverable reliability hazards such as specification drift, invocation errors, execution failures, output drift, and cross-source conflicts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.25819","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_toolbench-x_cb76a568"},"modality":null,"name":"ToolBench-X","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25819","source_url":"https://arxiv.org/abs/2606.25819"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25819","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25819","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"ToolFailBench evaluates tool-use failures in LLM agents across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. It labels traces with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and LLM judges.","descriptions":{"en":"ToolFailBench evaluates tool-use failures in LLM agents across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. It labels traces with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and LLM judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.04686","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","factuality"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_toolfailbench_999a8910"},"modality":null,"name":"ToolFailBench","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.04686","source_url":"https://arxiv.org/abs/2607.04686"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-04686","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04686","topics":["agents","cybersecurity","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","其他","Other","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ToolHop is a dataset specifically designed for rigorous evaluation of multi-hop tool use, which ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach. ToolHop是一个通过查询驱动构建的数据集，专门用于评测大模型的多跳工具使用能力，具备多样化的查询、有意义的相互依赖关系、本地可执行的工具、详细的反馈和可验证的答案五大特征。","descriptions":{"en":"ToolHop is a dataset specifically designed for rigorous evaluation of multi-hop tool use, which ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach.","zh":"ToolHop是一个通过查询驱动构建的数据集，专门用于评测大模型的多跳工具使用能力，具备多样化的查询、有意义的相互依赖关系、本地可执行的工具、详细的反馈和可验证的答案五大特征。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1712","languages":[],"library_direction":null,"modality":null,"name":"ToolHop","openness":"unknown","publisher":"复旦大学&字节跳动","released":"2025-01-05","released_reference":{"basis":"paper_first_version","note":"First version introducing the query-driven multi-hop tool-use benchmark.","source_key":"opencompass:1712","source_url":"https://arxiv.org/abs/2501.02506"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1712-toolhop","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ToolHop","topics":["agents"],"unit":null},{"aliases":["When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark that tests how tool-using LLM agents replan and recover from tool failures. It varies task structure (DAG-based complexity) and failure type (explicit/implicit, transient/permanent).","descriptions":{"en":"A benchmark that tests how tool-using LLM agents replan and recover from tool failures. It varies task structure (DAG-based complexity) and failure type (explicit/implicit, transient/permanent)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.05806","languages":[],"library_direction":null,"modality":null,"name":"ToolMaze","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05806","source_url":"https://arxiv.org/abs/2606.05806"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05806","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05806","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ToolMenuBench is a benchmark for evaluating tool-menu filtering strategies in multi-step LLM agents. It varies tool-menu size, distractor type, state-dependent structure, and risk exposure, and reports filter-level and downstream metrics such as task success, tool calls, and token usage.","descriptions":{"en":"ToolMenuBench is a benchmark for evaluating tool-menu filtering strategies in multi-step LLM agents. It varies tool-menu size, distractor type, state-dependent structure, and risk exposure, and reports filter-level and downstream metrics such as task success, tool calls, and token usage."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.15508","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_toolmenubench_fce3386d"},"modality":null,"name":"ToolMenuBench","openness":"unknown","publisher":null,"released":"2026-06-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15508","source_url":"https://arxiv.org/abs/2606.15508"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15508","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15508","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ToolRet is a heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks, and a corpus of 43k tools, collected from existing datasets. ToolRet是一个包含 7.6k 个不同检索任务的异构工具检索基准，以及从现有数据集中收集的 43k 个工具语料库。","descriptions":{"en":"ToolRet is a heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks, and a corpus of 43k tools, collected from existing datasets.","zh":"ToolRet是一个包含 7.6k 个不同检索任务的异构工具检索基准，以及从现有数据集中收集的 43k 个工具语料库。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1607","languages":[],"library_direction":null,"modality":null,"name":"ToolRet","openness":"unknown","publisher":"Shandong University, Baidu Inc, etc.","released":"2025-03-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1607-toolret","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ToolRet","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Code & Software","General AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates tool-calling agents under perturbations across the tool-use pipeline, attributing failures to selection, grounding, argument binding, and feedback handling.","descriptions":{"en":"Evaluates tool-calling agents under perturbations across the tool-use pipeline, attributing failures to selection, grounding, argument binding, and feedback handling."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23635","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_toolrobustbench_a4d79725"},"modality":null,"name":"ToolRobustBench","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23635","source_url":"https://arxiv.org/abs/2608.23635"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23635","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23635","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":293,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:topu-benchmark/topu-lbvs","source_record":"bm_topu-lbvs_d83acb6d"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"TopU-LBVS is a multi-target ligand-based virtual screening benchmark with 93 protein targets and hard-negative decoys, scored by enrichment factor metrics.","descriptions":{"en":"TopU-LBVS is a multi-target ligand-based virtual screening benchmark with 93 protein targets and hard-negative decoys, scored by enrichment factor metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.29740","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_topu-lbvs_d83acb6d"},"modality":null,"name":"TopU-LBVS","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.29740","source_url":"https://arxiv.org/abs/2609.29740"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-29740","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29740","topics":["science"],"unit":null},{"aliases":["Tables Decoded: DELTA for Structure, TARQA for Understanding"],"attention_signals":{"as_of":"2026-10-14","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tihiitborg/tables-decoded","source_record":"bm_tables-decoded-delta-for-structure-tarqa-f_d73b0786"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A proposed dataset named TORQUE for table reconstruction and table visual question answering, with a supporting table structure representation.","descriptions":{"en":"A proposed dataset named TORQUE for table reconstruction and table visual question answering, with a supporting table structure representation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.17458","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","visual-reasoning"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_tables-decoded-delta-for-structure-tarqa-f_d73b0786"},"modality":null,"name":"TORQUE","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17458","source_url":"https://arxiv.org/abs/2609.17458"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17458","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17458","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Multimodal","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"TouchSafeBench is a physics-grounded benchmark with 2,940 simulated episodes for evaluating collision grounding in vision-language models for safe human-robot collaboration.","descriptions":{"en":"TouchSafeBench is a physics-grounded benchmark with 2,940 simulated episodes for evaluating collision grounding in vision-language models for safe human-robot collaboration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.31196","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","visual-reasoning"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_touchsafebench_f153a852"},"modality":null,"name":"TouchSafeBench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31196","source_url":"https://arxiv.org/abs/2605.31196"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31196","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31196","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of roughly 800 backdoored LLMs across attack objectives, trigger mechanisms, poisoning rates, model scales, and training mechanisms. Evaluates whether a defender can recover a planted trigger given white-box access and behavior of concern.","descriptions":{"en":"A benchmark of roughly 800 backdoored LLMs across attack objectives, trigger mechanisms, poisoning rates, model scales, and training mechanisms. Evaluates whether a defender can recover a planted trigger given white-box access and behavior of concern."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.26849","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_toxscreen_d4a7771e"},"modality":null,"name":"ToxScreen","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26849","source_url":"https://arxiv.org/abs/2607.26849"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26849","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26849","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"TRACE is an evidence-grounded safety evaluation benchmark covering prompts, reasoning traces, and final responses of Large Reasoning Models (LRMs). It includes prompts in two languages, nine risk categories, ten attack strategies, and per-component safety annotations with supporting evidence. Evaluates 18 guardrail models on safety judgment and evidence extraction.","descriptions":{"en":"TRACE is an evidence-grounded safety evaluation benchmark covering prompts, reasoning traces, and final responses of Large Reasoning Models (LRMs). It includes prompts in two languages, nine risk categories, ten attack strategies, and per-component safety annotations with supporting evidence. Evaluates 18 guardrail models on safety judgment and evidence extraction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24232","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_trace_e2618c65"},"modality":null,"name":"TRACE","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24232","source_url":"https://arxiv.org/abs/2608.24232"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24232","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24232","topics":["reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":14,"join":"claire_record_id","repository":"gh:amuseum-whr/tracebench","source_record":"bm_trace-bench_6cb6dfd0"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-reference image generation by decomposing prompts into atomic operators (Anchor, Disentangle, Apply, Compose) and scoring operator-aligned capabilities and diagnostic failure localization.","descriptions":{"en":"Evaluates multi-reference image generation by decomposing prompts into atomic operators (Anchor, Disentangle, Apply, Compose) and scoring operator-aligned capabilities and diagnostic failure localization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.16765","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_trace-bench_6cb6dfd0"},"modality":null,"name":"TRACE-Bench","openness":"unknown","publisher":null,"released":"2026-08-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.16765","source_url":"https://arxiv.org/abs/2608.16765"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-16765","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16765","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":9,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:om-ai-lab/trace-bench","source_record":"bm_trace-temporal-audit-and-condition-aware-e_1c33473a"},"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Streaming video understanding evaluation with annotated timing and trigger conditions, scored on accuracy, timeliness, response behavior, workload, completion, and reliability.","descriptions":{"en":"Streaming video understanding evaluation with annotated timing and trigger conditions, scored on accuracy, timeliness, response behavior, workload, completion, and reliability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.30670","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_trace-temporal-audit-and-condition-aware-e_1c33473a"},"modality":null,"name":"TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30670","source_url":"https://arxiv.org/abs/2609.30670"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30670","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30670","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"A diagnostic benchmark for auditing citation-like artifacts in retrieval-augmented generation and agent-workflow evaluation, including content-equivalent pairs, citation ablations, correctness conflicts, quality gaps, prompt-strength ladders, decoupled judging, and workflow ranking.","descriptions":{"en":"A diagnostic benchmark for auditing citation-like artifacts in retrieval-augmented generation and agent-workflow evaluation, including content-equivalent pairs, citation ablations, correctness conflicts, quality gaps, prompt-strength ladders, decoupled judging, and workflow ranking."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.12439","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research","health-medicine"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_tracejudgebench_6d9b999a"},"modality":null,"name":"TraceJudgeBench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12439","source_url":"https://arxiv.org/abs/2609.12439"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12439","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12439","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"TradeVerse evaluates LLMs on longitudinal political trade negotiation understanding using reconstructed minutes of 1170 WTO meetings across 5 groups and 89 product groups, with three tasks: predicting HS codes, identifying responding countries, and generating final statements.","descriptions":{"en":"TradeVerse evaluates LLMs on longitudinal political trade negotiation understanding using reconstructed minutes of 1170 WTO meetings across 5 groups and 89 product groups, with three tasks: predicting HS codes, identifying responding countries, and generating final statements."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06549","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_tradeverse_c4be7871"},"modality":null,"name":"TradeVerse","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06549","source_url":"https://arxiv.org/abs/2608.06549"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06549","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06549","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","coding"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Trae Code Gen is a component of Trae Agent Bench that evaluates implementing new functionality across multiple programming languages in containerized, runnable repositories.","descriptions":{"en":"Trae Code Gen is a component of Trae Agent Bench that evaluates implementing new functionality across multiple programming languages in containerized, runnable repositories."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:trae-code-gen:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-code-gen?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:trae-code-gen","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","code-generation"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_96b952f2d4104005"},"modality":"text","name":"Trae Code Gen","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":62.4,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.624,"raw_min":0.597,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:trae-code-gen:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-code-gen?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-trae-code-gen","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-code-gen?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","coding"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Trae Error Fix is a component of Trae Agent Bench that evaluates fixing existing code across multiple programming languages in containerized, runnable repositories.","descriptions":{"en":"Trae Error Fix is a component of Trae Agent Bench that evaluates fixing existing code across multiple programming languages in containerized, runnable repositories."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:trae-error-fix:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-error-fix?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:trae-error-fix","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_ddff600660f07ae2"},"modality":"text","name":"Trae Error Fix","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":63.3,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.633,"raw_min":0.567,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:trae-error-fix:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-error-fix?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-trae-error-fix","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-error-fix?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Introduces a benchmark for counterfactual roadside traffic video generation, but no artifact evidence is provided.","descriptions":{"en":"Introduces a benchmark for counterfactual roadside traffic video generation, but no artifact evidence is provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.30722","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["factuality","image-video-generation"],"join":"claire_record_id","primary":"factuality","source_record":"bm_trafficimag_32cee8ed"},"modality":null,"name":"TrafficImag","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30722","source_url":"https://arxiv.org/abs/2609.30722"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30722","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30722","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates critical error detection in long-horizon agent trajectories using 486 annotated failed trajectories with error lifecycle labels.","descriptions":{"en":"Evaluates critical error detection in long-horizon agent trajectories using 486 annotated failed trajectories with error lifecycle labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.06346","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_trajerrbench_f533923d"},"modality":null,"name":"TrajErrBench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06346","source_url":"https://arxiv.org/abs/2608.06346"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06346","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06346","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension","合作共建","Co-Built"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TransBench is the first industry-oriented comprehensive multilingual translation evaluation system designed for industrial applications. It quantifies translation model performance across diverse industries and linguistic environments through meticulously curated datasets aligned with standards. TransBench is the first industry-oriented comprehensive multilingual translation evaluation system designed for industrial applications. It quantifies translation model performance across diverse industries and linguistic environments through meticulously curated datasets aligned with standards.","descriptions":{"en":"TransBench is the first industry-oriented comprehensive multilingual translation evaluation system designed for industrial applications. It quantifies translation model performance across diverse industries and linguistic environments through meticulously curated datasets aligned with standards.","zh":"TransBench is the first industry-oriented comprehensive multilingual translation evaluation system designed for industrial applications. It quantifies translation model performance across diverse industries and linguistic environments through meticulously curated datasets aligned with standards."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1855","languages":["English","Chinese","Japanese","French","Arabic","Multilingual"],"library_direction":null,"modality":null,"name":"TransBench","openness":"unknown","publisher":"Alibaba International Digital Commerce, Beijing Language and Culture University","released":"2025-05-20","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1855-transbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TransBench","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"TransClean is a controlled benchmark of 9,900 noisy-and-clean translation output pairs for detecting and extracting clean translations from large language model outputs, including 8,800 synthetic and 1,100 manually curated instances.","descriptions":{"en":"TransClean is a controlled benchmark of 9,900 noisy-and-clean translation output pairs for detecting and extracting clean translations from large language model outputs, including 8,800 synthetic and 1,100 manually curated instances."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11399","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_transclean_110295c3"},"modality":null,"name":"TransClean","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11399","source_url":"https://arxiv.org/abs/2609.11399"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11399","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11399","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":770,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_c9d70348c2c65bb3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"COMET-22 is an ensemble machine translation evaluation metric combining a COMET estimator model trained with Direct Assessments and a multitask model that predicts sentence-level scores and word-level OK/BAD tags. It demonstrates improved correlations compared to state-of-the-art metrics and increased robustness to critical errors.","descriptions":{"en":"COMET-22 is an ensemble machine translation evaluation metric combining a COMET estimator model trained with Direct Assessments and a multitask model that predicts sentence-level scores and word-level OK/BAD tags. It demonstrates improved correlations compared to state-of-the-art metrics and increased robustness to critical errors."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:translation-en→set1-comet22:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-comet22?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:translation-en→set1-comet22","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_c9d70348c2c65bb3"},"modality":"text","name":"Translation en→Set1 COMET22","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":89.1,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.891,"raw_min":0.885,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:translation-en→set1-comet22:nova-pro","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-comet22?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"}],"slug":"llm-stats-translation-en-set1-comet22","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-comet22?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":770,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_4f5d3b6b0451dacd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Translation evaluation using spBLEU (SentencePiece BLEU), a BLEU metric computed over text tokenized with a language-agnostic SentencePiece subword model. Introduced in the FLORES-101 evaluation benchmark for low-resource and multilingual machine translation.","descriptions":{"en":"Translation evaluation using spBLEU (SentencePiece BLEU), a BLEU metric computed over text tokenized with a language-agnostic SentencePiece subword model. Introduced in the FLORES-101 evaluation benchmark for low-resource and multilingual machine translation."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:translation-en→set1-spbleu:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-spbleu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:translation-en→set1-spbleu","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_4f5d3b6b0451dacd"},"modality":"text","name":"Translation en→Set1 spBleu","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":43.4,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.434,"raw_min":0.402,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:translation-en→set1-spbleu:nova-pro","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-spbleu?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"}],"slug":"llm-stats-translation-en-set1-spbleu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-spbleu?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":770,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_2c15ad0533b756e6"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"COMET-22 is a neural machine translation evaluation metric that uses an ensemble of two models: a COMET estimator trained with Direct Assessments and a multitask model that predicts sentence-level scores and word-level OK/BAD tags. It provides improved correlations with human judgments and increased robustness to critical errors compared to previous metrics.","descriptions":{"en":"COMET-22 is a neural machine translation evaluation metric that uses an ensemble of two models: a COMET estimator trained with Direct Assessments and a multitask model that predicts sentence-level scores and word-level OK/BAD tags. It provides improved correlations with human judgments and increased robustness to critical errors compared to previous metrics."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:translation-set1→en-comet22:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-comet22?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:translation-set1→en-comet22","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_2c15ad0533b756e6"},"modality":"text","name":"Translation Set1→en COMET22","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":89.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.89,"raw_min":0.887,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:translation-set1→en-comet22:nova-pro","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-comet22?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"}],"slug":"llm-stats-translation-set1-en-comet22","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-comet22?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":770,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_6b67cff18fbba972"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"spBLEU (SentencePiece BLEU) evaluation metric for machine translation quality assessment, using language-agnostic SentencePiece tokenization with BLEU scoring. Part of the FLORES-101 evaluation benchmark for low-resource and multilingual machine translation.","descriptions":{"en":"spBLEU (SentencePiece BLEU) evaluation metric for machine translation quality assessment, using language-agnostic SentencePiece tokenization with BLEU scoring. Part of the FLORES-101 evaluation benchmark for low-resource and multilingual machine translation."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:translation-set1→en-spbleu:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-spbleu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:translation-set1→en-spbleu","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_6b67cff18fbba972"},"modality":"text","name":"Translation Set1→en spBleu","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":44.4,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.444,"raw_min":0.426,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:translation-set1→en-spbleu:nova-pro","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-spbleu?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"}],"slug":"llm-stats-translation-set1-en-spbleu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-spbleu?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","智能体","Agent","指令跟随","Instruct","Legal AI","任务执行","Task Execution","语言理解","Comprehension","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"本文提出TransLaw——专为香港判例翻译设计的协同交互式多智能体框架。该框架创新性地将传统翻译流程拆解为翻译、错误标注及校对修正三大子任务，并分配三个智能体协同执行。为评估框架性能，我们构建了大规模双语基准数据集BJC Judgments，对13个开源与商业大语言模型（作为智能体）展开评测。实验结果验证了协同策略的有效性：在多智能体协作显著提升效果的同时，提供了具有参考价值的LLM性能横向对比。通过错误类型学分析，本研究进一步揭示了亟待解决的关键翻译挑战。未来工作将聚焦于优化智能体架构以应对这些挑战，同时开发更全面、低成本的评估基准。 本文提出TransLaw——专为香港判例翻译设计的协同交互式多智能体框架。该框架创新性地将传统翻译流程拆解为翻译、错误标注及校对修正三大子任务，并分配三个智能体协同执行。为评估框架性能，我们构建了大规模双语基准数据集BJC Judgments，对13个开源与商业大语言模型（作为智能体）展开评测。实验结果验证了协同策略的有效性：在多智能体协作显著提升效果的同时，提供了具有参考价值的LLM性能横向对比。通过错误类型学分析，本研究进一步揭示了亟待解决的关键翻译挑战。未来工作将聚焦于优化智能体架构以应对这些挑战，同时开发更全面、低成本的评估基准。","descriptions":{"en":"本文提出TransLaw——专为香港判例翻译设计的协同交互式多智能体框架。该框架创新性地将传统翻译流程拆解为翻译、错误标注及校对修正三大子任务，并分配三个智能体协同执行。为评估框架性能，我们构建了大规模双语基准数据集BJC Judgments，对13个开源与商业大语言模型（作为智能体）展开评测。实验结果验证了协同策略的有效性：在多智能体协作显著提升效果的同时，提供了具有参考价值的LLM性能横向对比。通过错误类型学分析，本研究进一步揭示了亟待解决的关键翻译挑战。未来工作将聚焦于优化智能体架构以应对这些挑战，同时开发更全面、低成本的评估基准。","zh":"本文提出TransLaw——专为香港判例翻译设计的协同交互式多智能体框架。该框架创新性地将传统翻译流程拆解为翻译、错误标注及校对修正三大子任务，并分配三个智能体协同执行。为评估框架性能，我们构建了大规模双语基准数据集BJC Judgments，对13个开源与商业大语言模型（作为智能体）展开评测。实验结果验证了协同策略的有效性：在多智能体协作显著提升效果的同时，提供了具有参考价值的LLM性能横向对比。通过错误类型学分析，本研究进一步揭示了亟待解决的关键翻译挑战。未来工作将聚焦于优化智能体架构以应对这些挑战，同时开发更全面、低成本的评估基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:2039","languages":[],"library_direction":null,"modality":null,"name":"TransLaw","openness":"unknown","publisher":"City University of Hong Kong","released":"2025-07-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2039-translaw","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TransLaw","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"TRAPSBench is a procedurally generated video benchmark with 1,404 physics pairs to test epistemic restraint in VLMs, using Penalized Epistemic Calibration Score (PECS).","descriptions":{"en":"TRAPSBench is a procedurally generated video benchmark with 1,404 physics pairs to test epistemic restraint in VLMs, using Penalized Epistemic Calibration Score (PECS)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.13167","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","video-understanding"],"join":"claire_record_id","primary":"world-models","source_record":"bm_trapsbench_2047e708"},"modality":null,"name":"TRAPSBench","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13167","source_url":"https://arxiv.org/abs/2608.13167"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13167","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13167","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Planning","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"TravelEval evaluates LLM-powered travel planning agents in a realistic sandbox with accommodation pricing and intercity transport data, using six dimensions: accuracy, compliance, temporality, spatiality, economy, and utility. It simulates complete plans with API-integrated geographic information and queuing time.","descriptions":{"en":"TravelEval evaluates LLM-powered travel planning agents in a realistic sandbox with accommodation pricing and intercity transport data, using six dimensions: accuracy, compliance, temporality, spatiality, economy, and utility. It simulates complete plans with API-integrated geographic information and queuing time."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01046","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","professional-work"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_traveleval_64e034c5"},"modality":null,"name":"TravelEval","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01046","source_url":"https://arxiv.org/abs/2606.01046"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01046","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01046","topics":["agents"],"unit":null},{"aliases":["Locating Hidden Failures Makes Long-Horizon Agents More Reliable"],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark of human-verified annotations for locating hidden failures in long-horizon agent trajectories.","descriptions":{"en":"Benchmark of human-verified annotations for locating hidden failures in long-horizon agent trajectories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17930","languages":[],"library_direction":null,"modality":null,"name":"Traverse","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17930","source_url":"https://arxiv.org/abs/2609.17930"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17930","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17930","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","Language & Knowledge","Mathematics & Formal Science","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"TREAT evaluates LLMs' ability to recognize theorem identities from equivalence-preserving formula transformations, with 737 identities and 29,480 transformed rows.","descriptions":{"en":"TREAT evaluates LLMs' ability to recognize theorem identities from equivalence-preserving formula transformations, with 737 identities and 29,480 transformed rows."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07540","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_treat_ece84fbf"},"modality":null,"name":"TREAT","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07540","source_url":"https://arxiv.org/abs/2608.07540"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07540","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07540","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":93,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":51,"join":"llm_stats_id","repository":"gh:haochen-wang409/treevgr","source_record":"catalog_b4f97d9fcd658e33"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","spatial_reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"TreeBench evaluates visual grounded reasoning, requiring models to localize and reason about fine-grained visual details.","descriptions":{"en":"TreeBench evaluates visual grounded reasoning, requiring models to localize and reason about fine-grained visual details."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:treebench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/treebench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:treebench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_b4f97d9fcd658e33"},"modality":"multimodal","name":"TreeBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":71.1,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.711,"raw_min":0.711,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:treebench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/treebench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-treebench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/treebench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Biology & Drug Discovery","Safety & Trustworthiness","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"TreeProbe is a dataset of 4,719 expert-adjudicated items covering 467 diseases and 10 subtasks across the Tibetan Tree of Medicine framework, used to evaluate cultural bias in LLMs.","descriptions":{"en":"TreeProbe is a dataset of 4,719 expert-adjudicated items covering 467 diseases and 10 subtasks across the Tibetan Tree of Medicine framework, used to evaluate cultural bias in LLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00640","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","safety-alignment"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_treeprobe_2c533886"},"modality":null,"name":"TreeProbe","openness":"unknown","publisher":null,"released":"2026-08-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00640","source_url":"https://arxiv.org/abs/2608.00640"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00640","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00640","topics":["safety","medical"],"unit":null},{"aliases":["TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning"],"attention_signals":null,"categories":["General AI","Language & Knowledge","Planning","Reasoning"],"collected_at":"2026-09-25T04:40:22Z","description":"TREK evaluates agents on synthesizing feasible travel itineraries under interacting entity, route, time, budget, and persona constraints.","descriptions":{"en":"TREK evaluates agents on synthesizing feasible travel itineraries under interacting entity, route, time, budget, and persona constraints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.26977","languages":[],"library_direction":null,"modality":null,"name":"TREK","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.26977","source_url":"https://arxiv.org/abs/2607.26977"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-26977","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26977","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.DC"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark that evaluates open-weight LLMs and a DeepLog baseline for root cause analysis using four public log datasets, with three retrieval-augmented strategies, on a local GPU setup.","descriptions":{"en":"A benchmark that evaluates open-weight LLMs and a DeepLog baseline for root cause analysis using four public log datasets, with three retrieval-augmented strategies, on a local GPU setup."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.14762","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_tricalrag_37af9d49"},"modality":null,"name":"TriCalRAG","openness":"unknown","publisher":null,"released":"2026-09-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.14762","source_url":"https://arxiv.org/abs/2609.14762"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-14762","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14762","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:kristenzhang/triggerbench-official","source_record":"bm_triggerbench_0852166e"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"TriggerBench is a benchmark for evaluating prospective memory (PM) in LLMs, spanning five dimensions across daily assistant and professional workflow scenarios, with matched retrospective memory (RM) controls, contrastive variants, and overloaded triggers, measuring proactive recall, false-alarm rate, and attentional robustness.","descriptions":{"en":"TriggerBench is a benchmark for evaluating prospective memory (PM) in LLMs, spanning five dimensions across daily assistant and professional workflow scenarios, with matched retrospective memory (RM) controls, contrastive variants, and overloaded triggers, measuring proactive recall, false-alarm rate, and attentional robustness."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.23459","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_triggerbench_0852166e"},"modality":null,"name":"TriggerBench","openness":"unknown","publisher":null,"released":"2026-06-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.23459","source_url":"https://arxiv.org/abs/2606.23459"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-23459","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23459","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":1,"hf_dataset_downloads":551,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:vanilladucky/trinity","source_record":"bm_trinity_35990632"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video highlight detection across three perspectives—Event, Emotion, and Nature—using precomputed CLIP features and corresponding annotations.","descriptions":{"en":"Evaluates video highlight detection across three perspectives—Event, Emotion, and Nature—using precomputed CLIP features and corresponding annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.29577","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_trinity_35990632"},"modality":null,"name":"TRINITY","openness":"unknown","publisher":null,"released":"2026-08-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.29577","source_url":"https://arxiv.org/abs/2608.29577"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-29577","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29577","topics":[],"unit":null},{"aliases":["From Context-Aware to Conflict-Aware: Generalizing Contrastive Decoding for Knowledge Conflict in LLMs"],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-09-25T04:40:22Z","description":"TriState-Bench evaluates how language models handle conflicts between retrieved context and parametric knowledge after calibrating each model's prior knowledge.","descriptions":{"en":"TriState-Bench evaluates how language models handle conflicts between retrieved context and parametric knowledge after calibrating each model's prior knowledge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.10298","languages":[],"library_direction":null,"modality":null,"name":"TriState-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10298","source_url":"https://arxiv.org/abs/2606.10298"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10298","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10298","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":341,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_d60b25609f91cc15"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A large-scale reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents (six per question on average) that provide high quality distant supervision for answering the questions. The dataset features relatively complex, compositional questions with considerable syntactic and lexical variability, requiring cross-sentence reasoning to find answers.","descriptions":{"en":"A large-scale reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents (six per question on average) that provide high quality distant supervision for answering the questions. The dataset features relatively complex, compositional questions with considerable syntactic and lexical variability, requiring cross-sentence reasoning to find answers."},"evidence_summary":{"document_count":1,"model_count":18,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:triviaqa:gemma-2-27b-it","reported_at":"2024-06-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/triviaqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:triviaqa","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","knowledge-expert-qa"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_d60b25609f91cc15"},"modality":"text","name":"TriviaQA","openness":"unknown","publication":{"evidence":[{"note":"The authors' university project names the ACL 2017 paper and provides both reading-comprehension and unfiltered releases. Its question-answer-evidence description matches this source record.","url":"https://nlp.cs.washington.edu/triviaqa/"}],"id":"triviaqa-original","links":[{"kind":"paper","title":"TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension","url":"https://arxiv.org/abs/1705.03551"},{"kind":"website","title":"TriviaQA project and data","url":"https://nlp.cs.washington.edu/triviaqa/"}],"record_keys":["llm-stats:triviaqa"],"reviewed_at":"2026-10-26","scope":{"en":"Original TriviaQA paper and University of Washington project. The project offers reading-comprehension and unfiltered data; the source record does not identify a specific split or filtering setup.","zh":"TriviaQA 原论文及华盛顿大学项目；项目分别提供阅读理解版和未过滤数据，当前来源记录未指明具体划分或过滤设置。"}},"publisher":null,"released":"2017-05-09","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:triviaqa","source_url":"https://arxiv.org/abs/1705.03551"},"repo_kind":null,"repo_resolution_status":null,"score_count":18,"score_direction":"higher_is_better","score_summary":{"display_max":85.1,"display_multiplier":100,"model_count":18,"model_count_basis":"source_model_id","numeric_count":18,"raw_max":0.851,"raw_min":0.592,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:triviaqa:kimi-k2-base","reported_date":"2025-07-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/triviaqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Base"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (14B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Base 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Base 2512)"},{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"},{"date":"2025-09-01","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2024-07-18","date_precision":"model_announcement","name":"Mistral NeMo Instruct"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Base"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Base"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1 24B Instruct"}],"slug":"llm-stats-triviaqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/triviaqa?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":341,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:1705.03551","repository":null,"source_record":"catalog_d60b25609f91cc15"},"categories":["知识","Knowledge","大语言模型","LLM","知识储备","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TriviaqQA is a reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaqQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. TriviaqQA是一个阅读理解数据集，包含超过65万个问题-答案-证据三元组。其包括95K个问答对，由冷知识爱好者和独立收集的事实性文档撰写，平均每个问题6个，为回答问题提供高质量的远程监督。","descriptions":{"en":"TriviaqQA is a reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaqQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions.","zh":"TriviaqQA是一个阅读理解数据集，包含超过65万个问题-答案-证据三元组。其包括95K个问答对，由冷知识爱好者和独立收集的事实性文档撰写，平均每个问题6个，为回答问题提供高质量的远程监督。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:512","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","knowledge-expert-qa"],"join":"artifact:arxiv:1705.03551","primary":"factuality","source_record":"catalog_d60b25609f91cc15"},"modality":null,"name":"TriviaQA","openness":"unknown","publisher":null,"released":"2017-05-09","released_reference":{"basis":"paper_first_version","note":"First version introducing the TriviaQA question-answering dataset.","source_key":"opencompass:512","source_url":"https://arxiv.org/abs/1705.03551"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-512-triviaqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TriviaQA","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Geometric reasoning","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"TriViewBench evaluates multimodal LLMs on multi-view structural reasoning using synthetic 3D scenes with controlled object count and occlusion. It comprises 1,923 scenes and over 14,000 QA pairs across four complexity levels and three reasoning categories: Local Decision, Object Counting, and Global Recovery.","descriptions":{"en":"TriViewBench evaluates multimodal LLMs on multi-view structural reasoning using synthetic 3D scenes with controlled object count and occlusion. It comprises 1,923 scenes and over 14,000 QA pairs across four complexity levels and three reasoning categories: Local Decision, Object Counting, and Global Recovery."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.26029","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_triviewbench_b0684034"},"modality":null,"name":"TriViewBench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26029","source_url":"https://arxiv.org/abs/2606.26029"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26029","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26029","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":235,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:triworldbench/triworldbench","source_record":"bm_triworldbench_95a17948"},"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates embodied world models on synchronized triple-view video generation using 19 metrics across consistency, alignment, physics, motion, temporal and visual quality.","descriptions":{"en":"Evaluates embodied world models on synchronized triple-view video generation using 19 metrics across consistency, alignment, physics, motion, temporal and visual quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.26314","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation","embodied-robotics"],"join":"claire_record_id","primary":"world-models","source_record":"bm_triworldbench_95a17948"},"modality":null,"name":"TriWorldBench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.26314","source_url":"https://arxiv.org/abs/2609.26314"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-26314","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26314","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":10,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":54,"join":"claire_record_id","repository":"gh:logo-cuhksz/trl-bench","source_record":"bm_trl-bench_b244de7b"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"TRL-Bench evaluates tabular encoders at the representation level by exporting row-, column-, or table-level embeddings through each encoder's supported wrapper and probing them with shared lightweight heads across three suites: TRL-CTbench (column/table), TRL-Rbench (row), and TRL-DLTE (compositional Data-Lake Table Enrichment) covering 16 tasks and 20 models.","descriptions":{"en":"TRL-Bench evaluates tabular encoders at the representation level by exporting row-, column-, or table-level embeddings through each encoder's supported wrapper and probing them with shared lightweight heads across three suites: TRL-CTbench (column/table), TRL-Rbench (row), and TRL-DLTE (compositional Data-Lake Table Enrichment) covering 16 tasks and 20 models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09323","languages":[],"library_direction":null,"modality":null,"name":"TRL-Bench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09323","source_url":"https://arxiv.org/abs/2606.09323"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09323","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09323","topics":[],"unit":null},{"aliases":["trust-me-bro-benchmark"],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:netraruntime/trust-me-bro-benchmark","source_record":"bm_trust-me-bro-benchmark_c3200c6b"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A statistical toolkit that compares answer distributions of API endpoints claiming to serve the same model using matched prompts and a two-sample permutation test.","descriptions":{"en":"A statistical toolkit that compares answer distributions of API endpoints claiming to serve the same model using matched prompts and a two-sample permutation test."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:netraruntime/trust-me-bro-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_trust-me-bro-benchmark_c3200c6b"},"modality":null,"name":"Trust Me Bro Benchmark","openness":"unknown","publisher":null,"released":"2026-09-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:netraruntime/trust-me-bro-benchmark","source_url":"https://github.com/NetraRuntime/trust-me-bro-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-netraruntime-trust-me-bro-benchmark","source":"claire_radar","source_url":"https://github.com/NetraRuntime/trust-me-bro-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:skyorca/trustdabench","source_record":"bm_trustdabench_96f4ae98"},"categories":["General AI","Robustness","Safety & Trustworthiness","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM reliability and robustness for structured data analysis using 2,340 human-verified perturbed instances derived from evidence-path perturbations, scoring refusal behavior and resistance to table representation changes.","descriptions":{"en":"Evaluates LLM reliability and robustness for structured data analysis using 2,340 human-verified perturbed instances derived from evidence-path perturbations, scoring refusal behavior and resistance to table representation changes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.24145","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","factuality"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_trustdabench_96f4ae98"},"modality":null,"name":"TrustDABench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24145","source_url":"https://arxiv.org/abs/2608.24145"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24145","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24145","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":951,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":"gh:sylinrl/truthfulqa","source_record":"catalog_76bf377f1a1ee3de"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning","finance","general","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"TruthfulQA is a benchmark to measure whether language models are truthful in generating answers to questions. It comprises 817 questions that span 38 categories, including health, law, finance and politics. The questions are crafted such that some humans would answer falsely due to a false belief or misconception, testing models' ability to avoid generating false answers learned from human texts.","descriptions":{"en":"TruthfulQA is a benchmark to measure whether language models are truthful in generating answers to questions. It comprises 817 questions that span 38 categories, including health, law, finance and politics. The questions are crafted such that some humans would answer falsely due to a false belief or misconception, testing models' ability to avoid generating false answers learned from human texts."},"evidence_summary":{"document_count":1,"model_count":18,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:truthfulqa:mistral-nemo-instruct-2407","reported_at":"2024-07-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/truthfulqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:truthfulqa","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","safety-alignment"],"join":"llm_stats_id","primary":"factuality","source_record":"catalog_76bf377f1a1ee3de"},"modality":"text","name":"TruthfulQA","openness":"unknown","publication":{"evidence":[{"note":"The repository names the paper's authors, links the introducing paper, and documents the generation and multiple-choice tasks plus the January 2025 binary-choice update. Its original benchmark description matches this record.","url":"https://github.com/sylinrl/TruthfulQA/blob/main/README.md"}],"id":"truthfulqa-original","links":[{"kind":"paper","title":"TruthfulQA: Measuring How Models Mimic Human Falsehoods","url":"https://arxiv.org/abs/2109.07958"},{"kind":"repo","title":"TruthfulQA authors' repository","url":"https://github.com/sylinrl/TruthfulQA"}],"record_keys":["llm-stats:truthfulqa"],"reviewed_at":"2026-10-26","scope":{"en":"Original TruthfulQA publication and authors' repository. Generation, MC1, MC2 and the later binary-choice update have distinct evaluation settings; this link review does not equate their scores.","zh":"TruthfulQA 原论文及作者仓库；生成式、MC1、MC2 与后续二选一版本的评测设置不同，本次出处核对不将这些成绩视为同一指标。"}},"publisher":null,"released":"2021-09-08","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: TruthfulQA: Measuring How Models Mimic Human Falsehoods. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:truthfulqa","source_url":"https://arxiv.org/abs/2109.07958"},"repo_kind":null,"repo_resolution_status":null,"score_count":18,"score_direction":"higher_is_better","score_summary":{"display_max":88.0,"display_multiplier":100,"model_count":18,"model_count_basis":"source_model_id","numeric_count":18,"raw_max":0.88,"raw_min":0.503,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:truthfulqa:mai-thinking-1","reported_date":"2026-06-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/truthfulqa?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-30","date_precision":"model_announcement","name":"Command R+"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Instruct"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2025-05-02","date_precision":"model_announcement","name":"IBM Granite 4.0 Tiny Preview"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"},{"date":"2026-06-02","date_precision":"model_announcement","name":"MAI-Thinking-1"},{"date":"2024-07-18","date_precision":"model_announcement","name":"Mistral NeMo Instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 14B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-truthfulqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/truthfulqa?top_n=500","topics":["reasoning-math","finance","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":951,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:2109.07958","repository":"gh:sylinrl/truthfulqa","source_record":"catalog_76bf377f1a1ee3de"},"categories":["知识","Knowledge","大语言模型","LLM","知识储备","事实可靠性","Factual Reliability","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TruthfulQA is  a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. TruthfulQA 用于测量语言模型在回答问题时的真实度。该基准包含 817 个问题，涵盖 38 个类别，包括健康、法律、金融和政治。","descriptions":{"en":"TruthfulQA is  a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics.","zh":"TruthfulQA 用于测量语言模型在回答问题时的真实度。该基准包含 817 个问题，涵盖 38 个类别，包括健康、法律、金融和政治。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1096","languages":[],"library_direction":{"basis":"model-review","categories":["factuality","safety-alignment"],"join":"artifact:arxiv:2109.07958","primary":"factuality","source_record":"catalog_76bf377f1a1ee3de"},"modality":null,"name":"TruthfulQA","openness":"unknown","publisher":"OpenAI","released":"2022-05-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1096-truthfulqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TruthfulQA","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:truthinsight-stack/truthinsightbench","source_record":"bm_truthinsightbench_fec89a32"},"categories":["AI Scientist","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"TruthInsightBench evaluates open-ended scientific discovery agents on 40 blind tasks across 10 scientific domains. Each task presents a neutral research objective with frozen data while source conclusions are withheld. A fixed LLM-based judge scores agents' claims along six dimensions with 29 artifact-grounded items, and automated aggregation produces overall scores. Data and scoring code are publicly available.","descriptions":{"en":"TruthInsightBench evaluates open-ended scientific discovery agents on 40 blind tasks across 10 scientific domains. Each task presents a neutral research objective with frozen data while source conclusions are withheld. A fixed LLM-based judge scores agents' claims along six dimensions with 29 artifact-grounded items, and automated aggregation produces overall scores. Data and scoring code are publicly available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.05079","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_truthinsightbench_fec89a32"},"modality":null,"name":"TruthInsightBench","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05079","source_url":"https://arxiv.org/abs/2609.05079"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05079","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05079","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":10,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ray-zyy/ts-fault","source_record":"bm_ts-fault_c6d5ff11"},"categories":["General AI","Robustness","Safety & Trustworthiness","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates time series forecasting models under four explicit structural fault modes (time-warped shock, dependency-fracture shock, regime-transition missingness, cascading sensor-to-system failure) injected into lookback windows, with paired clean/corrupt protocol and five difficulty levels across nine datasets and six domains.","descriptions":{"en":"Evaluates time series forecasting models under four explicit structural fault modes (time-warped shock, dependency-fracture shock, regime-transition missingness, cascading sensor-to-system failure) injected into lookback windows, with paired clean/corrupt protocol and five difficulty levels across nine datasets and six domains."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.18539","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_ts-fault_c6d5ff11"},"modality":null,"name":"TS-Fault","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18539","source_url":"https://arxiv.org/abs/2606.18539"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18539","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18539","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A controlled benchmark for evaluating analytical skills in time-series question answering: temporal scale selection, temporal localization, and cross-interval integration.","descriptions":{"en":"A controlled benchmark for evaluating analytical skills in time-series question answering: temporal scale selection, temporal localization, and cross-interval integration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.24703","languages":[],"library_direction":{"basis":"model-review-2","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_ts-skill_e0ee25de"},"modality":null,"name":"TS-Skill","openness":"unknown","publisher":null,"released":"2026-05-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.24703","source_url":"https://arxiv.org/abs/2605.24703"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-24703","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24703","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"TSAI-MetaFraud is a multimodal, multi-task benchmark dataset for fraud detection in virtual economies, integrating behavioral, transactional, and graph-structured data. It defines tasks including fraud detection, node classification, temporal link prediction, and weakly supervised fraud detection, with baseline evaluations.","descriptions":{"en":"TSAI-MetaFraud is a multimodal, multi-task benchmark dataset for fraud detection in virtual economies, integrating behavioral, transactional, and graph-structured data. It defines tasks including fraud detection, node classification, temporal link prediction, and weakly supervised fraud detection, with baseline evaluations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09528","languages":[],"library_direction":null,"modality":null,"name":"TSAI-MetaFraud","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09528","source_url":"https://arxiv.org/abs/2607.09528"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09528","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09528","topics":["multimodal","finance"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"TSC-Bench is a benchmark for triple-shot composition, generating establishing, medium, and close-up crops from human-centric images with shot descriptions. It contains 1.2k expert-annotated test cases.","descriptions":{"en":"TSC-Bench is a benchmark for triple-shot composition, generating establishing, medium, and close-up crops from human-centric images with shot descriptions. It contains 1.2k expert-annotated test cases."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05635","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","visual-reasoning"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_tsc-bench_7fff3bb6"},"modality":null,"name":"TSC-Bench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05635","source_url":"https://arxiv.org/abs/2606.05635"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05635","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05635","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"TSM-Bench is a multilingual, multi-generator, multi-task benchmark for evaluating machine-generated text detectors on real-world Wikipedia editing tasks.","descriptions":{"en":"TSM-Bench is a multilingual, multi-generator, multi-task benchmark for evaluating machine-generated text detectors on real-world Wikipedia editing tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.31113","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection","multilingual"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_tsm-bench_b1598d50"},"modality":null,"name":"TSM-Bench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31113","source_url":"https://arxiv.org/abs/2605.31113"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31113","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31113","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"TsuGO is a process-level reasoning benchmark for search efficiency in LLMs using Go life-and-death problems, parsing CoT into search trees and reporting diagnostics.","descriptions":{"en":"TsuGO is a process-level reasoning benchmark for search efficiency in LLMs using Go life-and-death problems, parsing CoT into search trees and reporting diagnostics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.13221","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_tsugo_41aa5b88"},"modality":null,"name":"TsuGO","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13221","source_url":"https://arxiv.org/abs/2608.13221"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13221","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13221","topics":["reasoning-math"],"unit":null},{"aliases":["TSWAP: A Multilingual Retrieval-Augmented Thai Wellness Advisor"],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":37,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_tswap-a-multilingual-retrieval-augmented-t_149c0a74"},"categories":["Biology & Drug Discovery","Factuality","Information retrieval","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"TSWAP Thai Wellness Benchmark is a retrieval-augmented question-answering benchmark for Thai traditional medicine and wellness advice. It includes 50 questions with gold document IDs, evaluated via Recall@5, and also releases production QA logs and a no-retrieval probe.","descriptions":{"en":"TSWAP Thai Wellness Benchmark is a retrieval-augmented question-answering benchmark for Thai traditional medicine and wellness advice. It includes 50 questions with gold document IDs, evaluated via Recall@5, and also releases production QA logs and a no-retrieval probe."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22917","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","deep-research"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_tswap-a-multilingual-retrieval-augmented-t_149c0a74"},"modality":null,"name":"TSWAP Thai Wellness Benchmark","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22917","source_url":"http://arxiv.org/abs/2608.22917v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22917","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22917v1","topics":["safety","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":49,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":48,"join":"claire_record_id","repository":"gh:facebookresearch/tua-bench","source_record":"bm_tua-bench_56e7e977"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates terminal-use agents on 120 real-world tasks across five families (document editing, email, web info, scientific/engineering workflows) using execution-based scoring.","descriptions":{"en":"Evaluates terminal-use agents on 120 real-world tasks across five families (document editing, email, web info, scientific/engineering workflows) using execution-based scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.28480","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","professional-work"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_tua-bench_56e7e977"},"modality":null,"name":"TUA-Bench","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.28480","source_url":"https://arxiv.org/abs/2606.28480"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-28480","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28480","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_tukabench_beb4bae4"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"TUKABENCH is a benchmark for jailbreak evaluation in seven African languages, extending JailbreakBench with human translations, cultural adaptations, and code-switched prompts. It assesses LLM safety in low-resource languages using metrics like Refused, Jailbroken, and Deflection, with human validation of LLM-as-a-judge.","descriptions":{"en":"TUKABENCH is a benchmark for jailbreak evaluation in seven African languages, extending JailbreakBench with human translations, cultural adaptations, and code-switched prompts. It assesses LLM safety in low-resource languages using metrics like Refused, Jailbroken, and Deflection, with human validation of LLM-as-a-judge."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01322","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_tukabench_beb4bae4"},"modality":null,"name":"TukaBench","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01322","source_url":"https://arxiv.org/abs/2606.01322"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01322","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01322","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kadirnar/turkish-llm-benchmark","source_record":"bm_turkish-llm-benchmark_fa8945d1"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates Turkish language quality, instruction following, latency, API cost, local throughput, and speculative decoding across OpenAI-compatible LLM endpoints.","descriptions":{"en":"Evaluates Turkish language quality, instruction following, latency, API cost, local throughput, and speculative decoding across OpenAI-compatible LLM endpoints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:kadirnar/turkish-llm-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_turkish-llm-benchmark_fa8945d1"},"modality":null,"name":"Turkish LLM Benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:kadirnar/turkish-llm-benchmark","source_url":"https://github.com/kadirnar/turkish-llm-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-kadirnar-turkish-llm-benchmark","source":"claire_radar","source_url":"https://github.com/kadirnar/turkish-llm-benchmark","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":51,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_turnbench_084201bf"},"categories":["General AI","Vision & 3D","eess.AS"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates end-of-turn and interruption detection in dyadic human conversation across six interaction styles using a 30-hour hand-labeled corpus and standardized metrics.","descriptions":{"en":"Evaluates end-of-turn and interruption detection in dyadic human conversation across six interaction styles using a 30-hour hand-labeled corpus and standardized metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25218","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_turnbench_084201bf"},"modality":null,"name":"TurnBench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25218","source_url":"https://arxiv.org/abs/2608.25218"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25218","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25218","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code generation","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"TurtleAI is a benchmark of 823 tasks for visual programming in Turtle Graphics, evaluating models on perceiving geometric patterns and synthesizing Python code.","descriptions":{"en":"TurtleAI is a benchmark of 823 tasks for visual programming in Turtle Graphics, evaluating models on perceiving geometric patterns and synthesizing Python code."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03626","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_turtleai_8dfa596d"},"modality":null,"name":"TurtleAI","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03626","source_url":"https://arxiv.org/abs/2606.03626"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03626","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03626","topics":["coding","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_221f38ca3f33898d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"TVBench is a temporal video understanding benchmark evaluating reasoning over actions, events, and temporal dynamics in videos.","descriptions":{"en":"TVBench is a temporal video understanding benchmark evaluating reasoning over actions, events, and temporal dynamics in videos."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tvbench:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tvbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tvbench","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_221f38ca3f33898d"},"modality":"multimodal","name":"TVBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":80.5,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.805,"raw_min":0.772,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tvbench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tvbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-tvbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tvbench?top_n=500","topics":["multimodal"],"unit":null},{"aliases":["Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?"],"attention_signals":null,"categories":["Agents","Agents & Tool Use","Geometric reasoning","Robotics & Embodied AI"],"collected_at":"2026-09-25T04:40:22Z","description":"A simulated indoor benchmark in which foundation models must actively move and turn to reproduce the viewpoint shown in a target image.","descriptions":{"en":"A simulated indoor benchmark in which foundation models must actively move and turn to reproduce the viewpoint shown in a target image."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.01247","languages":[],"library_direction":null,"modality":null,"name":"TVRBench","openness":"unknown","publisher":null,"released":"2026-05-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01247","source_url":"https://arxiv.org/abs/2606.01247"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01247","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01247","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Taiwan benchmark dataset with claimed leaderboard and methodology on an external site.","descriptions":{"en":"Taiwan benchmark dataset with claimed leaderboard and methodology on an external site."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:opentwbench/tw-bench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["math"],"join":"claire_record_id","primary":"math","source_record":"bm_tw-bench_62d8b3a2"},"modality":null,"name":"tw-bench","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:opentwbench/tw-bench","source_url":"https://huggingface.co/datasets/OpenTWBench/tw-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-opentwbench-tw-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/OpenTWBench/tw-bench","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Taiwan exam benchmark dataset with claimed leaderboard and methodology on an external site.","descriptions":{"en":"Taiwan exam benchmark dataset with claimed leaderboard and methodology on an external site."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:opentwbench/tw-exam-bench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","math"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_tw-exam-bench_0b096fe2"},"modality":null,"name":"tw-exam-bench","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:opentwbench/tw-exam-bench","source_url":"https://huggingface.co/datasets/OpenTWBench/tw-exam-bench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-opentwbench-tw-exam-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/OpenTWBench/tw-exam-bench","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:lab-flair/twinicl","source_record":"bm_twinicl_e8dc2c5c"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"A diagnostic benchmark pairing text and image versions of shape tasks to compare in-context learning across modalities.","descriptions":{"en":"A diagnostic benchmark pairing text and image versions of shape tasks to compare in-context learning across modalities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.15028","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","health-medicine"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_twinicl_e8dc2c5c"},"modality":null,"name":"TwinICL","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15028","source_url":"https://arxiv.org/abs/2609.15028"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15028","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15028","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":319,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_b8088edd09d095bd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multilingual question answering benchmark covering 11 typologically diverse languages with 204K question-answer pairs. Questions are written by people seeking genuine information and data is collected directly in each language without translation to test model generalization across diverse linguistic structures.","descriptions":{"en":"A multilingual question answering benchmark covering 11 typologically diverse languages with 204K question-answer pairs. Questions are written by people seeking genuine information and data is collected directly in each language without translation to test model generalization across diverse linguistic structures."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:tydiqa:llama-4-maverick","reported_at":"2025-04-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tydiqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:tydiqa","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","deep-research"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_b8088edd09d095bd"},"modality":"text","name":"TydiQA","openness":"unknown","publisher":null,"released":"2020-03-10","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:tydiqa","source_url":"https://arxiv.org/abs/2003.05002"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":31.7,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.317,"raw_min":0.315,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:tydiqa:llama-4-maverick","reported_date":"2025-04-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tydiqa?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"}],"slug":"llm-stats-tydiqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tydiqa?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":319,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"artifact:arxiv:2003.05002","repository":null,"source_record":"catalog_b8088edd09d095bd"},"categories":["知识","Knowledge","大语言模型","LLM","知识储备","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"TyDi QA is a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology -- the set of linguistic features that each language expresses. TyDi QA 是一个涵盖 11 种不同语言的问题回答数据集，包含 20.4 万个问题-答案对。TyDi QA 的语言种类多样，涵盖了语言学特征的各种类型。","descriptions":{"en":"TyDi QA is a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology -- the set of linguistic features that each language expresses.","zh":"TyDi QA 是一个涵盖 11 种不同语言的问题回答数据集，包含 20.4 万个问题-答案对。TyDi QA 的语言种类多样，涵盖了语言学特征的各种类型。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:508","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"artifact:arxiv:2003.05002","primary":"multilingual","source_record":"catalog_b8088edd09d095bd"},"modality":null,"name":"TyDiQA","openness":"unknown","publisher":null,"released":"2020-03-10","released_reference":{"basis":"paper_first_version","note":"First version introducing TyDi QA.","source_key":"opencompass:508","source_url":"https://arxiv.org/abs/2003.05002"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-508-tydiqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TyDiQA","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"System One models output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes, via a non-generative interface.","descriptions":{"en":"System One models output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes, via a non-generative interface."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11392","languages":[],"library_direction":{"basis":"model-review-daily","categories":["instruction-following"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_typedbench_445bfb57"},"modality":null,"name":"TypedBench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11392","source_url":"https://arxiv.org/abs/2610.11392"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11392","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11392","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["长文本","Long-Context","RAG","Needle In A Haystack","大语言模型","LLM","长上下文","Long Context","检索能力","Retrieval"],"collected_at":"2026-08-18T00:00:00+00:00","description":"U-NIAH is a framework unifying RAG and LLM for needle-in-a-haystack tasks, based on the fictional Starlight Academy dataset. It eliminates interference from pre-trained knowledge and supports diverse, complex scenarios (e.g., multi-needle, long-needle, \"needle-in-needle\"). U-NIAH是将 RAG 和 LLM 统一映射在大海捞针任务中的框架。所有任务基于一个虚构背景下的数据集Starlight Academy，涵盖了魔法系统、学术课程、校园生活、等多个方面，旨在消除预训练知识的干扰，从而能够独立于 LLMs 的先验知识。框架包含多种评估场景，支持多针（3、7、15个针）和长针（400-500 token）配置，还引入了“针中针”结构，进一步增加了复杂性。该数据集通过多样化的场景和合成生成的内容，能够从多个维度分析模型在长文本场景下的性能。同时通过模块化设计，U-NIAH可持续注入新的挑战场景。","descriptions":{"en":"U-NIAH is a framework unifying RAG and LLM for needle-in-a-haystack tasks, based on the fictional Starlight Academy dataset. It eliminates interference from pre-trained knowledge and supports diverse, complex scenarios (e.g., multi-needle, long-needle, \"needle-in-needle\").","zh":"U-NIAH是将 RAG 和 LLM 统一映射在大海捞针任务中的框架。所有任务基于一个虚构背景下的数据集Starlight Academy，涵盖了魔法系统、学术课程、校园生活、等多个方面，旨在消除预训练知识的干扰，从而能够独立于 LLMs 的先验知识。框架包含多种评估场景，支持多针（3、7、15个针）和长针（400-500 token）配置，还引入了“针中针”结构，进一步增加了复杂性。该数据集通过多样化的场景和合成生成的内容，能够从多个维度分析模型在长文本场景下的性能。同时通过模块化设计，U-NIAH可持续注入新的挑战场景。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1742","languages":[],"library_direction":null,"modality":null,"name":"U-NIAH","openness":"unknown","publisher":"同济大学","released":"2025-03-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1742-u-niah","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/U-NIAH","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":206,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_ua-legal-bench_a6ec4166"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"UA-Legal-Bench is a five-task benchmark for evaluating LLMs on Ukrainian legal reasoning using court decisions, covering classification, outcome prediction, and norm extraction.","descriptions":{"en":"UA-Legal-Bench is a five-task benchmark for evaluating LLMs on Ukrainian legal reasoning using court decisions, covering classification, outcome prediction, and norm extraction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.29170","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","multilingual"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_ua-legal-bench_a6ec4166"},"modality":null,"name":"UA-Legal-Bench","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.29170","source_url":"https://arxiv.org/abs/2605.29170"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-29170","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29170","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"UAV-OVO is an out-of-viewpoint generalization benchmark for UAV action recognition. It evaluates models on low-depression viewpoint training data and tests on high-depression viewpoint out-of-distribution data, with class-matched ID/OOD splits.","descriptions":{"en":"UAV-OVO is an out-of-viewpoint generalization benchmark for UAV action recognition. It evaluates models on low-depression viewpoint training data and tests on high-depression viewpoint out-of-distribution data, with class-matched ID/OOD splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.25615","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_uav-ovo_af2dbcd6"},"modality":null,"name":"UAV-OVO","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25615","source_url":"https://arxiv.org/abs/2605.25615"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25615","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25615","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:link-dev/uav3dcrop","source_record":"bm_uav3dcrop_4b4a17ef"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"UAV3DCrop is a benchmark of repeated multi-angle UAV crop surveys with 88,830 images across 91 scenes, evaluating 3D reconstruction methods on appearance, geometry, and canopy height.","descriptions":{"en":"UAV3DCrop is a benchmark of repeated multi-angle UAV crop surveys with 88,830 images across 91 scenes, evaluating 3D reconstruction methods on appearance, geometry, and canopy height."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.06404","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_uav3dcrop_4b4a17ef"},"modality":null,"name":"UAV3DCrop","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.06404","source_url":"https://arxiv.org/abs/2608.06404"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-06404","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06404","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","智能体","Agent","任务执行","Task Execution","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"OSWorld is a first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across operating systems. OSWorld 是一个首创的、可扩展的、真实计算机环境，用于多模态智能体，支持操作系统跨平台的任务设置、基于执行的评估和交互式学习。","descriptions":{"en":"OSWorld is a first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across operating systems.","zh":"OSWorld 是一个首创的、可扩展的、真实计算机环境，用于多模态智能体，支持操作系统跨平台的任务设置、基于执行的评估和交互式学习。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1625","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"artifact:arxiv:2404.07972","primary":"computer-use","source_record":"catalog_29abe33c53bcfbfc"},"modality":"multimodal","name":"ubuntu_osworld","openness":"unknown","publisher":"HKU, etc.","released":"2024-04-11","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1625-ubuntu-osworld","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ubuntu_osworld","topics":["computer-use","agents","multimodal","reasoning-math"],"unit":null},{"aliases":["Learning to Deny: Action Denial in Multimodal Large Language Models"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"A paired-video benchmark evaluating whether multimodal language models can distinguish action presence from action denial under matched contextual and motion cues.","descriptions":{"en":"A paired-video benchmark evaluating whether multimodal language models can distinguish action presence from action denial under matched contextual and motion cues."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.31187","languages":[],"library_direction":null,"modality":null,"name":"UCF101-AD","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31187","source_url":"https://arxiv.org/abs/2606.31187"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31187","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31187","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_uesf-bench_80fcb44c"},"categories":["Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"UESF-Bench evaluates embodied agents on unified language-guided human seeking and following in dynamic environments, covering semantic-guided exploration, behavior switching, and identity grounding across single- and multi-person settings. Scoring uses success metrics for both phases.","descriptions":{"en":"UESF-Bench evaluates embodied agents on unified language-guided human seeking and following in dynamic environments, covering semantic-guided exploration, behavior switching, and identity grounding across single- and multi-person settings. Scoring uses success metrics for both phases."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.13621","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_uesf-bench_80fcb44c"},"modality":null,"name":"UESF-Bench","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.13621","source_url":"https://arxiv.org/abs/2607.13621"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-13621","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13621","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety & Trustworthiness","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Face recognition bias evaluation using public videos of Brazilian politicians with race/color labels, tested across verification and identification.","descriptions":{"en":"Face recognition bias evaluation using public videos of Brazilian politicians with race/color labels, tested across verification and identification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.30688","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_ufpr-pes_1e62a53b"},"modality":null,"name":"UFPR-PEs","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30688","source_url":"https://arxiv.org/abs/2608.30688"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30688","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30688","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["知识","Knowledge","大语言模型","LLM","知识储备","事实可靠性","Factual Reliability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"UHGEval(Unconstrained Hallucination Generation Evaluation) benchmark contains hallucinations generated by LLMs with minimal restrictions. UHGEval 基准，包含由限制条件最小的大语言模型（LLMs）生成的幻觉。","descriptions":{"en":"UHGEval(Unconstrained Hallucination Generation Evaluation) benchmark contains hallucinations generated by LLMs with minimal restrictions.","zh":"UHGEval 基准，包含由限制条件最小的大语言模型（LLMs）生成的幻觉。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1088","languages":[],"library_direction":null,"modality":null,"name":"UHGEval","openness":"open","publisher":"Institute for Advanced Algorithms Research","released":"2024-05-24","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1088-uhgeval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/UHGEval","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"UHI-Bench is a benchmark for dual-source urban heat island (UHI) modeling that integrates dynamic meteorological drivers and static urban morphology. It evaluates over 20 baselines from four model families across five tasks on 20 cities in nine Köppen climate classes, covering land surface temperature UHI and near-surface air temperature UHI.","descriptions":{"en":"UHI-Bench is a benchmark for dual-source urban heat island (UHI) modeling that integrates dynamic meteorological drivers and static urban morphology. It evaluates over 20 baselines from four model families across five tasks on 20 cities in nine Köppen climate classes, covering land surface temperature UHI and near-surface air temperature UHI."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23857","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_uhi-bench_9c52740b"},"modality":null,"name":"UHI-Bench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23857","source_url":"https://arxiv.org/abs/2608.23857"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23857","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23857","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":"gh:chenmancm169/ui2app","source_record":"bm_ui2app_cf415439"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmarks visual interaction inference in executable web application generation. Contains 327 screenshots in 45 sets, evaluating executability, navigation reachability, visual fidelity, and interaction inference via the IIS metric.","descriptions":{"en":"Benchmarks visual interaction inference in executable web application generation. Contains 327 screenshots in 45 sets, evaluating executability, navigation reachability, visual fidelity, and interaction inference via the IIS metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.06306","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_ui2app_cf415439"},"modality":null,"name":"UI2App","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06306","source_url":"https://arxiv.org/abs/2607.06306"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06306","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06306","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Language & Knowledge","cs.IR"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for paragraph-level precedent retrieval in UK case law.","descriptions":{"en":"A benchmark for paragraph-level precedent retrieval in UK case law."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24613","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research","finance-legal"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_uk-prbench_3d9a1283"},"modality":null,"name":"UK-PRBENCH","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24613","source_url":"https://arxiv.org/abs/2609.24613"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24613","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24613","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Ultrasound vision foundation model benchmark; details withheld but no artifacts available.","descriptions":{"en":"Ultrasound vision foundation model benchmark; details withheld but no artifacts available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.28610","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_ultrabench-2_8ea75f5f"},"modality":null,"name":"UltraBench 2","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.28610","source_url":"https://arxiv.org/abs/2609.28610"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-28610","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28610","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zhuqh19/ultrag-bench","source_record":"bm_ultrag-bench_10250348"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language models on pixel-level evidence grounding in ultrasound via segmentation, VQA, and report generation tasks.","descriptions":{"en":"Evaluates vision-language models on pixel-level evidence grounding in ultrasound via segmentation, VQA, and report generation tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.30928","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_ultrag-bench_10250348"},"modality":null,"name":"UltraG-Bench","openness":"unknown","publisher":null,"released":"2026-09-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30928","source_url":"https://arxiv.org/abs/2609.30928"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30928","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30928","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_ultravr_947de071"},"categories":["Biology & Drug Discovery","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"UltraVR is a diagnostic benchmark for evidence-grounded visual reasoning over ultra-resolution images, spanning four domains: CCTV surveillance, remote sensing, whole-slide pathology, and industrial anomaly detection. It includes structured ground-truth chain-of-thought annotations for process-level diagnosis.","descriptions":{"en":"UltraVR is a diagnostic benchmark for evidence-grounded visual reasoning over ultra-resolution images, spanning four domains: CCTV surveillance, remote sensing, whole-slide pathology, and industrial anomaly detection. It includes structured ground-truth chain-of-thought annotations for process-level diagnosis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05576","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_ultravr_947de071"},"modality":null,"name":"UltraVR","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05576","source_url":"https://arxiv.org/abs/2606.05576"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05576","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05576","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_umi-bench_1bf436a3"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"UMI-Bench 1.0 is a benchmark for tabletop robotic manipulation policies using the Universal Manipulation Interface, with a unified protocol for data collection, reset, execution, and logging.","descriptions":{"en":"UMI-Bench 1.0 is a benchmark for tabletop robotic manipulation policies using the Universal Manipulation Interface, with a unified protocol for data collection, reset, execution, and logging."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.10382","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_umi-bench_1bf436a3"},"modality":null,"name":"UMI-Bench","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10382","source_url":"https://arxiv.org/abs/2606.10382"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10382","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10382","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":126,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jellyfish042/uncheatable_eval","source_record":"bm_uncheatable-eval_ae8744f9"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates base language models on newly collected internet data using compression-based negative log likelihood.","descriptions":{"en":"Evaluates base language models on newly collected internet data using compression-based negative log likelihood."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.27510","languages":[],"library_direction":null,"modality":null,"name":"Uncheatable Eval","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27510","source_url":"https://arxiv.org/abs/2609.27510"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27510","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27510","topics":[],"unit":null},{"aliases":["A Benchmark for Vehicle Attribute Classification in Cross-Domain Surveillance Scenarios"],"attention_signals":{"as_of":"2026-10-11","github_stars":6,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ufpr-ipasp-pr/uvib-vehicle-attributes","source_record":"bm_a-benchmark-for-vehicle-attribute-classifi_ff6d38c9"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vehicle attribute classification for front/rear orientation, VMMR suitability, and color clarity using binary annotations across images from seven Brazilian datasets.","descriptions":{"en":"Evaluates vehicle attribute classification for front/rear orientation, VMMR suitability, and color clarity using binary annotations across images from seven Brazilian datasets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.01584","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_a-benchmark-for-vehicle-attribute-classifi_ff6d38c9"},"modality":null,"name":"Unconstrained Vehicle Identification Benchmark","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01584","source_url":"https://arxiv.org/abs/2609.01584"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01584","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01584","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_underspecbench_dd8281f2"},"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"UnderSpecBench evaluates coding agents on DevOps tasks under varying instruction underspecification, measuring action-boundary violations such as wrong-target or over-scope actions.","descriptions":{"en":"UnderSpecBench evaluates coding agents on DevOps tasks under varying instruction underspecification, measuring action-boundary violations such as wrong-target or over-scope actions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.02294","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","safety-alignment"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_underspecbench_dd8281f2"},"modality":null,"name":"UnderSpecBench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.02294","source_url":"https://arxiv.org/abs/2607.02294"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-02294","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02294","topics":["coding","agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":22,"join":"claire_record_id","repository":"gh:tradertanmay/undobench","source_record":"bm_undobench_272ab9d1"},"categories":["Agents & Tool Use","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery.","descriptions":{"en":"Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.05622","languages":[],"library_direction":{"basis":"model-review-daily","categories":["tool-use","agent-security"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_undobench_272ab9d1"},"modality":null,"name":"UndoBench","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05622","source_url":"https://arxiv.org/abs/2610.05622"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05622","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05622","topics":["agents","safety"],"unit":null},{"aliases":["Uni-MMMU","UniMMMU"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Eight tasks that couple understanding and generation, each scored on a text channel and an image channel. One headline number hides which channel earned it, and a model that only understands or only generates is structurally capped on tasks that require both.","descriptions":{"en":"Eight tasks that couple understanding and generation, each scored on a text channel and an image channel. One headline number hides which channel earned it, and a model that only understands or only generates is structurally capped on tasks that require both."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:uni_mmmu","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["image-video-generation"],"join":"review:14H034160212:2026-10-15","primary":"image-video-generation","source_record":null},"modality":null,"name":"Uni-MMMU","openness":"unknown","publisher":null,"released":"2025-10-15","released_reference":{"basis":"paper_first_version","note":"First version of the paper introducing Uni-MMMU, the identifier its repository's citation block gives. The paper's later conference acceptance is not its release.","source_key":"model-reports:uni_mmmu","source_url":"https://arxiv.org/abs/2510.13759"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"uni_mmmu","source":"model_reports","source_url":"https://github.com/Vchitect/Uni-MMMU","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","NeurIPS 2024","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"UniBench is meant for evaluating VLMs' reasoning abilities. It is a unified implementation of 50+ VLM benchmarks spanning a comprehensive range of carefully categorized capabilities from object recognition to spatial awareness, counting, and much more. UniBench旨在评估VLM的推理能力，包括50个基准测试，涵盖对象识别、空间感知、计数等任务。","descriptions":{"en":"UniBench is meant for evaluating VLMs' reasoning abilities. It is a unified implementation of 50+ VLM benchmarks spanning a comprehensive range of carefully categorized capabilities from object recognition to spatial awareness, counting, and much more.","zh":"UniBench旨在评估VLM的推理能力，包括50个基准测试，涵盖对象识别、空间感知、计数等任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1332","languages":["English"],"library_direction":null,"modality":"multimodal","name":"UniBench","openness":"unknown","publisher":"Meta FAIR","released":"2024-08-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1332-unibench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/UniBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["CAD","Geometric reasoning","Manufacturing & Process Control","Science & Engineering"],"collected_at":"2026-10-14T09:44:51Z","description":"UniCAD is a benchmark for multi-modal CAD learning covering point-to-CAD reconstruction, text/image-to-CAD generation, and CAD question answering. It includes a universal model, UniCAD-MLLM, and reports state-of-the-art results on UniCAD and Fusion360 benchmarks.","descriptions":{"en":"UniCAD is a benchmark for multi-modal CAD learning covering point-to-CAD reconstruction, text/image-to-CAD generation, and CAD question answering. It includes a universal model, UniCAD-MLLM, and reports state-of-the-art results on UniCAD and Fusion360 benchmarks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05058","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","spatial-3d"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_unicad_f706b085"},"modality":null,"name":"UniCAD","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05058","source_url":"https://arxiv.org/abs/2606.05058"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05058","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05058","topics":["coding","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":40,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":34,"join":"claire_record_id","repository":"gh:hku-mmlab/uniclawbench","source_record":"bm_uniclawbench_4d04906d"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A capability-driven benchmark for proactive agents in real-world tasks, with 400 bilingual tasks across five capabilities. It evaluates agents in Docker containers using step-by-step checkpoints and a closed-loop strategy with executor, supervisor, and user agents.","descriptions":{"en":"A capability-driven benchmark for proactive agents in real-world tasks, with 400 bilingual tasks across five capabilities. It evaluates agents in Docker containers using step-by-step checkpoints and a closed-loop strategy with executor, supervisor, and user agents."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.08768","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_uniclawbench_4d04906d"},"modality":null,"name":"UniClawBench","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08768","source_url":"https://arxiv.org/abs/2607.08768"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08768","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08768","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":29,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":7,"join":"llm_stats_id","repository":"gh:mjbommar/gpt4-passes-the-bar","source_record":"catalog_5751aeb45604a440"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The Uniform Bar Examination (UBE) benchmark evaluates language models on the complete bar exam including multiple-choice Multistate Bar Examination (MBE), open-ended Multistate Essay Exam (MEE), and Multistate Performance Test (MPT) components. Used to assess legal reasoning capabilities across seven subject areas including Evidence, Torts, Constitutional Law, Contracts, Criminal Law and Procedure, Real Property, and Civil Procedure.","descriptions":{"en":"The Uniform Bar Examination (UBE) benchmark evaluates language models on the complete bar exam including multiple-choice Multistate Bar Examination (MBE), open-ended Multistate Essay Exam (MEE), and Multistate Performance Test (MPT) components. Used to assess legal reasoning capabilities across seven subject areas including Evidence, Torts, Constitutional Law, Contracts, Criminal Law and Procedure, Real Property, and Civil Procedure."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:uniform-bar-exam:gpt-4-0613","reported_at":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/uniform-bar-exam?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:uniform-bar-exam","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal","knowledge-expert-qa"],"join":"llm_stats_id","primary":"finance-legal","source_record":"catalog_5751aeb45604a440"},"modality":"text","name":"Uniform Bar Exam","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":90.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.9,"raw_min":0.9,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:uniform-bar-exam:gpt-4-0613","reported_date":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/uniform-bar-exam?top_n=500"},"unit":null},"scored_models":[{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"}],"slug":"llm-stats-uniform-bar-exam","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/uniform-bar-exam?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["Unified Hallucination Fuzzing for Multimodal Large Language Models"],"attention_signals":null,"categories":["General AI","Multimodal"],"collected_at":"2026-09-25T04:40:22Z","description":"UniHall evaluates multimodal-model hallucinations across object, instruction, and knowledge dimensions, alongside adaptive fuzzing stress tests.","descriptions":{"en":"UniHall evaluates multimodal-model hallucinations across object, instruction, and knowledge dimensions, alongside adaptive fuzzing stress tests."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.07525","languages":[],"library_direction":null,"modality":null,"name":"UniHall","openness":"unknown","publisher":null,"released":"2026-07-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07525","source_url":"https://arxiv.org/abs/2608.07525"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07525","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07525","topics":["multimodal"],"unit":null},{"aliases":["Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs"],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"UniKE evaluates whether textual knowledge edits in unified multimodal models transfer to image generation, using 2,971 subjects covering attribute and relation edits.","descriptions":{"en":"UniKE evaluates whether textual knowledge edits in unified multimodal models transfer to image generation, using 2,971 subjects covering attribute and relation edits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.00477","languages":[],"library_direction":null,"modality":null,"name":"UniKE","openness":"unknown","publisher":null,"released":"2026-05-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00477","source_url":"https://arxiv.org/abs/2606.00477"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00477","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00477","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jerrygao818/uniql","source_record":"bm_uniql_126b9bdb"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"UniQL is a human-verified benchmark for cross-dialect text-to-SQL, aligning 1,534 questions with executable SQL across 16 dialects, with dialect-specific evaluation tracks.","descriptions":{"en":"UniQL is a human-verified benchmark for cross-dialect text-to-SQL, aligning 1,534 questions with executable SQL across 16 dialects, with dialect-specific evaluation tracks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.08018","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","multilingual"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_uniql_126b9bdb"},"modality":null,"name":"UniQL","openness":"unknown","publisher":null,"released":"2026-06-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08018","source_url":"https://arxiv.org/abs/2606.08018"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08018","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08018","topics":["coding","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":15,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:fudancvl/unison","source_record":"bm_unison_ffbd3e3c"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates unified multimodal models on joint understanding and generation across four dimensions: internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement, using 2,169 task samples.","descriptions":{"en":"Evaluates unified multimodal models on joint understanding and generation across four dimensions: internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement, using 2,169 task samples."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.26984","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","visual-reasoning"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_unison_ffbd3e3c"},"modality":null,"name":"Unison","openness":"unknown","publisher":null,"released":"2026-06-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26984","source_url":"https://arxiv.org/abs/2606.26984"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26984","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26984","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"UnpredictaBench evaluates whether language-model samples reproduce specified target distributions across statistical, program-induced, and natural-language random processes.","descriptions":{"en":"UnpredictaBench evaluates whether language-model samples reproduce specified target distributions across statistical, program-induced, and natural-language random processes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.06622","languages":[],"library_direction":null,"modality":null,"name":"UnpredictaBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06622","source_url":"https://arxiv.org/abs/2606.06622"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06622","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06622","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"UOJ-Bench is a benchmark for evaluating LLMs in code generation, hacking, and repair, built from real-world submissions on the Universal Online Judge and evaluated through UOJ's judging infrastructure.","descriptions":{"en":"UOJ-Bench is a benchmark for evaluating LLMs in code generation, hacking, and repair, built from real-world submissions on the Universal Online Judge and evaluated through UOJ's judging infrastructure."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.12864","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_uoj-bench_c2013a7a"},"modality":null,"name":"UOJ-Bench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.12864","source_url":"https://arxiv.org/abs/2606.12864"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-12864","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12864","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Software & AI Compute","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark that evaluates uplift (conditional-average-treatment-effect) estimators across seven dataset families under a fixed, test-isolated, multi-objective protocol, with a public living leaderboard.","descriptions":{"en":"A benchmark that evaluates uplift (conditional-average-treatment-effect) estimators across seven dataset families under a fixed, test-isolated, multi-objective protocol, with a public living leaderboard."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.00915","languages":[],"library_direction":null,"modality":null,"name":"UpliftBench","openness":"unknown","publisher":null,"released":"2026-08-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00915","source_url":"https://arxiv.org/abs/2608.00915"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00915","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00915","topics":[],"unit":null},{"aliases":["TransGaze-Object: Transformer Based Driver Gaze Object Prediction Framework in Real Driving"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A driver gaze dataset with synchronized face and scene images, object bounding boxes, and gaze labels used to evaluate a proposed model.","descriptions":{"en":"A driver gaze dataset with synchronized face and scene images, object bounding boxes, and gaze labels used to evaluate a proposed model."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10139","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","embodied-robotics"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_transgaze-object-transformer-based-driver-_92c2348b"},"modality":null,"name":"Urban Driving-Face Scene Gaze (UD-FSG)","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10139","source_url":"https://arxiv.org/abs/2609.10139"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10139","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10139","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The benchmark is designed to evaluate whether video-large language models (Video-LLMs) can naturally process continuous first-person visual observations like humans, enabling recall, perception, reasoning, and navigation. UrbanVideo-Bench旨在评估视频大型语言模型（Video-LLMs）是否能够像人类一样自然地处理连续的第一人称视觉观察，实现回忆、感知、推理和导航。","descriptions":{"en":"The benchmark is designed to evaluate whether video-large language models (Video-LLMs) can naturally process continuous first-person visual observations like humans, enabling recall, perception, reasoning, and navigation.","zh":"UrbanVideo-Bench旨在评估视频大型语言模型（Video-LLMs）是否能够像人类一样自然地处理连续的第一人称视觉观察，实现回忆、感知、推理和导航。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1636","languages":[],"library_direction":null,"modality":"multimodal","name":"UrbanVideo-Bench","openness":"open","publisher":"THU","released":"2025-03-08","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1636-urbanvideo-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/UrbanVideo-Bench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":5,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:axin1301/urbanwell-benchmark","source_record":"bm_urbanwell_81901465"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"UrbanWell provides a dataset and evaluation protocol for assessing spatio-temporal reasoning in multimodal large language models using satellite and street view imagery across 38 cities, covering environmental, accessibility, urban form, vitality, and subjective perception indicators with tasks in static prediction, forecasting, and trend classification.","descriptions":{"en":"UrbanWell provides a dataset and evaluation protocol for assessing spatio-temporal reasoning in multimodal large language models using satellite and street view imagery across 38 cities, covering environmental, accessibility, urban form, vitality, and subjective perception indicators with tasks in static prediction, forecasting, and trend classification."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.15890","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_urbanwell_81901465"},"modality":null,"name":"UrbanWell","openness":"unknown","publisher":null,"released":"2026-06-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.15890","source_url":"https://arxiv.org/abs/2606.15890"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-15890","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15890","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_urdummlu_7d4881d1"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"UrduMMLU evaluates Urdu language understanding through 26,431 multiple-choice questions across 26 subjects and five domains, sourced from native educational materials. Accuracy under zero-shot and few-shot prompting protocols serves as the primary scoring metric.","descriptions":{"en":"UrduMMLU evaluates Urdu language understanding through 26,431 multiple-choice questions across 26 subjects and five domains, sourced from native educational materials. Accuracy under zero-shot and few-shot prompting protocols serves as the primary scoring metric."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07167","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","knowledge-expert-qa"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_urdummlu_7d4881d1"},"modality":null,"name":"UrduMMLU","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07167","source_url":"https://arxiv.org/abs/2606.07167"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07167","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07167","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eth-sri/matharena","source_record":"catalog_f98a3c865f045fb7"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"USAMO 2026 evaluates models on the six problems from the 2026 United States of America Mathematical Olympiad, requiring rigorous proof-based reasoning. Following the MathArena methodology, proofs are graded against human expert rubrics by dual strong judge models, with the minimum of the two scores taken as final. Maximum score is 42 points.","descriptions":{"en":"USAMO 2026 evaluates models on the six problems from the 2026 United States of America Mathematical Olympiad, requiring rigorous proof-based reasoning. Following the MathArena methodology, proofs are graded against human expert rubrics by dual strong judge models, with the minimum of the two scores taken as final. Maximum score is 42 points."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:usamo-2026:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo-2026?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:usamo-2026","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_f98a3c865f045fb7"},"modality":"text","name":"USAMO 2026","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":36.0,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":36.0,"raw_min":30.24,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:usamo-2026:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo-2026?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-usamo-2026","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo-2026?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":291,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:eth-sri/matharena","source_record":"catalog_2938cd779ea308ef"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The 2025 United States of America Mathematical Olympiad (USAMO) benchmark consists of six challenging mathematical problems requiring rigorous proof-based reasoning. USAMO is the most prestigious high school mathematics competition in the United States, serving as the final round of the American Mathematics Competitions series. This benchmark evaluates models on mathematical problem-solving capabilities beyond simple numerical computation, focusing on formal mathematical reasoning and proof generation.","descriptions":{"en":"The 2025 United States of America Mathematical Olympiad (USAMO) benchmark consists of six challenging mathematical problems requiring rigorous proof-based reasoning. USAMO is the most prestigious high school mathematics competition in the United States, serving as the final round of the American Mathematics Competitions series. This benchmark evaluates models on mathematical problem-solving capabilities beyond simple numerical computation, focusing on formal mathematical reasoning and proof generation."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:usamo25:grok-4","reported_at":"2025-07-09","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo25?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:usamo25","languages":[],"library_direction":{"basis":"model-review","categories":["math"],"join":"llm_stats_id","primary":"math","source_record":"catalog_2938cd779ea308ef"},"modality":"text","name":"USAMO25","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":97.6,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.976,"raw_min":0.375,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:usamo25:claude-mythos-preview","reported_date":"2026-04-07","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo25?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-07","date_precision":"model_announcement","name":"Claude Mythos Preview"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Grok-4 Heavy"}],"slug":"llm-stats-usamo25","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo25?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user.","descriptions":{"en":"Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38043","languages":[],"library_direction":null,"modality":null,"name":"UserProxyBench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38043","source_url":"https://arxiv.org/abs/2609.38043"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38043","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38043","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"UserToolBench evaluates personalized decision making in tool-use LLMs through inference of latent user preferences, clarification need, and user-aligned tool-call trajectories, built from privacy-sanitized interaction traces with 10 user profiles, 36 tool sets, and 1,065 turns.","descriptions":{"en":"UserToolBench evaluates personalized decision making in tool-use LLMs through inference of latent user preferences, clarification need, and user-aligned tool-call trajectories, built from privacy-sanitized interaction traces with 10 user profiles, 36 tool sets, and 1,065 turns."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10042","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","agent-memory"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_usertoolbench_99873053"},"modality":null,"name":"UserToolBench","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10042","source_url":"https://arxiv.org/abs/2608.10042"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10042","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10042","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","Code Agent","SWE-Bench","Automatic test augmentation","智能体","Agent","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"UTBoost generates unit tests using LLMs to augment the test cases for certain instances in SWE-Bench, enabling a more rigorous use of SWE-Bench to evaluate the performance of Code Agents. UTBoost通过LLM生成的单元测试，增强了SWE-Bench中一些instances的测试用例，能严谨的使用SWE-Bench来评估Code Agents的表现。","descriptions":{"en":"UTBoost generates unit tests using LLMs to augment the test cases for certain instances in SWE-Bench, enabling a more rigorous use of SWE-Bench to evaluate the performance of Code Agents.","zh":"UTBoost通过LLM生成的单元测试，增强了SWE-Bench中一些instances的测试用例，能严谨的使用SWE-Bench来评估Code Agents的表现。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2014","languages":[],"library_direction":null,"modality":null,"name":"UTBoost","openness":"unknown","publisher":null,"released":"2025-06-10","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2014-utboost","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/UTBoost","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:peijunallin/utilmem","source_record":"bm_utilmem_d6cec83c"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-term conversational memory systems on evidence utilization through 1,717 instances across five domains, testing reasoning, implicit retrieval, synthesis, and interference resistance.","descriptions":{"en":"Evaluates long-term conversational memory systems on evidence utilization through 1,717 instances across five domains, testing reasoning, implicit retrieval, synthesis, and interference resistance."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.30508","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_utilmem_d6cec83c"},"modality":null,"name":"UTILMEM","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.30508","source_url":"https://arxiv.org/abs/2608.30508"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-30508","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30508","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":18,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:mengze-hong/uxbench","source_record":"bm_uxbench_9ba76902"},"categories":["General AI","Language & Knowledge","Safety"],"collected_at":"2026-10-14T09:44:51Z","description":"UXBench evaluates user experience in AI assistants through three tasks: UX Judge (binary classification of response quality), UX Eval (response generation), and UX Recovery (repairing failed interactions). The dataset contains 7,400 test instances from 70K+ real interaction logs, covering 8 scenarios and 83 domains. Scoring uses accuracy for Judge and GRM-rated quality for generation tasks.","descriptions":{"en":"UXBench evaluates user experience in AI assistants through three tasks: UX Judge (binary classification of response quality), UX Eval (response generation), and UX Recovery (repairing failed interactions). The dataset contains 7,400 test instances from 70K+ real interaction logs, covering 8 scenarios and 83 domains. Scoring uses accuracy for Judge and GRM-rated quality for generation tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09570","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_uxbench_9ba76902"},"modality":null,"name":"UXBench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09570","source_url":"https://arxiv.org/abs/2606.09570"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09570","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09570","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"UXBench evaluates LLM-generated UX critiques through local web fixtures, coverage-gated exploration, and a downstream repair agent, measuring report actionability across seven rubric dimensions.","descriptions":{"en":"UXBench evaluates LLM-generated UX critiques through local web fixtures, coverage-gated exploration, and a downstream repair agent, measuring report actionability across seven rubric dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.16262","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","agentic-coding"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_uxbench_0fa77b90"},"modality":null,"name":"UXBench","openness":"unknown","publisher":null,"released":"2026-06-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.16262","source_url":"https://arxiv.org/abs/2606.16262"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-16262","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16262","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":715,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:penghao-wu/vstar","source_record":"catalog_4c94485e0c21ae6c"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A visual reasoning benchmark evaluating multimodal inference under challenging spatial and grounded tasks.","descriptions":{"en":"A visual reasoning benchmark evaluating multimodal inference under challenging spatial and grounded tasks."},"evidence_summary":{"document_count":1,"model_count":7,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:v-star:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/v-star?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:v-star","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_4c94485e0c21ae6c"},"modality":"multimodal","name":"V*","openness":"unknown","publisher":null,"released":"2023-12-21","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:v-star","source_url":"https://arxiv.org/abs/2312.14135"},"repo_kind":null,"repo_resolution_status":null,"score_count":7,"score_direction":"higher_is_better","score_summary":{"display_max":96.89999999999999,"display_multiplier":100,"model_count":7,"model_count_basis":"source_model_id","numeric_count":7,"raw_max":0.969,"raw_min":0.89,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:v-star:kimi-k2.6","reported_date":"2026-04-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/v-star?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"}],"slug":"llm-stats-v-star","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/v-star?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents & Tool Use","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video in-context learning for multimodal agents across 342 interactive tasks in 37 environments.","descriptions":{"en":"Evaluates video in-context learning for multimodal agents across 342 interactive tasks in 37 environments."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15683","languages":[],"library_direction":null,"modality":null,"name":"V-ICAL Bench","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15683","source_url":"https://arxiv.org/abs/2609.15683"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15683","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15683","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"V-STaR is a spatio-temporal reasoning benchmark for Video-LLMs, evaluating Video-LLM’s spatio-temporal reasoning ability in answering questions explicitly in the context of “when”, “where”, and “what”. V-STaR 是一个针对 Video-LLMs的空间时间推理基准，评估 Video-LLM在“何时”、“何地”和“何物”的上下文中明确回答问题的空间时间推理能力。","descriptions":{"en":"V-STaR is a spatio-temporal reasoning benchmark for Video-LLMs, evaluating Video-LLM’s spatio-temporal reasoning ability in answering questions explicitly in the context of “when”, “where”, and “what”.","zh":"V-STaR 是一个针对 Video-LLMs的空间时间推理基准，评估 Video-LLM在“何时”、“何地”和“何物”的上下文中明确回答问题的空间时间推理能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1654","languages":[],"library_direction":null,"modality":"multimodal","name":"V-STaR","openness":"restricted","publisher":"CV-Group, Queen Mary University of London, NJU, etc.","released":"2025-03-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1654-v-star","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/V-STaR","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_v2v-bench_86c97c1d"},"categories":["General AI","Safety","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"V2V-Bench evaluates video-to-video generation models across 11 dimensions in five categories: temporal alignment, structural fidelity, transformation quality, video quality, and semantic alignment. It pairs source videos with editing tasks and scores models on these dimensions.","descriptions":{"en":"V2V-Bench evaluates video-to-video generation models across 11 dimensions in five categories: temporal alignment, structural fidelity, transformation quality, video quality, and semantic alignment. It pairs source videos with editing tasks and scores models on these dimensions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.05665","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_v2v-bench_86c97c1d"},"modality":null,"name":"V2V-Bench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.05665","source_url":"https://arxiv.org/abs/2606.05665"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-05665","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05665","topics":["safety"],"unit":null},{"aliases":["V\\={a}kQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering"],"attention_signals":null,"categories":["General AI","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Telugu spoken factoid question answering benchmark with 2,001 QA pairs and speech audio.","descriptions":{"en":"Telugu spoken factoid question answering benchmark with 2,001 QA pairs and speech audio."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.19879","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio","knowledge-expert-qa"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_v-a-kqa-a-benchmark-and-evaluation-study-f_76029c55"},"modality":null,"name":"V\\={a}kQA","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19879","source_url":"https://arxiv.org/abs/2609.19879"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19879","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19879","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":59,"github_stars_30d":8,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:alibaba-velldepth/metavideoagent","source_record":"bm_va-evobench_24a7c4ca"},"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates long-form video understanding agents across eight video distributions with separate evolution and held-out splits.","descriptions":{"en":"Evaluates long-form video understanding agents across eight video distributions with separate evolution and held-out splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.04587","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","agent-skills-evolution"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_va-evobench_24a7c4ca"},"modality":null,"name":"VA-EvoBench","openness":"unknown","publisher":null,"released":"2026-08-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.04587","source_url":"https://arxiv.org/abs/2608.04587"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-04587","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04587","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":68,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"claire_record_id","repository":"gh:sharelab-sii/va-judger","source_record":"bm_va-judger-bench_7e8bd378"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VA-Judger-Bench evaluates reward models for joint video-audio generation. It contains paired comparisons of generated video-audio samples with human preference labels, covering in-domain and out-of-domain model outputs.","descriptions":{"en":"VA-Judger-Bench evaluates reward models for joint video-audio generation. It contains paired comparisons of generated video-audio samples with human preference labels, covering in-domain and out-of-domain model outputs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.18607","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","speech-audio"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_va-judger-bench_7e8bd378"},"modality":null,"name":"VA-Judger-Bench","openness":"unknown","publisher":null,"released":"2026-08-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.18607","source_url":"https://arxiv.org/abs/2608.18607"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-18607","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18607","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"Proposed benchmark for evaluating embodied spatial intelligence through visual demonstrations, active perception, and metric control.","descriptions":{"en":"Proposed benchmark for evaluating embodied spatial intelligence through visual demonstrations, active perception, and metric control."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.19554","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_vabench_9eab90fe"},"modality":null,"name":"VABench","openness":"unknown","publisher":null,"released":"2026-09-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.19554","source_url":"https://arxiv.org/abs/2609.19554"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-19554","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19554","topics":[],"unit":null},{"aliases":["VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies"],"attention_signals":{"as_of":"2026-10-13","github_stars":73,"github_stars_30d":6,"hf_dataset_downloads":1864,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ibm/vakra","source_record":"bm_vakra-evaluating-multi-hop-reasoning-acros_7a94433a"},"categories":["General AI","Information retrieval","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multi-hop tool-use agents across live executable APIs and retrieval, scoring via policy adherence, exact match, and groundedness judges.","descriptions":{"en":"Evaluates multi-hop tool-use agents across live executable APIs and retrieval, scoring via policy adherence, exact match, and groundedness judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.12282","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","deep-research"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_vakra-evaluating-multi-hop-reasoning-acros_7a94433a"},"modality":null,"name":"VAKRA","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.12282","source_url":"https://arxiv.org/abs/2608.12282"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-12282","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12282","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A pragmatic competence benchmark for Hindi, Punjabi, Tamil, and Malayalam.","descriptions":{"en":"A pragmatic competence benchmark for Hindi, Punjabi, Tamil, and Malayalam."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01788","languages":[],"library_direction":null,"modality":null,"name":"VakyArth","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01788","source_url":"https://arxiv.org/abs/2609.01788"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01788","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01788","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":18,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:veris-ai/riley-agent","source_record":"bm_vamos-bench_83d46046"},"categories":["Agents","General AI","Speech & Audio"],"collected_at":"2026-10-14T09:44:51Z","description":"VAmoS Bench evaluates complete voice-agent systems in a stateful customer-support task, with 100 scenarios, a simulated caller, real SQL tools, and binary assertions assessed against full interaction traces.","descriptions":{"en":"VAmoS Bench evaluates complete voice-agent systems in a stateful customer-support task, with 100 scenarios, a simulated caller, real SQL tools, and binary assertions assessed against full interaction traces."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.27453","languages":[],"library_direction":{"basis":"model-review","categories":["tool-use","speech-audio"],"join":"claire_record_id","primary":"tool-use","source_record":"bm_vamos-bench_83d46046"},"modality":null,"name":"VAmoS Bench","openness":"unknown","publisher":null,"released":"2026-07-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.27453","source_url":"https://arxiv.org/abs/2607.27453"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-27453","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27453","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Robustness","Safety & Trustworthiness","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"VanillaBench evaluates the clean accuracy gap between adversarially trained models and vanilla (non-robust) reference models across four threat models. It defines a protocol for comparing robustness-accuracy trade-offs using model accuracy on standard benchmarks.","descriptions":{"en":"VanillaBench evaluates the clean accuracy gap between adversarially trained models and vanilla (non-robust) reference models across four threat models. It defines a protocol for comparing robustness-accuracy trade-offs using model accuracy on standard benchmarks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.12545","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_vanillabench_f1c1ed7c"},"modality":null,"name":"VanillaBench","openness":"unknown","publisher":null,"released":"2026-07-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.12545","source_url":"https://arxiv.org/abs/2607.12545"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-12545","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12545","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":2210,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:clemson-capstone/vantage-bench","source_record":"bm_vantage-bench_f8c64682"},"categories":["Logistics & Operations","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates vision-language models on infrastructure camera video across logistics, transportation, and smart spaces using eight spatio-temporal tasks.","descriptions":{"en":"Evaluates vision-language models on infrastructure camera video across logistics, transportation, and smart spaces using eight spatio-temporal tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09396","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","spatial-3d"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_vantage-bench_f8c64682"},"modality":null,"name":"VANTAGE-BENCH","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09396","source_url":"https://arxiv.org/abs/2609.09396"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09396","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09396","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Speech & Audio"],"collected_at":"2026-10-14T09:44:51Z","description":"VARM-Bench evaluates verifiable structured reasoning in Chinese abusive-speech moderation. It uses field-anchored chain-of-thought rationales with six decision fields, and a deterministic protocol assessing field correctness, alignment, output validity, and record errors.","descriptions":{"en":"VARM-Bench evaluates verifiable structured reasoning in Chinese abusive-speech moderation. It uses field-anchored chain-of-thought rationales with six decision fields, and a deterministic protocol assessing field correctness, alignment, output validity, and record errors."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.15600","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","multilingual"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_varm-bench_2f35ad2f"},"modality":null,"name":"VARM-Bench","openness":"unknown","publisher":null,"released":"2026-08-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15600","source_url":"https://arxiv.org/abs/2608.15600"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15600","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15600","topics":["reasoning-math","language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:drivelineresearch/vast-benchmarking","source_record":"bm_vast-benchmarking_a1aad559"},"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Measures GPU throughput, effective CPU concurrency, memory bandwidth, and disk speed of rented Vast.ai containers under a fixed wall-clock budget.","descriptions":{"en":"Measures GPU throughput, effective CPU concurrency, memory bandwidth, and disk speed of rented Vast.ai containers under a fixed wall-clock budget."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:drivelineresearch/vast-benchmarking","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-memory"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_vast-benchmarking_a1aad559"},"modality":null,"name":"Vast Benchmarking","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:drivelineresearch/vast-benchmarking","source_url":"https://github.com/drivelineresearch/vast-benchmarking"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-drivelineresearch-vast-benchmarking","source":"claire_radar","source_url":"https://github.com/drivelineresearch/vast-benchmarking","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_e0956798955b98b1"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","language","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research. Contains over 41,250 videos and 825,000 captions in both English and Chinese, with over 206,000 English-Chinese parallel translation pairs. Supports multilingual video captioning and video-guided machine translation tasks.","descriptions":{"en":"VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research. Contains over 41,250 videos and 825,000 captions in both English and Chinese, with over 206,000 English-Chinese parallel translation pairs. Supports multilingual video captioning and video-guided machine translation tasks."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vatex:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vatex?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vatex","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","multilingual"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_e0956798955b98b1"},"modality":"multimodal","name":"VATEX","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":77.8,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.778,"raw_min":0.778,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vatex:nova-lite","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vatex?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"}],"slug":"llm-stats-vatex","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vatex?top_n=500","topics":["multimodal","language"],"unit":null},{"aliases":["VBench"],"attention_signals":{"as_of":"2026-10-13","github_stars":1818,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"artifact:gh:vchitect/vbench","repository":"gh:vchitect/vbench","source_record":"catalog_4e6fb22dbcf81150"},"categories":["multimodal"],"collected_at":null,"description":"Sixteen dimensions folded into a Quality Score and a Semantic Score and then into one Total Score. Every dimension is min-max normalized across the models on the leaderboard at the time, so a total states a model's standing within that pool rather than an absolute rate, and two totals read from different leaderboard revisions are not the same measurement.","descriptions":{"en":"Sixteen dimensions folded into a Quality Score and a Semantic Score and then into one Total Score. Every dimension is min-max normalized across the models on the leaderboard at the time, so a total states a model's standing within that pool rather than an absolute rate, and two totals read from different leaderboard revisions are not the same measurement."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":true,"key":"model-reports:vbench","languages":[],"library_direction":{"basis":"anchor-review","categories":["image-video-generation"],"join":"artifact:gh:vchitect/vbench","primary":"image-video-generation","source_record":"catalog_4e6fb22dbcf81150"},"modality":null,"name":"VBench","openness":"unknown","publisher":"Nanyang Technological University","released":"2023-11-29","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"vbench","source":"model_reports","source_url":"https://github.com/Vchitect/VBench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1818,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":9,"join":"artifact:arxiv:2311.17982","repository":"gh:vchitect/vbench","source_record":"catalog_4e6fb22dbcf81150"},"categories":["多模态","Multimodal","创作","Creation","多模态模型","VLM","视觉生成","Visual Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VBench is a comprehensive benchmark evaluates video generation quality. It comprises 16 dimensions in video generation, and also provides a dataset of human preference annotations. VBench用于评估多模态大模型的视频生成质量，包含16个视频生成维度及1个人类偏好注释数据集。","descriptions":{"en":"VBench is a comprehensive benchmark evaluates video generation quality. It comprises 16 dimensions in video generation, and also provides a dataset of human preference annotations.","zh":"VBench用于评估多模态大模型的视频生成质量，包含16个视频生成维度及1个人类偏好注释数据集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1375","languages":[],"library_direction":{"basis":"anchor-review","categories":["image-video-generation"],"join":"artifact:arxiv:2311.17982","primary":"image-video-generation","source_record":"catalog_4e6fb22dbcf81150"},"modality":"multimodal","name":"VBench","openness":"unknown","publisher":"Nanyang Technological University","released":"2023-11-29","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1375-vbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VBench","topics":["multimodal"],"unit":null},{"aliases":["VBench++","VBench-plus-plus"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Extends VBench to image-to-video generation and adds trustworthiness dimensions, so its totals are computed over a different dimension set. A VBench++ number is not a VBench number for the same model.","descriptions":{"en":"Extends VBench to image-to-video generation and adds trustworthiness dimensions, so its totals are computed over a different dimension set. A VBench++ number is not a VBench number for the same model."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:vbench_plus_plus","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["image-video-generation"],"join":"review:14H034160212:2026-10-15","primary":"image-video-generation","source_record":null},"modality":null,"name":"VBench++","openness":"unknown","publisher":null,"released":"2024-11-20","released_reference":{"basis":"paper_first_version","note":"First version of the paper introducing VBench++. It extends VBench to image-to-video generation over a different dimension set, so this dates a distinct instrument rather than the 2023 VBench release.","source_key":"model-reports:vbench_plus_plus","source_url":"https://arxiv.org/abs/2411.13503"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"vbench_plus_plus","source":"model_reports","source_url":"https://arxiv.org/abs/2411.13503","topics":["multimodal"],"unit":null},{"aliases":["VBench-2.0","VBench 2.0","VBench2.0"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Targets intrinsic faithfulness -- physics, commonsense and anatomy -- rather than the surface quality VBench 1.x measures, and is built to stay unsaturated where that suite tops out. Low scores here do not contradict high VBench scores; the two suites ask different questions.","descriptions":{"en":"Targets intrinsic faithfulness -- physics, commonsense and anatomy -- rather than the surface quality VBench 1.x measures, and is built to stay unsaturated where that suite tops out. Low scores here do not contradict high VBench scores; the two suites ask different questions."},"evidence_summary":{"document_count":null,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:vbench_2_0","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["image-video-generation"],"join":"review:14H034160212:2026-10-15","primary":"image-video-generation","source_record":null},"modality":null,"name":"VBench-2.0","openness":"unknown","publisher":null,"released":"2025-03-27","released_reference":{"basis":"paper_first_version","note":"First version of the paper introducing VBench-2.0. It measures intrinsic faithfulness where VBench 1.x measures surface quality, so it is its own release and not a revision of the earlier suite.","source_key":"model-reports:vbench_2_0","source_url":"https://arxiv.org/abs/2503.21755"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"vbench_2_0","source":"model_reports","source_url":"https://arxiv.org/abs/2503.21755","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":37,"github_stars_30d":7,"hf_dataset_downloads":null,"hf_paper_upvotes":150,"join":"claire_record_id","repository":"gh:video-reason/vbvr-pro","source_record":"bm_vbvr-pro_41ab73db"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Provides 300 procedurally generated tasks for native visual reasoning through generation, with verifiable reward scorers and controlled modality comparisons.","descriptions":{"en":"Provides 300 procedurally generated tasks for native visual reasoning through generation, with verifiable reward scorers and controlled modality comparisons."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.26105","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","image-video-generation"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_vbvr-pro_41ab73db"},"modality":null,"name":"VBVR-Pro","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26105","source_url":"https://arxiv.org/abs/2608.26105"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26105","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26105","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates complex instruction following for video understanding across content, format, style, and structure constraints with 306 test instructions.","descriptions":{"en":"Evaluates complex instruction following for video understanding across content, format, style, and structure constraints with 306 test instructions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.04588","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","video-understanding"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_vcifbench_33b2eb08"},"modality":null,"name":"VCIFBench","openness":"unknown","publisher":null,"released":"2026-06-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.04588","source_url":"https://arxiv.org/abs/2606.04588"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-04588","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04588","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Spatial reasoning is fundamental to embodied agents, yet it remains unclear whether spatial understanding can be carried forward to guide sequential interactions.","descriptions":{"en":"Spatial reasoning is fundamental to embodied agents, yet it remains unclear whether spatial understanding can be carried forward to guide sequential interactions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.34687","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","embodied-robotics"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_vcn-bench_14b584a1"},"modality":null,"name":"VCN-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.34687","source_url":"https://arxiv.org/abs/2609.34687"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-34687","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34687","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_76bd7780e0e532c3"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Visual Commonsense Reasoning (VCR) benchmark that tests higher-order cognition and commonsense reasoning beyond simple object recognition. Models must answer challenging questions about images and provide rationales justifying their answers. The benchmark measures the ability to infer people's actions, goals, and mental states from visual context.","descriptions":{"en":"Visual Commonsense Reasoning (VCR) benchmark that tests higher-order cognition and commonsense reasoning beyond simple object recognition. Models must answer challenging questions about images and provide rationales justifying their answers. The benchmark measures the ability to infer people's actions, goals, and mental states from visual context."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vcr-en-easy:qwen2-vl-72b","reported_at":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vcr-en-easy?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vcr-en-easy","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_76bd7780e0e532c3"},"modality":"multimodal","name":"VCR_en_easy","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":91.93,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.9193,"raw_min":0.9193,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vcr-en-easy:qwen2-vl-72b","reported_date":"2024-08-29","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vcr-en-easy?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-29","date_precision":"model_announcement","name":"Qwen2-VL-72B-Instruct"}],"slug":"llm-stats-vcr-en-easy","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vcr-en-easy?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Vector-Bench evaluates instruction-based SVG code editing with 40 repair tasks, deterministic specification rewards, and metrics including repair progress and unintended change rate, across 34 model endpoints.","descriptions":{"en":"Vector-Bench evaluates instruction-based SVG code editing with 40 repair tasks, deterministic specification rewards, and metrics including repair progress and unintended change rate, across 34 model endpoints."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.19056","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","image-video-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_vector-bench_6366f6e3"},"modality":null,"name":"Vector-Bench","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.19056","source_url":"https://arxiv.org/abs/2607.19056"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-19056","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19056","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":232,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_vehbench_bf410800"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"VEHBench is a diagnostic benchmark for LLM-assisted vibration energy harvester design, featuring 763 tasks across four design roles: specification triage, verifier-guided search, corrupted-state recovery, and policy-conditioned selection. It uses an analytical physical oracle for scoring.","descriptions":{"en":"VEHBench is a diagnostic benchmark for LLM-assisted vibration energy harvester design, featuring 763 tasks across four design roles: specification triage, verifier-guided search, corrupted-state recovery, and policy-conditioned selection. It uses an analytical physical oracle for scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.18181","languages":[],"library_direction":{"basis":"model-review","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_vehbench_bf410800"},"modality":null,"name":"VEHBench","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18181","source_url":"https://arxiv.org/abs/2607.18181"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18181","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18181","topics":["science"],"unit":null},{"aliases":["Vending Bench","Vending Bench 2"],"attention_signals":null,"categories":["agent"],"collected_at":null,"description":"Reported in simulated dollars, not a percentage, and run over long horizons where a single early failure dominates the total.","descriptions":{"en":"Reported in simulated dollars, not a percentage, and run over long horizons where a single early failure dominates the total."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000vending_bench\u0000zai_glm_5_model_card\u0000vending_bench_2\u0000runs conducted independently by Andon Labs\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"first_score_reported_at":"2026-02-11","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000vending_bench\u0000zai_glm_5_model_card\u0000vending_bench_2\u0000runs conducted independently by Andon Labs\u0000GLM-5","reported_at":"2026-02-11","source_url":"https://huggingface.co/zai-org/GLM-5"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:vending_bench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["multi-agent-games"],"join":"review:14H034160212:2026-10-15","primary":"multi-agent-games","source_record":null},"modality":null,"name":"Vending Bench","openness":"unknown","publisher":null,"released":"2025-02-18","released_reference":{"basis":"benchmark_release","source_key":"model-reports:vending_bench","source_url":"https://andonlabs.com/evals/vending-bench-2"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":5634.41,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":5634.41,"raw_min":4432.12,"source_reference":{"obs_id":"curated\u0000vending_bench\u0000zai_glm_5_1_model_card\u0000vending_bench_2\u0000as reported\u0000GLM-5.1","observation_id":"curated\u0000vending_bench\u0000zai_glm_5_1_model_card\u0000vending_bench_2\u0000as reported\u0000GLM-5.1","reported_at":"2026-04-03","reported_date":"2026-04-03","source_id":"zai_glm_5_1_model_card","source_url":"https://huggingface.co/zai-org/GLM-5.1"},"unit":"usd"},"scored_models":[{"date":"2026-02-11","date_precision":"document_publication","name":"GLM-5"},{"date":"2026-04-03","date_precision":"document_publication","name":"GLM-5.1"}],"slug":"vending_bench","source":"model_reports","source_url":"https://andonlabs.com/evals/vending-bench-2","topics":["agents"],"unit":"usd"},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Vending-Bench 2 tests longer horizon planning capabilities by evaluating how well AI models can manage a simulated vending machine business over extended periods. The benchmark measures a model's ability to maintain consistent tool usage and decision-making for a full simulated year of operation, driving higher returns without drifting off task.","descriptions":{"en":"Vending-Bench 2 tests longer horizon planning capabilities by evaluating how well AI models can manage a simulated vending machine business over extended periods. The benchmark measures a model's ability to maintain consistent tool usage and decision-making for a full simulated year of operation, driving higher returns without drifting off task."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vending-bench-2:gemini-3-pro-preview","reported_at":"2025-11-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vending-bench-2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vending-bench-2","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","professional-work"],"join":"llm_stats_id","primary":"multi-agent-games","source_record":"catalog_33a7ccc91a5843b6"},"modality":"text","name":"Vending-Bench 2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":8017.59,"display_multiplier":1,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":8017.59,"raw_min":3635.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vending-bench-2:claude-opus-4-6","reported_date":"2026-02-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vending-bench-2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1"}],"slug":"llm-stats-vending-bench-2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vending-bench-2?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sharayu-20/vendorbench-100","source_record":"bm_vendorbench-100_8cfd26ec"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates deepfake image detectors across three paradigms—commercial APIs, vision LLMs, and open-source detectors—on a fixed 100-image adversarial corpus. Uses a unified output schema and scores primarily by Matthews correlation coefficient with ROC-AUC.","descriptions":{"en":"Evaluates deepfake image detectors across three paradigms—commercial APIs, vision LLMs, and open-source detectors—on a fixed 100-image adversarial corpus. Uses a unified output schema and scores primarily by Matthews correlation coefficient with ROC-AUC."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.06254","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_vendorbench-100_8cfd26ec"},"modality":null,"name":"VendorBench-100","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06254","source_url":"https://arxiv.org/abs/2607.06254"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06254","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06254","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Materials & Chemistry","q-bio.QM"],"collected_at":"2026-10-14T09:44:51Z","description":"Existing reaction models primarily learn molecular transformations, whereas enzy- matic reactions depend jointly on molecular structure and catalytic function.","descriptions":{"en":"Existing reaction models primarily learn molecular transformations, whereas enzy- matic reactions depend jointly on molecular structure and catalytic function."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11694","languages":[],"library_direction":{"basis":"model-review-daily","categories":["science"],"join":"claire_record_id","primary":"science","source_record":"bm_venusrx-bench_13ac544c"},"modality":null,"name":"VenusRX-Bench","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11694","source_url":"https://arxiv.org/abs/2610.11694"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11694","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11694","topics":["science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Safety","Safety & Trustworthiness","Software & AI Compute"],"collected_at":"2026-09-25T04:40:22Z","description":"Vera-Bench is an executable safety benchmark for tool-using agents, with deterministic, artifact-grounded verification across diverse risks and execution settings.","descriptions":{"en":"Vera-Bench is an executable safety benchmark for tool-using agents, with deterministic, artifact-grounded verification across diverse risks and execution settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.01793","languages":[],"library_direction":null,"modality":null,"name":"Vera-Bench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01793","source_url":"https://arxiv.org/abs/2607.01793"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01793","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01793","topics":["agents","safety"],"unit":null},{"aliases":["VeriBugBench: An Empirically Grounded Framework for Constructing Verilog RTL Debugging Benchmarks"],"attention_signals":null,"categories":["Chip Design & EDA","Language & Knowledge","cs.AR"],"collected_at":"2026-10-14T09:44:51Z","description":"VeriBugBench is a framework for constructing Verilog RTL debugging benchmarks using empirically grounded fault construction, LLM-based testbench enhancement, and execution-based retention.","descriptions":{"en":"VeriBugBench is a framework for constructing Verilog RTL debugging benchmarks using empirically grounded fault construction, LLM-based testbench enhancement, and execution-based retention."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.18022","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["engineering-design"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_veribugbench-an-empirically-grounded-frame_9ddf8887"},"modality":null,"name":"VeriBugBench","openness":"unknown","publisher":null,"released":"2026-09-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.18022","source_url":"https://arxiv.org/abs/2609.18022"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-18022","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18022","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code & Software","Code generation","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable guarantees.","descriptions":{"en":"Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable guarantees."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.39568","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_vericodebench_ab5b6c1e"},"modality":null,"name":"VeriCodeBench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.39568","source_url":"https://arxiv.org/abs/2609.39568"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-39568","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39568","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zy001122/beyond-blind-compliance","source_record":"bm_veriocrbench_5a8d8252"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates OCR-grounded task verification with 1,800 human-verified samples including 1,600 trap-injected invalid tasks and 200 controls, scored via TVR, DA, and ORR.","descriptions":{"en":"Evaluates OCR-grounded task verification with 1,800 human-verified samples including 1,600 trap-injected invalid tasks and 200 controls, scored via TVR, DA, and ORR."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.00232","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_veriocrbench_5a8d8252"},"modality":null,"name":"VeriOCRBench","openness":"unknown","publisher":null,"released":"2026-08-31","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00232","source_url":"https://arxiv.org/abs/2609.00232"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00232","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00232","topics":["reasoning-math"],"unit":null},{"aliases":["To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech"],"attention_signals":null,"categories":["General AI","Information retrieval","Speech & Audio","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A probe benchmark for speech-based fact verification in Large Audio Language Models, derived from KVQA and Wikidata, with synthetic false claims and veracity labels.","descriptions":{"en":"A probe benchmark for speech-based fact verification in Large Audio Language Models, derived from KVQA and Wikidata, with synthetic false claims and veracity labels."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.30227","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio","visual-reasoning"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_to-trust-or-not-to-trust-retrieval-augment_4d69581d"},"modality":null,"name":"VeriSpeak","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.30227","source_url":"https://arxiv.org/abs/2609.30227"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-30227","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30227","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Planning","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"VeriTrip benchmarks travel planning agents on evidence-grounded reasoning over unstructured web corpora, with a verifiable knowledge base for cell-wise verification of factual reliability.","descriptions":{"en":"VeriTrip benchmarks travel planning agents on evidence-grounded reasoning over unstructured web corpora, with a verifiable knowledge base for cell-wise verification of factual reliability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.28683","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","tool-use"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_veritrip_b9f0bf63"},"modality":null,"name":"VeriTrip","openness":"unknown","publisher":null,"released":"2026-05-27","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.28683","source_url":"https://arxiv.org/abs/2605.28683"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-28683","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28683","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":108,"github_stars_30d":18,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:sunblaze-ucb/vero","source_record":"bm_vero_3ed8d572"},"categories":["Language & Knowledge","Software & AI Compute","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Vero evaluates repository-level verified code generation in Lean 4, with 43 multi-module instances, formal specifications, and proof-only and code-and-proof evaluation modes.","descriptions":{"en":"Vero evaluates repository-level verified code generation in Lean 4, with 43 multi-module instances, formal specifications, and proof-only and code-and-proof evaluation modes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.13522","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","math"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_vero_3ed8d572"},"modality":null,"name":"Vero","openness":"unknown","publisher":null,"released":"2026-08-13","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.13522","source_url":"https://arxiv.org/abs/2608.13522"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-13522","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13522","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"VertiCue-Bench evaluates multimodal large language models on geospatial reasoning using canopy height models to resolve 2D ambiguity in remote sensing natural scenes. It includes 1,534 instances across 17 tasks, testing height perception and semantic reasoning.","descriptions":{"en":"VertiCue-Bench evaluates multimodal large language models on geospatial reasoning using canopy height models to resolve 2D ambiguity in remote sensing natural scenes. It includes 1,534 instances across 17 tasks, testing height perception and semantic reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.25784","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","visual-reasoning"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_verticue-bench_6f5c5a90"},"modality":null,"name":"VertiCue-Bench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25784","source_url":"https://arxiv.org/abs/2605.25784"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25784","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25784","topics":["reasoning-math"],"unit":null},{"aliases":["Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization"],"attention_signals":null,"categories":["Code & Software","General AI","cs.SE"],"collected_at":"2026-09-25T04:40:22Z","description":"Verus-SpecBench has 581 tasks in which agents write formal Verus specifications for Codeforces problems. Specs are executed and tested against official tests and hack cases.","descriptions":{"en":"Verus-SpecBench has 581 tasks in which agents write formal Verus specifications for Codeforces problems. Specs are executed and tested against official tests and hack cases."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26457","languages":[],"library_direction":null,"modality":null,"name":"Verus-SpecBench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26457","source_url":"https://arxiv.org/abs/2605.26457"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26457","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26457","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Tests long-horizon video understanding with 600 Temporal Ordering and Event Counting questions, auditing whether decoded frames cover all evidence intervals.","descriptions":{"en":"Tests long-horizon video understanding with 600 Temporal Ordering and Event Counting questions, auditing whether decoded frames cover all evidence intervals."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22516","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_ves-bench_b925ca24"},"modality":null,"name":"VES-Bench","openness":"unknown","publisher":null,"released":"2026-08-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22516","source_url":"http://arxiv.org/abs/2608.22516v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22516","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22516v1","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":11,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:steven1518/vex-bench","source_record":"bm_vex-bench_83da0ca3"},"categories":["Language & Knowledge","Logistics & Operations","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"Consists of 75 real-world software supply chain vulnerability cases across Python, Java, and Go for evaluating LLM agents on exploitability assessment tasks.","descriptions":{"en":"Consists of 75 real-world software supply chain vulnerability cases across Python, Java, and Go for evaluating LLM agents on exploitability assessment tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.08040","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","agentic-coding"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_vex-bench_83da0ca3"},"modality":null,"name":"VEX-Bench","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08040","source_url":"https://arxiv.org/abs/2609.08040"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08040","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08040","topics":["coding","agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":6,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:hanxunh/vex-bench","source_record":"bm_vex-bench_b5a814bc"},"categories":["Language & Knowledge","Software & AI Compute","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"A unified benchmark for evaluating verification complexity of LLM-generated misinformation, covering multiple dimensions and using LLM-as-judge validated by content analysis.","descriptions":{"en":"A unified benchmark for evaluating verification complexity of LLM-generated misinformation, covering multiple dimensions and using LLM-as-judge validated by content analysis."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.35028","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","factuality"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_vex-bench_b5a814bc"},"modality":null,"name":"VEX-Bench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35028","source_url":"https://arxiv.org/abs/2609.35028"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35028","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35028","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":20,"github_stars_30d":-1,"hf_dataset_downloads":null,"hf_paper_upvotes":25,"join":"claire_record_id","repository":"gh:vg-gui-tasker/vg-gui-tasker","source_record":"bm_vg-guibench_23eefff5"},"categories":["Agents & Tool Use","General AI","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VG-GUI-Bench evaluates MLLM-based GUI agents on following video tutorials to complete interactive tasks, with 1,000 long-horizon test cases and four metrics.","descriptions":{"en":"VG-GUI-Bench evaluates MLLM-based GUI agents on following video tutorials to complete interactive tasks, with 1,000 long-horizon test cases and four metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.29445","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","video-understanding"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_vg-guibench_23eefff5"},"modality":null,"name":"VG-GUIBench","openness":"unknown","publisher":null,"released":"2026-06-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.29445","source_url":"https://arxiv.org/abs/2606.29445"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-29445","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29445","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":110,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_vga-benchv2_d6dfef90"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video generation quality and aesthetic value with 1,016 prompts, 60,000 videos, and 36,000 task-level annotations.","descriptions":{"en":"Evaluates video generation quality and aesthetic value with 1,016 prompts, 60,000 videos, and 36,000 task-level annotations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.25452","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_vga-benchv2_d6dfef90"},"modality":null,"name":"VGA-BenchV2","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25452","source_url":"https://arxiv.org/abs/2608.25452"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25452","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25452","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":14,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":22,"join":"claire_record_id","repository":"gh:zinosii/vgenst-bench","source_record":"bm_vgenst-bench_f8fb9973"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"VGenST-Bench evaluates spatio-temporal reasoning in multimodal large language models using 1,200 procedurally generated videos with controlled scene composition, camera trajectory, and reasoning targets. It covers 12 reasoning tasks across three spatial scales and 12 QA types across three reasoning levels, with multiple choice and open-ended question variants.","descriptions":{"en":"VGenST-Bench evaluates spatio-temporal reasoning in multimodal large language models using 1,200 procedurally generated videos with controlled scene composition, camera trajectory, and reasoning targets. It covers 12 reasoning tasks across three spatial scales and 12 QA types across three reasoning levels, with multiple choice and open-ended question variants."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.22570","languages":[],"library_direction":{"basis":"model-review","categories":["spatial-3d","video-understanding"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_vgenst-bench_f8fb9973"},"modality":null,"name":"VGenST-Bench","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22570","source_url":"https://arxiv.org/abs/2605.22570"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22570","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22570","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":14,"github_stars_30d":2,"hf_dataset_downloads":null,"hf_paper_upvotes":332,"join":"claire_record_id","repository":null,"source_record":"bm_vgi-bench_4a5d68ac"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Contains 27 tasks and 810 instances organized by task domains and skill tags to evaluate visual reasoning capabilities of video generation models.","descriptions":{"en":"Contains 27 tasks and 810 instances organized by task domains and skill tags to evaluate visual reasoning capabilities of video generation models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.19583","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","visual-reasoning"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_vgi-bench_4a5d68ac"},"modality":null,"name":"VGI-BENCH","openness":"unknown","publisher":null,"released":"2026-08-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.19583","source_url":"https://arxiv.org/abs/2608.19583"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-19583","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19583","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Code & Software","Software & AI Compute","cs.PL"],"collected_at":"2026-10-14T09:44:51Z","description":"Large Language Models (LLMs) are increasingly applied in hardware design automation, demonstrating strong potential in generating and understanding hardware description languages.","descriptions":{"en":"Large Language Models (LLMs) are increasingly applied in hardware design automation, demonstrating strong potential in generating and understanding hardware description languages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.05380","languages":[],"library_direction":{"basis":"model-review-daily","categories":["engineering-design","code-generation"],"join":"claire_record_id","primary":"engineering-design","source_record":"bm_vhdl-repobench_0a758628"},"modality":null,"name":"VHDL-REPOBENCH","openness":"unknown","publisher":null,"released":"2026-10-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.05380","source_url":"https://arxiv.org/abs/2610.05380"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-05380","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05380","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Chip Design & EDA","Language & Knowledge","cs.AR"],"collected_at":"2026-10-14T09:44:51Z","description":"VHDLSuite is a benchmark-centered infrastructure for VHDL generation evaluation, integrating automated benchmark synthesis, executable validation, and multi-model diagnostic analysis. It includes VHDLBench with over 200 VHDL problems with validated testbenches.","descriptions":{"en":"VHDLSuite is a benchmark-centered infrastructure for VHDL generation evaluation, integrating automated benchmark synthesis, executable validation, and multi-model diagnostic analysis. It includes VHDLBench with over 200 VHDL problems with validated testbenches."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13735","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_vhdlbench_8f92d226"},"modality":null,"name":"VHDLBench","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13735","source_url":"https://arxiv.org/abs/2606.13735"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13735","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13735","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VI-Bench is a benchmark for evaluating video prompt inversion from AIGC videos, using 16.1 million real-user prompts and 900 human-verified videos across three difficulty settings.","descriptions":{"en":"VI-Bench is a benchmark for evaluating video prompt inversion from AIGC videos, using 16.1 million real-user prompts and 900 human-verified videos across three difficulty settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.08079","languages":[],"library_direction":null,"modality":null,"name":"VI-Bench","openness":"unknown","publisher":null,"released":"2026-09-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.08079","source_url":"https://arxiv.org/abs/2609.08079"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-08079","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08079","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":8,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":"gh:mcg-nju/viabench","source_record":"bm_viabench_7ade7668"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VIABench evaluates multimodal large language models on three tasks from first-person videos of visually impaired individuals: proactive reminder, visual question answering, and vision-guided interaction. It includes 761 videos, 46.9 hours, and 14,526 annotations, with protocols for online and offline settings.","descriptions":{"en":"VIABench evaluates multimodal large language models on three tasks from first-person videos of visually impaired individuals: proactive reminder, visual question answering, and vision-guided interaction. It includes 761 videos, 46.9 hours, and 14,526 annotations, with protocols for online and offline settings."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.14660","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_viabench_7ade7668"},"modality":null,"name":"VIABench","openness":"unknown","publisher":null,"released":"2026-07-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.14660","source_url":"https://arxiv.org/abs/2607.14660"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-14660","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14660","topics":[],"unit":null},{"aliases":["A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark of over 300K judgment samples that evaluates VLM judges for visually impaired assistance on effectiveness, impartiality, and stability.","descriptions":{"en":"A benchmark of over 300K judgment samples that evaluates VLM judges for visually impaired assistance on effectiveness, impartiality, and stability."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.31351","languages":[],"library_direction":null,"modality":null,"name":"VIABLE","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.31351","source_url":"https://arxiv.org/abs/2605.31351"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-31351","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31351","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLM robustness to Vietnamese dialectal rewrites across emotion recognition, natural language inference, question answering, and multiple-choice QA, measuring performance degradation across six dialect groups.","descriptions":{"en":"Evaluates LLM robustness to Vietnamese dialectal rewrites across emotion recognition, natural language inference, question answering, and multiple-choice QA, measuring performance degradation across six dialect groups."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10414","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_vialectbench_c8a2ee22"},"modality":null,"name":"VialectBench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10414","source_url":"https://arxiv.org/abs/2608.10414"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10414","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10414","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"VIBE is a benchmark for entity-centered affective profiling of LLM outputs in Valence-Arousal-Dominance space, introducing a measurement contract with scalar favorability, response-level and target-directed VAD, and an Affective Passport reporting format.","descriptions":{"en":"VIBE is a benchmark for entity-centered affective profiling of LLM outputs in Valence-Arousal-Dominance space, introducing a measurement contract with scalar favorability, response-level and target-directed VAD, and an Affective Passport reporting format."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.03810","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","safety-alignment"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_vibe_30ded12f"},"modality":null,"name":"VIBE","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03810","source_url":"https://arxiv.org/abs/2608.03810"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03810","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03810","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Visual Interface Building Evaluation benchmark for UI/app generation","descriptions":{"en":"Visual Interface Building Evaluation benchmark for UI/app generation"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vibe:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vibe","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","safety-alignment"],"join":"llm_stats_id","primary":"chat-writing","source_record":"bm_vibe_30ded12f"},"modality":"text","name":"VIBE","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":88.6,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.886,"raw_min":0.886,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vibe:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-vibe","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"SVRPBench is an open and extensible benchmark for the Stochastic Vehicle Routing Problem (SVRP). It includes 500+ instances spanning small to large scales (10–1000 customers), designed to evaluate algorithms under realistic urban logistics conditions with uncertainty and operational constraints. SVRPBench是一个针对随机车辆路径问题（SVRP）的开放且可扩展的基准测试平台。它包含500多个实例，涵盖小到大规模（10-1000个客户），旨在评估算法在具有不确定性和操作约束的现实城市物流条件下的表现。","descriptions":{"en":"SVRPBench is an open and extensible benchmark for the Stochastic Vehicle Routing Problem (SVRP). It includes 500+ instances spanning small to large scales (10–1000 customers), designed to evaluate algorithms under realistic urban logistics conditions with uncertainty and operational constraints.","zh":"SVRPBench是一个针对随机车辆路径问题（SVRP）的开放且可扩展的基准测试平台。它包含500多个实例，涵盖小到大规模（10-1000个客户），旨在评估算法在具有不确定性和操作约束的现实城市物流条件下的表现。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1875","languages":[],"library_direction":null,"modality":null,"name":"VIBE","openness":"unknown","publisher":"MBZUAI, Abu Dhabi, UAE","released":"2025-05-29","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1875-vibe","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VIBE","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VIBE benchmark subset for Android application generation","descriptions":{"en":"VIBE benchmark subset for Android application generation"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vibe-android:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-android?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vibe-android","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_c5fd1801a2c3042f"},"modality":"text","name":"VIBE Android","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":89.7,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.897,"raw_min":0.897,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vibe-android:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-android?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-vibe-android","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-android?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VIBE benchmark subset for backend service generation","descriptions":{"en":"VIBE benchmark subset for backend service generation"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vibe-backend:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-backend?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vibe-backend","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_c1d1fee0593d8c3d"},"modality":"text","name":"VIBE Backend","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":86.7,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.867,"raw_min":0.867,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vibe-backend:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-backend?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-vibe-backend","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-backend?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VIBE benchmark subset for iOS application generation","descriptions":{"en":"VIBE benchmark subset for iOS application generation"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vibe-ios:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-ios?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vibe-ios","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_a13f25bf6c359279"},"modality":"text","name":"VIBE iOS","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":88.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.88,"raw_min":0.88,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vibe-ios:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-ios?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-vibe-ios","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-ios?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VIBE benchmark subset for simulation code generation","descriptions":{"en":"VIBE benchmark subset for simulation code generation"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vibe-simulation:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-simulation?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vibe-simulation","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_0fa3c47d9e1f8121"},"modality":"text","name":"VIBE Simulation","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":87.1,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.871,"raw_min":0.871,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vibe-simulation:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-simulation?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-vibe-simulation","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-simulation?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VIBE benchmark subset for web application generation","descriptions":{"en":"VIBE benchmark subset for web application generation"},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vibe-web:minimax-m2.1","reported_at":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-web?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vibe-web","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_58b102d24a7d44bc"},"modality":"text","name":"VIBE Web","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":91.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.915,"raw_min":0.915,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vibe-web:minimax-m2.1","reported_date":"2025-12-23","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-web?top_n=500"},"unit":null},"scored_models":[{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax M2.1"}],"slug":"llm-stats-vibe-web","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-web?top_n=500","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates personalized large language models on cross-concept preference reasoning using psychology-grounded tasks.","descriptions":{"en":"Evaluates personalized large language models on cross-concept preference reasoning using psychology-grounded tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.00921","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["agent-memory","chat-writing"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_vibe-bench_10014945"},"modality":null,"name":"VIBE-Bench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.00921","source_url":"https://arxiv.org/abs/2609.00921"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-00921","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00921","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":189,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_926ff83a1e9c46a4"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","general","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VIBE-Eval is a hard evaluation suite for measuring progress of multimodal language models, consisting of 269 visual understanding prompts with gold-standard responses authored by experts. The benchmark has dual objectives: vibe checking multimodal chat models for day-to-day tasks and rigorously testing frontier models, with the hard set containing >50% questions that all frontier models answer incorrectly.","descriptions":{"en":"VIBE-Eval is a hard evaluation suite for measuring progress of multimodal language models, consisting of 269 visual understanding prompts with gold-standard responses authored by experts. The benchmark has dual objectives: vibe checking multimodal chat models for day-to-day tasks and rigorously testing frontier models, with the hard set containing >50% questions that all frontier models answer incorrectly."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vibe-eval:gemini-1.5-flash-8b","reported_at":"2024-03-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-eval?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vibe-eval","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_926ff83a1e9c46a4"},"modality":"multimodal","name":"Vibe-Eval","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":67.2,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.672,"raw_min":0.409,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vibe-eval:gemini-2.5-pro-preview-06-05","reported_date":"2025-06-05","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-eval?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2024-12-01","date_precision":"model_announcement","name":"Gemini 2.0 Flash"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"}],"slug":"llm-stats-vibe-eval","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-eval?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VIBE-Pro is an advanced version of the VIBE (Visual & Interactive Benchmark for Execution) benchmark that evaluates LLMs on professional-grade full-stack application development tasks. It measures model performance across complex real-world development scenarios including web, mobile, and backend applications with higher difficulty than the standard VIBE benchmark.","descriptions":{"en":"VIBE-Pro is an advanced version of the VIBE (Visual & Interactive Benchmark for Execution) benchmark that evaluates LLMs on professional-grade full-stack application development tasks. It measures model performance across complex real-world development scenarios including web, mobile, and backend applications with higher difficulty than the standard VIBE benchmark."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vibe-pro:minimax-m2.5","reported_at":"2026-02-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-pro?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vibe-pro","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_dd66ecafddb88c16"},"modality":"text","name":"VIBE-Pro","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":55.60000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.556,"raw_min":0.542,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vibe-pro:minimax-m2.7","reported_date":"2026-03-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-pro?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax M2.7"}],"slug":"llm-stats-vibe-pro","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-pro?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VIBE-V2 is an internal benchmark covering pure front-end and full-stack Web, Android, and iOS projects with build-from-scratch tasks. It uses an Agent-as-a-Verifier paradigm to automatically verify program interaction logic and visual output, scoring models through a unified pipeline that includes a requirement set, containerized deployment, and a dynamic interaction environment.","descriptions":{"en":"VIBE-V2 is an internal benchmark covering pure front-end and full-stack Web, Android, and iOS projects with build-from-scratch tasks. It uses an Agent-as-a-Verifier paradigm to automatically verify program interaction logic and visual output, scoring models through a unified pipeline that includes a requirement set, containerized deployment, and a dynamic interaction environment."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vibe-v2:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-v2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vibe-v2","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_e15ee86f46c26205"},"modality":"text","name":"VIBE-V2","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":50.12,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.5012,"raw_min":0.5012,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vibe-v2:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-v2?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-vibe-v2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-v2?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":25,"github_stars_30d":8,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"claire_record_id","repository":"gh:evolvent-ai/vibelifebench","source_record":"bm_vibelifebench_bf7626dc"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"VibeLifeBench evaluates LLM agents on 200 long-horizon, multi-week tasks across 10 everyday-life domains in a simulated world of 22 mock services. Agents must manage silent world changes and implicit constraints, with fine-grained weighted checks on end state, timeliness, and constraint adherence.","descriptions":{"en":"VibeLifeBench evaluates LLM agents on 200 long-horizon, multi-week tasks across 10 everyday-life domains in a simulated world of 22 mock services. Agents must manage silent world changes and implicit constraints, with fine-grained weighted checks on end state, timeliness, and constraint adherence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.10875","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","professional-work"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_vibelifebench_bf7626dc"},"modality":null,"name":"VibeLifeBench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10875","source_url":"https://arxiv.org/abs/2608.10875"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10875","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10875","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_vibemembench_a0c52eec"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates memory systems for coding agents on 111 coding targets from 90 SWE-rebench V2 repositories using 3,634 history trajectories.","descriptions":{"en":"Evaluates memory systems for coding agents on 111 coding targets from 90 SWE-rebench V2 repositories using 3,634 history trajectories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.23570","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","agentic-coding"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_vibemembench_a0c52eec"},"modality":null,"name":"VibeMemBench","openness":"unknown","publisher":null,"released":"2026-09-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.23570","source_url":"https://arxiv.org/abs/2609.23570"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-23570","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23570","topics":["coding","agents"],"unit":null},{"aliases":["ViBench"],"attention_signals":null,"categories":["coding_agent"],"collected_at":null,"description":"End-to-end vibe-coding benchmark by authors at Replit and Georgian AI Lab, scoring web applications from the user's perspective. Tasks are derived from anonymized Replit production traces, so the benchmark is first-party to one of the agents it measures. It reaches this registry through a customer testimonial in Anthropic's Claude Fable 5 / Mythos 5 launch post, where it is described qualitatively and never scored in the comparison table. Distinct from VBench, an unrelated video-generation benchmark.","descriptions":{"en":"End-to-end vibe-coding benchmark by authors at Replit and Georgian AI Lab, scoring web applications from the user's perspective. Tasks are derived from anonymized Replit production traces, so the benchmark is first-party to one of the agents it measures. It reaches this registry through a customer testimonial in Anthropic's Claude Fable 5 / Mythos 5 launch post, where it is described qualitatively and never scored in the comparison table. Distinct from VBench, an unrelated video-generation benchmark."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:vibench","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["agentic-coding"],"join":"review:14H034160212:2026-10-15","primary":"agentic-coding","source_record":null},"modality":null,"name":"ViBench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"model-reports:vibench","source_url":"https://vibench.ai/"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"vibench","source":"model_reports","source_url":"https://vibench.ai/","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Benchmark for reasoning with spatio-temporal audio-visual evidence in long videos.","descriptions":{"en":"Benchmark for reasoning with spatio-temporal audio-visual evidence in long videos."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.17248","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding","speech-audio"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_video-holmesv2_8cddcbd7"},"modality":null,"name":"Video-HolmesV2","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.17248","source_url":"https://arxiv.org/abs/2609.17248"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-17248","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17248","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"claire_record_id","repository":"gh:alexios-hub/video-ifbench","source_record":"bm_video-ifbench_4fc7a0ab"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates instruction following of multimodal LLMs in video understanding with 1.5K samples across constraint categories.","descriptions":{"en":"Evaluates instruction following of multimodal LLMs in video understanding with 1.5K samples across constraint categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.25529","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","video-understanding"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_video-ifbench_4fc7a0ab"},"modality":null,"name":"Video-IFBench","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25529","source_url":"https://arxiv.org/abs/2608.25529"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25529","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25529","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":799,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:bradyfu/video-mme","source_record":"lib_video_mme"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Video-MME is the first-ever comprehensive evaluation benchmark of Multi-modal Large Language Models (MLLMs) in video analysis. It features 900 videos totaling 254 hours with 2,700 human-annotated question-answer pairs across 6 primary visual domains (Knowledge, Film & Television, Sports Competition, Life Record, Multilingual, and others) and 30 subfields. The benchmark evaluates models across diverse temporal dimensions (11 seconds to 1 hour), integrates multi-modal inputs including video frames, subtitles, and audio, and uses rigorous manual labeling by expert annotators for precise assessment.","descriptions":{"en":"Video-MME is the first-ever comprehensive evaluation benchmark of Multi-modal Large Language Models (MLLMs) in video analysis. It features 900 videos totaling 254 hours with 2,700 human-annotated question-answer pairs across 6 primary visual domains (Knowledge, Film & Television, Sports Competition, Life Record, Multilingual, and others) and 30 subfields. The benchmark evaluates models across diverse temporal dimensions (11 seconds to 1 hour), integrates multi-modal inputs including video frames, subtitles, and audio, and uses rigorous manual labeling by expert annotators for precise assessment."},"evidence_summary":{"document_count":1,"model_count":17,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:video-mme:gemini-1.5-flash-8b","reported_at":"2024-03-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:video-mme","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"lib_video_mme"},"modality":"multimodal","name":"Video-MME","openness":"unknown","publication":{"evidence":[{"note":"The benchmark team's repository links the paper and project and describes the original video collection and question set, matching the source record. It also documents subtitle and sampling choices.","url":"https://github.com/MME-Benchmarks/Video-MME/blob/main/README.md"}],"id":"video-mme-original","links":[{"kind":"paper","title":"Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis","url":"https://arxiv.org/abs/2405.21075"},{"kind":"repo","title":"Video-MME official repository","url":"https://github.com/MME-Benchmarks/Video-MME"},{"kind":"website","title":"Video-MME project","url":"https://video-mme.github.io/"}],"record_keys":["llm-stats:video-mme"],"reviewed_at":"2026-10-26","scope":{"en":"Original Video-MME, not Video-MME-v2 or Video-MME-Logical. Subtitle use and frame sampling remain properties of each reported evaluation.","zh":"原始 Video-MME，不包括 Video-MME-v2 或 Video-MME-Logical；字幕使用和抽帧方式仍以每项成绩的原始评测设置为准。"}},"publisher":null,"released":"2024-05-31","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:video-mme","source_url":"https://arxiv.org/abs/2405.21075"},"repo_kind":null,"repo_resolution_status":null,"score_count":17,"score_direction":"higher_is_better","score_summary":{"display_max":89.2,"display_multiplier":100,"model_count":17,"model_count_basis":"source_model_id","numeric_count":17,"raw_max":0.892,"raw_min":0.55,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:video-mme:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Nova 2 Omni"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi-4-multimodal-instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-video-mme","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["Video-MME","VideoMME"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Frame sampling rate dominates long-video results.","descriptions":{"en":"Frame sampling rate dominates long-video results."},"evidence_summary":{"document_count":4,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000video_mme\u0000qwen3_5_model_card\u0000video_mme\u0000thinking (temp 0.6, top_p 0.95, top_k 20), with subtitles\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000video_mme\u0000qwen3_5_model_card\u0000video_mme\u0000thinking (temp 0.6, top_p 0.95, top_k 20), with subtitles\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:video_mme","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["video-understanding"],"join":"review:14H034160212:2026-10-15","primary":"video-understanding","source_record":null},"modality":null,"name":"Video-MME","openness":"unknown","publisher":null,"released":"2024-05-31","released_reference":{"basis":"benchmark_release","source_key":"model-reports:video_mme","source_url":"https://video-mme.github.io/"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":90.0,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":90.0,"raw_min":83.7,"source_reference":{"obs_id":"curated\u0000video_mme\u0000moonshot_kimi_k3_model_card\u0000video_mme\u0000with subtitle, reasoning=max\u0000Kimi K3","observation_id":"curated\u0000video_mme\u0000moonshot_kimi_k3_model_card\u0000video_mme\u0000with subtitle, reasoning=max\u0000Kimi K3","reported_at":"2026-06-13","reported_date":"2026-06-13","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3"},"unit":"percent"},"scored_models":[{"date":"2026-06-13","date_precision":"document_publication","name":"Kimi K3"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"video_mme","source":"model_reports","source_url":"https://video-mme.github.io/","topics":["multimodal"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":799,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"artifact:gh:bradyfu/video-mme","repository":"gh:bradyfu/video-mme","source_record":"lib_video_mme"},"categories":["多模态","Multimodal","理解","Understanding","多模态模型","VLM","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Video-MME is an evaluation benchmark of multi-modal LLMs in video analysis, including 900 videos in various duration with a total of 254 hours which spans 6 primary visual domains with 30 subfields. Video-MME用于评估多模态大模型的视频分析能力，包含900个不同长度的视频，来自6个主要视觉领域和30个子领域，总时长达254小时。","descriptions":{"en":"Video-MME is an evaluation benchmark of multi-modal LLMs in video analysis, including 900 videos in various duration with a total of 254 hours which spans 6 primary visual domains with 30 subfields.","zh":"Video-MME用于评估多模态大模型的视频分析能力，包含900个不同长度的视频，来自6个主要视觉领域和30个子领域，总时长达254小时。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:1358","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"artifact:gh:bradyfu/video-mme","primary":"video-understanding","source_record":"lib_video_mme"},"modality":"multimodal","name":"Video-MME","openness":"unknown","publisher":"University of Science and Technology of China","released":"2024-03-31","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1358-video-mme","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Video-MME","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":799,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_621de856e8fd1d1d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Video-MME is the first-ever comprehensive evaluation benchmark for Multi-modal Large Language Models (MLLMs) in video analysis. This variant focuses on long-term videos (30min-60min) without subtitle inputs, testing robust contextual dynamics across 6 primary visual domains with 30 subfields including knowledge, film & television, sports competition, life record, and multilingual content.","descriptions":{"en":"Video-MME is the first-ever comprehensive evaluation benchmark for Multi-modal Large Language Models (MLLMs) in video analysis. This variant focuses on long-term videos (30min-60min) without subtitle inputs, testing robust contextual dynamics across 6 primary visual domains with 30 subfields including knowledge, film & television, sports competition, life record, and multilingual content."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:video-mme-(long,-no-subtitles):gpt-4.1-2025-04-14","reported_at":"2025-04-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme-%28long%2C-no-subtitles%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:video-mme-(long,-no-subtitles)","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_621de856e8fd1d1d"},"modality":"multimodal","name":"Video-MME (long, no subtitles)","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":72.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.72,"raw_min":0.72,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:video-mme-(long,-no-subtitles):gpt-4.1-2025-04-14","reported_date":"2025-04-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme-%28long%2C-no-subtitles%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"}],"slug":"llm-stats-video-mme-long-no-subtitles","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme-%28long%2C-no-subtitles%29?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":11,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":25,"join":"claire_record_id","repository":"gh:mrakas/video-mme-logical","source_record":"bm_video-mme-logical_db483a80"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video temporal-logical reasoning in multimodal LLMs across 25 fine-grained task categories, with difficulty-controlled final-answer scoring and intermediate-state diagnostics.","descriptions":{"en":"Evaluates video temporal-logical reasoning in multimodal LLMs across 25 fine-grained task categories, with difficulty-controlled final-answer scoring and intermediate-state diagnostics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.27828","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","abstract-reasoning"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_video-mme-logical_db483a80"},"modality":null,"name":"Video-MME-Logical","openness":"unknown","publisher":null,"released":"2026-06-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.27828","source_url":"https://arxiv.org/abs/2606.27828"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-27828","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27828","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":16,"join":"claire_record_id","repository":null,"source_record":"bm_video2skill_ec0548ab"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Manipulation behaviors vary widely across objects and scenes, but they share a small set of reusable skills, and planning with these skills helps embodied agents generalize to new tasks.","descriptions":{"en":"Manipulation behaviors vary widely across objects and scenes, but they share a small set of reusable skills, and planning with these skills helps embodied agents generalize to new tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.36691","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","agent-skills-evolution"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_video2skill_ec0548ab"},"modality":null,"name":"Video2Skill","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.36691","source_url":"https://arxiv.org/abs/2609.36691"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-36691","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36691","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics & Embodied AI","cs.RO"],"collected_at":"2026-10-14T09:44:51Z","description":"We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI.","descriptions":{"en":"We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38966","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","embodied-robotics"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_video2swimfish_a7faa842"},"modality":null,"name":"Video2SwimFish","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38966","source_url":"https://arxiv.org/abs/2609.38966"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38966","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38966","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":9,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:zzzmyyzeng/videoargus","source_record":"bm_videoargus-bench_f3e24463"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video generation and editing across five settings using agent-generated rubrics that score candidates with criterion-specific evidence and diagnostic reports.","descriptions":{"en":"Evaluates video generation and editing across five settings using agent-generated rubrics that score candidates with criterion-specific evidence and diagnostic reports."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05485","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_videoargus-bench_f3e24463"},"modality":null,"name":"VideoArgus-Bench","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05485","source_url":"https://arxiv.org/abs/2608.05485"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05485","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05485","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":null,"source_record":"bm_videofdb_d27be39c"},"categories":["Cybersecurity","Multimodal","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VideoFDB is a benchmark presented for evaluating full-duplex audio-visual conversational agents. It includes 237 dyadic clips with 11 nonverbal conversational dynamics from real-world video calls, along with a taxonomy and rubric-based LM-as-judge evaluation framework.","descriptions":{"en":"VideoFDB is a benchmark presented for evaluating full-duplex audio-visual conversational agents. It includes 237 dyadic clips with 11 nonverbal conversational dynamics from real-world video calls, along with a taxonomy and rubric-based LM-as-judge evaluation framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30256","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","video-understanding"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_videofdb_d27be39c"},"modality":null,"name":"VideoFDB","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30256","source_url":"https://arxiv.org/abs/2605.30256"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30256","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30256","topics":["multimodal","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":106,"github_stars_30d":-2,"hf_dataset_downloads":null,"hf_paper_upvotes":11,"join":"claire_record_id","repository":"gh:zfkarl/videogaia","source_record":"bm_videogaia_c89526e7"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VideoGAIA evaluates agentic video understanding through multi-turn, tool-augmented interactions where models must iteratively perceive videos, invoke external tools, and integrate multimodal evidence. It contains 271 human-verified tasks across diverse real-world scenarios.","descriptions":{"en":"VideoGAIA evaluates agentic video understanding through multi-turn, tool-augmented interactions where models must iteratively perceive videos, invoke external tools, and integrate multimodal evidence. It contains 271 human-verified tasks across diverse real-world scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.14718","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","tool-use","deep-research"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_videogaia_c89526e7"},"modality":null,"name":"VideoGAIA","openness":"unknown","publisher":null,"released":"2026-08-12","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.14718","source_url":"https://arxiv.org/abs/2608.14718"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-14718","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14718","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["创作","Creation","NeurIPS 2024","智能体","Agent","任务执行","Task Execution","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VideoGUI is designed to evaluate GUI assistants on visual-centric GUI tasks. Sourced from high-quality web instructional videos, it focuses on tasks involving professional and novel software and complex activities (e.g., video editing). VideoGUI旨在评估以视觉为中心的GUI任务上的GUI助手，来自高质量的网络教学视频，侧重于涉及专业和新颖软件和复杂活动（例如视频编辑）的任务。","descriptions":{"en":"VideoGUI is designed to evaluate GUI assistants on visual-centric GUI tasks. Sourced from high-quality web instructional videos, it focuses on tasks involving professional and novel software and complex activities (e.g., video editing).","zh":"VideoGUI旨在评估以视觉为中心的GUI任务上的GUI助手，来自高质量的网络教学视频，侧重于涉及专业和新颖软件和复杂活动（例如视频编辑）的任务。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1277","languages":[],"library_direction":null,"modality":null,"name":"VideoGUI","openness":"unknown","publisher":"National University of Singapore","released":"2024-06-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1277-videogui","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VideoGUI","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":101,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":29,"join":"llm_stats_id","repository":"gh:tencentarc/video-holmes","source_record":"catalog_281510454193b470"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","video"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VideoHolmes evaluates video understanding and reasoning capabilities in multimodal models.","descriptions":{"en":"VideoHolmes evaluates video understanding and reasoning capabilities in multimodal models."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:videoholmes:mimo-v2.5","reported_at":"2026-04-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videoholmes?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:videoholmes","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_281510454193b470"},"modality":"multimodal","name":"VideoHolmes","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":68.2,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.682,"raw_min":0.64,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:videoholmes:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videoholmes?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-videoholmes","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videoholmes?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","多模态模型","VLM","逻辑推理","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VideoMathQA is a benchmark designed to evaluate mathematical reasoning in real-world educational videos. It requires models to interpret and integrate information from three modalities, visuals, audio, and text, across time. VideoMathQA是一个旨在评估实际教育视频中数学推理能力的基准。它要求模型解释和整合来自三种模态(视觉、音频和文本)随时间变化的信息。该基准解决了\"多模态针堆\"问题,即关键信息稀疏且分散在视频的不同模态和时刻。","descriptions":{"en":"VideoMathQA is a benchmark designed to evaluate mathematical reasoning in real-world educational videos. It requires models to interpret and integrate information from three modalities, visuals, audio, and text, across time.","zh":"VideoMathQA是一个旨在评估实际教育视频中数学推理能力的基准。它要求模型解释和整合来自三种模态(视觉、音频和文本)随时间变化的信息。该基准解决了\"多模态针堆\"问题,即关键信息稀疏且分散在视频的不同模态和时刻。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1926","languages":[],"library_direction":null,"modality":"multimodal","name":"VideoMathQA","openness":"unknown","publisher":"MBZUAI,University of California Merced,Google Research,etc","released":"2025-06-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1926-videomathqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VideoMathQA","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":799,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_4dd9ff92b3fe2f97"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The first-ever comprehensive evaluation benchmark of Multi-modal LLMs in Video analysis. Features 900 videos (254 hours) with 2,700 question-answer pairs covering 6 primary visual domains and 30 subfields. Evaluates temporal understanding across short (11 seconds) to long (1 hour) videos with multi-modal inputs including video frames, subtitles, and audio.","descriptions":{"en":"The first-ever comprehensive evaluation benchmark of Multi-modal LLMs in Video analysis. Features 900 videos (254 hours) with 2,700 question-answer pairs covering 6 primary visual domains and 30 subfields. Evaluates temporal understanding across short (11 seconds) to long (1 hour) videos with multi-modal inputs including video frames, subtitles, and audio."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:videomme-w-sub.:qwen2.5-vl-7b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-sub.?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:videomme-w-sub.","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_4dd9ff92b3fe2f97"},"modality":"multimodal","name":"VideoMME w sub.","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":90.4,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.904,"raw_min":0.716,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:videomme-w-sub.:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-sub.?top_n=500"},"unit":null},"scored_models":[{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-videomme-w-sub","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-sub.?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":799,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":null,"source_record":"catalog_c5faee41a30faa01"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Video-MME is a comprehensive evaluation benchmark for multi-modal large language models in video analysis. It features 900 videos across 6 primary visual domains with 30 subfields, ranging from 11 seconds to 1 hour in duration, with 2,700 question-answer pairs. The benchmark evaluates MLLMs' capabilities in processing sequential visual data and multi-modal content including video frames, subtitles, and audio.","descriptions":{"en":"Video-MME is a comprehensive evaluation benchmark for multi-modal large language models in video analysis. It features 900 videos across 6 primary visual domains with 30 subfields, ranging from 11 seconds to 1 hour in duration, with 2,700 question-answer pairs. The benchmark evaluates MLLMs' capabilities in processing sequential visual data and multi-modal content including video frames, subtitles, and audio."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:videomme-w-o-sub.:qwen2.5-vl-72b","reported_at":"2025-01-26","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-o-sub.?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:videomme-w-o-sub.","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_c5faee41a30faa01"},"modality":"multimodal","name":"VideoMME w/o sub.","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":83.89999999999999,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.839,"raw_min":0.651,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:videomme-w-o-sub.:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-o-sub.?top_n=500"},"unit":null},"scored_models":[{"date":"2025-02-28","date_precision":"model_announcement","name":"Qwen2.5 VL 32B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 72B Instruct"},{"date":"2025-01-26","date_precision":"model_announcement","name":"Qwen2.5 VL 7B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-videomme-w-o-sub","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-o-sub.?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":73,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":24,"join":"llm_stats_id","repository":"gh:evolvinglmms-lab/videommmu","source_record":"catalog_3f084ff3bbed7170"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Video-MMMU evaluates Large Multimodal Models' ability to acquire knowledge from expert-level professional videos across six disciplines through three cognitive stages: perception, comprehension, and adaptation. Contains 300 videos and 900 human-annotated questions spanning Art, Business, Science, Medicine, Humanities, and Engineering.","descriptions":{"en":"Video-MMMU evaluates Large Multimodal Models' ability to acquire knowledge from expert-level professional videos across six disciplines through three cognitive stages: perception, comprehension, and adaptation. Contains 300 videos and 900 human-annotated questions spanning Art, Business, Science, Medicine, Humanities, and Engineering."},"evidence_summary":{"document_count":1,"model_count":26,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:videommmu:gpt-4o-2024-08-06","reported_at":"2024-08-06","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videommmu?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"llm-stats:videommmu","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","knowledge-expert-qa"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_3f084ff3bbed7170"},"modality":"multimodal","name":"VideoMMMU","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":"benchmark_source","repo_resolution_status":"resolved","score_count":26,"score_direction":"higher_is_better","score_summary":{"display_max":87.6,"display_multiplier":100,"model_count":26,"model_count_basis":"source_model_id","numeric_count":26,"raw_max":0.876,"raw_min":0.562,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:videommmu:gemini-3-pro-preview","reported_date":"2025-11-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videommmu?top_n=500"},"unit":null},"scored_models":[{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro Preview 06-05"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-videommmu","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videommmu?top_n=500","topics":["multimodal","reasoning-math","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":null,"source_record":"bm_videoodyssey_004b7012"},"categories":["General AI","Long Context","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"VideoOdyssey evaluates models on ultra-long-context video understanding using videos averaging 109 minutes across 11 domains, with two subsets for visual and audio-visual understanding. Tasks include question answering with continuous certificates averaging 16 and 12.8 minutes respectively.","descriptions":{"en":"VideoOdyssey evaluates models on ultra-long-context video understanding using videos averaging 109 minutes across 11 domains, with two subsets for visual and audio-visual understanding. Tasks include question answering with continuous certificates averaging 16 and 12.8 minutes respectively."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.22907","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","long-context"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_videoodyssey_004b7012"},"modality":null,"name":"VideoOdyssey","openness":"unknown","publisher":null,"released":"2026-05-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.22907","source_url":"https://arxiv.org/abs/2605.22907"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-22907","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22907","topics":["long-context"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","多模态","Multimodal","video reasoning","MLLMs","多模态模型","VLM","逻辑推理","Reasoning","视频理解","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VideoReasonBench is designed to evaluate vision-centric complex video reasoning. VideoReasonBench是一个用于评测视觉为中心、复杂视频推理的基准。","descriptions":{"en":"VideoReasonBench is designed to evaluate vision-centric complex video reasoning.","zh":"VideoReasonBench是一个用于评测视觉为中心、复杂视频推理的基准。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1900","languages":[],"library_direction":null,"modality":"multimodal","name":"VideoReasonBench","openness":"restricted","publisher":"Peking University","released":"2025-06-05","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1900-videoreasonbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VideoReasonBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_videorover-bench_f87e9ddf"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"VideoRover-Bench is a benchmark for open-world video reasoning that combines video understanding with deep research, stratified by video duration and research difficulty. It evaluates the capability of models to locate sparse visual evidence and acquire external knowledge through coordinated video cropping, multimodal search, and webpage browsing.","descriptions":{"en":"VideoRover-Bench is a benchmark for open-world video reasoning that combines video understanding with deep research, stratified by video duration and research difficulty. It evaluates the capability of models to locate sparse visual evidence and acquire external knowledge through coordinated video cropping, multimodal search, and webpage browsing."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23329","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","deep-research"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_videorover-bench_f87e9ddf"},"modality":null,"name":"VideoRover-Bench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23329","source_url":"http://arxiv.org/abs/2608.23329v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23329","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23329v1","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":14,"join":"llm_stats_id","repository":null,"source_record":"catalog_8d535515b5577d52"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","knowledge","video","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VideoSimpleQA evaluates factual knowledge grounded in video content, measuring how accurately models answer short, fact-seeking questions about videos.","descriptions":{"en":"VideoSimpleQA evaluates factual knowledge grounded in video content, measuring how accurately models answer short, fact-seeking questions about videos."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:videosimpleqa:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videosimpleqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:videosimpleqa","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","factuality"],"join":"llm_stats_id","primary":"video-understanding","source_record":"catalog_8d535515b5577d52"},"modality":"multimodal","name":"VideoSimpleQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":76.4,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.764,"raw_min":0.714,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:videosimpleqa:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videosimpleqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-videosimpleqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videosimpleqa?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates video-grounded diagnostic understanding of one-shot website generation. Approximately 1.7K Video QA instances from 6,338 verified failures across semantic-logical, visual-motion, structural-temporal, and functional categories. Scoring is based on model accuracy in multi-agent system V2Lens compared against baseline Video MLLMs.","descriptions":{"en":"Evaluates video-grounded diagnostic understanding of one-shot website generation. Approximately 1.7K Video QA instances from 6,338 verified failures across semantic-logical, visual-motion, structural-temporal, and functional categories. Scoring is based on model accuracy in multi-agent system V2Lens compared against baseline Video MLLMs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.09573","languages":[],"library_direction":{"basis":"model-review-2","categories":["video-understanding","code-generation"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_videovibe_8a828619"},"modality":null,"name":"VideoVIBE","openness":"unknown","publisher":null,"released":"2026-08-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.09573","source_url":"https://arxiv.org/abs/2608.09573"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-09573","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09573","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":42,"github_stars_30d":7,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jianhuiwei7/videoweaver","source_record":"bm_videoweaver_b86f354a"},"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"VideoWeaver is an agent harness and benchmark for long video generation, with 16 task categories and 285 cases, evaluating agents via evidence-grounded judge on process and output.","descriptions":{"en":"VideoWeaver is an agent harness and benchmark for long video generation, with 16 task categories and 285 cases, evaluating agents via evidence-grounded judge on process and output."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.08091","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_videoweaver_b86f354a"},"modality":null,"name":"VideoWeaver","openness":"unknown","publisher":null,"released":"2026-06-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08091","source_url":"https://arxiv.org/abs/2606.08091"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08091","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08091","topics":["agents"],"unit":null},{"aliases":["Can We Trust Video Hallucination Detectors? VidHalLoc for Evaluating the Evaluators"],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_can-we-trust-video-hallucination-detectors_9edaaba4"},"categories":["Software & AI Compute","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark with 2,000 adversarial video hallucination samples for evaluating hallucination detectors.","descriptions":{"en":"A benchmark with 2,000 adversarial video hallucination samples for evaluating hallucination detectors."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.09895","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","factuality"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_can-we-trust-video-hallucination-detectors_9edaaba4"},"modality":null,"name":"VidHalLoc","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.09895","source_url":"https://arxiv.org/abs/2609.09895"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-09895","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09895","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:iyttor/vidmsg","source_record":"bm_vidmsg_63a1ae32"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VidMsg is a benchmark for implicit message inference in short videos, containing 400 clips across 9 topic areas with retrieval and multiple-choice QA tasks.","descriptions":{"en":"VidMsg is a benchmark for implicit message inference in short videos, containing 400 clips across 9 topic areas with retrieval and multiple-choice QA tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.03635","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_vidmsg_63a1ae32"},"modality":null,"name":"VidMsg","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03635","source_url":"https://arxiv.org/abs/2606.03635"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03635","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03635","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates fine-grained video understanding by verifying events in dense video captions across complexity and duration.","descriptions":{"en":"Evaluates fine-grained video understanding by verifying events in dense video captions across complexity and duration."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.21521","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_vidomni-bench_3ae45976"},"modality":null,"name":"VidOmni-Bench","openness":"unknown","publisher":null,"released":"2026-09-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.21521","source_url":"https://arxiv.org/abs/2609.21521"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-21521","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21521","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:hng0303/vietfashion","source_record":"bm_vietfashion_31e76260"},"categories":["General AI","Information retrieval","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VietFashion is a benchmark for sketch-text composed image retrieval centered on the Ao Dai, a traditional Vietnamese garment. It includes 650 sketches expanded to over 21,000 photorealistic images with captions, and adopts a multi-target retrieval setting.","descriptions":{"en":"VietFashion is a benchmark for sketch-text composed image retrieval centered on the Ao Dai, a traditional Vietnamese garment. It includes 650 sketches expanded to over 21,000 photorealistic images with captions, and adopts a multi-target retrieval setting."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.13427","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_vietfashion_31e76260"},"modality":null,"name":"VietFashion","openness":"unknown","publisher":null,"released":"2026-06-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13427","source_url":"https://arxiv.org/abs/2606.13427"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13427","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13427","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":545,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:shim0114/vif-bench","source_record":"bm_vif-bench_c4b82e83"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Recent multimodal image generation models can take multiple images and textual instructions as input, enabling reference-based generation guided not only by text but also by visual instructions such as layouts, arrows, and pose cues.","descriptions":{"en":"Recent multimodal image generation models can take multiple images and textual instructions as input, enabling reference-based generation guided not only by text but also by visual instructions such as layouts, arrows, and pose cues."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.37709","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_vif-bench_c4b82e83"},"modality":null,"name":"VIF-Bench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37709","source_url":"https://arxiv.org/abs/2609.37709"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37709","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37709","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","vision-language","math","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ViLBench is a benchmark designed to evaluate vision-language models. It features 600 examples from 5 datasets, selected based on the criterion that process reward models offer greater improvements over output reward models in guiding generations. ViLBench 是一项旨在评估视觉-语言模型的数据集，其强调对模型进行细粒度的逐步推理能力测试。该基准共包含600个经过严格筛选的样本，来源于五个不同的视觉-语言数据集，筛选标准是在模型答案选择过程中，过程奖励模型（process-reward model）相较于输出奖励模型（output-reward model）具有更显著的性能提升效果。","descriptions":{"en":"ViLBench is a benchmark designed to evaluate vision-language models. It features 600 examples from 5 datasets, selected based on the criterion that process reward models offer greater improvements over output reward models in guiding generations.","zh":"ViLBench 是一项旨在评估视觉-语言模型的数据集，其强调对模型进行细粒度的逐步推理能力测试。该基准共包含600个经过严格筛选的样本，来源于五个不同的视觉-语言数据集，筛选标准是在模型答案选择过程中，过程奖励模型（process-reward model）相较于输出奖励模型（output-reward model）具有更显著的性能提升效果。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1717","languages":[],"library_direction":null,"modality":"multimodal","name":"ViLBench","openness":"unknown","publisher":"UC Santa Cruz","released":"2025-03-26","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1717-vilbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ViLBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":10,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:mahmoodlab/viper","source_record":"bm_viper_dbe65c04"},"categories":["Biology & Drug Discovery","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"VIPER evaluates vision-language models on toxicologic pathology via 1,251 questions over 419 H&E-stained rat histology images across organ systems, covering multiple-choice, KPrim, and free-text formats with domain-expert validation.","descriptions":{"en":"VIPER evaluates vision-language models on toxicologic pathology via 1,251 questions over 419 H&E-stained rat histology images across organ systems, covering multiple-choice, KPrim, and free-text formats with domain-expert validation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.26382","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"claire_record_id","primary":"health-medicine","source_record":"bm_viper_dbe65c04"},"modality":null,"name":"VIPER","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.26382","source_url":"https://arxiv.org/abs/2608.26382"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-26382","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26382","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Autonomous Driving","Planning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates cooperative autonomous driving in vehicle-to-infrastructure settings using pseudo-simulation scenarios.","descriptions":{"en":"Evaluates cooperative autonomous driving in vehicle-to-infrastructure settings using pseudo-simulation scenarios."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02462","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multi-agent-games","embodied-robotics"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_vips_678db551"},"modality":null,"name":"VIPS","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02462","source_url":"https://arxiv.org/abs/2609.02462"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02462","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02462","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":15,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:qianjinydx/virobench","source_record":"bm_virobench_ca7b2c7e"},"categories":["Biology & Drug Discovery","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"ViroBench evaluates nucleotide foundation models on viral genomics tasks across two axes: biological understanding (taxonomy classification, host prediction) and latent biosecurity risk (genome modeling, CDS completion). It includes 18 tasks, 58,314 viral samples, multiple split strategies (genus-disjoint, temporal), and 11 metrics, with a unified evaluation protocol and leaderboard.","descriptions":{"en":"ViroBench evaluates nucleotide foundation models on viral genomics tasks across two axes: biological understanding (taxonomy classification, host prediction) and latent biosecurity risk (genome modeling, CDS completion). It includes 18 tasks, 58,314 viral samples, multiple split strategies (genus-disjoint, temporal), and 11 metrics, with a unified evaluation protocol and leaderboard."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.25388","languages":[],"library_direction":{"basis":"model-review","categories":["science","safety-alignment"],"join":"claire_record_id","primary":"science","source_record":"bm_virobench_ca7b2c7e"},"modality":null,"name":"ViroBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25388","source_url":"https://arxiv.org/abs/2605.25388"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25388","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25388","topics":["safety","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_c00b13289041b12b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Virology Capabilities Test (VCT) is an expert-level multiple-choice benchmark measuring the capability to troubleshoot complex virology laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development.","descriptions":{"en":"Virology Capabilities Test (VCT) is an expert-level multiple-choice benchmark measuring the capability to troubleshoot complex virology laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vct:grok-4.1-thinking-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vct?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vct","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","science"],"join":"llm_stats_id","primary":"safety-alignment","source_record":"catalog_c00b13289041b12b"},"modality":"text","name":"Virology Capabilities Test","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":61.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.61,"raw_min":0.61,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vct:grok-4.1-thinking-2025-11-17","reported_date":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vct?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-vct","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vct?top_n=500","topics":["safety","medical"],"unit":null},{"aliases":["virtualv_llm"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:virtuanalytica/virtualv_llm","source_record":"bm_virtualv-llm_8ee22c0f"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Hardware-aware evaluation suite for GGUF/llama.cpp and 1Cat-vLLM workloads on heterogeneous GPU pools.","descriptions":{"en":"Hardware-aware evaluation suite for GGUF/llama.cpp and 1Cat-vLLM workloads on heterogeneous GPU pools."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:virtuanalytica/virtualv_llm","languages":[],"library_direction":null,"modality":null,"name":"VirtualV LLM Testsuite","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:virtuanalytica/virtualv_llm","source_url":"https://github.com/virtuanalytica/virtualv_llm"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-virtuanalytica-virtualv-llm","source":"claire_radar","source_url":"https://github.com/virtuanalytica/virtualv_llm","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_visanombench_44ae350c"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"VisAnomBench is a benchmark assembled from public time-series datasets for anomaly detection, augmented with natural-language explanations selected from large vision-language models. It supports fine-tuning a parameter-efficient VLM called VisAnomReasoner for grounded anomaly detection decisions.","descriptions":{"en":"VisAnomBench is a benchmark assembled from public time-series datasets for anomaly detection, augmented with natural-language explanations selected from large vision-language models. It supports fine-tuning a parameter-efficient VLM called VisAnomReasoner for grounded anomaly detection decisions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2605.30344","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting","visual-reasoning"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_visanombench_44ae350c"},"modality":null,"name":"VisAnomBench","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30344","source_url":"https://arxiv.org/abs/2605.30344"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30344","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30344","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","知识","Knowledge","self-critique","VLM","VLM reasoning","多模态模型","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VISCO aims to evaluate the critique and correction capabilities of VLMs, which are two essential building blocks towards VLM self-improvement. VISCO requires VLMs to critique the correctness of each step in CoT, provide natural language explanation, and corrects the CoT based on the critique. VISCO 旨在评估 VLM 的评判（critique）和纠正（correction）能力，这两个能力是 VLM 自主提升推理性能的基础。对于一个视觉推理问题，给定模型生成的 CoT，VISCO 评测集要求 VLM 评判 CoT 中每个步骤的正确性，提供自然语言解释，并根据评判结果对 CoT 进行修正。","descriptions":{"en":"VISCO aims to evaluate the critique and correction capabilities of VLMs, which are two essential building blocks towards VLM self-improvement. VISCO requires VLMs to critique the correctness of each step in CoT, provide natural language explanation, and corrects the CoT based on the critique.","zh":"VISCO 旨在评估 VLM 的评判（critique）和纠正（correction）能力，这两个能力是 VLM 自主提升推理性能的基础。对于一个视觉推理问题，给定模型生成的 CoT，VISCO 评测集要求 VLM 评判 CoT 中每个步骤的正确性，提供自然语言解释，并根据评判结果对 CoT 进行修正。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2063","languages":[],"library_direction":null,"modality":"multimodal","name":"VISCO","openness":"unknown","publisher":"UCLA","released":"2024-12-03","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2063-visco","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VISCO","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:vis-nlp/viseditbench","source_record":"bm_viseditbench_e7b0411b"},"categories":["General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"VisEditBench is a benchmark comprising 1,395 human-annotated visualization code-editing tasks across two settings: feedback-guided repair and reference-guided restyling. Models are evaluated on their ability to revise existing visualization code based on multimodal feedback such as buggy or marked charts with textual instructions, and target chart images. Scoring is based on pass rates of generated code executions.","descriptions":{"en":"VisEditBench is a benchmark comprising 1,395 human-annotated visualization code-editing tasks across two settings: feedback-guided repair and reference-guided restyling. Models are evaluated on their ability to revise existing visualization code based on multimodal feedback such as buggy or marked charts with textual instructions, and target chart images. Scoring is based on pass rates of generated code executions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.10408","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_viseditbench_e7b0411b"},"modality":null,"name":"VisEditBench","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10408","source_url":"https://arxiv.org/abs/2608.10408"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10408","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10408","topics":["coding","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_ef79cc168e97f387"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VisFactor is a benchmark evaluating fine-grained visual factor perception and reasoning over images.","descriptions":{"en":"VisFactor is a benchmark evaluating fine-grained visual factor perception and reasoning over images."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:visfactor:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visfactor?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:visfactor","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_ef79cc168e97f387"},"modality":"multimodal","name":"VisFactor","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":51.4,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.514,"raw_min":0.428,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:visfactor:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visfactor?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-visfactor","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visfactor?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoning"],"attention_signals":null,"categories":["Biology & Drug Discovery","Multimodal","Reasoning","Robotics & Embodied AI","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Proposed benchmark framework to evaluate robustness of vision-language reasoning via flip-direction steering, constructing evaluation groups for a target model and visual variation.","descriptions":{"en":"Proposed benchmark framework to evaluate robustness of vision-language reasoning via flip-direction steering, constructing evaluation groups for a target model and visual variation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.28851","languages":[],"library_direction":null,"modality":null,"name":"VisFlip","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.28851","source_url":"https://arxiv.org/abs/2609.28851"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-28851","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28851","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Finance","Speech & Audio","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates forecasting of vishing call trajectories toward terminal harms using a 35,340-utterance synthetic Chinese call corpus labeled with a 6-tactic taxonomy.","descriptions":{"en":"Evaluates forecasting of vishing call trajectories toward terminal harms using a 35,340-utterance synthetic Chinese call corpus labeled with a 6-tactic taxonomy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.07151","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["forecasting","agentic-coding","multilingual"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_vishing-tactics-bench_5dc83880"},"modality":null,"name":"Vishing-Tactics-Bench","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07151","source_url":"https://arxiv.org/abs/2609.07151"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07151","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07151","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":81,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":44,"join":"llm_stats_id","repository":null,"source_record":"catalog_71bd6f729f139564"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","code","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Vision2Web evaluates multimodal models on converting visual designs and screenshots into functional web pages, measuring end-to-end design-to-code capability.","descriptions":{"en":"Vision2Web evaluates multimodal models on converting visual designs and screenshots into functional web pages, measuring end-to-end design-to-code capability."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vision2web:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vision2web?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vision2web","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","visual-reasoning"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_71bd6f729f139564"},"modality":"multimodal","name":"Vision2Web","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":69.0,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.69,"raw_min":0.31,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vision2web:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vision2web?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-vision2web","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vision2web?top_n=500","topics":["coding","multimodal"],"unit":null},{"aliases":["VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents"],"attention_signals":null,"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark for LLM coding agents that build web apps from visual specs. It covers five input conditions and scores structure, behavior, and visual fidelity.","descriptions":{"en":"A benchmark for LLM coding agents that build web apps from visual specs. It covers five input conditions and scores structure, behavior, and visual fidelity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26144","languages":[],"library_direction":null,"modality":null,"name":"VISTA (VIsual Spec-To-App Benchmark)","openness":"unknown","publisher":null,"released":"2026-05-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26144","source_url":"https://arxiv.org/abs/2605.26144"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26144","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26144","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Image translation is a fundamental capability of multimodal models for multilingual applications, requiring visual understanding and meaning preservation across languages.","descriptions":{"en":"Image translation is a fundamental capability of multimodal models for multilingual applications, requiring visual understanding and meaning preservation across languages."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.37287","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","visual-reasoning","document-ocr"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_vista-bench_8b26df34"},"modality":null,"name":"VISTA-Bench","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.37287","source_url":"https://arxiv.org/abs/2609.37287"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-37287","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37287","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_vistahop_3279a79d"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VistaHop is a benchmark for long-horizon Visual DeepSearch, evaluating repeated image inspection, visual-anchor grounding, and evidence traversal across 600 tasks.","descriptions":{"en":"VistaHop is a benchmark for long-horizon Visual DeepSearch, evaluating repeated image inspection, visual-anchor grounding, and evidence traversal across 600 tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.03273","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research","visual-reasoning"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_vistahop_3279a79d"},"modality":null,"name":"VistaHop","openness":"unknown","publisher":null,"released":"2026-06-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.03273","source_url":"https://arxiv.org/abs/2606.03273"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-03273","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03273","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Information retrieval","Multimodal","Planning"],"collected_at":"2026-10-14T09:44:51Z","description":"VisTarget-Bench is a 150-task human-verified benchmark pairing each multimodal search question with a held-out target image to separate image-retrieval failures from visual-perception failures.","descriptions":{"en":"VisTarget-Bench is a 150-task human-verified benchmark pairing each multimodal search question with a held-out target image to separate image-retrieval failures from visual-perception failures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.28062","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["visual-reasoning","deep-research"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_vistarget-bench_aaa3c19c"},"modality":null,"name":"VisTarget-Bench","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.28062","source_url":"https://arxiv.org/abs/2608.28062"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-28062","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28062","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","创作","Creation","多模态模型","VLM","视觉生成","Visual Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ViStoryBench is a benchmark designed to rigorously evaluate the capabilities of multimodal generative models (e.g., diffusion models, LLM-based agents) in synthesizing visually coherent image sequences from textual narratives and reference images. ViStoryBench 是一个面向故事可视化任务的综合性评测基准，旨在评估多模态生成模型（如扩散模型视频生成模型等）根据给定叙事文本和参考图像生成视觉连贯且情节一致的图像序列的能力。","descriptions":{"en":"ViStoryBench is a benchmark designed to rigorously evaluate the capabilities of multimodal generative models (e.g., diffusion models, LLM-based agents) in synthesizing visually coherent image sequences from textual narratives and reference images.","zh":"ViStoryBench 是一个面向故事可视化任务的综合性评测基准，旨在评估多模态生成模型（如扩散模型视频生成模型等）根据给定叙事文本和参考图像生成视觉连贯且情节一致的图像序列的能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1993","languages":["English","Chinese"],"library_direction":null,"modality":"multimodal","name":"ViStoryBench","openness":"open","publisher":"Shanghai Tech University , StepFun , AIGC Research , AGI Lab, Westlake Universit","released":"2025-06-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1993-vistorybench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ViStoryBench","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"An evaluation suite of 1,340 video QA pairs across 15 subtasks assessing MLLM qualitative reasoning in dynamic scenes, covering motion perception, spatial relations, outcome prediction, and physical dynamics.","descriptions":{"en":"An evaluation suite of 1,340 video QA pairs across 15 subtasks assessing MLLM qualitative reasoning in dynamic scenes, covering motion perception, spatial relations, outcome prediction, and physical dynamics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20868","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding","world-models"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_vistr-bench_5dcce2bf"},"modality":null,"name":"ViSTR-Bench","openness":"unknown","publisher":null,"released":"2026-07-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20868","source_url":"https://arxiv.org/abs/2607.20868"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20868","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20868","topics":["reasoning-math"],"unit":null},{"aliases":["visual-rag-benchmark"],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:james-a-watson/visual-rag-benchmark","source_record":"bm_visual-rag-benchmark_df083a55"},"categories":["Finance","Information retrieval","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Visual RAG Benchmark evaluates end-to-end retrieval-augmented generation on reference tables from three EU Water Framework Directive guidance documents, scoring answers on 103 and 154 question sets split into text and table question types via LLM-as-a-judge correctness, ROUGE-1, and retrieval accuracy across seven indexing and retrieval pipelines.","descriptions":{"en":"Visual RAG Benchmark evaluates end-to-end retrieval-augmented generation on reference tables from three EU Water Framework Directive guidance documents, scoring answers on 103 and 154 question sets split into text and table question types via LLM-as-a-judge correctness, ROUGE-1, and retrieval accuracy across seven indexing and retrieval pipelines."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:james-a-watson/visual-rag-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["deep-research","computer-use","document-ocr"],"join":"claire_record_id","primary":"deep-research","source_record":"bm_visual-rag-benchmark_df083a55"},"modality":null,"name":"Visual RAG Benchmark","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:james-a-watson/visual-rag-benchmark","source_url":"https://github.com/james-a-watson/visual-rag-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-james-a-watson-visual-rag-benchmark","source":"claire_radar","source_url":"https://github.com/james-a-watson/visual-rag-benchmark","topics":["finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:didizhu-judy/visualflip","source_record":"bm_visualflip_556bf533"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"VisualFLIP evaluates multimodal LLMs on visual reasoning with 1,374 images in paired perturbation tasks. Each pair has a fixed question but minimally changed visual evidence so the answer flips. Scoring uses pair accuracy and Collapse Rate to test evidence dependence in capabilities like cardinality, attribute, spatial, and logic.","descriptions":{"en":"VisualFLIP evaluates multimodal LLMs on visual reasoning with 1,374 images in paired perturbation tasks. Each pair has a fixed question but minimally changed visual evidence so the answer flips. Scoring uses pair accuracy and Collapse Rate to test evidence dependence in capabilities like cardinality, attribute, spatial, and logic."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07872","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_visualflip_556bf533"},"modality":null,"name":"VisualFLIP","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07872","source_url":"https://arxiv.org/abs/2606.07872"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07872","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07872","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"VisualLeakBench is a 500-image benchmark for evaluating action-boundary propagation failures in vision-language agents, with stratified subsets and oracle diagnostics.","descriptions":{"en":"VisualLeakBench is a 500-image benchmark for evaluating action-boundary propagation failures in vision-language agents, with stratified subsets and oracle diagnostics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.07595","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","visual-reasoning"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_visualleakbench_6618ffcf"},"modality":null,"name":"VisualLeakBench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07595","source_url":"https://arxiv.org/abs/2606.07595"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07595","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07595","topics":["agents","multimodal","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["强推理","Strong Reasoning","多模态","Multimodal","推理","Reasoning","多模态模型","VLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VisualPuzzles is a benchmark that targets visual reasoning while deliberately minimizing reliance on specialized knowledge. VisualPuzzles consists of 1168 diverse questions spanning five categories: algorithmic, analogical, deductive, inductive, and spatial reasoning. LLM 能考公务员吗？我们做了个测试…\n\n近年来，大模型（LLM）的能力突飞猛进，似乎“越来越聪明”了。但有一个关键问题仍然摆在眼前：\n🤔 它们真的会“推理”吗？\n\n🚀 我们设计了一个名为 VisualPuzzles 🧩 的全新数据集，专门用来回答这个问题：\n脱离专业知识的支持，大模型能靠逻辑本身解题吗？\n我们从多个来源精心挑选或改编了 1168 道图文逻辑题，其中一个重要来源便是中国国家公务员考试行测中的逻辑推理题（没错，真·考公难度）🎯","descriptions":{"en":"VisualPuzzles is a benchmark that targets visual reasoning while deliberately minimizing reliance on specialized knowledge. VisualPuzzles consists of 1168 diverse questions spanning five categories: algorithmic, analogical, deductive, inductive, and spatial reasoning.","zh":"LLM 能考公务员吗？我们做了个测试…\n\n近年来，大模型（LLM）的能力突飞猛进，似乎“越来越聪明”了。但有一个关键问题仍然摆在眼前：\n🤔 它们真的会“推理”吗？\n\n🚀 我们设计了一个名为 VisualPuzzles 🧩 的全新数据集，专门用来回答这个问题：\n脱离专业知识的支持，大模型能靠逻辑本身解题吗？\n我们从多个来源精心挑选或改编了 1168 道图文逻辑题，其中一个重要来源便是中国国家公务员考试行测中的逻辑推理题（没错，真·考公难度）🎯"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1748","languages":[],"library_direction":null,"modality":"multimodal","name":"VisualPuzzles","openness":"unknown","publisher":"Carnegie Mellon University","released":"2025-04-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1748-visualpuzzles","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VisualPuzzles","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning","事实可靠性","Factual Reliability"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VisualSimpleQA is a multimodal fact-seeking benchmark with two key features. VisualSimpleQA 是一个多模态事实寻求基准，具有两个关键特性。首先，它使视觉和语言模态中 LVLMs 的评估更加简化和解耦。其次，它纳入了明确的难度标准，以指导人工标注并促进提取具有挑战性的子集，即 VisualSimpleQA-hard。","descriptions":{"en":"VisualSimpleQA is a multimodal fact-seeking benchmark with two key features.","zh":"VisualSimpleQA 是一个多模态事实寻求基准，具有两个关键特性。首先，它使视觉和语言模态中 LVLMs 的评估更加简化和解耦。其次，它纳入了明确的难度标准，以指导人工标注并促进提取具有挑战性的子集，即 VisualSimpleQA-hard。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":true,"key":"opencompass:1634","languages":[],"library_direction":null,"modality":"multimodal","name":"VisualSimpleQA","openness":"restricted","publisher":"Zhongguancun Laboratory, RUC, Tencent, etc.","released":"2025-03-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1634-visualsimpleqa","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VisualSimpleQA","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":68,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_e656cd850fb42bd5"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","frontend_development","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A multimodal benchmark designed to assess the capabilities of multimodal large language models (MLLMs) across web page understanding and grounding tasks. Comprises 7 tasks (captioning, webpage QA, heading OCR, element OCR, element grounding, action prediction, and action grounding) with 1.5K human-curated instances from 139 real websites across 87 sub-domains.","descriptions":{"en":"A multimodal benchmark designed to assess the capabilities of multimodal large language models (MLLMs) across web page understanding and grounding tasks. Comprises 7 tasks (captioning, webpage QA, heading OCR, element OCR, element grounding, action prediction, and action grounding) with 1.5K human-curated instances from 139 real websites across 87 sub-domains."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:visualwebbench:nova-lite","reported_at":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visualwebbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:visualwebbench","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","visual-reasoning"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_e656cd850fb42bd5"},"modality":"multimodal","name":"VisualWebBench","openness":"unknown","publisher":null,"released":"2024-04-09","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:visualwebbench","source_url":"https://arxiv.org/abs/2404.05955"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":79.7,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.797,"raw_min":0.777,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:visualwebbench:nova-pro","reported_date":"2024-11-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visualwebbench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-11-20","date_precision":"model_announcement","name":"Nova Pro"}],"slug":"llm-stats-visualwebbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visualwebbench?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":38,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":78,"join":"llm_stats_id","repository":null,"source_record":"catalog_f1add9362ddfe9cd"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VisuLogic evaluates logical reasoning capabilities in visual contexts.","descriptions":{"en":"VisuLogic evaluates logical reasoning capabilities in visual contexts."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:visulogic:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visulogic?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:visulogic","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","abstract-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_f1add9362ddfe9cd"},"modality":"multimodal","name":"VisuLogic","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":0.543,"display_multiplier":1,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.543,"raw_min":0.344,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:visulogic:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visulogic?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-visulogic","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visulogic?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":71,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"llm_stats_id","repository":"gh:meituan-longcat/vitabench-2.0","source_record":"bm_vitabench_d7ecb50f"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VITA-Bench evaluates AI agents on real-world virtual task automation, measuring their ability to complete complex multi-step tasks in simulated environments.","descriptions":{"en":"VITA-Bench evaluates AI agents on real-world virtual task automation, measuring their ability to complete complex multi-step tasks in simulated environments."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vita-bench:qwen3.5-397b-a17b","reported_at":"2026-02-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vita-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vita-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","tool-use"],"join":"llm_stats_id","primary":"agent-memory","source_record":"bm_vitabench_d7ecb50f"},"modality":"text","name":"VITA-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":49.7,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.497,"raw_min":0.22,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vita-bench:qwen3.5-397b-a17b","reported_date":"2026-02-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vita-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-vita-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vita-bench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":71,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":17,"join":"claire_record_id","repository":"gh:meituan-longcat/vitabench-2.0","source_record":"bm_vitabench_d7ecb50f"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"VitaBench 2.0 evaluates personalized and proactive agent behavior in long-term, multi-session user interactions across food delivery, in-store consumption, and online travel domains. Tasks are per-user sequences of subtasks requiring agents to infer, utilize, and update user preferences from fragmented interaction history, with an extensible memory interface for controlled comparison across memory architectures.","descriptions":{"en":"VitaBench 2.0 evaluates personalized and proactive agent behavior in long-term, multi-session user interactions across food delivery, in-store consumption, and online travel domains. Tasks are per-user sequences of subtasks requiring agents to infer, utilize, and update user preferences from fragmented interaction history, with an extensible memory interface for controlled comparison across memory architectures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.27141","languages":[],"library_direction":{"basis":"model-review","categories":["agent-memory","tool-use"],"join":"claire_record_id","primary":"agent-memory","source_record":"bm_vitabench_d7ecb50f"},"modality":null,"name":"VitaBench","openness":"unknown","publisher":null,"released":"2026-05-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.27141","source_url":"https://arxiv.org/abs/2605.27141"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-27141","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27141","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","视频理解","Video-Understanding","Visual Knowledge","Physics","Psychology","物理智能","Embodied AI","逻辑推理","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"尽管多模态大型语言模型（MLLMs）在识别物体方面已相当熟练，但它们往往缺乏对世界潜在物理和社会原则的直觉的、类似人类的理解。这种高级的、基于视觉的语义，我们称之为视觉知识，其构成了感知与推理之间的桥梁。为了系统地评估这种能力，我们提出了 VKnowU，一个包含 1,249 个视频、1,680 个问题的综合基准，涵盖了 8 种核心类型的视觉知识，既包括以世界为中心的（例如直觉物理），也包括以人类为中心的任务（例如主观意图）。 尽管多模态大型语言模型（MLLMs）在识别物体方面已相当熟练，但它们往往缺乏对世界潜在物理和社会原则的直觉的、类似人类的理解。这种高级的、基于视觉的语义，我们称之为视觉知识，其构成了感知与推理之间的桥梁。为了系统地评估这种能力，我们提出了 VKnowU，一个包含 1,249 个视频、1,680 个问题的综合基准，涵盖了 8 种核心类型的视觉知识，既包括以世界为中心的（例如直觉物理），也包括以人类为中心的任务（例如主观意图）。","descriptions":{"en":"尽管多模态大型语言模型（MLLMs）在识别物体方面已相当熟练，但它们往往缺乏对世界潜在物理和社会原则的直觉的、类似人类的理解。这种高级的、基于视觉的语义，我们称之为视觉知识，其构成了感知与推理之间的桥梁。为了系统地评估这种能力，我们提出了 VKnowU，一个包含 1,249 个视频、1,680 个问题的综合基准，涵盖了 8 种核心类型的视觉知识，既包括以世界为中心的（例如直觉物理），也包括以人类为中心的任务（例如主观意图）。","zh":"尽管多模态大型语言模型（MLLMs）在识别物体方面已相当熟练，但它们往往缺乏对世界潜在物理和社会原则的直觉的、类似人类的理解。这种高级的、基于视觉的语义，我们称之为视觉知识，其构成了感知与推理之间的桥梁。为了系统地评估这种能力，我们提出了 VKnowU，一个包含 1,249 个视频、1,680 个问题的综合基准，涵盖了 8 种核心类型的视觉知识，既包括以世界为中心的（例如直觉物理），也包括以人类为中心的任务（例如主观意图）。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:2388","languages":[],"library_direction":null,"modality":"multimodal","name":"VKnowU","openness":"restricted","publisher":"Shanghai AI Laboratory","released":"2025-11-25","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2388-vknowu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VKnowU","topics":["multimodal","reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":42,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:depth2world/vladbench","source_record":"catalog_a7ef41637e381635"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VLADBench is a vision-language autonomous-driving benchmark evaluating understanding of dynamic traffic scenes and participants.","descriptions":{"en":"VLADBench is a vision-language autonomous-driving benchmark evaluating understanding of dynamic traffic scenes and participants."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vladbench:qwen3.7-plus","reported_at":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vladbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vladbench","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","visual-reasoning"],"join":"llm_stats_id","primary":"embodied-robotics","source_record":"catalog_a7ef41637e381635"},"modality":"multimodal","name":"VLADBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":77.2,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.772,"raw_min":0.772,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vladbench:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vladbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-vladbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vladbench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:jiuyixu25/vlaquantbench","source_record":"bm_vlaquantbench_94c7d46c"},"categories":["Multimodal","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates post-training quantization of vision-language-action models by rolling out quantized policies on simulation benchmarks and recording per-episode success and diagnostic metrics.","descriptions":{"en":"Evaluates post-training quantization of vision-language-action models by rolling out quantized policies on simulation benchmarks and recording per-episode success and diagnostic metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.25376","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_vlaquantbench_94c7d46c"},"modality":null,"name":"VLAQuantBench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.25376","source_url":"https://arxiv.org/abs/2609.25376"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-25376","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25376","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VLM²-Bench is the first comprehensive benchmark that evaluates vision-language models' (VLMs) ability to visually link matching cues across multi-image sequences and videos. The benchmark consists of 9 subtasks with over 3,000 test cases. VLM²-Bench 是第一个全面评估视觉语言模型（VLMs）在多图像序列和视频中视觉链接匹配线索能力的基准。该基准包括 9 个子任务，超过 3000 个测试案例，旨在评估人类日常使用的根本视觉链接能力。","descriptions":{"en":"VLM²-Bench is the first comprehensive benchmark that evaluates vision-language models' (VLMs) ability to visually link matching cues across multi-image sequences and videos. The benchmark consists of 9 subtasks with over 3,000 test cases.","zh":"VLM²-Bench 是第一个全面评估视觉语言模型（VLMs）在多图像序列和视频中视觉链接匹配线索能力的基准。该基准包括 9 个子任务，超过 3000 个测试案例，旨在评估人类日常使用的根本视觉链接能力。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1541","languages":[],"library_direction":null,"modality":"multimodal","name":"VLM2-Bench","openness":"restricted","publisher":"HKUST; CMU; MIT","released":"2025-02-17","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1541-vlm2-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VLM2-Bench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":114,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":23,"join":"llm_stats_id","repository":null,"source_record":"catalog_bd6ed6044f5fa2e0"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VLMsAreBiased evaluates whether vision-language models rely on visual evidence or fall back on language priors when answering.","descriptions":{"en":"VLMsAreBiased evaluates whether vision-language models rely on visual evidence or fall back on language priors when answering."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vlmsarebiased:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsarebiased?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vlmsarebiased","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_bd6ed6044f5fa2e0"},"modality":"multimodal","name":"VLMsAreBiased","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":83.6,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.836,"raw_min":0.683,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vlmsarebiased:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsarebiased?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-vlmsarebiased","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsarebiased?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":144,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":84,"join":"llm_stats_id","repository":"gh:anguyen8/vision-llms-are-blind","source_record":"catalog_b5d612c1a1f2eede"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A vision-language benchmark that probes blind spots and brittle reasoning in multimodal models.","descriptions":{"en":"A vision-language benchmark that probes blind spots and brittle reasoning in multimodal models."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vlmsareblind:qwen3.5-122b-a10b","reported_at":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsareblind?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vlmsareblind","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_b5d612c1a1f2eede"},"modality":"multimodal","name":"VLMsAreBlind","openness":"unknown","publisher":null,"released":"2024-07-09","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Vision language models are blind: Failing to translate detailed visual features into words. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:vlmsareblind","source_url":"https://arxiv.org/abs/2407.06581"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":97.0,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.97,"raw_min":0.967,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vlmsareblind:qwen3.5-35b-a3b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsareblind?top_n=500"},"unit":null},"scored_models":[{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"}],"slug":"llm-stats-vlmsareblind","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsareblind?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates text-to-image models on their ability to generate images that convey visual metaphors, using a set of metaphor prompts and comparative scoring.","descriptions":{"en":"Evaluates text-to-image models on their ability to generate images that convey visual metaphors, using a set of metaphor prompts and comparative scoring."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.02502","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_vmetaphor-bench_19432c0c"},"modality":null,"name":"VMetaphor-Bench","openness":"unknown","publisher":null,"released":"2026-09-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.02502","source_url":"https://arxiv.org/abs/2609.02502"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-02502","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02502","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:vnish-global/vnish-global-operator-routing-benchmark","source_record":"bm_vnish-global-operator-routing-benchmark_1814945a"},"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"Tests multilingual AI assistant routing across 80 cases in ten locales, requiring correct owned-source routing, hard stops on missing evidence, and citation precision.","descriptions":{"en":"Tests multilingual AI assistant routing across 80 cases in ten locales, requiring correct owned-source routing, hard stops on missing evidence, and citation precision."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:vnish-global/vnish-global-operator-routing-benchmark","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_vnish-global-operator-routing-benchmark_1814945a"},"modality":null,"name":"VNISH GLOBAL Multilingual Operator Routing Benchmark","openness":"unknown","publisher":null,"released":"2026-08-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:vnish-global/vnish-global-operator-routing-benchmark","source_url":"https://github.com/vnish-global/vnish-global-operator-routing-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-vnish-global-vnish-global-operator-routing-benchmark","source":"claire_radar","source_url":"https://github.com/vnish-global/vnish-global-operator-routing-benchmark","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":171,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:yuangongnd/vocalsound","source_record":"catalog_5d8477303965671d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["audio"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A dataset for improving human vocal sounds recognition, containing over 21,000 crowdsourced recordings of laughter, sighs, coughs, throat clearing, sneezes, and sniffs from 3,365 unique subjects. Used for audio event classification and recognition of human non-speech vocalizations.","descriptions":{"en":"A dataset for improving human vocal sounds recognition, containing over 21,000 crowdsourced recordings of laughter, sighs, coughs, throat clearing, sneezes, and sniffs from 3,365 unique subjects. Used for audio event classification and recognition of human non-speech vocalizations."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vocalsound:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vocalsound?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vocalsound","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_5d8477303965671d"},"modality":"audio","name":"VocalSound","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":93.89999999999999,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.939,"raw_min":0.939,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vocalsound:qwen2.5-omni-7b","reported_date":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vocalsound?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-vocalsound","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vocalsound?top_n=500","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-14","github_stars":3,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:kadirnar/voice-agent-next","source_record":"bm_voice-agent-next_344f5e74"},"categories":["Agents","General AI","Speech & Audio"],"collected_at":"2026-10-14T09:44:51Z","description":"A framework for building real-time speech-to-speech voice agents with a built-in benchmark suite measuring latency.","descriptions":{"en":"A framework for building real-time speech-to-speech voice agents with a built-in benchmark suite measuring latency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:kadirnar/voice-agent-next","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["speech-audio"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_voice-agent-next_344f5e74"},"modality":null,"name":"voice-agent-next","openness":"unknown","publisher":null,"released":"2026-09-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:kadirnar/voice-agent-next","source_url":"https://github.com/kadirnar/voice-agent-next"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-kadirnar-voice-agent-next","source":"claire_radar","source_url":"https://github.com/kadirnar/voice-agent-next","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":403,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"llm_stats_id","repository":"gh:matthewcym/voicebench","source_record":"catalog_fb9ba53aee44d463"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","safety","speech_to_text","general","communication"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VoiceBench is the first benchmark designed to provide a multi-faceted evaluation of LLM-based voice assistants, evaluating capabilities including general knowledge, instruction-following, reasoning, and safety using both synthetic and real spoken instruction data with diverse speaker characteristics and environmental conditions.","descriptions":{"en":"VoiceBench is the first benchmark designed to provide a multi-faceted evaluation of LLM-based voice assistants, evaluating capabilities including general knowledge, instruction-following, reasoning, and safety using both synthetic and real spoken instruction data with diverse speaker characteristics and environmental conditions."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:voicebench-avg:qwen2.5-omni-7b","reported_at":"2025-03-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/voicebench-avg?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:voicebench-avg","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio"],"join":"llm_stats_id","primary":"speech-audio","source_record":"catalog_fb9ba53aee44d463"},"modality":"multimodal","name":"VoiceBench Avg","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":90.10000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.901,"raw_min":0.7412,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:voicebench-avg:inkling-small","reported_date":"2026-07-30","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/voicebench-avg?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling-Small"},{"date":"2025-03-27","date_precision":"model_announcement","name":"Qwen2.5-Omni-7B"}],"slug":"llm-stats-voicebench-avg","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/voicebench-avg?top_n=500","topics":["multimodal","reasoning-math","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_voxsumm_b7118a8e"},"categories":["General AI","Speech & Audio","cs.SD"],"collected_at":"2026-10-14T09:44:51Z","description":"VoxSumm evaluates joint summarization and translation of long-form spoken news. It comprises 10,045 BBC article-summary pairs across 24 languages and about 703 hours of speech, with scoring based on summarization and translation quality.","descriptions":{"en":"VoxSumm evaluates joint summarization and translation of long-form spoken news. It comprises 10,045 BBC article-summary pairs across 24 languages and about 703 hours of speech, with scoring based on summarization and translation quality."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.10359","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_voxsumm_b7118a8e"},"modality":null,"name":"VoxSumm","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10359","source_url":"https://arxiv.org/abs/2608.10359"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10359","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10359","topics":["language"],"unit":null},{"aliases":["VPRef: A Cross-Domain Benchmark for Referring Remote Sensing Image Segmentation"],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:quanweiliu/vpref","source_record":"bm_vpref-a-cross-domain-benchmark-for-referri_a0fee8f4"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates referring remote sensing image segmentation across visual and textual domain shifts using the Vaihingen-Potsdam Referring dataset with 46,972 triplets and a three-tier linguistic hierarchy.","descriptions":{"en":"Evaluates referring remote sensing image segmentation across visual and textual domain shifts using the Vaihingen-Potsdam Referring dataset with 46,972 triplets and a three-tier linguistic hierarchy."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.16486","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","spatial-3d"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_vpref-a-cross-domain-benchmark-for-referri_a0fee8f4"},"modality":null,"name":"VPRef","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.16486","source_url":"https://arxiv.org/abs/2609.16486"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-16486","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16486","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","image_to_text","healthcare","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VQA-RAD (Visual Question Answering in Radiology) is the first manually constructed dataset of medical visual question answering containing 3,515 clinically generated visual questions and answers about radiology images. The dataset includes questions created by clinical trainees on 315 radiology images from MedPix covering head, chest, and abdominal scans, designed to support AI development for medical image analysis and improve patient care.","descriptions":{"en":"VQA-RAD (Visual Question Answering in Radiology) is the first manually constructed dataset of medical visual question answering containing 3,515 clinically generated visual questions and answers about radiology images. The dataset includes questions created by clinical trainees on 315 radiology images from MedPix covering head, chest, and abdominal scans, designed to support AI development for medical image analysis and improve patient care."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vqa-rad:medgemma-4b-it","reported_at":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqa-rad?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vqa-rad","languages":[],"library_direction":{"basis":"model-review","categories":["health-medicine","visual-reasoning"],"join":"llm_stats_id","primary":"health-medicine","source_record":"catalog_6c0328a0f8c820fb"},"modality":"multimodal","name":"VQA-Rad","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":49.9,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.499,"raw_min":0.499,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vqa-rad:medgemma-4b-it","reported_date":"2025-05-20","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqa-rad?top_n=500"},"unit":null},"scored_models":[{"date":"2025-05-20","date_precision":"model_announcement","name":"MedGemma 4B IT"}],"slug":"llm-stats-vqa-rad","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqa-rad?top_n=500","topics":["multimodal","medical"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates 12 image preprocessing techniques for cloud VLM-based visual question answering across 3 VQA datasets and 4 commercial models, measuring accuracy, cost, and latency.","descriptions":{"en":"Evaluates 12 image preprocessing techniques for cloud VLM-based visual question answering across 3 VQA datasets and 4 commercial models, measuring accuracy, cost, and latency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07861","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_vqabench_22600a6b"},"modality":null,"name":"VQABench","openness":"unknown","publisher":null,"released":"2026-08-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07861","source_url":"https://arxiv.org/abs/2608.07861"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07861","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07861","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_d55072af4a2137af"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","image_to_text","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VQAv2 is a balanced Visual Question Answering dataset that addresses language bias by providing complementary images for each question, forcing models to rely on visual understanding rather than language priors. It contains approximately twice the number of image-question pairs compared to the original VQA dataset.","descriptions":{"en":"VQAv2 is a balanced Visual Question Answering dataset that addresses language bias by providing complementary images for each question, forcing models to rely on visual understanding rather than language priors. It contains approximately twice the number of image-question pairs compared to the original VQA dataset."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vqav2:pixtral-12b-2409","reported_at":"2024-09-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vqav2","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_d55072af4a2137af"},"modality":"multimodal","name":"VQAv2","openness":"unknown","publisher":null,"released":"2016-12-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:vqav2","source_url":"https://arxiv.org/abs/1612.00837"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":80.9,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.809,"raw_min":0.781,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vqav2:pixtral-large","reported_date":"2024-11-18","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 90B Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-17","date_precision":"model_announcement","name":"Pixtral-12B"}],"slug":"llm-stats-vqav2","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_e7a89b60e2a2e710"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","image_to_text","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VQA v2.0 (Visual Question Answering v2.0) is a balanced dataset designed to counter language priors in visual question answering. It consists of complementary image pairs where the same question yields different answers, forcing models to rely on visual understanding rather than language bias. The dataset contains 1,105,904 questions across 204,721 COCO images, requiring understanding of vision, language, and commonsense knowledge.","descriptions":{"en":"VQA v2.0 (Visual Question Answering v2.0) is a balanced dataset designed to counter language priors in visual question answering. It consists of complementary image pairs where the same question yields different answers, forcing models to rely on visual understanding rather than language bias. The dataset contains 1,105,904 questions across 204,721 COCO images, requiring understanding of vision, language, and commonsense knowledge."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vqav2-(test):llama-3.2-11b-instruct","reported_at":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28test%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vqav2-(test)","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_e7a89b60e2a2e710"},"modality":"multimodal","name":"VQAv2 (test)","openness":"unknown","publisher":null,"released":"2016-12-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:vqav2-(test)","source_url":"https://arxiv.org/abs/1612.00837"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":75.2,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.752,"raw_min":0.752,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vqav2-(test):llama-3.2-11b-instruct","reported_date":"2024-09-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28test%29?top_n=500"},"unit":null},"scored_models":[{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 11B Instruct"}],"slug":"llm-stats-vqav2-test","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28test%29?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_afc35271db09b185"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","image_to_text","language","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"VQAv2 is a balanced Visual Question Answering dataset containing open-ended questions about images that require understanding of vision, language, and commonsense knowledge to answer. VQAv2 addresses bias issues from the original VQA dataset by collecting complementary images such that every question is associated with similar images that result in different answers, forcing models to actually understand visual content rather than relying on language priors.","descriptions":{"en":"VQAv2 is a balanced Visual Question Answering dataset containing open-ended questions about images that require understanding of vision, language, and commonsense knowledge to answer. VQAv2 addresses bias issues from the original VQA dataset by collecting complementary images such that every question is associated with similar images that result in different answers, forcing models to actually understand visual content rather than relying on language priors."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:vqav2-(val):gemma-3-12b-it","reported_at":"2025-03-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28val%29?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:vqav2-(val)","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_afc35271db09b185"},"modality":"multimodal","name":"VQAv2 (val)","openness":"unknown","publisher":null,"released":"2016-12-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:vqav2-(val)","source_url":"https://arxiv.org/abs/1612.00837"},"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":71.6,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.716,"raw_min":0.624,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:vqav2-(val):gemma-3-12b-it","reported_date":"2025-03-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28val%29?top_n=500"},"unit":null},"scored_models":[{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"}],"slug":"llm-stats-vqav2-val","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28val%29?top_n=500","topics":["multimodal","reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","视频理解","Video-Understanding","多步推理","视频推理","多语种","多模态模型","VLM","逻辑推理","Video Understanding"],"collected_at":"2026-08-18T00:00:00+00:00","description":"VRBench is the first evaluation benchmark specifically designed to assess multi-step reasoning capabilities over long videos. The benchmark comprises 1,010 long videos with an average duration of 1.6 hours, covering 8 languages and 7 video types. It includes annotations for 9,468 reasoning steps. VRBench是首个专门针对长视频多步推理能力设计的评测基准VRBench。该基准包含 1010 个平均时长1.6小时的长视频，覆盖 8 种语言和 7 种视频类型，标注了 9468 个多步推理问答对和超过 3 万个详细推理步骤。本评测集能够同时对模型的推理过程和推理结果进行评分，是首个同时具备多步推理链标注和评测能力的视频评测集。","descriptions":{"en":"VRBench is the first evaluation benchmark specifically designed to assess multi-step reasoning capabilities over long videos. The benchmark comprises 1,010 long videos with an average duration of 1.6 hours, covering 8 languages and 7 video types. It includes annotations for 9,468 reasoning steps.","zh":"VRBench是首个专门针对长视频多步推理能力设计的评测基准VRBench。该基准包含 1010 个平均时长1.6小时的长视频，覆盖 8 种语言和 7 种视频类型，标注了 9468 个多步推理问答对和超过 3 万个详细推理步骤。本评测集能够同时对模型的推理过程和推理结果进行评分，是首个同时具备多步推理链标注和评测能力的视频评测集。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:2383","languages":[],"library_direction":null,"modality":"multimodal","name":"VRBench","openness":"open","publisher":"上海人工智能实验室","released":"2025-06-12","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2383-vrbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VRBench","topics":["multimodal","reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates agents on computer control tasks under finite trial budgets.","descriptions":{"en":"Evaluates agents on computer control tasks under finite trial budgets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.12404","languages":[],"library_direction":null,"modality":null,"name":"VRL-Bench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12404","source_url":"https://arxiv.org/abs/2609.12404"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12404","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12404","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Robustness","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"A large-scale dataset for visual speech recognition in Romanian, with 200 hours of video and annotations. It studies supervision quality, robustness under domain shift, and multimodal fusion.","descriptions":{"en":"A large-scale dataset for visual speech recognition in Romanian, with 200 hours of video and annotations. It studies supervision quality, robustness under domain shift, and multimodal fusion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.08112","languages":[],"library_direction":{"basis":"model-review","categories":["speech-audio","multilingual","video-understanding"],"join":"claire_record_id","primary":"speech-audio","source_record":"bm_vsro-200_f4255395"},"modality":null,"name":"VSRo-200","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08112","source_url":"https://arxiv.org/abs/2607.08112"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08112","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08112","topics":["multimodal","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"VTC-Bench evaluates multiple LLM generations across five domains using Validated Task Coverage (VTC), which measures the number of distinct useful results obtained within k attempts. Tasks are selected from real data where output quality and task-relevant distinctness can be checked automatically without model-based judges.","descriptions":{"en":"VTC-Bench evaluates multiple LLM generations across five domains using Validated Task Coverage (VTC), which measures the number of distinct useful results obtained within k attempts. Tasks are selected from real data where output quality and task-relevant distinctness can be checked automatically without model-based judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.24228","languages":[],"library_direction":null,"modality":null,"name":"VTC-Bench","openness":"unknown","publisher":null,"released":"2026-08-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.24228","source_url":"https://arxiv.org/abs/2608.24228"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-24228","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24228","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code & Software","Cybersecurity","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Vul4Py evaluates automated vulnerability repair in Python across 100 real vulnerabilities from 60 open-source projects, with paired exploit and functional oracles.","descriptions":{"en":"Vul4Py evaluates automated vulnerability repair in Python across 100 real vulnerabilities from 60 open-source projects, with paired exploit and functional oracles."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00692","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","code-generation"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_vul4py_70948b69"},"modality":null,"name":"Vul4Py","openness":"unknown","publisher":null,"released":"2026-08-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00692","source_url":"https://arxiv.org/abs/2608.00692"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00692","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00692","topics":["coding","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code generation","Cybersecurity","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"VULBENCH-CPP evaluates the safety of AI-generated C++ code using multi-tier verification including functional testing, static analysis, dynamic analysis, and bounded model checking. It includes 8,918 programs from three LLMs and human authors.","descriptions":{"en":"VULBENCH-CPP evaluates the safety of AI-generated C++ code using multi-tier verification including functional testing, static analysis, dynamic analysis, and bounded model checking. It includes 8,918 programs from three LLMs and human authors."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.00107","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","code-generation"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_vulbench-cpp_84620b34"},"modality":null,"name":"VULBENCH-CPP","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00107","source_url":"https://arxiv.org/abs/2607.00107"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00107","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00107","topics":["coding","safety","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","Software & AI Compute","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark of 500 real-world vulnerabilities requiring models to locate security-relevant files in repository snapshots, also testing appropriate abstention on patched versions.","descriptions":{"en":"A benchmark of 500 real-world vulnerabilities requiring models to locate security-relevant files in repository snapshots, also testing appropriate abstention on patched versions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.15939","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","agentic-coding"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_vulnerability-localization-benchmark_9c6a0320"},"modality":null,"name":"Vulnerability Localization Benchmark","openness":"unknown","publisher":null,"released":"2026-09-14","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.15939","source_url":"https://arxiv.org/abs/2609.15939"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-15939","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15939","topics":["coding","agents","cybersecurity"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-09-25T04:40:22Z","description":"A safety benchmark for image-to-video systems exposed to categorized attacks encoded through visual prompts.","descriptions":{"en":"A safety benchmark for image-to-video systems exposed to categorized attacks encoded through visual prompts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.25592","languages":[],"library_direction":null,"modality":null,"name":"VVA-Bench","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.25592","source_url":"https://arxiv.org/abs/2606.25592"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-25592","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25592","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:hunter-will/vvm-tuning","source_record":"bm_vvm-bench_539536ab"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"VVM-Bench evaluates Large Multimodal Models on semantic perception and modality understanding across six real and synthetic modalities, using multiple-choice questions and generation tasks to assess zero-shot generalization to unseen visual modalities.","descriptions":{"en":"VVM-Bench evaluates Large Multimodal Models on semantic perception and modality understanding across six real and synthetic modalities, using multiple-choice questions and generation tasks to assess zero-shot generalization to unseen visual modalities."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.10308","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_vvm-bench_539536ab"},"modality":null,"name":"VVM-Bench","openness":"unknown","publisher":null,"released":"2026-07-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.10308","source_url":"https://arxiv.org/abs/2607.10308"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-10308","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10308","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_vvrbench_bac45a60"},"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates image generation models on 10,000 synthetic scene tasks with 32 verifiable constraint types measuring precise visual instruction following.","descriptions":{"en":"Evaluates image generation models on 10,000 synthetic scene tasks with 32 verifiable constraint types measuring precise visual instruction following."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35641","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_vvrbench_bac45a60"},"modality":null,"name":"VVRBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35641","source_url":"https://arxiv.org/abs/2609.35641"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35641","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35641","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":603,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_vwg-bench_fa3de0ff"},"categories":["General AI","Reasoning","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"VWG-Bench evaluates image-to-video generation models on rule-following, physical and commonsense constraints, and goal realization over time, using 380 annotated initial images across 9 reasoning dimensions and 38 ten-sample task groups, scored with bundled 1–5 metrics and an evaluation toolkit.","descriptions":{"en":"VWG-Bench evaluates image-to-video generation models on rule-following, physical and commonsense constraints, and goal realization over time, using 380 annotated initial images across 9 reasoning dimensions and 38 ten-sample task groups, scored with bundled 1–5 metrics and an evaluation toolkit."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.11242","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","world-models"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_vwg-bench_fa3de0ff"},"modality":null,"name":"VWG-Bench","openness":"unknown","publisher":null,"released":"2026-09-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.11242","source_url":"https://arxiv.org/abs/2609.11242"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-11242","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11242","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"A reasoning-annotated benchmark for floating waste detection and grounding, with 2,167 images, 13,608 bounding boxes, and ten categories, each with recognition rules.","descriptions":{"en":"A reasoning-annotated benchmark for floating waste detection and grounding, with 2,167 images, 13,608 bounding boxes, and ten categories, each with recognition rules."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22950","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_wade_93cc3c8f"},"modality":null,"name":"WADE","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22950","source_url":"http://arxiv.org/abs/2608.22950v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22950","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22950v1","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:tstanczyk95/waspmot","source_record":"bm_waspmot_953ea5b1"},"categories":["Software & AI Compute","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for long-term multi-object tracking of Trichogramma wasps in controlled ecological experiments, with 10 sequences of ~12,000 frames each and dense annotations. It evaluates identity preservation over extended durations.","descriptions":{"en":"A benchmark for long-term multi-object tracking of Trichogramma wasps in controlled ecological experiments, with 10 sequences of ~12,000 frames each and dense annotations. It evaluates identity preservation over extended durations."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.08729","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_waspmot_953ea5b1"},"modality":null,"name":"WaspMOT","openness":"unknown","publisher":null,"released":"2026-07-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.08729","source_url":"https://arxiv.org/abs/2607.08729"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-08729","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08729","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:baiqi-li/watchact","source_record":"bm_watchact_107de5f7"},"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"WatchAct evaluates robot manipulation from observed human behavior. Each instance pairs a human-action video and a language instruction with an aligned simulator scene and an executable LIBERO task. It covers 3,000 long-horizon instances across 14 tasks in four capability domains: Event Grounding, Procedural Reasoning, Implicit Intent Inference, and Episodic Reasoning. The evaluation protocol separately measures video-to-plan reasoning, policy execution under oracle plans, and full task completion, in simulation and on a Franka Research 3 robot.","descriptions":{"en":"WatchAct evaluates robot manipulation from observed human behavior. Each instance pairs a human-action video and a language instruction with an aligned simulator scene and an executable LIBERO task. It covers 3,000 long-horizon instances across 14 tasks in four capability domains: Event Grounding, Procedural Reasoning, Implicit Intent Inference, and Episodic Reasoning. The evaluation protocol separately measures video-to-plan reasoning, policy execution under oracle plans, and full task completion, in simulation and on a Franka Research 3 robot."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.26443","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","video-understanding"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_watchact_107de5f7"},"modality":null,"name":"WatchAct","openness":"unknown","publisher":null,"released":"2026-06-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.26443","source_url":"https://arxiv.org/abs/2606.26443"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-26443","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26443","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Language & Knowledge","Reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"WaveformQA is a QA benchmark for LLM temporal reasoning over digital waveforms, comprising 360 questions with programmatically generated ground truths across eight categories, including multi-signal correlation and event ordering, with waveforms generated from open-source designs.","descriptions":{"en":"WaveformQA is a QA benchmark for LLM temporal reasoning over digital waveforms, comprising 360 questions with programmatically generated ground truths across eight categories, including multi-signal correlation and event ordering, with waveforms generated from open-source designs."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.20638","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_waveformqa_f89d742c"},"modality":null,"name":"WaveformQA","openness":"unknown","publisher":null,"released":"2026-07-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.20638","source_url":"https://arxiv.org/abs/2607.20638"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-20638","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20638","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":249,"github_stars_30d":18,"hf_dataset_downloads":null,"hf_paper_upvotes":81,"join":"claire_record_id","repository":"gh:meituan-longcat/wbench","source_record":"bm_wbench_de4d593c"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"WBench evaluates interactive video world models across five dimensions: video quality, setting adherence, interaction adherence, consistency, and physics compliance. It includes 289 test cases and 1,058 multi-turn interaction sequences, covering diverse scenes and control types, with 22 automatic sub-metrics validated against human judgment.","descriptions":{"en":"WBench evaluates interactive video world models across five dimensions: video quality, setting adherence, interaction adherence, consistency, and physics compliance. It includes 289 test cases and 1,058 multi-turn interaction sequences, covering diverse scenes and control types, with 22 automatic sub-metrics validated against human judgment."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.25874","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_wbench_de4d593c"},"modality":null,"name":"WBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.25874","source_url":"https://arxiv.org/abs/2605.25874"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-25874","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25874","topics":[],"unit":null},{"aliases":["FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches"],"attention_signals":null,"categories":["Agents & Tool Use","Finance","cs.LG"],"collected_at":"2026-09-25T04:40:22Z","description":"WC2026-Agents evaluates autonomous LLM forecasting on 104 FIFA World Cup matches using probability forecasts, virtual wagers, and post-match reflections.","descriptions":{"en":"WC2026-Agents evaluates autonomous LLM forecasting on 104 FIFA World Cup matches using probability forecasts, virtual wagers, and post-match reflections."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.17765","languages":[],"library_direction":null,"modality":null,"name":"WC2026-Agents","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.17765","source_url":"https://arxiv.org/abs/2607.17765"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-17765","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17765","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":134,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":81,"join":"llm_stats_id","repository":null,"source_record":"catalog_02608907f64603da"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["math","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"We-Math evaluates multimodal models on visual mathematical reasoning, requiring models to understand and solve math problems presented with visual elements such as diagrams, charts, and geometric figures.","descriptions":{"en":"We-Math evaluates multimodal models on visual mathematical reasoning, requiring models to understand and solve math problems presented with visual elements such as diagrams, charts, and geometric figures."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:we-math:qwen3.6-plus","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/we-math?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:we-math","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","math"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_02608907f64603da"},"modality":"multimodal","name":"We-Math","openness":"unknown","publisher":null,"released":"2024-07-01","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:we-math","source_url":"https://arxiv.org/abs/2407.01284"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":89.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.89,"raw_min":0.89,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:we-math:qwen3.6-plus","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/we-math?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"}],"slug":"llm-stats-we-math","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/we-math?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"WearableQA evaluates AI reasoning over longitudinal wearable sensor data. It comprises 4,084 multiple-choice questions derived from 200 real users' wearable time series, blood biomarkers, and demographics. Questions are organized into 16 types along data versus health reasoning and single- versus cross-signal axes. Scoring uses multiple-choice accuracy with a known chance baseline.","descriptions":{"en":"WearableQA evaluates AI reasoning over longitudinal wearable sensor data. It comprises 4,084 multiple-choice questions derived from 200 real users' wearable time series, blood biomarkers, and demographics. Questions are organized into 16 types along data versus health reasoning and single- versus cross-signal axes. Scoring uses multiple-choice accuracy with a known chance baseline."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.05405","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["knowledge-expert-qa","forecasting","health-medicine"],"join":"claire_record_id","primary":"knowledge-expert-qa","source_record":"bm_wearableqa_b2b26908"},"modality":null,"name":"WearableQA","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05405","source_url":"https://arxiv.org/abs/2609.05405"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05405","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05405","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":170,"github_stars_30d":7,"hf_dataset_downloads":1940,"hf_paper_upvotes":49,"join":"claire_record_id","repository":"gh:weavebench/weavebench","source_record":"bm_weavebench_27cb863c"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"WeaveBench evaluates computer-use agents on 114 long-horizon tasks across 8 real-world work domains. Each task interleaves GUI interaction with command-line and code operations in a single trajectory, with scoring based on a trajectory-aware judge that detects fabricated evidence.","descriptions":{"en":"WeaveBench evaluates computer-use agents on 114 long-horizon tasks across 8 real-world work domains. Each task interleaves GUI interaction with command-line and code operations in a single trajectory, with scoring based on a trajectory-aware judge that detects fabricated evidence."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.09426","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","professional-work","agentic-coding"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_weavebench_27cb863c"},"modality":null,"name":"WeaveBench","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.09426","source_url":"https://arxiv.org/abs/2606.09426"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-09426","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09426","topics":["coding","computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":294,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_013cc061eb74a965"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","coding"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Web Bench evaluates agents on realistic web-development engineering tasks, measuring end-to-end implementation in browser-based workflows.","descriptions":{"en":"Web Bench evaluates agents on realistic web-development engineering tasks, measuring end-to-end implementation in browser-based workflows."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:web-bench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/web-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:web-bench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_013cc061eb74a965"},"modality":"text","name":"Web Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":78.4,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.784,"raw_min":0.736,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:web-bench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/web-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-web-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/web-bench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-12","github_stars":1626,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:web-arena-x/webarena","source_record":"lib_webarena_verified"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WebArena-Verified evaluates browser agents on realistic web tasks using a verified task set and execution-based grading.","descriptions":{"en":"WebArena-Verified evaluates browser agents on realistic web tasks using a verified task set and execution-based grading."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:webarena-verified:qwen3.8-27b","reported_at":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webarena-verified?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:webarena-verified","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"lib_webarena_verified"},"modality":"multimodal","name":"WebArena-Verified","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":64.8,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.648,"raw_min":0.648,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:webarena-verified:qwen3.8-27b","reported_date":"2026-08-14","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webarena-verified?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8-27B"}],"slug":"llm-stats-webarena-verified","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webarena-verified?top_n=500","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","frontend_development","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WebDev Arena is a leaderboard for evaluating AI models on web development tasks, including zero-shot generation, complex prompts, and interactive web UI creation. Models are ranked using Elo ratings based on their performance in coding and web development challenges.","descriptions":{"en":"WebDev Arena is a leaderboard for evaluating AI models on web development tasks, including zero-shot generation, complex prompts, and interactive web UI creation. Models are ranked using Elo ratings based on their performance in coding and web development challenges."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:webdev-arena:gemini-3.7-flash","reported_at":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webdev-arena?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:webdev-arena","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","chat-writing"],"join":"llm_stats_id","primary":"code-generation","source_record":"catalog_161771f831374fa6"},"modality":"text","name":"WebDev Arena","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":1588.0,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":1588.0,"raw_min":1588.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:webdev-arena:gemini-3.7-flash","reported_date":"2026-08-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webdev-arena?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash"}],"slug":"llm-stats-webdev-arena","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webdev-arena?top_n=500","topics":["coding","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for evaluating agent skills in web development, comparing matched conditions with length-matched controls and ablation studies on 31 skills across 50 projects.","descriptions":{"en":"A benchmark for evaluating agent skills in web development, comparing matched conditions with length-matched controls and ablation studies on 31 skills across 50 projects."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.23067","languages":[],"library_direction":{"basis":"model-review","categories":["agent-skills-evolution","agentic-coding"],"join":"claire_record_id","primary":"agent-skills-evolution","source_record":"bm_webdev-skills-bench_00be1a53"},"modality":null,"name":"WebDev-Skills-Bench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.23067","source_url":"http://arxiv.org/abs/2608.23067v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-23067","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23067v1","topics":["coding","agents"],"unit":null},{"aliases":["WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing"],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_weagent-mmgenedit-a-full-stack-recipe-for-_1dc7d757"},"categories":["Agents","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal agentic image generation and editing, focusing on knowledge-intensive generation and multi-image editing across bilingual prompts.","descriptions":{"en":"Evaluates multimodal agentic image generation and editing, focusing on knowledge-intensive generation and multi-image editing across bilingual prompts."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.05171","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_weagent-mmgenedit-a-full-stack-recipe-for-_1dc7d757"},"modality":null,"name":"WeBench-MMGenEdit","openness":"unknown","publisher":null,"released":"2026-09-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.05171","source_url":"https://arxiv.org/abs/2609.05171"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-05171","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05171","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Code","Code generation","Multimodal","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"WebIGBench evaluates code generation for interactive webpages with 103 complex examples and 871 distinct actions, proposing an automated evaluation pipeline for interactive consistency.","descriptions":{"en":"WebIGBench evaluates code generation for interactive webpages with 103 complex examples and 871 distinct actions, proposing an automated evaluation pipeline for interactive consistency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.00154","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_webigbench_7eee6092"},"modality":null,"name":"WebIGBench","openness":"unknown","publisher":null,"released":"2026-05-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.00154","source_url":"https://arxiv.org/abs/2606.00154"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-00154","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00154","topics":["coding","multimodal"],"unit":null},{"aliases":["Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents"],"attention_signals":null,"categories":["Agents & Tool Use","General AI","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"Offline benchmark with 541 tasks and 5,293 steps derived from successful WebArena trajectories, with audited test labels and a deterministic protocol.","descriptions":{"en":"Offline benchmark with 541 tasks and 5,293 steps derived from successful WebArena trajectories, with audited test labels and a deterministic protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27353","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_guides-that-cause-actions-an-offline-study_c4d01dab"},"modality":null,"name":"WebMRE","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27353","source_url":"https://arxiv.org/abs/2609.27353"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27353","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27353","topics":["agents","multimodal"],"unit":null},{"aliases":["WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"An open framework for evaluating web agents, with 152 tasks across six instrumented mock sites, using event-log verification and controlled UI variants.","descriptions":{"en":"An open framework for evaluating web agents, with 152 tasks across six instrumented mock sites, using event-log verification and controlled UI variants."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.35026","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_webpagebench-event-level-verification-and-_cc9a0cb1"},"modality":null,"name":"WebPageBench","openness":"unknown","publisher":null,"released":"2026-09-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.35026","source_url":"https://arxiv.org/abs/2609.35026"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-35026","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35026","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_webretriever_b067311d"},"categories":["Agents","General AI","Vision & 3D"],"collected_at":"2026-10-14T09:44:51Z","description":"Introduces a benchmark with 800 websites and 1,550 tasks for web agent evaluation, plus the NavEval LLM-as-Judge framework and three evaluation protocols.","descriptions":{"en":"Introduces a benchmark with 800 websites and 1,550 tasks for web agent evaluation, plus the NavEval LLM-as-Judge framework and three evaluation protocols."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.06118","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","deep-research"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_webretriever_b067311d"},"modality":null,"name":"WebRetriever","openness":"unknown","publisher":null,"released":"2026-07-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.06118","source_url":"https://arxiv.org/abs/2607.06118"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-06118","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06118","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","智能体","Agent","任务执行","Task Execution","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WebUIBench is a comprehensive benchmark designed to evaluate Multimodal Large Language Models (MLLMs) in Web UI-to-code generation tasks across four key capabilities: UI perception, HTML programming, UI-code understanding, and end-to-end transformation. WebUIBench 是一个面向多模态大语言模型（MLLMs）的综合性评测基准，旨在系统评估模型在 Web UI 到代码生成任务中的四个关键能力：界面感知、HTML 编程、界面-代码理解以及整体转换能力。该基准包含来自 700 多个真实网站的 21,000 个高质量问答对，支持对模型在各阶段的细粒度能力分析。","descriptions":{"en":"WebUIBench is a comprehensive benchmark designed to evaluate Multimodal Large Language Models (MLLMs) in Web UI-to-code generation tasks across four key capabilities: UI perception, HTML programming, UI-code understanding, and end-to-end transformation.","zh":"WebUIBench 是一个面向多模态大语言模型（MLLMs）的综合性评测基准，旨在系统评估模型在 Web UI 到代码生成任务中的四个关键能力：界面感知、HTML 编程、界面-代码理解以及整体转换能力。该基准包含来自 700 多个真实网站的 21,000 个高质量问答对，支持对模型在各阶段的细粒度能力分析。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1982","languages":[],"library_direction":null,"modality":null,"name":"WebUI-Bench","openness":"open","publisher":"The Chinese University of HongKong,HongKong SAR,China ,etc.","released":"2025-06-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1982-webui-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WebUI-Bench","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Agents & Tool Use","Geometric reasoning","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that mis…","descriptions":{"en":"Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that mis…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.02617","languages":[],"library_direction":{"basis":"model-review-daily","categories":["code-generation","computer-use"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_webuiproof_ffd9c399"},"modality":null,"name":"WebUIProof","openness":"unknown","publisher":null,"released":"2026-10-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.02617","source_url":"https://arxiv.org/abs/2610.02617"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-02617","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02617","topics":["coding","computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":1127,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":32,"join":"llm_stats_id","repository":null,"source_record":"catalog_7532f2cb612216a0"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WebVoyager evaluates an agent's ability to navigate and complete tasks on real websites by perceiving page screenshots and executing browser actions.","descriptions":{"en":"WebVoyager evaluates an agent's ability to navigate and complete tasks on real websites by perceiving page screenshots and executing browser actions."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:webvoyager:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webvoyager?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:webvoyager","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use"],"join":"llm_stats_id","primary":"computer-use","source_record":"catalog_7532f2cb612216a0"},"modality":"multimodal","name":"WebVoyager","openness":"unknown","publisher":null,"released":"2024-01-25","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:webvoyager","source_url":"https://arxiv.org/abs/2401.13919"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":88.5,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.885,"raw_min":0.885,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:webvoyager:glm-5v-turbo","reported_date":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webvoyager?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"}],"slug":"llm-stats-webvoyager","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webvoyager?top_n=500","topics":["computer-use","agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":8,"join":"claire_record_id","repository":"gh:kingofspace0wzz/weclawarena","source_record":"bm_weclawarena_f743496b"},"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"WeClawArena is a benchmark and runtime sandbox for multi-party owned-agent collaboration over personal workspaces. It contains 124 base tasks across six domains, expanded into 620 scenario variants with benign and attack-vector conditions, and reports task success and attack success separately.","descriptions":{"en":"WeClawArena is a benchmark and runtime sandbox for multi-party owned-agent collaboration over personal workspaces. It contains 124 base tasks across six domains, expanded into 620 scenario variants with benign and attack-vector conditions, and reports task success and attack success separately."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.03499","languages":[],"library_direction":{"basis":"model-review","categories":["multi-agent-games","professional-work"],"join":"claire_record_id","primary":"multi-agent-games","source_record":"bm_weclawarena_f743496b"},"modality":null,"name":"WeClawArena","openness":"unknown","publisher":null,"released":"2026-08-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.03499","source_url":"https://arxiv.org/abs/2608.03499"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-03499","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03499","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_wegenbench_4908a1e1"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"WeGenBench evaluates text-to-image generation across 4,000 bilingual prompts, with multi-dimensional tags and novel VLM-based metrics. It assesses generation quality on scene classification and specific sub-categories.","descriptions":{"en":"WeGenBench evaluates text-to-image generation across 4,000 bilingual prompts, with multi-dimensional tags and novel VLM-based metrics. It assesses generation quality on scene classification and specific sub-categories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.20100","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation","multilingual"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_wegenbench_4908a1e1"},"modality":null,"name":"WeGenBench","openness":"unknown","publisher":null,"released":"2026-06-18","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.20100","source_url":"https://arxiv.org/abs/2606.20100"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-20100","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20100","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Cybersecurity","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"WeSCE is a benchmark for quantifying security drift in LLM-driven code editing. It consists of 400 executable programs derived from real-world code, covering feature addition, removal, bug fixing, and refactoring. The benchmark proposes a continuous risk representation and drift measures that capture changes in overall risk, worst-case severity, and vulnerability distribution.","descriptions":{"en":"WeSCE is a benchmark for quantifying security drift in LLM-driven code editing. It consists of 400 executable programs derived from real-world code, covering feature addition, removal, bug fixing, and refactoring. The benchmark proposes a continuous risk representation and drift measures that capture changes in overall risk, worst-case severity, and vulnerability distribution."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.15092","languages":[],"library_direction":{"basis":"model-review","categories":["cybersecurity","code-generation"],"join":"claire_record_id","primary":"cybersecurity","source_record":"bm_wesce_79f8889e"},"modality":null,"name":"WeSCE","openness":"unknown","publisher":null,"released":"2026-08-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15092","source_url":"https://arxiv.org/abs/2608.15092"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15092","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15092","topics":["coding","cybersecurity"],"unit":null},{"aliases":["The Illusion of $\\textit{What If}$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs"],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:zju-gt/whatifbench","source_record":"bm_the-illusion-of-textit-what-if-evaluating-_2b6da5c9"},"categories":["General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"WhatIfBench is an open-domain, open-form, long-horizon counterfactual causal reasoning benchmark with 220 what-if questions across STEM, HSS, and Hybrid scenarios, evaluated using the PRISM framework.","descriptions":{"en":"WhatIfBench is an open-domain, open-form, long-horizon counterfactual causal reasoning benchmark with 220 what-if questions across STEM, HSS, and Hybrid scenarios, evaluated using the PRISM framework."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.27953","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_the-illusion-of-textit-what-if-evaluating-_2b6da5c9"},"modality":null,"name":"WhatIfBench","openness":"unknown","publisher":null,"released":"2026-08-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.27953","source_url":"https://arxiv.org/abs/2608.27953"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-27953","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27953","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:ethanning/whatworkedbench","source_record":"bm_whatworkedbench_d3b365df"},"categories":["Language & Knowledge","Software & AI Compute","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI agents on predicting component-change effects in 36 executable workflow tasks across 8 families using 30 data sources and 1,248 native outcomes.","descriptions":{"en":"Evaluates AI agents on predicting component-change effects in 36 executable workflow tasks across 8 families using 30 data sources and 1,248 native outcomes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:ethanning/whatworkedbench","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_whatworkedbench_d3b365df"},"modality":null,"name":"WhatWorkedBench","openness":"unknown","publisher":null,"released":"2026-09-22","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:ethanning/whatworkedbench","source_url":"https://github.com/EthanNing/WhatWorkedBench"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-ethanning-whatworkedbench","source":"claire_radar","source_url":"https://github.com/EthanNing/WhatWorkedBench","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_whisperbench_c407b81a"},"categories":["General AI","Language & Knowledge","cs.CR"],"collected_at":"2026-10-14T09:44:51Z","description":"WhisperBench evaluates stealth memory injection attacks on persistent personal agents through a 108-case benchmark spanning five risk categories with fact and preference poisoning, using an IMAP/SMTP workflow and an email agent skill.","descriptions":{"en":"WhisperBench evaluates stealth memory injection attacks on persistent personal agents through a 108-case benchmark spanning five risk categories with fact and preference poisoning, using an IMAP/SMTP workflow and an email agent skill."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.05189","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security","agent-memory"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_whisperbench_c407b81a"},"modality":null,"name":"WhisperBench","openness":"unknown","publisher":null,"released":"2026-07-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.05189","source_url":"https://arxiv.org/abs/2607.05189"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-05189","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05189","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_who-when-pro_d21cef2d"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Who&When Pro is a benchmark for automated failure attribution in agentic systems, containing 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks. It uses a controlled pipeline that injects failures after replaying successful prefixes.","descriptions":{"en":"Who&When Pro is a benchmark for automated failure attribution in agentic systems, containing 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks. It uses a controlled pipeline that injects failures after replaying successful prefixes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09996","languages":[],"library_direction":null,"modality":null,"name":"Who&When Pro","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09996","source_url":"https://arxiv.org/abs/2607.09996"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09996","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09996","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","NeurIPS 2024","任务执行","Task Execution"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WhodunitBench is used to evaluate large multimodal agent under complex tasks and dynamic scenarios. WhodunitBench用于评估大型多模式代理在复杂任务场景下的动态评估。","descriptions":{"en":"WhodunitBench is used to evaluate large multimodal agent under complex tasks and dynamic scenarios.","zh":"WhodunitBench用于评估大型多模式代理在复杂任务场景下的动态评估。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"opencompass:1279","languages":[],"library_direction":null,"modality":null,"name":"WhodunitBench","openness":"unknown","publisher":"The Chinese University of Hong Kong","released":"2024-09-26","released_reference":{"basis":"paper_first_version","note":"OpenReview's public publication date for the murder-mystery benchmark, before the later camera-ready modification.","source_key":"opencompass:1279","source_url":"https://openreview.net/forum?id=qmvtDIfbmS"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1279-whodunitbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WhodunitBench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WiC is a benchmark for the evaluation of context-sensitive word embeddings. WiC is framed as a binary classification task. Each instance in WiC has a target word w, either a verb or a noun, for which two contexts are provided. Each of these contexts triggers a specific meaning of w. The task is to identify if the occurrences of w in the two contexts correspond to the same meaning or not. In fact, the dataset can also be viewed as an application of Word Sense Disambiguation in practise. Word-in-Context是一个词义消歧任务，被视为句子对的二元分类。给定两个文本片段和一个在两个句子中都出现的多义词，任务是确定该词在两个句子中是否具有相同的含义。","descriptions":{"en":"WiC is a benchmark for the evaluation of context-sensitive word embeddings. WiC is framed as a binary classification task. Each instance in WiC has a target word w, either a verb or a noun, for which two contexts are provided. Each of these contexts triggers a specific meaning of w. The task is to identify if the occurrences of w in the two contexts correspond to the same meaning or not. In fact, the dataset can also be viewed as an application of Word Sense Disambiguation in practise.","zh":"Word-in-Context是一个词义消歧任务，被视为句子对的二元分类。给定两个文本片段和一个在两个句子中都出现的多义词，任务是确定该词在两个句子中是否具有相同的含义。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:504","languages":[],"library_direction":null,"modality":null,"name":"WiC","openness":"unknown","publisher":null,"released":"2018-08-28","released_reference":{"basis":"paper_first_version","note":"First version introducing Word-in-Context.","source_key":"opencompass:504","source_url":"https://arxiv.org/abs/1808.09121"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-504-wic","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WiC","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":155,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":113,"join":"llm_stats_id","repository":null,"source_record":"catalog_d91422f7df97f946"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","search","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WideSearch is an agentic search benchmark that evaluates models' ability to perform broad, parallel search operations across multiple sources. It tests wide-coverage information retrieval and synthesis capabilities.","descriptions":{"en":"WideSearch is an agentic search benchmark that evaluates models' ability to perform broad, parallel search operations across multiple sources. It tests wide-coverage information retrieval and synthesis capabilities."},"evidence_summary":{"document_count":1,"model_count":10,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:widesearch:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/widesearch?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:widesearch","languages":[],"library_direction":{"basis":"model-review","categories":["deep-research"],"join":"llm_stats_id","primary":"deep-research","source_record":"catalog_d91422f7df97f946"},"modality":"text","name":"WideSearch","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":10,"score_direction":"higher_is_better","score_summary":{"display_max":81.89999999999999,"display_multiplier":100,"model_count":10,"model_count_basis":"source_model_id","numeric_count":10,"raw_max":0.819,"raw_min":0.571,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:widesearch:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/widesearch?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"}],"slug":"llm-stats-widesearch","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/widesearch?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["WideSearch"],"attention_signals":null,"categories":["agent"],"collected_at":null,"description":"Wide-coverage web collection; scored on completeness of an enumerated answer set.","descriptions":{"en":"Wide-coverage web collection; scored on completeness of an enumerated answer set."},"evidence_summary":{"document_count":2,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000widesearch\u0000qwen3_5_model_card\u0000widesearch\u0000thinking (temp 0.6, top_p 0.95, top_k 20), 256k context, no context management\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"first_score_reported_at":"2026-02-16","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000widesearch\u0000qwen3_5_model_card\u0000widesearch\u0000thinking (temp 0.6, top_p 0.95, top_k 20), 256k context, no context management\u0000Qwen3.5-397B-A17B","reported_at":"2026-02-16","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"model-reports:widesearch","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["deep-research"],"join":"review:14H034160212:2026-10-15","primary":"deep-research","source_record":null},"modality":null,"name":"WideSearch","openness":"unknown","publisher":null,"released":"2025-08-11","released_reference":{"basis":"benchmark_release","source_key":"model-reports:widesearch","source_url":"https://widesearch-seed.github.io/"},"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":83.9,"display_multiplier":1,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":83.9,"raw_min":74.0,"source_reference":{"obs_id":"curated\u0000widesearch\u0000tencent_hy4_preview\u0000widesearch\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000widesearch\u0000tencent_hy4_preview\u0000widesearch\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"},{"date":"2026-02-16","date_precision":"document_publication","name":"Qwen3.5-397B-A17B"}],"slug":"widesearch","source":"model_reports","source_url":"https://widesearch-seed.github.io/","topics":["agents"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":1,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:hiverwin/widgetva","source_record":"bm_widgetvabench_6624b08d"},"categories":["General AI","Language & Knowledge","cs.HC"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates agentic visual analytics tasks involving single- and multi-widget interactions, with benchmark instances released as data archives and a widget-centered runtime for comparisons.","descriptions":{"en":"Evaluates agentic visual analytics tasks involving single- and multi-widget interactions, with benchmark instances released as data archives and a widget-centered runtime for comparisons."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.24094","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","computer-use"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_widgetvabench_6624b08d"},"modality":null,"name":"WidgetVABench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24094","source_url":"https://arxiv.org/abs/2609.24094"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24094","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24094","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","NeurIPS 2024","大语言模型","LLM","事实可靠性","Factual Reliability","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WikiContradict is a benchmark consisting of 253 high-quality, human-annotated instances designed to assess LLM performance when augmented with retrieved passages containing real-world knowledge conflicts. WikiContradict旨在评估LLM遇到包含真实世界知识冲突的段落检索增强时的性能，由253个高质量的人工注释实例组成。","descriptions":{"en":"WikiContradict is a benchmark consisting of 253 high-quality, human-annotated instances designed to assess LLM performance when augmented with retrieved passages containing real-world knowledge conflicts.","zh":"WikiContradict旨在评估LLM遇到包含真实世界知识冲突的段落检索增强时的性能，由253个高质量的人工注释实例组成。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1318","languages":["English"],"library_direction":null,"modality":null,"name":"WikiContradict","openness":"unknown","publisher":"IBM Research","released":"2024-06-19","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1318-wikicontradict","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WikiContradict","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["其他","Other","大语言模型","LLM","代码工程","Code"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WikiSQL is a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia that is an order of magnitude larger than comparable datasets. WikiSQL 是一个包含 80,654 个手动标注示例的问题和 SQL 查询的数据集，分布在来自维基百科的 24,241 个表格中。","descriptions":{"en":"WikiSQL is a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia that is an order of magnitude larger than comparable datasets.","zh":"WikiSQL 是一个包含 80,654 个手动标注示例的问题和 SQL 查询的数据集，分布在来自维基百科的 24,241 个表格中。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1129","languages":[],"library_direction":null,"modality":null,"name":"WikiSQL","openness":"unknown","publisher":"Salesforce Research","released":"2017-11-09","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1129-wikisql","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WikiSQL","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":258,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":28,"join":"llm_stats_id","repository":null,"source_record":"catalog_60336cacc4ba5437"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","communication"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WildBench is an automated evaluation framework that benchmarks large language models using 1,024 challenging, real-world tasks selected from over one million human-chatbot conversation logs. It introduces two evaluation metrics (WB-Reward and WB-Score) that achieve high correlation with human preferences and uses task-specific checklists for systematic evaluation.","descriptions":{"en":"WildBench is an automated evaluation framework that benchmarks large language models using 1,024 challenging, real-world tasks selected from over one million human-chatbot conversation logs. It introduces two evaluation metrics (WB-Reward and WB-Score) that achieve high correlation with human preferences and uses task-specific checklists for systematic evaluation."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:wild-bench:jamba-1.5-large","reported_at":"2024-08-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wild-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:wild-bench","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_60336cacc4ba5437"},"modality":"text","name":"Wild Bench","openness":"unknown","publisher":null,"released":"2024-06-07","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:wild-bench","source_url":"https://arxiv.org/abs/2406.04770"},"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":68.5,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.685,"raw_min":0.424,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:wild-bench:ministral-3-14b-instruct-2512","reported_date":"2025-12-04","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wild-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Large"},{"date":"2024-08-22","date_precision":"model_announcement","name":"Jamba 1.5 Mini"},{"date":"2025-12-04","date_precision":"model_announcement","name":"MiniStral 3 (14B Instruct 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (3B Instruct 2512)"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Ministral 3 (8B Instruct 2512)"},{"date":"2025-09-01","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3 24B Instruct"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2 24B Instruct"}],"slug":"llm-stats-wild-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wild-bench?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":258,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":28,"join":"artifact:arxiv:2406.04770","repository":null,"source_record":"catalog_60336cacc4ba5437"},"categories":["理解","Understanding","大语言模型","LLM","语言生成","Generation","指令遵循","Instruction Following"],"collected_at":"2026-08-18T00:00:00+00:00","description":"Weintroduce WildBench, an automated evaluation framework designed to bench-mark large language models (LLMs) using challenging, real-world user queries, which consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. WildBench推出自动评估框架和数据集，基于真实用户难题评测大语言模型，包含从逾百万人机对话日志中精选的1,024个任务样本。","descriptions":{"en":"Weintroduce WildBench, an automated evaluation framework designed to bench-mark large language models (LLMs) using challenging, real-world user queries, which consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs.","zh":"WildBench推出自动评估框架和数据集，基于真实用户难题评测大语言模型，包含从逾百万人机对话日志中精选的1,024个任务样本。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1555","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"artifact:arxiv:2406.04770","primary":"chat-writing","source_record":"catalog_60336cacc4ba5437"},"modality":null,"name":"WildBench","openness":"restricted","publisher":"Allen Institute for AI, University of Washington","released":"2024-06-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1555-wildbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WildBench","topics":["language"],"unit":null},{"aliases":["WildBox: A Dataset and Benchmark for Aerial Monocular 3D Detection of African Savanna Wildlife"],"attention_signals":null,"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"WildBox evaluates monocular 3D wildlife detection from drone video across six benchmark classes.","descriptions":{"en":"WildBox evaluates monocular 3D wildlife detection from drone video across six benchmark classes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.21309","languages":[],"library_direction":null,"modality":null,"name":"WildBox","openness":"unknown","publisher":null,"released":"2026-06-19","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.21309","source_url":"https://arxiv.org/abs/2606.21309"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-21309","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21309","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":531,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":37,"join":"llm_stats_id","repository":null,"source_record":"catalog_98f32ff7d2707501"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","coding"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WildClawBench is an agentic coding benchmark from InternLM/Claw-Eval that reports overall model performance on real-world tool-using development tasks.","descriptions":{"en":"WildClawBench is an agentic coding benchmark from InternLM/Claw-Eval that reports overall model performance on real-world tool-using development tasks."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:wildclawbench:mimo-v2.5-pro","reported_at":"2026-04-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wildclawbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:wildclawbench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding","tool-use"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_98f32ff7d2707501"},"modality":"text","name":"WildClawBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":62.8,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.628,"raw_min":0.43,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:wildclawbench:seed-2.1-turbo","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wildclawbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer-30B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-wildclawbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wildclawbench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["智能体","Agent","代码","Code","创作","Creation","Agent Evaluation","In-the-Wild Tasks","Multimodal & Tool-Use Reasoning","代码工程","语言生成","Generation","任务执行","Task Execution","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WildClawBench evaluates AI agents across six task categories: productivity flow, code intelligence, social interaction, search & retrieval, creative synthesis, and safety alignment. It comprises 60 hand-authored tasks inside a live environment with real tools — browser, bash, file system, etc. WildClawBench 从六个任务维度评测 AI 智能体：生产力流程、代码智能、社交互动、搜索与检索、创意合成和安全对齐。基准包含 60 个人工设计的任务，运行在配备真实工具（浏览器、bash、文件系统、电子邮件等）的真实环境中。","descriptions":{"en":"WildClawBench evaluates AI agents across six task categories: productivity flow, code intelligence, social interaction, search & retrieval, creative synthesis, and safety alignment. It comprises 60 hand-authored tasks inside a live environment with real tools — browser, bash, file system, etc.","zh":"WildClawBench 从六个任务维度评测 AI 智能体：生产力流程、代码智能、社交互动、搜索与检索、创意合成和安全对齐。基准包含 60 个人工设计的任务，运行在配备真实工具（浏览器、bash、文件系统、电子邮件等）的真实环境中。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":true,"has_size":true,"key":"opencompass:2445","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"WildClawBench","openness":"open","publisher":"上海人工智能实验室","released":"2026-04-07","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-2445-wildclawbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WildClawBench","topics":["coding","agents","multimodal","reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark containing 500 handwritten documents across three structures, four languages, and nine real-world scenarios, with a Prior-Driven Error metric to quantify reliance on language priors.","descriptions":{"en":"A benchmark containing 500 handwritten documents across three structures, four languages, and nine real-world scenarios, with a Prior-Driven Error metric to quantify reliance on language priors."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.22959","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["document-ocr","multilingual"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_wildhandbench_d9a9447e"},"modality":null,"name":"WildHandBench","openness":"unknown","publisher":null,"released":"2026-08-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.22959","source_url":"http://arxiv.org/abs/2608.22959v1"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-22959","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22959v1","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_wildtrace_951518d8"},"categories":["General AI","Language & Knowledge","Long Context","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"WildTrace evaluates long-context reasoning over naturally occurring long-form sources, focusing on integrating evidence dispersed across distant passages. It includes 481 tasks over 214 sources with seven source-internal evidence geometries, with multi-stage validation ensuring answer groundedness and clue necessity.","descriptions":{"en":"WildTrace evaluates long-context reasoning over naturally occurring long-form sources, focusing on integrating evidence dispersed across distant passages. It includes 481 tasks over 214 sources with seven source-internal evidence geometries, with multi-stage validation ensuring answer groundedness and clue necessity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.09328","languages":[],"library_direction":{"basis":"model-review","categories":["long-context"],"join":"claire_record_id","primary":"long-context","source_record":"bm_wildtrace_951518d8"},"modality":null,"name":"WildTrace","openness":"unknown","publisher":null,"released":"2026-07-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.09328","source_url":"https://arxiv.org/abs/2607.09328"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-09328","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09328","topics":["reasoning-math","long-context"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"llm_stats_id","repository":null,"source_record":"catalog_031cd2068b79e93e"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WinoGrande: An Adversarial Winograd Schema Challenge at Scale. A large-scale dataset of 44,000 pronoun resolution problems designed to test machine commonsense reasoning. Uses adversarial filtering to reduce spurious biases and provides a more robust evaluation of whether AI systems truly understand commonsense or exploit statistical shortcuts. Current best AI methods achieve 59.4-79.1% accuracy, significantly below human performance of 94.0%.","descriptions":{"en":"WinoGrande: An Adversarial Winograd Schema Challenge at Scale. A large-scale dataset of 44,000 pronoun resolution problems designed to test machine commonsense reasoning. Uses adversarial filtering to reduce spurious biases and provides a more robust evaluation of whether AI systems truly understand commonsense or exploit statistical shortcuts. Current best AI methods achieve 59.4-79.1% accuracy, significantly below human performance of 94.0%."},"evidence_summary":{"document_count":1,"model_count":22,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:winogrande:gpt-4-0613","reported_at":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/winogrande?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"llm-stats:winogrande","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_031cd2068b79e93e"},"modality":"text","name":"Winogrande","openness":"unknown","publisher":"Allen Institute for Artificial Intelligence","released":"2019-11-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":22,"score_direction":"higher_is_better","score_summary":{"display_max":87.5,"display_multiplier":100,"model_count":22,"model_count_basis":"source_model_id","numeric_count":22,"raw_max":0.875,"raw_min":0.513,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:winogrande:gpt-4-0613","reported_date":"2023-06-13","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/winogrande?top_n=500"},"unit":null},"scored_models":[{"date":"2024-08-30","date_precision":"model_announcement","name":"Command R+"},{"date":"2025-06-25","date_precision":"model_announcement","name":"ERNIE 4.5"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 27B"},{"date":"2024-06-27","date_precision":"model_announcement","name":"Gemma 2 9B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2023-06-13","date_precision":"model_announcement","name":"GPT-4"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B Base"},{"date":"2024-08-15","date_precision":"model_announcement","name":"Hermes 3 70B"},{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"},{"date":"2026-04-27","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2024-10-16","date_precision":"model_announcement","name":"Ministral 8B Instruct"},{"date":"2024-07-18","date_precision":"model_announcement","name":"Mistral NeMo Instruct"},{"date":"2025-02-01","date_precision":"model_announcement","name":"Phi 4 Mini"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-mini-instruct"},{"date":"2024-08-23","date_precision":"model_announcement","name":"Phi-3.5-MoE-instruct"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Qwen2 72B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 32B Instruct"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5-Coder 7B Instruct"}],"slug":"llm-stats-winogrande","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/winogrande?top_n=500","topics":["reasoning-math","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"artifact:arxiv:1907.10641","repository":null,"source_record":"catalog_031cd2068b79e93e"},"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理","开源收录","Open-Source","支持","Supported"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WINOGRANDE is a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. WINOGRANDE 包含 44,000 个问题，受到 WSC 设计的启发，但进行了调整，以提高数据集的规模和难度。","descriptions":{"en":"WINOGRANDE is a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset.","zh":"WINOGRANDE 包含 44,000 个问题，受到 WSC 设计的启发，但进行了调整，以提高数据集的规模和难度。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1109","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"artifact:arxiv:1907.10641","primary":"abstract-reasoning","source_record":"catalog_031cd2068b79e93e"},"modality":null,"name":"WinoGrande","openness":"unknown","publisher":"Allen Institute for Artificial Intelligence","released":"2019-11-21","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1109-winogrande","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WinoGrande","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"WireCraft is a simulation benchmark for industrial deformable linear object manipulation, featuring three task families (connector insertion, clip routing, channel seating) with configurable difficulty, two DLO physics models, and shared evaluation metrics for RL, IL, and VLA policies across simulation and physical UR5 trajectories.","descriptions":{"en":"WireCraft is a simulation benchmark for industrial deformable linear object manipulation, featuring three task families (connector insertion, clip routing, channel seating) with configurable difficulty, two DLO physics models, and shared evaluation metrics for RL, IL, and VLA policies across simulation and physical UR5 trajectories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18097","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_wirecraft_e44247e4"},"modality":null,"name":"WireCraft","openness":"unknown","publisher":null,"released":"2026-06-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18097","source_url":"https://arxiv.org/abs/2606.18097"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18097","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18097","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":186,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"llm_stats_id","repository":"gh:centerforaisafety/wmdp","source_record":"catalog_31909bc8f19a3e93"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety","healthcare","biology","chemistry"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Weapons of Mass Destruction (WMDP) is a multiple-choice benchmark on dual-use biology, chemistry, and cyber knowledge. It measures a model's capacity to enable malicious actors to design, synthesize, acquire, or use chemical, biological, radiological, or nuclear (CBRN) weapons.","descriptions":{"en":"Weapons of Mass Destruction (WMDP) is a multiple-choice benchmark on dual-use biology, chemistry, and cyber knowledge. It measures a model's capacity to enable malicious actors to design, synthesize, acquire, or use chemical, biological, radiological, or nuclear (CBRN) weapons."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:wmdp:grok-4.1-thinking-2025-11-17","reported_at":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmdp?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:wmdp","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment","cybersecurity","science"],"join":"llm_stats_id","primary":"safety-alignment","source_record":"catalog_31909bc8f19a3e93"},"modality":"text","name":"WMDP","openness":"unknown","publisher":null,"released":"2024-03-05","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:wmdp","source_url":"https://arxiv.org/abs/2403.03218"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":84.0,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.84,"raw_min":0.84,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:wmdp:grok-4.1-thinking-2025-11-17","reported_date":"2025-11-17","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmdp?top_n=500"},"unit":null},"scored_models":[{"date":"2025-11-17","date_precision":"model_announcement","name":"Grok-4.1 Thinking"}],"slug":"llm-stats-wmdp","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmdp?top_n=500","topics":["safety","medical","science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":27,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"llm_stats_id","repository":"gh:wmt-conference/wmt-format-tools","source_record":"catalog_d5fdceb2154bab3d"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language","healthcare"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"The Eighth Conference on Machine Translation (WMT23) benchmark evaluating machine translation systems across 8 language pairs (14 translation directions) including general, biomedical, literary, and low-resource language translation tasks. Features specialized shared tasks for quality estimation, metrics evaluation, sign language translation, and discourse-level literary translation with professional human assessment.","descriptions":{"en":"The Eighth Conference on Machine Translation (WMT23) benchmark evaluating machine translation systems across 8 language pairs (14 translation directions) including general, biomedical, literary, and low-resource language translation tasks. Features specialized shared tasks for quality estimation, metrics evaluation, sign language translation, and discourse-level literary translation with professional human assessment."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:wmt23:gemini-1.0-pro","reported_at":"2024-02-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt23?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:wmt23","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_d5fdceb2154bab3d"},"modality":"text","name":"WMT23","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":75.1,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.751,"raw_min":0.717,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:wmt23:gemini-1.5-pro","reported_date":"2024-05-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt23?top_n=500"},"unit":null},"scored_models":[{"date":"2024-02-15","date_precision":"model_announcement","name":"Gemini 1.0 Pro"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"}],"slug":"llm-stats-wmt23","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt23?top_n=500","topics":["language","medical"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"llm_stats_id","repository":null,"source_record":"catalog_7ce3657bfc8882e2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WMT24++ is a comprehensive multilingual machine translation benchmark that expands the WMT24 dataset to cover 55 languages and dialects. It includes human-written references and post-edits across four domains (literary, news, social, and speech) to evaluate machine translation systems and large language models across diverse linguistic contexts.","descriptions":{"en":"WMT24++ is a comprehensive multilingual machine translation benchmark that expands the WMT24 dataset to cover 55 languages and dialects. It includes human-written references and post-edits across four domains (literary, news, social, and speech) to evaluate machine translation systems and large language models across diverse linguistic contexts."},"evidence_summary":{"document_count":1,"model_count":23,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:wmt24++:gemma-3-12b-it","reported_at":"2025-03-12","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt24%2B%2B?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:wmt24++","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_7ce3657bfc8882e2"},"modality":"text","name":"WMT24++","openness":"unknown","publication":{"evidence":[{"note":"The Google dataset card names the WMT24++ paper, describes the English-to-55-language/dialect expansion, and documents post-edited versus original targets. These details match this exact record's expansion and reference description, rather than the base WMT24 benchmark.","url":"https://huggingface.co/datasets/google/wmt24pp"}],"id":"wmt24pp-original","links":[{"kind":"paper","title":"WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects","url":"https://arxiv.org/abs/2502.12404"},{"kind":"dataset","title":"WMT24++ official dataset","url":"https://huggingface.co/datasets/google/wmt24pp"}],"record_keys":["llm-stats:wmt24++"],"reviewed_at":"2026-10-26","scope":{"en":"Original WMT24++ expansion of English translation evaluation to 55 languages and dialects. Language selection, reference treatment and scoring metric remain properties of each reported evaluation.","zh":"WMT24++ 原始扩展版，评估英语译入 55 种语言及方言；具体语言组合、参考译文处理方式和计分指标仍以各项成绩的评测设置为准。"}},"publisher":null,"released":null,"released_reference":null,"repo_kind":"not_found","repo_resolution_status":"not_found","score_count":23,"score_direction":"higher_is_better","score_summary":{"display_max":86.67,"display_multiplier":100,"model_count":23,"model_count_basis":"source_model_id","numeric_count":23,"raw_max":0.8667,"raw_min":0.272,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:wmt24++:nemotron-3-super-120b-a12b","reported_date":"2026-03-11","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt24%2B%2B?top_n=500"},"unit":null},"scored_models":[{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 1B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E2B Instructed LiteRT (Preview)"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemma 3n E4B Instructed LiteRT Preview"},{"date":"2025-12-15","date_precision":"model_announcement","name":"Nemotron 3 Nano (30B A3B)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super (120B A12B)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra (550B A55B)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-0.8B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-2B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5-397B-A17B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-4B"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5-9B"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"}],"slug":"llm-stats-wmt24","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt24%2B%2B?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-09-25T04:40:22Z","description":"WorkBench evaluates workplace agents by task completion and unintended harmful actions.","descriptions":{"en":"WorkBench evaluates workplace agents by task completion and unintended harmful actions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.13715","languages":[],"library_direction":null,"modality":null,"name":"WorkBench","openness":"unknown","publisher":null,"released":"2026-06-10","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.13715","source_url":"https://arxiv.org/abs/2606.13715"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-13715","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13715","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Safety","Safety & Trustworthiness"],"collected_at":"2026-10-14T09:44:51Z","description":"LLM-based agents now interact with their environments continuously, shaped by such organizational channels as user instructions, peer messages, and long-term memory.","descriptions":{"en":"LLM-based agents now interact with their environments continuously, shaped by such organizational channels as user instructions, peer messages, and long-term memory."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.11561","languages":[],"library_direction":{"basis":"model-review-daily","categories":["professional-work","agent-memory"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_workerville_f2ffa6bb"},"modality":null,"name":"Workerville","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.11561","source_url":"https://arxiv.org/abs/2610.11561"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-11561","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11561","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":221,"join":"claire_record_id","repository":null,"source_record":"bm_workflow-gym_72fcdcd4"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Workflow-GYM evaluates AI agents on long-horizon GUI tasks in professional domains, using specialized software environments and economically valuable workflows. Tasks require end-to-end operation of graphical user interfaces, with success rates measured by task completion.","descriptions":{"en":"Workflow-GYM evaluates AI agents on long-horizon GUI tasks in professional domains, using specialized software environments and economically valuable workflows. Tasks require end-to-end operation of graphical user interfaces, with success rates measured by task completion."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.11042","languages":[],"library_direction":{"basis":"model-review","categories":["computer-use","professional-work"],"join":"claire_record_id","primary":"computer-use","source_record":"bm_workflow-gym_72fcdcd4"},"modality":null,"name":"Workflow-GYM","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11042","source_url":"https://arxiv.org/abs/2606.11042"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11042","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11042","topics":["computer-use","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"llm_stats_id","repository":null,"source_record":"catalog_7a2bde6e6f86bc45"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Workspace Bench evaluates AI agents on high-economic-value workplace tasks that span multi-step planning, file processing, and tool use across realistic office and productivity workflows.","descriptions":{"en":"Workspace Bench evaluates AI agents on high-economic-value workplace tasks that span multi-step planning, file processing, and tool use across realistic office and productivity workflows."},"evidence_summary":{"document_count":1,"model_count":3,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:workspace-bench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/workspace-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:workspace-bench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_7a2bde6e6f86bc45"},"modality":"text","name":"Workspace Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":3,"score_direction":"higher_is_better","score_summary":{"display_max":67.7,"display_multiplier":100,"model_count":3,"model_count_basis":"source_model_id","numeric_count":3,"raw_max":0.677,"raw_min":0.53,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:workspace-bench:qwen3.8-max","reported_date":"2026-08-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/workspace-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-08-02","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-workspace-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/workspace-bench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":["Workspace-Bench","Workspace Bench","Workspace-Bench 1.0"],"attention_signals":{"as_of":"2026-10-10","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":12,"join":"artifact:arxiv:2605.03596","repository":null,"source_record":"catalog_7a2bde6e6f86bc45"},"categories":["professional"],"collected_at":null,"description":"Large-scale file-workspace tasks; the environment and file dependencies are part of the measurement.","descriptions":{"en":"Large-scale file-workspace tasks; the environment and file dependencies are part of the measurement."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000workspacebench\u0000tencent_hy4_preview\u0000workspacebench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"first_score_reported_at":"2026-08-28","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000workspacebench\u0000tencent_hy4_preview\u0000workspacebench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","source_url":"https://hy.tencent.ai/research/hy4-preview"},"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"model-reports:workspacebench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","tool-use"],"join":"artifact:arxiv:2605.03596","primary":"professional-work","source_record":"catalog_7a2bde6e6f86bc45"},"modality":null,"name":"WorkspaceBench","openness":"unknown","publisher":null,"released":"2026-05-05","released_reference":{"basis":"benchmark_release","source_key":"model-reports:workspacebench","source_url":"https://arxiv.org/abs/2605.03596"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":60.2,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":60.2,"raw_min":60.2,"source_reference":{"obs_id":"curated\u0000workspacebench\u0000tencent_hy4_preview\u0000workspacebench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","observation_id":"curated\u0000workspacebench\u0000tencent_hy4_preview\u0000workspacebench\u0000as printed in Table 3 (Large Models)\u0000Hy4 preview","reported_at":"2026-08-28","reported_date":"2026-08-28","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview"},"unit":"percent"},"scored_models":[{"date":"2026-08-28","date_precision":"document_publication","name":"Hy4 preview"}],"slug":"workspacebench","source":"model_reports","source_url":"https://arxiv.org/abs/2605.03596","topics":[],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":11,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:haolpku/worksurface-bench","source_record":"bm_worksurface-bench_7866ac7c"},"categories":["Factuality","General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"WorkSurface-Bench evaluates enterprise agents on knowledge routing across documents, tables, and graphs. It includes 1,151 atomic tasks with auditable reference answers and scoring for route, evidence, answer, and efficiency.","descriptions":{"en":"WorkSurface-Bench evaluates enterprise agents on knowledge routing across documents, tables, and graphs. It includes 1,151 atomic tasks with auditable reference answers and scoring for route, evidence, answer, and efficiency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.25765","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","deep-research"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_worksurface-bench_7866ac7c"},"modality":null,"name":"WorkSurface-Bench","openness":"unknown","publisher":null,"released":"2026-07-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.25765","source_url":"https://arxiv.org/abs/2607.25765"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-25765","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25765","topics":[],"unit":null},{"aliases":["World Models' Last Exam in Physics","Physics-Last-Exam","phys-last-exam"],"attention_signals":null,"categories":["multimodal"],"collected_at":null,"description":"Forty controlled image-to-video tasks across nine physical families. Overall is a video-weighted composite score, S = 0.15C + 0.85P × I(C ≥ 80), on 0–100; it is not a success rate. Four samples are generated per model-task pair, with different resolutions and frame counts. Seedance uses sample labels rather than transmitted random seeds. Visibility and measurement robustness limit interpretation; empirical difficulty groups are derived from the evaluated models.","descriptions":{"en":"Forty controlled image-to-video tasks across nine physical families. Overall is a video-weighted composite score, S = 0.15C + 0.85P × I(C ≥ 80), on 0–100; it is not a success rate. Four samples are generated per model-task pair, with different resolutions and frame counts. Seedance uses sample labels rather than transmitted random seeds. Visibility and measurement robustness limit interpretation; empirical difficulty groups are derived from the evaluated models."},"editorial_guide":{"id":"world-models-last-exam-physics","record_keys":["model-reports:world_models_last_exam_physics"],"reviewed_at":"2026-10-27","scope":{"en":"The paper v1 evaluation of eight image-to-video models on 40 controlled physical tasks. The published overall measure is a composite score on a 0–100 scale.","zh":"对应论文 v1 中 8 个图生视频模型在 40 道受控物理任务上的评测。总榜指标是 0–100 的复合分数。"},"sections":[{"id":"tasks","source_ids":["paper","project"],"text":{"en":"The 40 tasks cover nine physical families, including mechanics, optics, buoyancy, phase change, electromagnetism and fluids. Examples compare pendulum periods, communicating-vessel levels and capillary rise. Models receive the same initial image and prompt for each task; evaluation does not require a reference video.","zh":"40 道任务覆盖九类物理现象，包括力学、光学、浮力、相变、电磁学与流体。例如比较摆的周期、连通器液面和毛细上升。每道题给各模型相同的首帧及提示词，判分不依赖参考视频。"},"title":{"en":"Controlled physical experiments","zh":"受控的物理实验"}},{"id":"measurement","source_ids":["paper"],"text":{"en":"The evaluator separates consistency and observability C from independently measured physical correctness P. Tracking and image-analysis tools extract quantities such as trajectories, angles, periods or event timing. Fixed task-specific rules map deviations from expected relationships into physical scores.","zh":"评估器将一致性及可观察性 C 与独立测量的物理正确性 P 分开。跟踪和图像分析工具提取轨迹、角度、周期或事件时序，再用固定的任务规则把物理关系偏差映射成分数。"},"title":{"en":"Measure the physical evidence","zh":"从可观察证据测量物理关系"}},{"id":"scoring","source_ids":["paper","results"],"text":{"en":"Each video uses S = 0.15C + 0.85P × I(C ≥ 80), then the overall score averages videos. A failed consistency gate removes the physical contribution, not necessarily the entire score. The table checkmark means all available samples in a model–task pair satisfy C ≥ 80; it is not a task-success label.","zh":"每段视频按 S = 0.15C + 0.85P × I(C ≥ 80) 计分，再对视频求总平均。一致性门槛未通过时去除的是物理分量，并非一定将总分归零。表格中的勾表示该模型在该题的全部可用样本均达到 C ≥ 80，不是任务成功标记。"},"title":{"en":"Read the composite score correctly","zh":"正确理解复合分数"}},{"id":"protocol","source_ids":["paper","generation"],"text":{"en":"The published table covers 1,280 videos: eight models × 40 tasks × four samples. Seeds 42–45 are fixed where supported, but the Seedance API adapter uses them only as sample labels. Resolutions and frame counts vary by model, so these generation settings belong with the reported score.","zh":"公开表格覆盖 1,280 段视频，即 8 个模型 × 40 道题 × 4 次样本。支持随机种子的模型使用 42–45，但 Seedance API 适配器仅将其作为样本标签。各模型分辨率和帧数不同，阅读成绩时应保留这些生成设置。"},"title":{"en":"Four samples, model-specific video settings","zh":"每题四次样本，视频设置因模型而异"}},{"id":"limits","source_ids":["paper"],"text":{"en":"Occlusion, unclear geometry or incomplete events can make physical evidence unreliable. Synthetic-video checks validate measurement under controlled conditions, not every complex scene. The 15 easy, 15 medium and 10 hard groups are empirical labels derived from the evaluated models, and do not constitute an independent difficulty standard.","zh":"遮挡、几何信息模糊或事件不完整，都可能影响物理证据。合成视频校验支持受控条件下的测量有效性，不能代表所有复杂场景。15 道简单、15 道中等和 10 道困难题，是依据本次模型表现划分的经验分组，并非独立难度标准。"},"title":{"en":"Visibility and measurement remain limits","zh":"可见性与测量仍是边界"}}],"sources":[{"id":"project","title":"World Models’ Last Exam in Physics official project","url":"https://lab.einsia.ai/phys-last-exam/"},{"id":"paper","title":"World Models’ Last Exam in Physics paper v1","url":"https://arxiv.org/html/2610.08791v1"},{"id":"results","title":"Official paper results JSON","url":"https://lab.einsia.ai/phys-last-exam/paper-results.json"},{"id":"repository","title":"Official repository","url":"https://github.com/Einsia/phys-last-exam"},{"id":"generation","title":"Generation protocol and model settings","url":"https://github.com/Einsia/phys-last-exam/blob/main/generation/README.md"}],"summary":{"en":"Does a generated video preserve measurable physical relationships, beyond looking plausible? Each task starts from a controlled image and checks a specific physical process.","zh":"视频看起来合理，是否也满足可测量的物理关系？每道任务从受控首帧出发，检查一个具体的物理过程。"},"title":"World Models’ Last Exam in Physics"},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000world_models_last_exam_physics\u0000world_models_last_exam_physics_v1_results_table\u0000World Models’ Last Exam in Physics: 40-task paper v1\u000040 common initial-frame/prompt tasks; 4 API samples; labels 42-45; seed not transmitted; generation 1270x726, 121 frames; automatic consistency C gate at 80; S=0.15C+0.85P*I(C>=80); video-weighted mean S (0-100)\u0000Seedance 2.5","reported_at":"2026-10-06","source_url":"https://arxiv.org/html/2610.08791v1#S3.T2"},"first_score_reported_at":"2026-10-06","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000world_models_last_exam_physics\u0000world_models_last_exam_physics_v1_results_table\u0000World Models’ Last Exam in Physics: 40-task paper v1\u000040 common initial-frame/prompt tasks; 4 API samples; labels 42-45; seed not transmitted; generation 1270x726, 121 frames; automatic consistency C gate at 80; S=0.15C+0.85P*I(C>=80); video-weighted mean S (0-100)\u0000Seedance 2.5","reported_at":"2026-10-06","source_url":"https://arxiv.org/html/2610.08791v1#S3.T2"},"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"model-reports:world_models_last_exam_physics","languages":[],"library_direction":null,"modality":null,"name":"World Models’ Last Exam in Physics","openness":"unknown","publication":{"evidence":[{"note":"The author project names this benchmark and links its introducing paper and available release resources. This exact-record review preserves the documented task version, metrics and evaluation setup.","url":"https://lab.einsia.ai/phys-last-exam/"},{"note":"Navers Lab / Einsia research index links this named project. Method papers and memory software in the same index are not added as benchmarks.","url":"https://lab.einsia.ai/research/"}],"id":"einsia-world-models-last-exam-physics-original","links":[{"kind":"website","title":"World Models’ Last Exam in Physics · official project","url":"https://lab.einsia.ai/phys-last-exam/"},{"kind":"paper","title":"World Models’ Last Exam in Physics · introducing paper","url":"https://arxiv.org/abs/2610.08791"},{"kind":"repo","title":"World Models’ Last Exam in Physics · official repository","url":"https://github.com/Einsia/phys-last-exam"}],"record_keys":["model-reports:world_models_last_exam_physics"],"reviewed_at":"2026-10-27","scope":{"en":"The paper v1 evaluation of eight image-to-video models on 40 controlled physical tasks. The published overall measure is a composite score on a 0–100 scale.","zh":"对应论文 v1 中 8 个图生视频模型在 40 道受控物理任务上的评测。总榜指标是 0–100 的复合分数。"}},"publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"model-reports:world_models_last_exam_physics","source_url":"https://lab.einsia.ai/phys-last-exam/"},"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":57.76,"display_multiplier":1,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":57.76,"raw_min":18.81,"source_reference":{"obs_id":"curated\u0000world_models_last_exam_physics\u0000world_models_last_exam_physics_v1_results_table\u0000World Models’ Last Exam in Physics: 40-task paper v1\u000040 common initial-frame/prompt tasks; 4 API samples; labels 42-45; seed not transmitted; generation 1270x726, 121 frames; automatic consistency C gate at 80; S=0.15C+0.85P*I(C>=80); video-weighted mean S (0-100)\u0000Seedance 2.5","observation_id":"curated\u0000world_models_last_exam_physics\u0000world_models_last_exam_physics_v1_results_table\u0000World Models’ Last Exam in Physics: 40-task paper v1\u000040 common initial-frame/prompt tasks; 4 API samples; labels 42-45; seed not transmitted; generation 1270x726, 121 frames; automatic consistency C gate at 80; S=0.15C+0.85P*I(C>=80); video-weighted mean S (0-100)\u0000Seedance 2.5","reported_at":"2026-10-06","reported_date":"2026-10-06","source_id":"world_models_last_exam_physics_v1_results_table","source_url":"https://arxiv.org/html/2610.08791v1#S3.T2"},"unit":"points"},"scored_models":[{"date":"2026-10-06","date_precision":"document_publication","name":"CogVideoX 1.5-5B"},{"date":"2026-10-06","date_precision":"document_publication","name":"Cosmos 3 Super"},{"date":"2026-10-06","date_precision":"document_publication","name":"Hunyuan 1.5"},{"date":"2026-10-06","date_precision":"document_publication","name":"LingBot 30B-A3B"},{"date":"2026-10-06","date_precision":"document_publication","name":"MiniMax H3"},{"date":"2026-10-06","date_precision":"document_publication","name":"Seedance 2.5"},{"date":"2026-10-06","date_precision":"document_publication","name":"VBVR Wan2.2"},{"date":"2026-10-06","date_precision":"document_publication","name":"Wan 2.2-A14B"}],"slug":"world_models_last_exam_physics","source":"model_reports","source_url":"https://lab.einsia.ai/phys-last-exam/","topics":["multimodal","science"],"unit":"points"},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":4,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":106,"join":"claire_record_id","repository":"gh:ucsb-nlp-chang/worldauditbench","source_record":"bm_worldauditbench_3e62aaa6"},"categories":["Agents & Tool Use","General AI","Geometric reasoning","Multimodal"],"collected_at":"2026-10-14T09:44:51Z","description":"As interactive 3D worlds are increasingly used to study intelligent behavior, it becomes important to develop efficient pipelines for identifying anomalies in these simulated environments, such as floating objects, traversable walls, or ob…","descriptions":{"en":"As interactive 3D worlds are increasingly used to study intelligent behavior, it becomes important to develop efficient pipelines for identifying anomalies in these simulated environments, such as floating objects, traversable walls, or ob…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.40325","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","spatial-3d"],"join":"claire_record_id","primary":"world-models","source_record":"bm_worldauditbench_3e62aaa6"},"modality":null,"name":"WorldAuditBench","openness":"unknown","publisher":null,"released":"2026-09-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.40325","source_url":"https://arxiv.org/abs/2609.40325"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-40325","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.40325","topics":["agents","multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":8,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":4,"join":"claire_record_id","repository":"gh:zlab-princeton/worldbench","source_record":"bm_worldbench_a7ecd2c0"},"categories":["General AI","Multimodal","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"WorldBench evaluates multimodal large language models on visually diverse reasoning questions, with accuracy as the metric on a curated dataset.","descriptions":{"en":"WorldBench evaluates multimodal large language models on visually diverse reasoning questions, with accuracy as the metric on a curated dataset."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.06538","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_worldbench_a7ecd2c0"},"modality":null,"name":"WorldBench","openness":"unknown","publisher":null,"released":"2026-06-04","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.06538","source_url":"https://arxiv.org/abs/2606.06538"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-06538","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06538","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"A multilingual benchmark of persona-grounded everyday workflows for agents, evaluated via sandbox actions.","descriptions":{"en":"A multilingual benchmark of persona-grounded everyday workflows for agents, evaluated via sandbox actions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.01056","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["chat-writing","multilingual"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_worldbench_7c7f4e1b"},"modality":null,"name":"WorldBench","openness":"unknown","publisher":null,"released":"2026-09-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.01056","source_url":"https://arxiv.org/abs/2609.01056"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-01056","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01056","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","knowledge","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WorldBench evaluates real-world visual knowledge and understanding across diverse everyday scenes.","descriptions":{"en":"WorldBench evaluates real-world visual knowledge and understanding across diverse everyday scenes."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:worldbench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/worldbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:worldbench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["chat-writing","multilingual"],"join":"llm_stats_id","primary":"chat-writing","source_record":"bm_worldbench_7c7f4e1b"},"modality":"multimodal","name":"WorldBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":67.60000000000001,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.676,"raw_min":0.637,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:worldbench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/worldbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-worldbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/worldbench?top_n=500","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":2,"join":"claire_record_id","repository":null,"source_record":"bm_worldcoder-bench_8f868de5"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates autonomous physically grounded 3D world synthesis from natural language, with 2,026 expert-curated tasks across Simulation, Rendering, and Application scenarios, using execution-based verification via StateProbe to check hidden behavioral contracts over runtime states.","descriptions":{"en":"Evaluates autonomous physically grounded 3D world synthesis from natural language, with 2,026 expert-curated tasks across Simulation, Rendering, and Application scenarios, using execution-based verification via StateProbe to check hidden behavioral contracts over runtime states."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.01869","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","image-video-generation","world-models"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_worldcoder-bench_8f868de5"},"modality":null,"name":"WorldCoder-Bench","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.01869","source_url":"https://arxiv.org/abs/2606.01869"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-01869","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01869","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":25,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":5,"join":"claire_record_id","repository":"gh:wzk1015/worldcuparena","source_record":"bm_worldcuparena_3034c352"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"WorldCupArena evaluates language models and deep-research agents on football forecasting across multiple tasks including result, score, player events, statistics, and tournament outcomes. It uses a composite score and supports adding new schedules for future events.","descriptions":{"en":"WorldCupArena evaluates language models and deep-research agents on football forecasting across multiple tasks including result, score, player events, statistics, and tournament outcomes. It uses a composite score and supports adding new schedules for future events."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.18084","languages":[],"library_direction":{"basis":"model-review","categories":["forecasting","deep-research"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_worldcuparena_3034c352"},"modality":null,"name":"WorldCupArena","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.18084","source_url":"https://arxiv.org/abs/2607.18084"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-18084","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18084","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":22,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":34,"join":"claire_record_id","repository":"gh:yuxueyang1204/worldexam","source_record":"bm_worldexam_7f2f166e"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"WorldExam evaluates controllable video generation models as world models across four levels: Visual Quality, Control Adherence, Spatial Consistency, and World Reactivity, with 1,474 cases across eight tasks.","descriptions":{"en":"WorldExam evaluates controllable video generation models as world models across four levels: Visual Quality, Control Adherence, Spatial Consistency, and World Reactivity, with 1,474 cases across eight tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.02603","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_worldexam_7f2f166e"},"modality":null,"name":"WorldExam","openness":"unknown","publisher":null,"released":"2026-08-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.02603","source_url":"https://arxiv.org/abs/2608.02603"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-02603","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02603","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["多模态","Multimodal","推理","Reasoning","知识","Knowledge","多模态模型","VLM","视觉生成","Visual Generation","知识储备"],"collected_at":"2026-08-18T00:00:00+00:00","description":"","descriptions":{},"editorial_guide":{"id":"worldgenbench","record_keys":["opencompass:1938"],"reviewed_at":"2026-10-23","scope":{"en":"Cited research guide; original source measurements remain independent.","zh":"附来源的研究导读；原始记录的测量保持独立。"},"sections":[{"id":"tasks","source_ids":["paper"],"text":{"en":"Prompts require factual grounding and contextual inference together. Attractive images can still omit or contradict the implied knowledge.","zh":"题目同时要求事实依据与情境推断。即使图像美观，也可能遗漏或违背隐含知识。"},"title":{"en":"What an image must get right","zh":"图像需要体现什么"}},{"id":"metrics","source_ids":["paper"],"text":{"en":"The proposed score checks whether explicit and implicit semantic elements appear faithfully in the generated image. It measures a different concern from visual appeal alone.","zh":"论文提出的评分检查图像是否忠实体现显式与隐式语义要素，关注点不同于单纯的视觉美感。"},"title":{"en":"Knowledge Checklist Score","zh":"知识清单评分"}},{"id":"analysis","source_ids":["paper"],"text":{"en":"The paper examines semantic hallucinations and gaps in cultural and scientific knowledge. Read category examples before interpreting an aggregate result.","zh":"论文分析语义幻觉及文化、科学知识方面的缺口。理解总成绩前，应查看具体类别的样例。"},"title":{"en":"Inspect the failure cases","zh":"查看失败类型"}},{"id":"limits","source_ids":["paper"],"text":{"en":"This guide uses the linked paper revision. The original OpenCompass record retains its own collection date and access evidence; this is not a fresh model evaluation.","zh":"导读依据所链接的论文版本。OpenCompass 原记录仍保留自身采集时间与访问证据；本次补充不是重新评测模型。"},"title":{"en":"Version and source boundaries","zh":"版本与来源边界"}}],"sources":[{"id":"paper","title":"WorldGenBench — original paper","url":"https://arxiv.org/abs/2505.01490v2"},{"id":"project","title":"WorldGenBench — authors’ project","url":"https://dwanzhang-ai.github.io/WorldGenBench/"}],"summary":{"en":"Whether text-to-image systems combine world knowledge with inference about unstated visual details.","zh":"考察文生图系统能否结合世界知识，推断提示词中未明说的视觉细节。"},"title":"WorldGenBench"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":false,"has_size":false,"key":"opencompass:1938","languages":[],"library_direction":null,"modality":"multimodal","name":"WorldGenBench","openness":"unknown","publisher":null,"released":"2025-06-16","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1938-worldgenbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WorldGenBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"claire_record_id","repository":null,"source_record":"bm_worldlines_ada0e776"},"categories":["Robotics & Embodied AI","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"WorldLines evaluates long-horizon stateful embodied agents in household environments through Memory QA and Embodied Task Planning, using temporally extended household traces with dialogues and state changes.","descriptions":{"en":"WorldLines evaluates long-horizon stateful embodied agents in household environments through Memory QA and Embodied Task Planning, using temporally extended household traces with dialogues and state changes."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.18847","languages":[],"library_direction":{"basis":"model-review","categories":["embodied-robotics","agent-memory"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_worldlines_ada0e776"},"modality":null,"name":"WorldLines","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.18847","source_url":"https://arxiv.org/abs/2606.18847"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-18847","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18847","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":57,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":33,"join":"claire_record_id","repository":"gh:alibaba-damo-academy/worldolympiad","source_record":"bm_worldolympiad_08793185"},"categories":["Robotics & Embodied AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"WorldOlympiad is a triathlon-style benchmark for video-based world models, evaluating physical faithfulness, geometric consistency, and interaction fidelity across gaming, robotics, and general real-world scenarios. It includes 1,000 long videos and interpretable automatic metrics.","descriptions":{"en":"WorldOlympiad is a triathlon-style benchmark for video-based world models, evaluating physical faithfulness, geometric consistency, and interaction fidelity across gaming, robotics, and general real-world scenarios. It includes 1,000 long videos and interpretable automatic metrics."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.11129","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_worldolympiad_08793185"},"modality":null,"name":"WorldOlympiad","openness":"unknown","publisher":null,"released":"2026-06-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.11129","source_url":"https://arxiv.org/abs/2606.11129"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-11129","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11129","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_worldroambench_ba7d4093"},"categories":["General AI","Geometric reasoning","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"WorldRoamBench evaluates interactive world models on long-horizon stability across action, vision, physics, and memory dimensions. It includes 600+ test cases in various scenes and views with continuous interaction.","descriptions":{"en":"WorldRoamBench evaluates interactive world models on long-horizon stability across action, vision, physics, and memory dimensions. It includes 600+ test cases in various scenes and views with continuous interaction."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.31672","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","image-video-generation"],"join":"claire_record_id","primary":"world-models","source_record":"bm_worldroambench_ba7d4093"},"modality":null,"name":"WorldRoamBench","openness":"unknown","publisher":null,"released":"2026-06-30","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.31672","source_url":"https://arxiv.org/abs/2606.31672"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-31672","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31672","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":317,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"artifact:arxiv:2504.00983","repository":null,"source_record":"catalog_9508baebc45d3362"},"categories":["创作","Creation","其他","Other","视频生成","多模态模型","VLM","视觉生成","Visual Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WorldScore benchmark is the first unified benchmark for world generation. WorldScore基准测试，这是首个用于世界生成的统一基准测试。","descriptions":{"en":"WorldScore benchmark is the first unified benchmark for world generation.","zh":"WorldScore基准测试，这是首个用于世界生成的统一基准测试。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1735","languages":[],"library_direction":{"basis":"anchor-review","categories":["world-models","image-video-generation"],"join":"artifact:arxiv:2504.00983","primary":"world-models","source_record":"catalog_9508baebc45d3362"},"modality":null,"name":"WorldScore","openness":"open","publisher":"Stanford University","released":"2025-04-01","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1735-worldscore","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WorldScore","topics":["multimodal"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sirujiang/worldsolver","source_record":"bm_worldsolver_70adb317"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"LLM-based agents are increasingly advancing scientific and engineering problem solving, with physics simulation emerging as a challenging yet practical testbed for reproducing complex physical phenomena with application in embodied AI, gam…","descriptions":{"en":"LLM-based agents are increasingly advancing scientific and engineering problem solving, with physics simulation emerging as a challenging yet practical testbed for reproducing complex physical phenomena with application in embodied AI, gam…"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2610.08720","languages":[],"library_direction":{"basis":"model-review-daily","categories":["world-models","science"],"join":"claire_record_id","primary":"world-models","source_record":"bm_worldsolver_70adb317"},"modality":null,"name":"WorldSolver","openness":"unknown","publisher":null,"released":"2026-10-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.08720","source_url":"https://arxiv.org/abs/2610.08720"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-08720","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08720","topics":["science"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":123,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":6,"join":"llm_stats_id","repository":null,"source_record":"catalog_4db4aada87823846"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"WorldVQA is a benchmark designed to evaluate atomic vision-centric world knowledge. It assesses models' ability to understand and reason about visual elements representing real-world knowledge.","descriptions":{"en":"WorldVQA is a benchmark designed to evaluate atomic vision-centric world knowledge. It assesses models' ability to understand and reason about visual elements representing real-world knowledge."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:worldvqa:kimi-k2.5","reported_at":"2026-01-27","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/worldvqa?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:worldvqa","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","factuality"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_4db4aada87823846"},"modality":"multimodal","name":"WorldVQA","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":61.1,"display_multiplier":100,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.611,"raw_min":0.463,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:worldvqa:qwen3.7-plus","reported_date":"2026-05-31","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/worldvqa?top_n=500"},"unit":null},"scored_models":[{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-05-31","date_precision":"model_announcement","name":"Qwen3.7-Plus"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-worldvqa","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/worldvqa?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_woven_dfbe3ac8"},"categories":["Multimodal","Reasoning","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning.","descriptions":{"en":"Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2610.12417","languages":[],"library_direction":{"basis":"model-review-daily","categories":["spatial-3d","world-models"],"join":"claire_record_id","primary":"spatial-3d","source_record":"bm_woven_dfbe3ac8"},"modality":null,"name":"WOVEN","openness":"unknown","publisher":null,"released":"2026-10-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2610.12417","source_url":"https://arxiv.org/abs/2610.12417"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2610-12417","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12417","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Energy & Grid","Language & Knowledge","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"Wind power forecasting benchmark covering tasks and dataset scope detailed in the paper.","descriptions":{"en":"Wind power forecasting benchmark covering tasks and dataset scope detailed in the paper."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.24444","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["forecasting"],"join":"claire_record_id","primary":"forecasting","source_record":"bm_wpbench_c29b38e1"},"modality":null,"name":"WPBench","openness":"unknown","publisher":null,"released":"2026-09-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.24444","source_url":"https://arxiv.org/abs/2609.24444"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-24444","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24444","topics":[],"unit":null},{"aliases":["wps-ai-benchmark-v6"],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":142,"hf_paper_upvotes":null,"join":"claire_record_id","repository":null,"source_record":"bm_wps-ai-benchmark-v6_1845882a"},"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates AI systems on applying Women, Peace and Security principles when answering operational advice prompts via 30 scenarios and an 11-criterion rubric with LLM judges.","descriptions":{"en":"Evaluates AI systems on applying Women, Peace and Security principles when answering operational advice prompts via 30 scenarios and an 11-criterion rubric with LLM judges."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":false,"has_repo":false,"has_size":false,"key":"claire-radar:huggingface:oursecurefuture/wps-ai-benchmark-v6","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"claire_record_id","primary":"safety-alignment","source_record":"bm_wps-ai-benchmark-v6_1845882a"},"modality":null,"name":"WPS AI Benchmark","openness":"unknown","publisher":null,"released":"2026-09-15","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:huggingface:oursecurefuture/wps-ai-benchmark-v6","source_url":"https://huggingface.co/datasets/OurSecureFuture/wps-ai-benchmark-v6"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-huggingface-oursecurefuture-wps-ai-benchmark-v6","source":"claire_radar","source_url":"https://huggingface.co/datasets/OurSecureFuture/wps-ai-benchmark-v6","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":202,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":22,"join":"llm_stats_id","repository":null,"source_record":"catalog_3712248b2b66bf45"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["legal","finance","communication","creativity","writing"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"A comprehensive benchmark for evaluating large language models' generative writing capabilities across 6 core writing domains (Academic & Engineering, Finance & Business, Politics & Law, Literature & Art, Education, Advertising & Marketing) and 100 subdomains. Contains 1,239 queries with a query-dependent evaluation framework that dynamically generates 5 instance-specific assessment criteria for each writing task, using a fine-tuned critic model to score responses on style, format, and length dimensions.","descriptions":{"en":"A comprehensive benchmark for evaluating large language models' generative writing capabilities across 6 core writing domains (Academic & Engineering, Finance & Business, Politics & Law, Literature & Art, Education, Advertising & Marketing) and 100 subdomains. Contains 1,239 queries with a query-dependent evaluation framework that dynamically generates 5 instance-specific assessment criteria for each writing task, using a fine-tuned critic model to score responses on style, format, and length dimensions."},"evidence_summary":{"document_count":1,"model_count":15,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:writingbench:qwen3-235b-a22b-instruct-2507","reported_at":"2025-07-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/writingbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:writingbench","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"catalog_3712248b2b66bf45"},"modality":"text","name":"WritingBench","openness":"unknown","publication":{"evidence":[{"note":"The repository links the paper and dates the first 1,239-query release to March 10, 2025. Its April 29 update changes the benchmark to 1,000 queries and revised criteria, followed by further evaluator updates. The record's 1,239-query description identifies the initial release without proving the revision used by every stored score.","url":"https://github.com/X-PLUG/WritingBench/blob/main/README.md"},{"note":"The first paper version presents WritingBench with the original six-domain, 100-subdomain generative-writing design, providing a version-specific publication alongside the evolving author repository.","url":"https://arxiv.org/abs/2503.05244v1"}],"id":"writingbench-initial-release","links":[{"kind":"paper","title":"WritingBench: A Comprehensive Benchmark for Generative Writing","url":"https://arxiv.org/abs/2503.05244v1"},{"kind":"repo","title":"WritingBench official repository and release history","url":"https://github.com/X-PLUG/WritingBench"}],"record_keys":["llm-stats:writingbench"],"reviewed_at":"2026-10-26","scope":{"en":"The record describes WritingBench's initial 1,239-query release, so the paper link selects its first version. The repository documents later revisions to 1,000 queries and evaluator updates; each score's release and judge must be read from its source.","zh":"本条描述的是 WritingBench 最初的 1,239 道问题，因此论文链接指向初版。仓库另有改为 1,000 道问题及评判模型更新的记录；每项成绩的版本和评判模型仍须查阅其来源。"}},"publisher":null,"released":"2025-03-07","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: WritingBench: A Comprehensive Benchmark for Generative Writing. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:writingbench","source_url":"https://arxiv.org/abs/2503.05244"},"repo_kind":null,"repo_resolution_status":null,"score_count":15,"score_direction":"higher_is_better","score_summary":{"display_max":88.3,"display_multiplier":100,"model_count":15,"model_count_basis":"source_model_id","numeric_count":15,"raw_max":0.883,"raw_min":0.738,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:writingbench:qwen3-235b-a22b-thinking-2507","reported_date":"2025-07-25","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/writingbench?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Thinking-0905"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 32B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 4B Thinking"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 8B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Thinking-2507"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Instruct"},{"date":"2025-09-10","date_precision":"model_announcement","name":"Qwen3-Next-80B-A3B-Thinking"}],"slug":"llm-stats-writingbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/writingbench?top_n=500","topics":["language","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":202,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":22,"join":"artifact:arxiv:2503.05244","repository":null,"source_record":"catalog_3712248b2b66bf45"},"categories":["长文本","Long-Context","创作","Creation","大语言模型","LLM","长上下文","Long Context","语言生成","Generation"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WritingBench: A Comprehensive Benchmark for Generative Writing WritingBench: A Comprehensive Benchmark for Generative Writing","descriptions":{"en":"WritingBench: A Comprehensive Benchmark for Generative Writing","zh":"WritingBench: A Comprehensive Benchmark for Generative Writing"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1701","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing"],"join":"artifact:arxiv:2503.05244","primary":"chat-writing","source_record":"catalog_3712248b2b66bf45"},"modality":null,"name":"WritingBench","openness":"unknown","publisher":"Alibaba Group; Renmin University of China; Shanghai Jiao Tong University","released":"2025-03-07","released_reference":{"basis":"paper_first_version","note":"First version introducing WritingBench.","source_key":"opencompass:1701","source_url":"https://arxiv.org/abs/2503.05244"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1701-writingbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WritingBench","topics":["long-context","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":8,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:sufe-ailab/wsadbench","source_record":"bm_wsadbench_d5d448b3"},"categories":["General AI","Vision & 3D","cs.LG"],"collected_at":"2026-10-14T09:44:51Z","description":"WSADBench is a benchmark for weakly supervised anomaly detection (WSAD) that unifies evaluation across incomplete, inexact, and inaccurate supervision scenarios. It evaluates 36 algorithms across 4 modalities (tabular, video, image features, text embeddings) by systematically varying label quantity, granularity, and quality, with protocols and code provided.","descriptions":{"en":"WSADBench is a benchmark for weakly supervised anomaly detection (WSAD) that unifies evaluation across incomplete, inexact, and inaccurate supervision scenarios. It evaluates 36 algorithms across 4 modalities (tabular, video, image features, text embeddings) by systematically varying label quantity, granularity, and quality, with protocols and code provided."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.26068","languages":[],"library_direction":null,"modality":null,"name":"WSADBench","openness":"unknown","publisher":null,"released":"2026-05-25","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.26068","source_url":"https://arxiv.org/abs/2605.26068"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-26068","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26068","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"WSC is a pronoun disambiguation task, which requires to determine which noun the pronoun refers to according to the context. WSC是一个代词消歧任务，要求根据上下文判断代词指代的是哪个名词。","descriptions":{"en":"WSC is a pronoun disambiguation task, which requires to determine which noun the pronoun refers to according to the context.","zh":"WSC是一个代词消歧任务，要求根据上下文判断代词指代的是哪个名词。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":"2019-05-02","first_score_source_reference":{"basis":"score_publication","date_precision":"score_publication","note":"The original benchmark release day is absent from the crawl. Table 2 provides the earliest dated numeric language-model evaluation retained in this date archive for the source-linked WSC task; it is not a benchmark release date.","reported_at":"2019-05-02","score_evidence":{"locator":"Table 2, BERT row, WSC column","metric":"accuracy","model":"BERT-large-cased (fine-tuned)","unit":"percent","value":68.5},"source_key":"opencompass:507","source_url":"https://arxiv.org/abs/1905.00537v1"},"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"opencompass:507","languages":[],"library_direction":null,"modality":null,"name":"WSC","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-507-wsc","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WSC","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","General AI","Language & Knowledge"],"collected_at":"2026-10-14T09:44:51Z","description":"WSE-bench evaluates LLM storytelling in open-ended world simulations, assessing three capacities: Generation Coverage (proportion of planned narrative steps), Consistency (canon coherence), and Richness (meaningful development). Uses a process benchmark to compare agent architectures.","descriptions":{"en":"WSE-bench evaluates LLM storytelling in open-ended world simulations, assessing three capacities: Generation Coverage (proportion of planned narrative steps), Consistency (canon coherence), and Richness (meaningful development). Uses a process benchmark to compare agent architectures."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.15654","languages":[],"library_direction":{"basis":"model-review","categories":["chat-writing","multi-agent-games"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_wse-bench_2966278a"},"modality":null,"name":"WSE-bench","openness":"unknown","publisher":null,"released":"2026-08-16","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.15654","source_url":"https://arxiv.org/abs/2608.15654"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-15654","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15654","topics":["agents","language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"WUICC-bench evaluates image change captioning for web UI visual regression testing, using natural language descriptions of UI changes. Scoring includes caption quality metrics and assesses suppression of non-meaningful noise.","descriptions":{"en":"WUICC-bench evaluates image change captioning for web UI visual regression testing, using natural language descriptions of UI changes. Scoring includes caption quality metrics and assesses suppression of non-meaningful noise."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.01728","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning","agentic-coding"],"join":"claire_record_id","primary":"visual-reasoning","source_record":"bm_wuicc-bench_284cff3d"},"modality":null,"name":"WUICC-bench","openness":"unknown","publisher":null,"released":"2026-07-02","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.01728","source_url":"https://arxiv.org/abs/2607.01728"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-01728","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01728","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"WuYu-EnvLE-Bench evaluates LLMs on environmental law enforcement with 2,521 instances across 14 tasks and 12 pollution-medium subdomains, covering pre-, in-, and post-enforcement workflows. It uses Absolute Environmental Enforcement Score (AES) and Intelligent Enforcement Index (IEI) for evaluation.","descriptions":{"en":"WuYu-EnvLE-Bench evaluates LLMs on environmental law enforcement with 2,521 instances across 14 tasks and 12 pollution-medium subdomains, covering pre-, in-, and post-enforcement workflows. It uses Absolute Environmental Enforcement Score (AES) and Intelligent Enforcement Index (IEI) for evaluation."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.17745","languages":[],"library_direction":{"basis":"model-review","categories":["finance-legal"],"join":"claire_record_id","primary":"finance-legal","source_record":"bm_wuyu-envle-bench_a002dca0"},"modality":null,"name":"WuYu-EnvLE-Bench","openness":"unknown","publisher":null,"released":"2026-07-20","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.17745","source_url":"https://arxiv.org/abs/2607.17745"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-17745","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17745","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Factuality","General AI","Language & Knowledge","Reasoning"],"collected_at":"2026-10-14T09:44:51Z","description":"A multi-level benchmark evaluates LLMs in solid waste management across foundational knowledge, domain reasoning, and expert decision-making using closed-ended and open-ended questions.","descriptions":{"en":"A multi-level benchmark evaluates LLMs in solid waste management across foundational knowledge, domain reasoning, and expert decision-making using closed-ended and open-ended questions."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.07529","languages":[],"library_direction":{"basis":"model-review","categories":["science","knowledge-expert-qa"],"join":"claire_record_id","primary":"science","source_record":"bm_wuyueval_f4ea028f"},"modality":null,"name":"WuYuEval","openness":"unknown","publisher":null,"released":"2026-07-24","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.07529","source_url":"https://arxiv.org/abs/2608.07529"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-07529","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07529","topics":["reasoning-math","science"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"X+Slides evaluates audience-conditioned slide generation from source documents, using a dynamic framework of 8,133 source-grounded probes across 113 topics and seven presentation scenes, reporting metrics like Audience Coverage and Efficiency.","descriptions":{"en":"X+Slides evaluates audience-conditioned slide generation from source documents, using a dynamic framework of 8,133 source-grounded probes across 113 topics and seven presentation scenes, reporting metrics like Audience Coverage and Efficiency."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.19256","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","document-ocr"],"join":"claire_record_id","primary":"professional-work","source_record":"bm_x-slides_283eef58"},"modality":null,"name":"X+Slides","openness":"unknown","publisher":null,"released":"2026-06-17","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.19256","source_url":"https://arxiv.org/abs/2606.19256"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-19256","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19256","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":36,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":31,"join":"claire_record_id","repository":"gh:peiwensun2000/x-stream","source_record":"bm_x-stream_b79ec653"},"categories":["Autonomous Driving","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multimodal large language models on multi-stream streaming understanding, with 4,220 QA pairs across 932 videos covering 11 subtasks in multi-window, multi-view, and multi-device scenarios, using a dual-verification construction pipeline and online inference under a fixed average video-token rate.","descriptions":{"en":"Evaluates multimodal large language models on multi-stream streaming understanding, with 4,220 QA pairs across 932 videos covering 11 subtasks in multi-window, multi-view, and multi-device scenarios, using a dual-verification construction pipeline and online inference under a fixed average video-token rate."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.02482","languages":[],"library_direction":{"basis":"model-review","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_x-stream_b79ec653"},"modality":null,"name":"X-Stream","openness":"unknown","publisher":null,"released":"2026-06-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.02482","source_url":"https://arxiv.org/abs/2606.02482"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-02482","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02482","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robot manipulation","Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Simulation benchmark for real-world robot policy evaluation; no artifacts available.","descriptions":{"en":"Simulation benchmark for real-world robot policy evaluation; no artifacts available."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.27449","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["embodied-robotics"],"join":"claire_record_id","primary":"embodied-robotics","source_record":"bm_x2real_41389678"},"modality":null,"name":"X2Real","openness":"unknown","publisher":null,"released":"2026-09-23","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.27449","source_url":"https://arxiv.org/abs/2609.27449"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-27449","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27449","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["代码","Code","ACL 2024","大语言模型","LLM","代码工程"],"collected_at":"2026-08-18T00:00:00+00:00","description":"xCodeEval is the largest executable multilingual multitask benchmark to date consisting of 25M document-level coding examples (16.5B tokens) from about 7.5K unique problems covering up to 11 programming languages with execution-level parallelism. xCodeEval 是迄今为止最大的可执行多语言多任务基准，包含 2500 万个文档级编码示例（165 亿个标记），来自约 7500 个独特问题，涵盖多达 11 种编程语言。它包括 7 个任务，涉及代码理解、生成、翻译和检索。","descriptions":{"en":"xCodeEval is the largest executable multilingual multitask benchmark to date consisting of 25M document-level coding examples (16.5B tokens) from about 7.5K unique problems covering up to 11 programming languages with execution-level parallelism.","zh":"xCodeEval 是迄今为止最大的可执行多语言多任务基准，包含 2500 万个文档级编码示例（165 亿个标记），来自约 7500 个独特问题，涵盖多达 11 种编程语言。它包括 7 个任务，涉及代码理解、生成、翻译和检索。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1072","languages":["Multilingual"],"library_direction":null,"modality":null,"name":"xCodeEval","openness":"open","publisher":"NTU-NLP","released":"2023-11-06","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1072-xcodeeval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/xCodeEval","topics":["coding"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on 248 tasks across 51 scenarios in personal life, white-collar work, learning and research, and cross-domain activities using binary rubrics for explicit and implicit requirements.","descriptions":{"en":"Evaluates LLMs on 248 tasks across 51 scenarios in personal life, white-collar work, learning and research, and cross-domain activities using binary rubrics for explicit and implicit requirements."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.07784","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["chat-writing"],"join":"claire_record_id","primary":"chat-writing","source_record":"bm_xdailybench_fe82cbbd"},"modality":null,"name":"xDailyBench","openness":"unknown","publisher":null,"released":"2026-09-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.07784","source_url":"https://arxiv.org/abs/2609.07784"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-07784","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07784","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning","general","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"xDailyBench evaluates AI agents on white-collar office work, covering everyday professional tasks such as document handling, consultation, and multi-step productivity workflows.","descriptions":{"en":"xDailyBench evaluates AI agents on white-collar office work, covering everyday professional tasks such as document handling, consultation, and multi-step productivity workflows."},"evidence_summary":{"document_count":1,"model_count":2,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:xdailybench:seed-2.1-pro","reported_at":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xdailybench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:xdailybench","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["chat-writing"],"join":"llm_stats_id","primary":"chat-writing","source_record":"bm_xdailybench_fe82cbbd"},"modality":"text","name":"xDailyBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":2,"score_direction":"higher_is_better","score_summary":{"display_max":61.0,"display_multiplier":100,"model_count":2,"model_count_basis":"source_model_id","numeric_count":2,"raw_max":0.61,"raw_min":0.564,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:xdailybench:seed-2.1-pro","reported_date":"2026-06-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xdailybench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-xdailybench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xdailybench?top_n=500","topics":["agents","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Robotics","Robotics & Embodied AI"],"collected_at":"2026-10-14T09:44:51Z","description":"XEWorld is a testbed for evaluating cross-embodiment generalization of action-conditioned world models, but it is primarily a research probe without a defined public benchmark protocol.","descriptions":{"en":"XEWorld is a testbed for evaluating cross-embodiment generalization of action-conditioned world models, but it is primarily a research probe without a defined public benchmark protocol."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.05799","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","embodied-robotics"],"join":"claire_record_id","primary":"world-models","source_record":"bm_xeworld_c6f125ea"},"modality":null,"name":"XEWorld","openness":"unknown","publisher":null,"released":"2026-08-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.05799","source_url":"https://arxiv.org/abs/2608.05799"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-05799","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05799","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":2,"github_stars_30d":0,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":"gh:g1moon/language-shapes-ih","source_record":"bm_xih-bench_b0306814"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates instruction hierarchy compliance in multilingual LLMs using same-language and cross-language conflicts across six languages, four domains (rule-following, safety, task-execution, persona), and three hierarchy types (system-user, system-tool, user-tool).","descriptions":{"en":"Evaluates instruction hierarchy compliance in multilingual LLMs using same-language and cross-language conflicts across six languages, four domains (rule-following, safety, task-execution, persona), and three hierarchy types (system-user, system-tool, user-tool)."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2607.23545","languages":[],"library_direction":{"basis":"model-review","categories":["instruction-following","multilingual","safety-alignment"],"join":"claire_record_id","primary":"instruction-following","source_record":"bm_xih-bench_b0306814"},"modality":null,"name":"XIH-Bench","openness":"unknown","publisher":null,"released":"2026-07-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.23545","source_url":"https://arxiv.org/abs/2607.23545"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-23545","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23545","topics":["language","safety"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":3,"join":"claire_record_id","repository":null,"source_record":"bm_xl-docbench_d4b9f759"},"categories":["Biology & Drug Discovery","Finance","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"XL-DocBench evaluates evidence-grounded long-document understanding with 1,519 human-verified questions from six professional domains, contexts up to 2,303 pages, multi-page evidence, and typed reasoning rules.","descriptions":{"en":"XL-DocBench evaluates evidence-grounded long-document understanding with 1,519 human-verified questions from six professional domains, contexts up to 2,303 pages, multi-page evidence, and typed reasoning rules."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2608.00036","languages":[],"library_direction":{"basis":"model-review","categories":["document-ocr","long-context"],"join":"claire_record_id","primary":"document-ocr","source_record":"bm_xl-docbench_d4b9f759"},"modality":null,"name":"XL-DocBench","openness":"unknown","publisher":null,"released":"2026-07-21","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.00036","source_url":"https://arxiv.org/abs/2608.00036"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-00036","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00036","topics":["long-context","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":279,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":"gh:csebuetnlp/xl-sum","source_record":"catalog_1b9f8b7eb2079c7b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["summarization","language"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"Large-scale multilingual abstractive summarization dataset comprising 1 million professionally annotated article-summary pairs from BBC, covering 44 languages. XL-Sum is highly abstractive, concise, and of high quality, designed to encourage research on multilingual abstractive summarization tasks.","descriptions":{"en":"Large-scale multilingual abstractive summarization dataset comprising 1 million professionally annotated article-summary pairs from BBC, covering 44 languages. XL-Sum is highly abstractive, concise, and of high quality, designed to encourage research on multilingual abstractive summarization tasks."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:xlsum-english:llama-3.1-nemotron-70b-instruct","reported_at":"2024-10-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xlsum-english?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:xlsum-english","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual","chat-writing"],"join":"llm_stats_id","primary":"multilingual","source_record":"catalog_1b9f8b7eb2079c7b"},"modality":"text","name":"XLSum English","openness":"unknown","publisher":null,"released":"2021-06-25","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:xlsum-english","source_url":"https://arxiv.org/abs/2106.13822"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":31.61,"display_multiplier":100,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":0.3161,"raw_min":0.3161,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:xlsum-english:llama-3.1-nemotron-70b-instruct","reported_date":"2024-10-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xlsum-english?top_n=500"},"unit":null},"scored_models":[{"date":"2024-10-01","date_precision":"model_announcement","name":"Llama 3.1 Nemotron 70B Instruct"}],"slug":"llm-stats-xlsum-english","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xlsum-english?top_n=500","topics":["language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":null,"source_record":"bm_xplainverse_db4c59ee"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"XPlainVerse evaluates deepfake detection and explanation quality, pairing real images with forgeries from twelve models and providing technical and simplified explanations, with metrics EntityScore and EvidenceScore for reasoning fidelity.","descriptions":{"en":"XPlainVerse evaluates deepfake detection and explanation quality, pairing real images with forgeries from twelve models and providing technical and simplified explanations, with metrics EntityScore and EvidenceScore for reasoning fidelity."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.03562","languages":[],"library_direction":{"basis":"model-review","categories":["content-detection","visual-reasoning"],"join":"claire_record_id","primary":"content-detection","source_record":"bm_xplainverse_db4c59ee"},"modality":null,"name":"XPlainVerse","openness":"unknown","publisher":null,"released":"2026-07-03","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.03562","source_url":"https://arxiv.org/abs/2607.03562"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-03562","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03562","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":167,"github_stars_30d":3,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:solis-team/xrepotest","source_record":"bm_xrepotest_04d9b86c"},"categories":["Code & Software","Software & AI Compute","cs.SE"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates multilingual repository-level unit test generation across Rust, Go, Julia, PHP, and Ruby using containerized execution and context augmentation strategies.","descriptions":{"en":"Evaluates multilingual repository-level unit test generation across Rust, Go, Julia, PHP, and Ruby using containerized execution and context augmentation strategies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.25939","languages":[],"library_direction":{"basis":"model-review","categories":["code-generation","multilingual"],"join":"claire_record_id","primary":"code-generation","source_record":"bm_xrepotest_04d9b86c"},"modality":null,"name":"XREPOTEST","openness":"unknown","publisher":null,"released":"2026-08-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.25939","source_url":"https://arxiv.org/abs/2608.25939"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-25939","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25939","topics":["coding","language"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":148,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_524e7cc577548477"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["safety"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"XSTest is a test suite designed to identify exaggerated safety behaviours in large language models. It comprises 450 prompts: 250 safe prompts across ten prompt types that well-calibrated models should not refuse to comply with, and 200 unsafe prompts as contrasts that models should refuse. The benchmark systematically evaluates whether models refuse to respond to clearly safe prompts due to overly cautious safety mechanisms.","descriptions":{"en":"XSTest is a test suite designed to identify exaggerated safety behaviours in large language models. It comprises 450 prompts: 250 safe prompts across ten prompt types that well-calibrated models should not refuse to comply with, and 200 unsafe prompts as contrasts that models should refuse. The benchmark systematically evaluates whether models refuse to respond to clearly safe prompts due to overly cautious safety mechanisms."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:xstest:gemini-1.5-flash-8b","reported_at":"2024-03-15","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xstest?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:xstest","languages":[],"library_direction":{"basis":"model-review","categories":["safety-alignment"],"join":"llm_stats_id","primary":"safety-alignment","source_record":"catalog_524e7cc577548477"},"modality":"text","name":"XSTest","openness":"unknown","publisher":null,"released":"2023-08-02","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:xstest","source_url":"https://arxiv.org/abs/2308.01263"},"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":98.8,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.988,"raw_min":0.926,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:xstest:gemini-1.5-pro","reported_date":"2024-05-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xstest?top_n=500"},"unit":null},"scored_models":[{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Flash"},{"date":"2024-03-15","date_precision":"model_announcement","name":"Gemini 1.5 Flash 8B"},{"date":"2024-05-01","date_precision":"model_announcement","name":"Gemini 1.5 Pro"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Shieldstral 1.0 (3B)"}],"slug":"llm-stats-xstest","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xstest?top_n=500","topics":["safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["理解","Understanding","大语言模型","LLM","语言理解","Comprehension","开源收录","Open-Source"],"collected_at":"2026-08-18T00:00:00+00:00","description":"XSum is a single-document summarization task which does not favor extractive strategies and calls for an abstractive modeling approach. The idea is to create a short, one-sentence news summary answering the question “What is the article about?”. The dataset collects real-world, large scale data by harvesting online articles from the British Broadcasting Corporation (BBC). XSum是一个单文档摘要任务，不支持抽取式策略，需要采用抽象建模方法。其思想是创建一个简短的一句话新闻摘要，回答“这篇文章是关于什么的？”的问题。该数据集通过从英国广播公司（BBC）收集在线文章，得到了大量的现实数据。","descriptions":{"en":"XSum is a single-document summarization task which does not favor extractive strategies and calls for an abstractive modeling approach. The idea is to create a short, one-sentence news summary answering the question “What is the article about?”. The dataset collects real-world, large scale data by harvesting online articles from the British Broadcasting Corporation (BBC).","zh":"XSum是一个单文档摘要任务，不支持抽取式策略，需要采用抽象建模方法。其思想是创建一个简短的一句话新闻摘要，回答“这篇文章是关于什么的？”的问题。该数据集通过从英国广播公司（BBC）收集在线文章，得到了大量的现实数据。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:521","languages":[],"library_direction":null,"modality":null,"name":"XSum","openness":"unknown","publisher":null,"released":"2018-08-27","released_reference":{"basis":"paper_first_version","note":"First version of the paper introducing the Extreme Summarization dataset.","source_key":"opencompass:521","source_url":"https://arxiv.org/abs/1808.08745"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-521-xsum","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/XSum","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["推理","Reasoning","大语言模型","LLM","逻辑推理"],"collected_at":"2026-08-18T00:00:00+00:00","description":"xVerify, an efficient answer verifier for reasoning model evaluations. xVerify demonstrates strong capability in equivalence judgment, enabling it to effectively determine whether the answers produced by reasoning models are equivalent to reference answers across various types of objective questions xVerify，这是一种用于推理模型评估的高效答案验证器。xVerify 在等价判断方面表现出强大的能力，使其能够有效地确定推理模型生成的答案是否等同于各种类型客观问题的参考答案。","descriptions":{"en":"xVerify, an efficient answer verifier for reasoning model evaluations. xVerify demonstrates strong capability in equivalence judgment, enabling it to effectively determine whether the answers produced by reasoning models are equivalent to reference answers across various types of objective questions","zh":"xVerify，这是一种用于推理模型评估的高效答案验证器。xVerify 在等价判断方面表现出强大的能力，使其能够有效地确定推理模型生成的答案是否等同于各种类型客观问题的参考答案。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1784","languages":["English","Chinese"],"library_direction":null,"modality":null,"name":"xVerify","openness":"unknown","publisher":"Peking University，Research Institute of China Telecom，etc.","released":"2025-04-14","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1784-xverify","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/xVerify","topics":["reasoning-math"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A benchmark for Arabic morphological generation across verbs, nouns, adjectives, cliticized forms, and invalid configurations, with over 600K entries.","descriptions":{"en":"A benchmark for Arabic morphological generation across verbs, nouns, adjectives, cliticized forms, and invalid configurations, with over 600K entries."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10153","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["abstract-reasoning","multilingual","document-ocr"],"join":"claire_record_id","primary":"abstract-reasoning","source_record":"bm_yallamorph_68dfa730"},"modality":null,"name":"YallaMorph","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10153","source_url":"https://arxiv.org/abs/2609.10153"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10153","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10153","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":135,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"llm_stats_id","repository":null,"source_record":"catalog_e2642e3ad267e6f8"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["finance","agents"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"YC-Bench evaluates agents on long-horizon, open-ended business and investment decision-making. The reported metric is the final assets (fund value, in US dollars) accumulated by the agent over the course of the simulation.","descriptions":{"en":"YC-Bench evaluates agents on long-horizon, open-ended business and investment decision-making. The reported metric is the final assets (fund value, in US dollars) accumulated by the agent over the course of the simulation."},"evidence_summary":{"document_count":1,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:yc-bench:minimax-m3","reported_at":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/yc-bench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:yc-bench","languages":[],"library_direction":{"basis":"model-review","categories":["professional-work","finance-legal"],"join":"llm_stats_id","primary":"professional-work","source_record":"catalog_e2642e3ad267e6f8"},"modality":"text","name":"YC-Bench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":2100000.0,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":2100000.0,"raw_min":2100000.0,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:yc-bench:minimax-m3","reported_date":"2026-06-01","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/yc-bench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax M3"}],"slug":"llm-stats-yc-bench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/yc-bench?top_n=500","topics":["agents","finance"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":38,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":51,"join":"claire_record_id","repository":"gh:youzhe0305/yocausal","source_record":"bm_yocausal_0e0be7af"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"YoCausal is a two-level benchmark that evaluates video diffusion models' understanding of temporal causality using the Violation of Expectation paradigm. It temporally reverses real-world videos as counterfactual samples and introduces two metrics: the Reverse Surprise Index (RSI) for arrow-of-time perception and the Causality Cognition Index (CCI) for disentangling genuine causal reasoning from temporal bias.","descriptions":{"en":"YoCausal is a two-level benchmark that evaluates video diffusion models' understanding of temporal causality using the Violation of Expectation paradigm. It temporally reverses real-world videos as counterfactual samples and introduces two metrics: the Reverse Surprise Index (RSI) for arrow-of-time perception and the Causality Cognition Index (CCI) for disentangling genuine causal reasoning from temporal bias."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2605.30346","languages":[],"library_direction":{"basis":"model-review","categories":["world-models","video-understanding"],"join":"claire_record_id","primary":"world-models","source_record":"bm_yocausal_0e0be7af"},"modality":null,"name":"YoCausal","openness":"unknown","publisher":null,"released":"2026-05-28","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2605.30346","source_url":"https://arxiv.org/abs/2605.30346"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2605-30346","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30346","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"YOMI-Bench evaluates kanji reading and phonological understanding in LLMs for Japanese through four tasks. It is used in a study assessing multilingual and Japanese-specific models.","descriptions":{"en":"YOMI-Bench evaluates kanji reading and phonological understanding in LLMs for Japanese through four tasks. It is used in a study assessing multilingual and Japanese-specific models."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2607.00664","languages":[],"library_direction":{"basis":"model-review","categories":["multilingual"],"join":"claire_record_id","primary":"multilingual","source_record":"bm_yomi-bench_e60b9cad"},"modality":null,"name":"YOMI-Bench","openness":"unknown","publisher":null,"released":"2026-07-01","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2607.00664","source_url":"https://arxiv.org/abs/2607.00664"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2607-00664","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00664","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["语言","Language","大语言模型","LLM","语言理解","Comprehension"],"collected_at":"2026-08-18T00:00:00+00:00","description":"The benchmarks introduced for evaluating large language models (LLMs) on Cantonese include Yue-TruthfulQA, Yue-GSM8K, Yue-ARC-C, Yue-MMLU, and Yue-TRANS. Each of these benchmarks focuses on different aspects of language understanding and generation in Cantonese, offering a comprehensive means of assessing the capabilities of LLMs in handling this language. Yue_Benchmark用于评估粤语大型语言模型（LLMs）。该评测集包含：Yue TruthtyQA、Yue-GSM8K、Yue-ARC-C、Yue MMLU和Yue TRANS，侧重于粤语语言理解和生成的不同方面，为评估LLM粤语能力提供了一种全面的方法。","descriptions":{"en":"The benchmarks introduced for evaluating large language models (LLMs) on Cantonese include Yue-TruthfulQA, Yue-GSM8K, Yue-ARC-C, Yue-MMLU, and Yue-TRANS. Each of these benchmarks focuses on different aspects of language understanding and generation in Cantonese, offering a comprehensive means of assessing the capabilities of LLMs in handling this language.","zh":"Yue_Benchmark用于评估粤语大型语言模型（LLMs）。该评测集包含：Yue TruthtyQA、Yue-GSM8K、Yue-ARC-C、Yue MMLU和Yue TRANS，侧重于粤语语言理解和生成的不同方面，为评估LLM粤语能力提供了一种全面的方法。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":true,"key":"opencompass:1017","languages":["English"],"library_direction":null,"modality":null,"name":"Yue_Benchmark","openness":"open","publisher":null,"released":"2024-08-31","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1017-yue-benchmark","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Yue_Benchmark","topics":["language"],"unit":null},{"aliases":[],"attention_signals":null,"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["agents","code"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ZClawBench evaluates Claw-style agent task execution quality, measuring a model's ability to autonomously complete complex multi-step coding tasks in real-world environments.","descriptions":{"en":"ZClawBench evaluates Claw-style agent task execution quality, measuring a model's ability to autonomously complete complex multi-step coding tasks in real-world environments."},"evidence_summary":{"document_count":1,"model_count":4,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:zclawbench:glm-5v-turbo","reported_at":"2026-04-02","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zclawbench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:zclawbench","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"llm_stats_id","primary":"agentic-coding","source_record":"catalog_5e2260ba16607194"},"modality":"text","name":"ZClawBench","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":4,"score_direction":"higher_is_better","score_summary":{"display_max":64.3,"display_multiplier":100,"model_count":4,"model_count_basis":"source_model_id","numeric_count":4,"raw_max":0.643,"raw_min":0.526,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:zclawbench:qwen3.7-max","reported_date":"2026-05-19","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zclawbench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-04-02","date_precision":"model_announcement","name":"GLM-5V-Turbo"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Qwen3.6-27B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"}],"slug":"llm-stats-zclawbench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zclawbench?top_n=500","topics":["coding","agents"],"unit":null},{"aliases":["Disparity Has a Sign: Stereo Matching Beyond the Zero-Disparity Plane"],"attention_signals":null,"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"ZDPShift evaluates stereo matching on 21,495 pairs with dense signed ground truth from seven open movies.","descriptions":{"en":"ZDPShift evaluates stereo matching on 21,495 pairs with dense signed ground truth from seven open movies."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.06809","languages":[],"library_direction":{"basis":"keyword-provisional","categories":["video-understanding"],"join":"claire_record_id","primary":"video-understanding","source_record":"bm_disparity-has-a-sign-stereo-matching-beyon_351f0330"},"modality":null,"name":"ZDPShift","openness":"unknown","publisher":null,"released":"2026-09-06","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.06809","source_url":"https://arxiv.org/abs/2609.06809"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-06809","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06809","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":21,"join":"llm_stats_id","repository":null,"source_record":"catalog_39277112cb15425b"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["reasoning"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ZebraLogic is an evaluation framework for assessing large language models' logical reasoning capabilities through logic grid puzzles derived from constraint satisfaction problems (CSPs). The benchmark consists of 1,000 programmatically generated puzzles with controllable and quantifiable complexity, revealing a 'curse of complexity' where model accuracy declines significantly as problem complexity grows.","descriptions":{"en":"ZebraLogic is an evaluation framework for assessing large language models' logical reasoning capabilities through logic grid puzzles derived from constraint satisfaction problems (CSPs). The benchmark consists of 1,000 programmatically generated puzzles with controllable and quantifiable complexity, revealing a 'curse of complexity' where model accuracy declines significantly as problem complexity grows."},"evidence_summary":{"document_count":1,"model_count":8,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:zebralogic:minimax-m1-40k","reported_at":"2025-06-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zebralogic?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:zebralogic","languages":[],"library_direction":{"basis":"model-review","categories":["abstract-reasoning"],"join":"llm_stats_id","primary":"abstract-reasoning","source_record":"catalog_39277112cb15425b"},"modality":"text","name":"ZebraLogic","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":8,"score_direction":"higher_is_better","score_summary":{"display_max":97.3,"display_multiplier":100,"model_count":8,"model_count_basis":"source_model_id","numeric_count":8,"raw_max":0.973,"raw_min":0.801,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:zebralogic:qwen3-vl-235b-a22b-thinking","reported_date":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zebralogic?top_n=500"},"unit":null},"scored_models":[{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2 Instruct"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2-Instruct-0905"},{"date":"2025-08-29","date_precision":"model_announcement","name":"LongCat-Flash-Chat"},{"date":"2025-09-22","date_precision":"model_announcement","name":"LongCat-Flash-Thinking"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 40K"},{"date":"2025-06-16","date_precision":"model_announcement","name":"MiniMax M1 80K"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3-235B-A22B-Instruct-2507"}],"slug":"llm-stats-zebralogic","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zebralogic?top_n=500","topics":["reasoning-math"],"unit":null},{"aliases":["zero-trust-agent-benchmark"],"attention_signals":{"as_of":"2026-10-13","github_stars":0,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":null,"join":"claire_record_id","repository":"gh:zero-trust-agent-benchmark/zero-trust-agent-benchmark","source_record":"bm_zero-trust-agent-benchmark_8d897d96"},"categories":["Agents","Language & Knowledge","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"A reproducible benchmark of synthetic JSONL agent tool-call traces in which a defense returns allow or deny per step; the evaluator scores attack blocking, benign false positives, and canary leaks over fixed dev and test splits.","descriptions":{"en":"A reproducible benchmark of synthetic JSONL agent tool-call traces in which a defense returns allow or deny per step; the evaluator scores attack blocking, benign false positives, and canary leaks over fixed dev and test splits."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"claire-radar:github:zero-trust-agent-benchmark/zero-trust-agent-benchmark","languages":[],"library_direction":{"basis":"model-review","categories":["agent-security"],"join":"claire_record_id","primary":"agent-security","source_record":"bm_zero-trust-agent-benchmark_8d897d96"},"modality":null,"name":"Zero Trust Agent Benchmark","openness":"unknown","publisher":null,"released":"2026-09-26","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:github:zero-trust-agent-benchmark/zero-trust-agent-benchmark","source_url":"https://github.com/zero-trust-agent-benchmark/zero-trust-agent-benchmark"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-github-zero-trust-agent-benchmark-zero-trust-agent-benchmark","source":"claire_radar","source_url":"https://github.com/zero-trust-agent-benchmark/zero-trust-agent-benchmark","topics":["agents","safety"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["Agents","Code & Software","Software & AI Compute"],"collected_at":"2026-10-14T09:44:51Z","description":"Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks.","descriptions":{"en":"Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.38269","languages":[],"library_direction":{"basis":"model-review","categories":["agentic-coding"],"join":"claire_record_id","primary":"agentic-coding","source_record":"bm_zero2repo_b9221863"},"modality":null,"name":"Zero2Repo","openness":"unknown","publisher":null,"released":"2026-09-29","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.38269","source_url":"https://arxiv.org/abs/2609.38269"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-38269","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38269","topics":["coding","agents"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":71,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":43,"join":"llm_stats_id","repository":null,"source_record":"catalog_702c994cae574a63"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ZEROBench is a challenging vision benchmark designed to test models on zero-shot visual understanding tasks.","descriptions":{"en":"ZEROBench is a challenging vision benchmark designed to test models on zero-shot visual understanding tasks."},"evidence_summary":{"document_count":1,"model_count":9,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:zerobench:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:zerobench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_702c994cae574a63"},"modality":"image","name":"ZEROBench","openness":"unknown","publisher":null,"released":"2025-02-13","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"llm-stats:zerobench","source_url":"https://arxiv.org/abs/2502.09696"},"repo_kind":null,"repo_resolution_status":null,"score_count":9,"score_direction":"higher_is_better","score_summary":{"display_max":0.41,"display_multiplier":1,"model_count":9,"model_count_basis":"source_model_id","numeric_count":9,"raw_max":0.41,"raw_min":0.04,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:zerobench:kimi-k3","reported_date":"2026-07-16","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench?top_n=500"},"unit":null},"scored_models":[{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Pro"},{"date":"2026-06-24","date_precision":"model_announcement","name":"Seed 2.1 Turbo"}],"slug":"llm-stats-zerobench","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["ZeroBench","ZeroBench-main"],"attention_signals":{"as_of":"2026-10-12","github_stars":71,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":43,"join":"artifact:gh:jonathan-roberts1/zerobench","repository":"gh:jonathan-roberts1/zerobench","source_record":"catalog_f5593566c5676379"},"categories":["multimodal"],"collected_at":null,"description":"Challenging visual reasoning benchmark designed to be impossible for frontier models at release; evaluates complex reasoning over images.","descriptions":{"en":"Challenging visual reasoning benchmark designed to be impossible for frontier models at release; evaluates complex reasoning over images."},"evidence_summary":{"document_count":2,"model_count":1,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"document_publication","obs_id":"curated\u0000zerobench\u0000bytedance_seed2_0_model_card\u0000zerobench_main\u0000accuracy on main and sub questions\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"first_score_reported_at":"2026-02-14","first_score_source_reference":{"date_precision":"document_publication","obs_id":"curated\u0000zerobench\u0000bytedance_seed2_0_model_card\u0000zerobench_main\u0000accuracy on main and sub questions\u0000Seed2.0 Pro","reported_at":"2026-02-14","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"has_dataset":false,"has_paper":false,"has_repo":true,"has_size":false,"key":"model-reports:zerobench","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:gh:jonathan-roberts1/zerobench","primary":"visual-reasoning","source_record":"catalog_f5593566c5676379"},"modality":null,"name":"ZeroBench","openness":"unknown","publisher":null,"released":"2025-02-13","released_reference":{"basis":"paper_first_version","note":"First version of the introducing paper: ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models. The source-record description identifies this benchmark or its original partition; no dates or scores are joined by name at runtime.","source_key":"model-reports:zerobench","source_url":"https://arxiv.org/abs/2502.09696"},"repo_kind":null,"repo_resolution_status":null,"score_count":1,"score_direction":"higher_is_better","score_summary":{"display_max":12.0,"display_multiplier":1,"model_count":1,"model_count_basis":"source_model_id","numeric_count":1,"raw_max":12.0,"raw_min":12.0,"source_reference":{"obs_id":"curated\u0000zerobench\u0000bytedance_seed2_0_model_card\u0000zerobench_main\u0000accuracy on main and sub questions\u0000Seed2.0 Pro","observation_id":"curated\u0000zerobench\u0000bytedance_seed2_0_model_card\u0000zerobench_main\u0000accuracy on main and sub questions\u0000Seed2.0 Pro","reported_at":"2026-02-14","reported_date":"2026-02-14","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf"},"unit":"percent"},"scored_models":[{"date":"2026-02-14","date_precision":"document_publication","name":"Seed2.0 Pro"}],"slug":"zerobench","source":"model_reports","source_url":"https://github.com/jonathan-roberts1/zerobench","topics":["multimodal"],"unit":"percent"},{"aliases":[],"attention_signals":{"as_of":"2026-10-10","github_stars":71,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":43,"join":"artifact:arxiv:2502.09696","repository":null,"source_record":"catalog_702c994cae574a63"},"categories":["多模态","Multimodal","多模态模型","VLM","跨模态推理","Cross-modal Reasoning"],"collected_at":"2026-08-18T00:00:00+00:00","description":"ZeroBench is a challenging visual reasoning benchmark for LMMs. It consists of a main set of 100 high-quality, manually curated questions covering numerous domains, reasoning types and image type. Questions have been designed and calibrated to be beyond the capabilities of current frontier models. ZeroBench 是针对多模态模型（LMMs）的具有挑战性的视觉推理基准。它由一组主要的 100 个高质量人工问题组成，涵盖多个领域、推理类型和图像类型。ZeroBench 中的问题经过设计和校准，已经超出了当前前沿模型的能力范围。","descriptions":{"en":"ZeroBench is a challenging visual reasoning benchmark for LMMs. It consists of a main set of 100 high-quality, manually curated questions covering numerous domains, reasoning types and image type. Questions have been designed and calibrated to be beyond the capabilities of current frontier models.","zh":"ZeroBench 是针对多模态模型（LMMs）的具有挑战性的视觉推理基准。它由一组主要的 100 个高质量人工问题组成，涵盖多个领域、推理类型和图像类型。ZeroBench 中的问题经过设计和校准，已经超出了当前前沿模型的能力范围。"},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":true,"has_paper":true,"has_repo":true,"has_size":false,"key":"opencompass:1532","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"artifact:arxiv:2502.09696","primary":"visual-reasoning","source_record":"catalog_702c994cae574a63"},"modality":"multimodal","name":"ZeroBench","openness":"restricted","publisher":"University of Cambridge, University of Alberta,etc.","released":"2025-02-13","released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"opencompass-1532-zerobench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ZeroBench","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-11","github_stars":71,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":43,"join":"llm_stats_id","repository":null,"source_record":"catalog_c670958d92dab5d2"},"attribution":{"name":"LLM Stats","terms_url":"https://llm-stats.com/legal/terms-of-service","url":"https://llm-stats.com/"},"categories":["multimodal","reasoning","vision"],"collected_at":"2026-08-17T18:49:35.489925+00:00","description":"ZEROBench-Sub is a subset of the ZEROBench benchmark.","descriptions":{"en":"ZEROBench-Sub is a subset of the ZEROBench benchmark."},"evidence_summary":{"document_count":1,"model_count":5,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"llm_stats:zerobench-sub:qwen3-vl-235b-a22b-thinking","reported_at":"2025-09-22","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench-sub?top_n=500"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"llm-stats:zerobench-sub","languages":[],"library_direction":{"basis":"model-review","categories":["visual-reasoning"],"join":"llm_stats_id","primary":"visual-reasoning","source_record":"catalog_c670958d92dab5d2"},"modality":"image","name":"ZEROBench-Sub","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":5,"score_direction":"higher_is_better","score_summary":{"display_max":0.362,"display_multiplier":1,"model_count":5,"model_count_basis":"source_model_id","numeric_count":5,"raw_max":0.362,"raw_min":0.277,"source_reference":{"crawled_at":"2026-08-17T18:49:35.489925+00:00","obs_id":"llm_stats:zerobench-sub:qwen3.5-122b-a10b","reported_date":"2026-02-24","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench-sub?top_n=500"},"unit":null},"scored_models":[{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Thinking"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-122B-A10B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-27B"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5-35B-A3B"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6-35B-A3B"}],"slug":"llm-stats-zerobench-sub","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench-sub?top_n=500","topics":["multimodal","reasoning-math"],"unit":null},{"aliases":["Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasets"],"attention_signals":null,"categories":["General AI","Information retrieval","Vision & 3D","cs.CV"],"collected_at":"2026-09-25T04:40:22Z","description":"A zero-shot composed-image-retrieval benchmark built from temporally controlled video frames, with evaluation based on ranking positive and negative targets.","descriptions":{"en":"A zero-shot composed-image-retrieval benchmark built from temporally controlled video frames, with evaluation based on ranking positive and negative targets."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.07032","languages":[],"library_direction":null,"modality":null,"name":"ZeroSight","openness":"unknown","publisher":null,"released":"2026-06-05","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.07032","source_url":"https://arxiv.org/abs/2606.07032"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-07032","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07032","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":null,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":1,"join":"claire_record_id","repository":null,"source_record":"bm_zipbench_2610de18"},"categories":["General AI","Vision & 3D","cs.AI"],"collected_at":"2026-10-14T09:44:51Z","description":"ZIPBench is a zero-shot personalization benchmark with 1.5K users, graph-mined personas, and 40K generated images for evaluating text-to-image personalization.","descriptions":{"en":"ZIPBench is a zero-shot personalization benchmark with 1.5K users, graph-mined personas, and 40K generated images for evaluating text-to-image personalization."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2606.08841","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_zipbench_2610de18"},"modality":null,"name":"ZIPBench","openness":"unknown","publisher":null,"released":"2026-06-07","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.08841","source_url":"https://arxiv.org/abs/2606.08841"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-08841","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08841","topics":[],"unit":null},{"aliases":[],"attention_signals":{"as_of":"2026-10-13","github_stars":6,"github_stars_30d":null,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:milkthink-lab/zipbench","source_record":"bm_zipbench_638d2f5b"},"categories":["General AI","Language & Knowledge","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"A collection of compressed benchmark versions for text, multimodal, and agent tasks, with tools to evaluate models or create new compact benchmarks.","descriptions":{"en":"A collection of compressed benchmark versions for text, multimodal, and agent tasks, with tools to evaluate models or create new compact benchmarks."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2609.12475","languages":[],"library_direction":null,"modality":null,"name":"ZipBench","openness":"unknown","publisher":null,"released":"2026-09-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.12475","source_url":"https://arxiv.org/abs/2609.12475"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-12475","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12475","topics":[],"unit":null},{"aliases":["$\\pi$-SUB: A Physics-Informed Synthetic Underwater Benchmark Dataset for Underwater Image Enhancement"],"attention_signals":{"as_of":"2026-10-12","github_stars":1,"github_stars_30d":1,"hf_dataset_downloads":null,"hf_paper_upvotes":0,"join":"claire_record_id","repository":"gh:airl-iisc/pi-sub","source_record":"bm_pi-sub-a-physics-informed-synthetic-underw_d13ec116"},"categories":["General AI","Vision & 3D","cs.CV"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates underwater image enhancement models using a synthetic paired dataset generated by a physics-informed framework with depth-dependent irradiance and Jerlov water types.","descriptions":{"en":"Evaluates underwater image enhancement models using a synthetic paired dataset generated by a physics-informed framework with depth-dependent irradiance and Jerlov water types."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2608.10589","languages":[],"library_direction":{"basis":"model-review","categories":["image-video-generation"],"join":"claire_record_id","primary":"image-video-generation","source_record":"bm_pi-sub-a-physics-informed-synthetic-underw_d13ec116"},"modality":null,"name":"π-SUB","openness":"unknown","publisher":null,"released":"2026-08-11","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2608.10589","source_url":"https://arxiv.org/abs/2608.10589"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2608-10589","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10589","topics":[],"unit":null},{"aliases":["$\\tau$-Rec: A Verifiable Benchmark for Agentic Recommender Systems"],"attention_signals":null,"categories":["General AI","Language & Knowledge","cs.IR"],"collected_at":"2026-09-25T04:40:22Z","description":"A benchmark for multi-turn agentic recommender systems that exposes dialogue constraints through reveal-tagged elicitation and verifies outcomes against structured catalog predicates.","descriptions":{"en":"A benchmark for multi-turn agentic recommender systems that exposes dialogue constraints through reveal-tagged elicitation and verifies outcomes against structured catalog predicates."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":true,"has_size":false,"key":"claire-radar:2606.10156","languages":[],"library_direction":null,"modality":null,"name":"τ-Rec","openness":"unknown","publisher":null,"released":"2026-06-08","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2606.10156","source_url":"https://arxiv.org/abs/2606.10156"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2606-10156","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10156","topics":[],"unit":null},{"aliases":[],"attention_signals":null,"categories":["agentic","tool-use"],"collected_at":"2026-08-25T10:41:06Z","description":"Agentic tool use","descriptions":{"en":"Agentic tool use"},"evidence_summary":{"document_count":1,"model_count":440,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:tau2-bench-telecom:217b34ec-5920-4fc1-8886-6a70a324837d","reported_at":"2023-09-27","source_url":"https://artificialanalysis.ai/evaluations/tau2-bench"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:tau2-bench-telecom","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["tool-use"],"join":"review:14H034160212:2026-10-15","primary":"tool-use","source_record":null},"modality":null,"name":"τ²-Bench Telecom","openness":"unknown","publisher":null,"released":"2025-06-12","released_reference":{"basis":"release_announcement","note":"Version 0.1.0 is labelled the initial public release and explicitly includes the Telecom domain. This is not the older tau-bench airline or retail release.","source_key":"artificial-analysis:tau2-bench-telecom","source_url":"https://github.com/sierra-research/tau2-bench/blob/main/RELEASE_NOTES.md"},"repo_kind":null,"repo_resolution_status":null,"score_count":440,"score_direction":"higher_is_better","score_summary":{"display_max":99.1228070175439,"display_multiplier":100,"model_count":440,"model_count_basis":"source_model_id","numeric_count":440,"raw_max":0.991228070175439,"raw_min":0.0,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:tau2-bench-telecom:0121d27b-5b8a-4901-8684-80589cc6d40d","reported_date":"2026-05-14","source_url":"https://artificialanalysis.ai/evaluations/tau2-bench"},"unit":null},"scored_models":[{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 70B Instruct"},{"date":"2025-09-02","date_precision":"model_announcement","name":"Apertus 8B Instruct"},{"date":"2025-09-30","date_precision":"model_announcement","name":"Apriel-v1.5-15B-Thinker"},{"date":"2025-11-25","date_precision":"model_announcement","name":"Apriel-v1.6-15B-Thinker"},{"date":"2024-03-04","date_precision":"model_announcement","name":"Claude 3 Haiku"},{"date":"2024-10-22","date_precision":"model_announcement","name":"Claude 3.5 Haiku"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Non-reasoning)"},{"date":"2025-02-24","date_precision":"model_announcement","name":"Claude 3.7 Sonnet (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Opus (Reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Non-reasoning)"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"Claude 4.1 Opus (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Non-reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Non-reasoning)"},{"date":"2025-11-24","date_precision":"model_announcement","name":"Claude Opus 4.5 (Reasoning)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"Claude Opus 4.6 (Non-reasoning, High Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Non-reasoning, High Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, High Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Non-reasoning, Low Effort)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Command A"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2025-05-28","date_precision":"model_announcement","name":"DeepSeek R1 0528 (May '25)"},{"date":"2025-05-29","date_precision":"model_announcement","name":"DeepSeek R1 0528 Qwen3 8B"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 Distill Llama 70B"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Non-reasoning)"},{"date":"2025-08-21","date_precision":"model_announcement","name":"DeepSeek V3.1 (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Non-reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Non-reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Non-reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"DeepSeek V3.2 Exp (Reasoning)"},{"date":"2025-12-01","date_precision":"model_announcement","name":"DeepSeek V3.2 Speciale"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Non-reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Medium"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Devstral Small (Jul '25)"},{"date":"2025-05-21","date_precision":"model_announcement","name":"Devstral Small (May '25)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"Doubao Seed Code"},{"date":"2025-06-30","date_precision":"model_announcement","name":"ERNIE 4.5 300B A47B"},{"date":"2025-11-13","date_precision":"model_announcement","name":"ERNIE 5.0 Thinking Preview"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"Exaone 4.0 1.2B (Reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Non-reasoning)"},{"date":"2025-07-15","date_precision":"model_announcement","name":"EXAONE 4.0 32B (Reasoning)"},{"date":"2026-04-09","date_precision":"model_announcement","name":"EXAONE 4.5 33B"},{"date":"2026-01-04","date_precision":"model_announcement","name":"Falcon-H1R-7B"},{"date":"2025-02-05","date_precision":"model_announcement","name":"Gemini 2.0 Flash (Feb '25)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Non-reasoning)"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Gemini 2.5 Flash (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash Preview (Sep '25) (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Non-reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite (Reasoning)"},{"date":"2025-09-25","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Non-reasoning)"},{"date":"2025-09-08","date_precision":"model_announcement","name":"Gemini 2.5 Flash-Lite Preview (Sep '25) (Reasoning)"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Non-reasoning)"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (high)"},{"date":"2025-11-18","date_precision":"model_announcement","name":"Gemini 3 Pro Preview (low)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (medium)"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (minimal)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-13","date_precision":"model_announcement","name":"Gemma 3 1B Instruct"},{"date":"2025-08-14","date_precision":"model_announcement","name":"Gemma 3 270M"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E2B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Non-reasoning)"},{"date":"2026-06-03","date_precision":"model_announcement","name":"Gemma 4 12B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 E2B (Reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Non-reasoning)"},{"date":"2026-04-03","date_precision":"model_announcement","name":"Gemma 4 E4B (Reasoning)"},{"date":"2026-04-01","date_precision":"model_announcement","name":"GLM 5V Turbo (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5 (Reasoning)"},{"date":"2025-07-28","date_precision":"model_announcement","name":"GLM-4.5-Air"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Non-reasoning)"},{"date":"2025-08-11","date_precision":"model_announcement","name":"GLM-4.5V (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Non-reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Non-reasoning)"},{"date":"2025-12-08","date_precision":"model_announcement","name":"GLM-4.6V (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Non-reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Non-reasoning)"},{"date":"2026-01-19","date_precision":"model_announcement","name":"GLM-4.7-Flash (Reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Non-reasoning)"},{"date":"2026-02-11","date_precision":"model_announcement","name":"GLM-5 (Reasoning)"},{"date":"2026-03-15","date_precision":"model_announcement","name":"GLM-5-Turbo"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-08-06","date_precision":"model_announcement","name":"GPT-4o (Aug '24)"},{"date":"2024-11-20","date_precision":"model_announcement","name":"GPT-4o (Nov '24)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (ChatGPT)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (low)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (minimal)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"GPT-5 Codex (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (minimal)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (medium)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 nano (minimal)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (Non-reasoning)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 Codex mini (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (medium)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (xhigh)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 Codex (xhigh)"},{"date":"2026-02-05","date_precision":"model_announcement","name":"GPT-5.3 Codex (xhigh)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (low)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (medium)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (Non-Reasoning)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-05-05","date_precision":"model_announcement","name":"GPT-5.5 Instant (May 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (low)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"Granite 3.3 8B (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 350M"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 1B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"Granite 4.0 H 350M"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 H Small"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Granite 4.0 Micro"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 30B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Granite 4.1 8B"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3"},{"date":"2025-02-19","date_precision":"model_announcement","name":"Grok 3 mini Reasoning (high)"},{"date":"2025-07-10","date_precision":"model_announcement","name":"Grok 4"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Non-reasoning)"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Grok 4 Fast (Reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Non-reasoning)"},{"date":"2025-11-19","date_precision":"model_announcement","name":"Grok 4.1 Fast (Reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Non-reasoning)"},{"date":"2026-03-10","date_precision":"model_announcement","name":"Grok 4.20 0309 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Non-reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"Grok 4.20 0309 v2 (Reasoning)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (low)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (medium)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2025-08-28","date_precision":"model_announcement","name":"Grok Code Fast 1"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 405B (Reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Non-reasoning)"},{"date":"2025-08-27","date_precision":"model_announcement","name":"Hermes 4 - Llama-3.1 70B (Reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Hy3-preview (Reasoning)"},{"date":"2025-12-26","date_precision":"model_announcement","name":"HyperCLOVA X SEED Think (32B)"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2025-11-27","date_precision":"model_announcement","name":"INTELLECT-3"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Large"},{"date":"2025-07-07","date_precision":"model_announcement","name":"Jamba 1.7 Mini"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Jamba Reasoning 3B"},{"date":"2026-05-14","date_precision":"model_announcement","name":"JT-35B-Flash"},{"date":"2026-04-15","date_precision":"model_announcement","name":"JT-MINI"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Non-reasoning)"},{"date":"2025-12-31","date_precision":"model_announcement","name":"K-EXAONE (Reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"K2 Think V2"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (high)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (low)"},{"date":"2025-12-05","date_precision":"model_announcement","name":"K2-V2 (medium)"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2025-11-11","date_precision":"model_announcement","name":"KAT-Coder-Pro V1"},{"date":"2025-07-11","date_precision":"model_announcement","name":"Kimi K2"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Kimi K2 0905"},{"date":"2025-11-06","date_precision":"model_announcement","name":"Kimi K2 Thinking"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Non-reasoning)"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6 (Non-reasoning)"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2025-10-30","date_precision":"model_announcement","name":"Kimi Linear 48B A3B Instruct"},{"date":"2025-07-10","date_precision":"model_announcement","name":"LFM2 1.2B"},{"date":"2025-09-23","date_precision":"model_announcement","name":"LFM2 2.6B"},{"date":"2026-02-25","date_precision":"model_announcement","name":"LFM2 24B A2B"},{"date":"2025-10-07","date_precision":"model_announcement","name":"LFM2 8B A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-1.2B-Instruct"},{"date":"2026-01-20","date_precision":"model_announcement","name":"LFM2.5-1.2B-Thinking"},{"date":"2026-05-28","date_precision":"model_announcement","name":"LFM2.5-8B-A1B"},{"date":"2026-01-05","date_precision":"model_announcement","name":"LFM2.5-VL-1.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2025-10-08","date_precision":"model_announcement","name":"Ling-1T"},{"date":"2026-04-23","date_precision":"model_announcement","name":"Ling-2.6-1T"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Ling-flash-2.0"},{"date":"2025-09-09","date_precision":"model_announcement","name":"Ling-mini-2.0"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 70B"},{"date":"2024-04-18","date_precision":"model_announcement","name":"Llama 3 Instruct 8B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 405B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 70B"},{"date":"2024-07-23","date_precision":"model_announcement","name":"Llama 3.1 Instruct 8B"},{"date":"2024-10-15","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Instruct 70B"},{"date":"2025-05-20","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Nano 4B v1.1 (Reasoning)"},{"date":"2025-04-07","date_precision":"model_announcement","name":"Llama 3.1 Nemotron Ultra 253B v1 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 11B (Vision)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 1B"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Llama 3.2 Instruct 3B"},{"date":"2024-12-06","date_precision":"model_announcement","name":"Llama 3.3 Instruct 70B"},{"date":"2025-03-18","date_precision":"model_announcement","name":"Llama 3.3 Nemotron Super 49B v1 (Reasoning)"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Non-reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Llama Nemotron Super 49B v1.5 (Reasoning)"},{"date":"2026-01-28","date_precision":"model_announcement","name":"LongCat Flash Lite"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Medium 1"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-06-10","date_precision":"model_announcement","name":"Magistral Small 1"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Mi:dm K 2.5 Pro Preview"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Feb 2026)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Non-reasoning)"},{"date":"2025-12-16","date_precision":"model_announcement","name":"MiMo-V2-Flash (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"MiMo-V2-Omni"},{"date":"2026-03-27","date_precision":"model_announcement","name":"MiMo-V2-Omni-0327"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiMo-V2-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro (Non-reasoning)"},{"date":"2026-05-11","date_precision":"model_announcement","name":"MiniCPM-V 4.6 1.3B"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Non-reasoning)"},{"date":"2026-05-25","date_precision":"model_announcement","name":"MiniCPM5-1B (Reasoning)"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 40k"},{"date":"2025-06-17","date_precision":"model_announcement","name":"MiniMax M1 80k"},{"date":"2025-10-26","date_precision":"model_announcement","name":"MiniMax-M2"},{"date":"2025-12-23","date_precision":"model_announcement","name":"MiniMax-M2.1"},{"date":"2026-02-12","date_precision":"model_announcement","name":"MiniMax-M2.5"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2023-09-27","date_precision":"model_announcement","name":"Mistral 7B Instruct"},{"date":"2024-07-24","date_precision":"model_announcement","name":"Mistral Large 2 (Jul '24)"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Mistral Large 2 (Nov '24)"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-05-07","date_precision":"model_announcement","name":"Mistral Medium 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-01-30","date_precision":"model_announcement","name":"Mistral Small 3"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Non-reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2024-09-25","date_precision":"model_announcement","name":"Molmo 7B-D"},{"date":"2025-12-11","date_precision":"model_announcement","name":"Molmo2-8B"},{"date":"2025-12-04","date_precision":"model_announcement","name":"Motif-2-12.7B-Reasoning"},{"date":"2026-04-08","date_precision":"model_announcement","name":"Muse Spark"},{"date":"2026-02-11","date_precision":"model_announcement","name":"Nanbeige4.1-3B"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Nemotron 3 Nano Omni 30B A3B Reasoning"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-03-19","date_precision":"model_announcement","name":"Nemotron Cascade 2 30B A3B"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (high)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (low)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (medium)"},{"date":"2025-10-29","date_precision":"model_announcement","name":"Nova 2.0 Lite (Non-reasoning)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (low)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (medium)"},{"date":"2025-11-26","date_precision":"model_announcement","name":"Nova 2.0 Omni (Non-reasoning)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (low)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (medium)"},{"date":"2025-11-27","date_precision":"model_announcement","name":"Nova 2.0 Pro Preview (Non-reasoning)"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Lite"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Micro"},{"date":"2025-04-30","date_precision":"model_announcement","name":"Nova Premier"},{"date":"2024-12-03","date_precision":"model_announcement","name":"Nova Pro"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning)"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2026-03-16","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 4B"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning)"},{"date":"2025-10-28","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 12B v2 VL (Reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Non-reasoning)"},{"date":"2025-08-18","date_precision":"model_announcement","name":"NVIDIA Nemotron Nano 9B V2 (Reasoning)"},{"date":"2024-12-05","date_precision":"model_announcement","name":"o1"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o3"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-16","date_precision":"model_announcement","name":"o4-mini (high)"},{"date":"2025-03-13","date_precision":"model_announcement","name":"OLMo 2 32B"},{"date":"2024-11-26","date_precision":"model_announcement","name":"OLMo 2 7B"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 32B Think"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Instruct"},{"date":"2025-11-20","date_precision":"model_announcement","name":"Olmo 3 7B Think"},{"date":"2026-01-13","date_precision":"model_announcement","name":"Olmo 3.1 32B Instruct"},{"date":"2025-12-12","date_precision":"model_announcement","name":"Olmo 3.1 32B Think"},{"date":"2024-04-23","date_precision":"model_announcement","name":"Phi-3 Mini Instruct 3.8B"},{"date":"2024-12-12","date_precision":"model_announcement","name":"Phi-4"},{"date":"2024-02-26","date_precision":"model_announcement","name":"Phi-4 Mini Instruct"},{"date":"2024-11-18","date_precision":"model_announcement","name":"Pixtral Large"},{"date":"2024-09-19","date_precision":"model_announcement","name":"Qwen2.5 Instruct 72B"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 0.6B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 1.7B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 235B A22B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-21","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 30B A3B (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-07-29","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 4B (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 (Reasoning)"},{"date":"2025-08-06","date_precision":"model_announcement","name":"Qwen3 4B 2507 Instruct"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Non-reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2025-07-31","date_precision":"model_announcement","name":"Qwen3 Coder 30B A3B Instruct"},{"date":"2025-07-22","date_precision":"model_announcement","name":"Qwen3 Coder 480B A35B Instruct"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 Max"},{"date":"2025-09-05","date_precision":"model_announcement","name":"Qwen3 Max (Preview)"},{"date":"2026-01-26","date_precision":"model_announcement","name":"Qwen3 Max Thinking"},{"date":"2025-11-03","date_precision":"model_announcement","name":"Qwen3 Max Thinking (Preview)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B Instruct"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B (Reasoning)"},{"date":"2025-09-22","date_precision":"model_announcement","name":"Qwen3 Omni 30B A3B Instruct"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B (Reasoning)"},{"date":"2025-09-23","date_precision":"model_announcement","name":"Qwen3 VL 235B A22B Instruct"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B (Reasoning)"},{"date":"2025-10-03","date_precision":"model_announcement","name":"Qwen3 VL 30B A3B Instruct"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B (Reasoning)"},{"date":"2025-10-21","date_precision":"model_announcement","name":"Qwen3 VL 32B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 4B Instruct"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B (Reasoning)"},{"date":"2025-10-14","date_precision":"model_announcement","name":"Qwen3 VL 8B Instruct"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 0.8B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 27B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 2B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Flash"},{"date":"2026-03-30","date_precision":"model_announcement","name":"Qwen3.5 Omni Plus"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Qwen3.6 Max Preview"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2025-03-10","date_precision":"model_announcement","name":"Reka Flash 3"},{"date":"2025-10-13","date_precision":"model_announcement","name":"Ring-1T"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2025-09-19","date_precision":"model_announcement","name":"Ring-flash-2.0"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 105B (high)"},{"date":"2026-03-06","date_precision":"model_announcement","name":"Sarvam 30B (high)"},{"date":"2025-05-23","date_precision":"model_announcement","name":"Sarvam M (Reasoning)"},{"date":"2025-08-20","date_precision":"model_announcement","name":"Seed-OSS-36B-Instruct"},{"date":"2024-01-25","date_precision":"model_announcement","name":"Solar Mini"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Solar Open 100B (Reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Non-reasoning)"},{"date":"2025-07-09","date_precision":"model_announcement","name":"Solar Pro 2 (Reasoning)"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-02-02","date_precision":"model_announcement","name":"Step 3.5 Flash"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Step 3.5 Flash 2603"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-01-20","date_precision":"model_announcement","name":"Step3 VL 10B"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Tiny Aya Global"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-Think"},{"date":"2026-02-10","date_precision":"model_announcement","name":"Tri-21B-think Preview"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-tau2-bench-telecom","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/tau2-bench","topics":["agents"],"unit":null},{"aliases":[],"attention_signals":null,"categories":["intelligence-index","agentic","tool-use"],"collected_at":"2026-08-25T10:41:06Z","description":"Agentic tool use","descriptions":{"en":"Agentic tool use"},"evidence_summary":{"document_count":1,"model_count":178,"model_count_basis":"source_model_id"},"first_score_record":{"date_precision":"model_announcement","obs_id":"artificial_analysis:tau3-banking:b5c1c91a-7474-4409-9a9c-9c2ac45d9eb6","reported_at":"2024-07-18","source_url":"https://artificialanalysis.ai/evaluations/tau3-banking"},"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":false,"has_repo":false,"has_size":false,"key":"artificial-analysis:tau3-banking","languages":[],"library_direction":{"basis":"reviewer_asserted","categories":["tool-use"],"join":"review:14H034160212:2026-10-15","primary":"tool-use","source_record":null},"modality":null,"name":"τ³-Banking","openness":"unknown","publisher":null,"released":null,"released_reference":null,"repo_kind":null,"repo_resolution_status":null,"score_count":178,"score_direction":"higher_is_better","score_summary":{"display_max":51.340206185567006,"display_multiplier":100,"model_count":178,"model_count_basis":"source_model_id","numeric_count":178,"raw_max":0.51340206185567,"raw_min":0.00206185567010309,"source_reference":{"crawled_at":"2026-08-25T10:41:06Z","obs_id":"artificial_analysis:tau3-banking:5e5b4ce7-bc54-47b2-b911-21b9cad8394c","reported_date":"2026-08-03","source_url":"https://artificialanalysis.ai/evaluations/tau3-banking"},"unit":null},"scored_models":[{"date":"2026-08-12","date_precision":"model_announcement","name":"A.X-K2"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Agnes 2.5 Pro Alpha"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Celeris-1"},{"date":"2025-05-22","date_precision":"model_announcement","name":"Claude 4 Sonnet (Reasoning)"},{"date":"2025-10-15","date_precision":"model_announcement","name":"Claude 4.5 Haiku (Reasoning)"},{"date":"2025-09-29","date_precision":"model_announcement","name":"Claude 4.5 Sonnet (Reasoning)"},{"date":"2026-06-09","date_precision":"model_announcement","name":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Claude Opus 4.7 (Adaptive Reasoning, Max Effort)"},{"date":"2026-05-28","date_precision":"model_announcement","name":"Claude Opus 4.8 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, High Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Low Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Medium Effort)"},{"date":"2026-07-24","date_precision":"model_announcement","name":"Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)"},{"date":"2026-02-17","date_precision":"model_announcement","name":"Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Adaptive Reasoning, Max Effort)"},{"date":"2026-06-30","date_precision":"model_announcement","name":"Claude Sonnet 5 (Non-reasoning, High Effort)"},{"date":"2026-05-20","date_precision":"model_announcement","name":"Command A+"},{"date":"2025-01-20","date_precision":"model_announcement","name":"DeepSeek R1 (Jan '25)"},{"date":"2024-12-26","date_precision":"model_announcement","name":"DeepSeek V3 (Dec '24)"},{"date":"2025-03-25","date_precision":"model_announcement","name":"DeepSeek V3 0324"},{"date":"2025-09-22","date_precision":"model_announcement","name":"DeepSeek V3.1 Terminus (Reasoning)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Flash (Reasoning, Max Effort)"},{"date":"2026-07-31","date_precision":"model_announcement","name":"DeepSeek V4 Flash 0731 (Reasoning, Max Effort)"},{"date":"2026-08-21","date_precision":"model_announcement","name":"DeepSeek V4 Flash Vision (Reasoning, Max Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, High Effort)"},{"date":"2026-04-24","date_precision":"model_announcement","name":"DeepSeek V4 Pro (Reasoning, Max Effort)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"DeepSeek V4 Pro 0813 (Reasoning, Max Effort)"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral 2"},{"date":"2025-12-09","date_precision":"model_announcement","name":"Devstral Small 2"},{"date":"2026-08-20","date_precision":"model_announcement","name":"G9v3-39A5B"},{"date":"2025-06-05","date_precision":"model_announcement","name":"Gemini 2.5 Pro"},{"date":"2025-12-17","date_precision":"model_announcement","name":"Gemini 3 Flash Preview (Reasoning)"},{"date":"2026-03-03","date_precision":"model_announcement","name":"Gemini 3.1 Flash-Lite"},{"date":"2026-02-19","date_precision":"model_announcement","name":"Gemini 3.1 Pro Preview"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Gemini 3.5 Flash (high)"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.5 Flash-Lite"},{"date":"2026-07-21","date_precision":"model_announcement","name":"Gemini 3.6 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (high)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (low)"},{"date":"2026-08-13","date_precision":"model_announcement","name":"Gemini 3.7 Flash (medium)"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 12B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 27B Instruct"},{"date":"2025-03-12","date_precision":"model_announcement","name":"Gemma 3 4B Instruct"},{"date":"2025-06-26","date_precision":"model_announcement","name":"Gemma 3n E4B Instruct"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 26B A4B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Non-reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Gemma 4 31B (Reasoning)"},{"date":"2025-09-30","date_precision":"model_announcement","name":"GLM-4.6 (Reasoning)"},{"date":"2025-12-22","date_precision":"model_announcement","name":"GLM-4.7 (Reasoning)"},{"date":"2026-04-07","date_precision":"model_announcement","name":"GLM-5.1 (Reasoning)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (max)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"GLM-5.2 (Non-reasoning)"},{"date":"2026-08-18","date_precision":"model_announcement","name":"GLM-5.3 (max)"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 mini"},{"date":"2025-04-14","date_precision":"model_announcement","name":"GPT-4.1 nano"},{"date":"2024-07-18","date_precision":"model_announcement","name":"GPT-4o mini"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 (high)"},{"date":"2025-08-07","date_precision":"model_announcement","name":"GPT-5 mini (high)"},{"date":"2025-11-13","date_precision":"model_announcement","name":"GPT-5.1 (high)"},{"date":"2025-12-11","date_precision":"model_announcement","name":"GPT-5.2 (Non-reasoning)"},{"date":"2026-03-05","date_precision":"model_announcement","name":"GPT-5.4 (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 mini (xhigh)"},{"date":"2026-03-17","date_precision":"model_announcement","name":"GPT-5.4 nano (xhigh)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (high)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (low)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (medium)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (Non-reasoning)"},{"date":"2026-04-23","date_precision":"model_announcement","name":"GPT-5.5 (xhigh)"},{"date":"2026-06-25","date_precision":"model_announcement","name":"GPT-5.5 Instant (June 2026)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Luna (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Sol (xhigh)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (low)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (max)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (medium)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (Non-reasoning)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"GPT-5.6 Terra (xhigh)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (high)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-120b (low)"},{"date":"2025-08-05","date_precision":"model_announcement","name":"gpt-oss-20b (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (high)"},{"date":"2026-04-30","date_precision":"model_announcement","name":"Grok 4.3 (Non-reasoning)"},{"date":"2026-07-08","date_precision":"model_announcement","name":"Grok 4.5 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (high)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (low)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (medium)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Grok 4.6 (xhigh)"},{"date":"2026-06-16","date_precision":"model_announcement","name":"Grok Build 0.1 0616"},{"date":"2026-07-06","date_precision":"model_announcement","name":"Hy3"},{"date":"2026-05-26","date_precision":"model_announcement","name":"HyperNova 60B 2605"},{"date":"2026-07-15","date_precision":"model_announcement","name":"Inkling (xhigh)"},{"date":"2026-07-30","date_precision":"model_announcement","name":"Inkling Small"},{"date":"2026-08-12","date_precision":"model_announcement","name":"K-EXAONE 2.0 0803"},{"date":"2026-03-27","date_precision":"model_announcement","name":"KAT Coder Pro V2"},{"date":"2026-01-27","date_precision":"model_announcement","name":"Kimi K2.5 (Reasoning)"},{"date":"2026-04-20","date_precision":"model_announcement","name":"Kimi K2.6"},{"date":"2026-06-12","date_precision":"model_announcement","name":"Kimi K2.7 Code"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (low)"},{"date":"2026-07-16","date_precision":"model_announcement","name":"Kimi K3 (max)"},{"date":"2026-08-04","date_precision":"model_announcement","name":"LFM2.5-2.6B"},{"date":"2026-04-21","date_precision":"model_announcement","name":"Ling 2.6 Flash"},{"date":"2026-08-04","date_precision":"model_announcement","name":"Ling 3.0 Flash"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Ling 3.0 Tiny"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Maverick"},{"date":"2025-04-05","date_precision":"model_announcement","name":"Llama 4 Scout"},{"date":"2026-06-29","date_precision":"model_announcement","name":"LongCat 2.0"},{"date":"2025-09-18","date_precision":"model_announcement","name":"Magistral Medium 1.2"},{"date":"2025-09-17","date_precision":"model_announcement","name":"Magistral Small 1.2"},{"date":"2026-02-20","date_precision":"model_announcement","name":"Mercury 2"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5"},{"date":"2026-04-22","date_precision":"model_announcement","name":"MiMo-V2.5-Pro"},{"date":"2026-03-18","date_precision":"model_announcement","name":"MiniMax-M2.7"},{"date":"2026-06-01","date_precision":"model_announcement","name":"MiniMax-M3"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 14B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 3B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Ministral 3 8B"},{"date":"2025-12-02","date_precision":"model_announcement","name":"Mistral Large 3"},{"date":"2025-08-12","date_precision":"model_announcement","name":"Mistral Medium 3.1"},{"date":"2026-04-29","date_precision":"model_announcement","name":"Mistral Medium 3.5"},{"date":"2025-03-17","date_precision":"model_announcement","name":"Mistral Small 3.1"},{"date":"2025-06-20","date_precision":"model_announcement","name":"Mistral Small 3.2"},{"date":"2026-03-16","date_precision":"model_announcement","name":"Mistral Small 4 (Reasoning)"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Motif 3"},{"date":"2026-08-10","date_precision":"model_announcement","name":"Muse Glimmer (high)"},{"date":"2026-07-09","date_precision":"model_announcement","name":"Muse Spark 1.1 (xhigh)"},{"date":"2026-08-05","date_precision":"model_announcement","name":"Muse Spark 1.2 (xhigh)"},{"date":"2026-03-11","date_precision":"model_announcement","name":"Nemotron 3 Super 120B A12B (Reasoning)"},{"date":"2026-06-04","date_precision":"model_announcement","name":"Nemotron 3 Ultra 550B A55B (Reasoning)"},{"date":"2026-08-11","date_precision":"model_announcement","name":"Nemotron 3.5 Lightning"},{"date":"2026-06-02","date_precision":"model_announcement","name":"Nex-N2-Pro"},{"date":"2026-06-09","date_precision":"model_announcement","name":"North Mini Code"},{"date":"2025-12-15","date_precision":"model_announcement","name":"NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)"},{"date":"2025-01-31","date_precision":"model_announcement","name":"o3-mini (high)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 14B (Reasoning)"},{"date":"2025-07-25","date_precision":"model_announcement","name":"Qwen3 235B A22B 2507 (Reasoning)"},{"date":"2025-07-30","date_precision":"model_announcement","name":"Qwen3 30B A3B 2507 (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 32B (Reasoning)"},{"date":"2025-04-28","date_precision":"model_announcement","name":"Qwen3 8B (Reasoning)"},{"date":"2026-02-03","date_precision":"model_announcement","name":"Qwen3 Coder Next"},{"date":"2025-09-11","date_precision":"model_announcement","name":"Qwen3 Next 80B A3B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Non-reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 122B A10B (Reasoning)"},{"date":"2026-02-24","date_precision":"model_announcement","name":"Qwen3.5 35B A3B (Non-reasoning)"},{"date":"2026-02-16","date_precision":"model_announcement","name":"Qwen3.5 397B A17B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Non-reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 4B (Reasoning)"},{"date":"2026-03-02","date_precision":"model_announcement","name":"Qwen3.5 9B (Reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Non-reasoning)"},{"date":"2026-04-22","date_precision":"model_announcement","name":"Qwen3.6 27B (Reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Non-reasoning)"},{"date":"2026-04-16","date_precision":"model_announcement","name":"Qwen3.6 35B A3B (Reasoning)"},{"date":"2026-04-02","date_precision":"model_announcement","name":"Qwen3.6 Plus"},{"date":"2026-05-19","date_precision":"model_announcement","name":"Qwen3.7 Max"},{"date":"2026-06-01","date_precision":"model_announcement","name":"Qwen3.7 Plus"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Qwen3.8 2.4T A95B"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (low)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (medium)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (Non-reasoning)"},{"date":"2026-08-14","date_precision":"model_announcement","name":"Qwen3.8 27B (xhigh)"},{"date":"2026-08-03","date_precision":"model_announcement","name":"Qwen3.8 Max"},{"date":"2026-05-08","date_precision":"model_announcement","name":"Ring-2.6-1T"},{"date":"2026-08-12","date_precision":"model_announcement","name":"Solar Open2 250B"},{"date":"2026-04-06","date_precision":"model_announcement","name":"Solar Pro 3"},{"date":"2026-08-06","date_precision":"model_announcement","name":"Solar Pro 4"},{"date":"2026-05-29","date_precision":"model_announcement","name":"Step 3.7 Flash"},{"date":"2026-04-01","date_precision":"model_announcement","name":"Trinity Large Thinking"}],"slug":"artificial-analysis-tau3-banking","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/tau3-banking","topics":["agents"],"unit":null},{"aliases":["$\\Phi$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?"],"attention_signals":{"as_of":"2026-10-12","github_stars":67,"github_stars_30d":10,"hf_dataset_downloads":null,"hf_paper_upvotes":18,"join":"claire_record_id","repository":null,"source_record":"bm_phi-bench-can-large-language-models-engine_4e5601b7"},"categories":["Language & Knowledge","Software & AI Compute","cs.CL"],"collected_at":"2026-10-14T09:44:51Z","description":"Evaluates LLMs on tasks across the LLM infrastructure stack, from kernel completion to end-to-end system optimization using real-world code repositories.","descriptions":{"en":"Evaluates LLMs on tasks across the LLM infrastructure stack, from kernel completion to end-to-end system optimization using real-world code repositories."},"evidence_summary":{"document_count":1,"model_count":null,"model_count_basis":null},"first_score_record":null,"first_score_reported_at":null,"first_score_source_reference":null,"has_dataset":false,"has_paper":true,"has_repo":false,"has_size":false,"key":"claire-radar:2609.10226","languages":[],"library_direction":{"basis":"model-review","categories":["ai-rnd","code-generation"],"join":"claire_record_id","primary":"ai-rnd","source_record":"bm_phi-bench-can-large-language-models-engine_4e5601b7"},"modality":null,"name":"Φ-Bench","openness":"unknown","publisher":null,"released":"2026-09-09","released_reference":{"basis":"benchmark_release","source_key":"claire-radar:2609.10226","source_url":"https://arxiv.org/abs/2609.10226"},"repo_kind":null,"repo_resolution_status":null,"score_count":0,"score_direction":null,"score_summary":null,"scored_models":[],"slug":"claire-radar-2609-10226","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10226","topics":["coding"],"unit":null}],"count":3772,"document_registry":{"benchmark_count":3772,"document_count":3672,"documents":[{"benchmarks":[{"benchmark_id":"artificial-analysis-aa-analystagent","domain":"agentic","name":"AA-AnalystAgent","released":null,"url":"https://artificialanalysis.ai/evaluations/aa-analyst-agent"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/aa-analyst-agent","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/aa-analyst-agent","title":"AA-AnalystAgent"},{"benchmarks":[{"benchmark_id":"artificial-analysis-aa-briefcase","domain":"agentic","name":"AA-Briefcase","released":"2026-06-18","url":"https://artificialanalysis.ai/evaluations/aa-briefcase"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/aa-briefcase","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/aa-briefcase","title":"AA-Briefcase"},{"benchmarks":[{"benchmark_id":"artificial-analysis-aime-2025","domain":"reasoning","name":"AIME 2025","released":"2025-02-12","url":"https://artificialanalysis.ai/evaluations/aime-2025"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/aime-2025","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/aime-2025","title":"AIME 2025"},{"benchmarks":[{"benchmark_id":"artificial-analysis-apex-agents-aa","domain":"agentic","name":"APEX-Agents-AA","released":null,"url":"https://artificialanalysis.ai/evaluations/apex-agents-aa"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/apex-agents-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/apex-agents-aa","title":"APEX-Agents-AA"},{"benchmarks":[{"benchmark_id":"artificial-analysis-aa-lcr","domain":"intelligence-index","name":"AA-LCR","released":"2025-08-05","url":"https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning","title":"AA-LCR"},{"benchmarks":[{"benchmark_id":"artificial-analysis-automationbench-aa","domain":"agentic","name":"AutomationBench-AA","released":null,"url":"https://artificialanalysis.ai/evaluations/automationbench-aa"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/automationbench-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/automationbench-aa","title":"AutomationBench-AA"},{"benchmarks":[{"benchmark_id":"artificial-analysis-critpt","domain":"intelligence-index","name":"CritPt","released":"2025-11-21","url":"https://artificialanalysis.ai/evaluations/critpt"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/critpt","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/critpt","title":"CritPt"},{"benchmarks":[{"benchmark_id":"artificial-analysis-enterpriseops-gym-aa","domain":"agentic","name":"EnterpriseOps-Gym-AA","released":null,"url":"https://artificialanalysis.ai/evaluations/enterprise-ops-gym-aa"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/enterprise-ops-gym-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/enterprise-ops-gym-aa","title":"EnterpriseOps-Gym-AA"},{"benchmarks":[{"benchmark_id":"artificial-analysis-gdpval-aa-v2-normalized-score","domain":"intelligence-index","name":"GDPval-AA v2 (normalized score)","released":"2026-06-15","url":"https://artificialanalysis.ai/evaluations/gdpval-aa"},{"benchmark_id":"artificial-analysis-gdpval-aa-v2-raw-elo","domain":"intelligence-index","name":"GDPval-AA v2 (Elo)","released":"2026-06-15","url":"https://artificialanalysis.ai/evaluations/gdpval-aa"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/gdpval-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/gdpval-aa","title":"GDPval-AA v2 (normalized score)"},{"benchmarks":[{"benchmark_id":"artificial-analysis-gpqa-diamond","domain":"intelligence-index","name":"GPQA Diamond","released":"2023-11-20","url":"https://artificialanalysis.ai/evaluations/gpqa-diamond"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/gpqa-diamond","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond","title":"GPQA Diamond"},{"benchmarks":[{"benchmark_id":"artificial-analysis-harvey-lab-aa","domain":"agentic","name":"Harvey LAB-AA","released":null,"url":"https://artificialanalysis.ai/evaluations/harvey-lab-aa"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/harvey-lab-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/harvey-lab-aa","title":"Harvey LAB-AA"},{"benchmarks":[{"benchmark_id":"artificial-analysis-humanitys-last-exam","domain":"intelligence-index","name":"Humanity’s Last Exam","released":"2025-01-23","url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/humanitys-last-exam","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam","title":"Humanity’s Last Exam"},{"benchmarks":[{"benchmark_id":"artificial-analysis-ifbench","domain":"instruction-following","name":"IFBench","released":"2025-07-03","url":"https://artificialanalysis.ai/evaluations/ifbench"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/ifbench","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/ifbench","title":"IFBench"},{"benchmarks":[{"benchmark_id":"artificial-analysis-itbench-aa","domain":"agentic","name":"ITBench-AA","released":null,"url":"https://artificialanalysis.ai/evaluations/itbench-aa"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/itbench-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/itbench-aa","title":"ITBench-AA"},{"benchmarks":[{"benchmark_id":"artificial-analysis-livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://artificialanalysis.ai/evaluations/livecodebench"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/livecodebench","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/livecodebench","title":"LiveCodeBench"},{"benchmarks":[{"benchmark_id":"artificial-analysis-mlcr-aa","domain":"reasoning","name":"MLCR-AA","released":null,"url":"https://artificialanalysis.ai/evaluations/mlcr-aa"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/mlcr-aa","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/mlcr-aa","title":"MLCR-AA"},{"benchmarks":[{"benchmark_id":"artificial-analysis-mmmu-pro","domain":"multimodal","name":"MMMU-Pro","released":"2024-09-05","url":"https://artificialanalysis.ai/evaluations/mmmu-pro"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/mmmu-pro","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/mmmu-pro","title":"MMMU-Pro"},{"benchmarks":[{"benchmark_id":"artificial-analysis-aa-omniscience-accuracy","domain":"intelligence-index","name":"AA-Omniscience Accuracy","released":"2025-11-16","url":"https://artificialanalysis.ai/evaluations/omniscience"},{"benchmark_id":"artificial-analysis-aa-omniscience-non-hallucination","domain":"intelligence-index","name":"AA-Omniscience Non-Hallucination Rate","released":"2025-11-16","url":"https://artificialanalysis.ai/evaluations/omniscience"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/omniscience","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/omniscience","title":"AA-Omniscience Accuracy"},{"benchmarks":[{"benchmark_id":"artificial-analysis-scicode","domain":"intelligence-index","name":"SciCode","released":"2024-07-18","url":"https://artificialanalysis.ai/evaluations/scicode"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/scicode","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/scicode","title":"SciCode"},{"benchmarks":[{"benchmark_id":"artificial-analysis-tau2-bench-telecom","domain":"agentic","name":"τ²-Bench Telecom","released":"2025-06-12","url":"https://artificialanalysis.ai/evaluations/tau2-bench"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/tau2-bench","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/tau2-bench","title":"τ²-Bench Telecom"},{"benchmarks":[{"benchmark_id":"artificial-analysis-tau3-banking","domain":"intelligence-index","name":"τ³-Banking","released":null,"url":"https://artificialanalysis.ai/evaluations/tau3-banking"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/tau3-banking","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/tau3-banking","title":"τ³-Banking"},{"benchmarks":[{"benchmark_id":"artificial-analysis-terminal-bench-hard","domain":"coding","name":"Terminal-Bench Hard","released":"2025-09-02","url":"https://artificialanalysis.ai/evaluations/terminalbench-hard"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/terminalbench-hard","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard","title":"Terminal-Bench Hard"},{"benchmarks":[{"benchmark_id":"artificial-analysis-terminal-bench-v2-1","domain":"intelligence-index","name":"Terminal-Bench v2.1","released":"2026-05-06","url":"https://artificialanalysis.ai/evaluations/terminalbench-v2-1"}],"document_type":"registry_page","id":"artificial_analysis:https://artificialanalysis.ai/evaluations/terminalbench-v2-1","source":"artificial_analysis","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-v2-1","title":"Terminal-Bench v2.1"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-21714","domain":"General AI","name":"LëtzCross","released":"2026-08-22","url":"http://arxiv.org/abs/2608.21714v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.21714v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21714v1","title":"LëtzCross"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-21827","domain":"General AI","name":"Peony","released":"2026-08-22","url":"http://arxiv.org/abs/2608.21827v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.21827v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21827v1","title":"Peony"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-21832","domain":"General AI","name":"GUI-Primitives","released":"2026-08-22","url":"http://arxiv.org/abs/2608.21832v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.21832v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21832v1","title":"GUI-Primitives"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-21833","domain":"General AI","name":"GameXpert-Bench","released":"2026-08-22","url":"http://arxiv.org/abs/2608.21833v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.21833v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21833v1","title":"GameXpert-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-21839","domain":"General AI","name":"FIRM-Video-Bench","released":"2026-08-22","url":"http://arxiv.org/abs/2608.21839v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.21839v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21839v1","title":"FIRM-Video-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-21880","domain":"General AI","name":"BanglaVeilGuard","released":"2026-08-22","url":"http://arxiv.org/abs/2608.21880v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.21880v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21880v1","title":"BanglaVeilGuard"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-21928","domain":"Robotics & Embodied AI","name":"GuardianBench","released":"2026-08-22","url":"http://arxiv.org/abs/2608.21928v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.21928v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.21928v1","title":"GuardianBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22059","domain":"Biology & Drug Discovery","name":"CRS-Bench","released":"2026-08-22","url":"http://arxiv.org/abs/2608.22059v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22059v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22059v1","title":"CRS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22103","domain":"General AI","name":"Hack-Verifiable Terminal Bench","released":"2026-08-22","url":"http://arxiv.org/abs/2608.22103v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22103v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22103v1","title":"Hack-Verifiable Terminal Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22236","domain":"General AI","name":"MRMAD","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22236v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22236v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22236v1","title":"MRMAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22301","domain":"Robotics","name":"The Imitator Game","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22301v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22301v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22301v1","title":"The Imitator Game"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22309","domain":"General AI","name":"StocBench","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22309v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22309v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22309v1","title":"StocBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22323","domain":"Biology & Drug Discovery","name":"MedReaMM","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22323v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22323v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22323v1","title":"MedReaMM"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22335","domain":"General AI","name":"BanglaSafe","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22335v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22335v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22335v1","title":"BanglaSafe"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22390","domain":"General AI","name":"SchemaGUI","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22390v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22390v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22390v1","title":"SchemaGUI"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22460","domain":"General AI","name":"MASH-Bench","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22460v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22460v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22460v1","title":"MASH-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22465","domain":"General AI","name":"M$^3$ISR","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22465v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22465v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22465v1","title":"M$^3$ISR"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22510","domain":"Agents","name":"ClawProBench","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22510v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22510v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22510v1","title":"ClawProBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22516","domain":"General AI","name":"VES-Bench","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22516v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22516v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22516v1","title":"VES-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22637","domain":"Geometric reasoning","name":"OmniCAD","released":"2026-08-23","url":"http://arxiv.org/abs/2608.22637v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22637v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22637v1","title":"OmniCAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22753","domain":"Factuality","name":"RuleWorld","released":"2026-08-24","url":"http://arxiv.org/abs/2608.22753v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22753v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22753v1","title":"RuleWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22770","domain":"Finance","name":"DelistBench","released":"2026-08-24","url":"http://arxiv.org/abs/2608.22770v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22770v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22770v1","title":"DelistBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22808","domain":"Agents","name":"CatchBench","released":"2026-08-24","url":"http://arxiv.org/abs/2608.22808v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22808v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22808v1","title":"CatchBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22842","domain":"Finance","name":"FinixDocBench","released":"2026-08-24","url":"http://arxiv.org/abs/2608.22842v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22842v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22842v1","title":"FinixDocBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22917","domain":"Biology & Drug Discovery","name":"TSWAP Thai Wellness Benchmark","released":"2026-08-24","url":"http://arxiv.org/abs/2608.22917v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22917v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22917v1","title":"TSWAP Thai Wellness Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22948","domain":"General AI","name":"Lit2Test","released":"2026-08-24","url":"http://arxiv.org/abs/2608.22948v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22948v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22948v1","title":"Lit2Test"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22950","domain":"General AI","name":"WADE","released":"2026-08-24","url":"http://arxiv.org/abs/2608.22950v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22950v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22950v1","title":"WADE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22959","domain":"General AI","name":"WildHandBench","released":"2026-08-24","url":"http://arxiv.org/abs/2608.22959v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22959v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22959v1","title":"WildHandBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22990","domain":"Robot manipulation","name":"InstructMove","released":"2026-08-24","url":"http://arxiv.org/abs/2608.22990v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.22990v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.22990v1","title":"InstructMove"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23065","domain":"General AI","name":"Cultural Moment Benchmark","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23065v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23065v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23065v1","title":"Cultural Moment Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23067","domain":"Agents","name":"WebDev-Skills-Bench","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23067v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23067v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23067v1","title":"WebDev-Skills-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23329","domain":"General AI","name":"VideoRover-Bench","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23329v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23329v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23329v1","title":"VideoRover-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23390","domain":"General AI","name":"CLAW-4L","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23390v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23390v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23390v1","title":"CLAW-4L"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23435","domain":"General AI","name":"BasketballBench","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23435v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23435v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23435v1","title":"BasketballBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23468","domain":"Cybersecurity","name":"Relational Anomaly Detection Benchmark","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23468v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23468v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23468v1","title":"Relational Anomaly Detection Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23475","domain":"General AI","name":"StrategyBench","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23475v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23475v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23475v1","title":"StrategyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23507","domain":"General AI","name":"MHER","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23507v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23507v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23507v1","title":"MHER"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23525","domain":"General AI","name":"EarthVerse","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23525v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23525v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23525v1","title":"EarthVerse"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23564","domain":"Language & Knowledge","name":"SWE Refactor Bench","released":"2026-08-24","url":"http://arxiv.org/abs/2608.23564v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.23564v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.23564v1","title":"SWE Refactor Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27923","domain":"General AI","name":"PCBnet","released":"2026-08-28","url":"http://arxiv.org/abs/2608.27923v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.27923v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.27923v1","title":"PCBnet"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28021","domain":"Cybersecurity","name":"GenIaC-SecBench","released":"2026-08-28","url":"http://arxiv.org/abs/2608.28021v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.28021v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28021v1","title":"GenIaC-SecBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28053","domain":"Cybersecurity","name":"CNeo-Bench","released":"2026-08-28","url":"http://arxiv.org/abs/2608.28053v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.28053v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28053v1","title":"CNeo-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28206","domain":"General AI","name":"NumBench","released":"2026-08-28","url":"http://arxiv.org/abs/2608.28206v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.28206v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28206v1","title":"NumBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28266","domain":"Agents","name":"CoCoBench","released":"2026-08-28","url":"http://arxiv.org/abs/2608.28266v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.28266v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28266v1","title":"CoCoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28281","domain":"Agents","name":"LoopArena","released":"2026-08-28","url":"http://arxiv.org/abs/2608.28281v1"}],"document_type":"registry_page","id":"claire_radar:http://arxiv.org/abs/2608.28281v1","source":"claire_radar","source_url":"http://arxiv.org/abs/2608.28281v1","title":"LoopArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-21858","domain":"General AI","name":"HyperAlign-Bench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.21858"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.21858","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.21858","title":"HyperAlign-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-21919","domain":"General AI","name":"SDGBiasBench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.21919"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.21919","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.21919","title":"SDGBiasBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-21974","domain":"Factuality","name":"CSVFidelity-Bench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.21974"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.21974","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.21974","title":"CSVFidelity-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22034","domain":"General AI","name":"AgroVG","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22034"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22034","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22034","title":"AgroVG"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22068","domain":"General AI","name":"COCOTree","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22068"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22068","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22068","title":"COCOTree"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22079","domain":"General AI","name":"Ishigaki-IDS-Bench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22079"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22079","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22079","title":"Ishigaki-IDS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22080","domain":"Biology & Drug Discovery","name":"JMed48k","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22080"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22080","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22080","title":"JMed48k"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22100","domain":"General AI","name":"MPDocBench-Parse","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22100"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22100","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22100","title":"MPDocBench-Parse"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22175","domain":"Code","name":"SWE-Mutation","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22175"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22175","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22175","title":"SWE-Mutation"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22219","domain":"General AI","name":"SGR-Bench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22219"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22219","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22219","title":"SGR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22247","domain":"General AI","name":"IdioLink","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22247"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22247","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22247","title":"IdioLink"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22304","domain":"Factuality","name":"KGI-Bench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22304"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22304","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22304","title":"KGI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22321","domain":"Agents","name":"A3S-Bench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22321"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22321","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22321","title":"A3S-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22366","domain":"General AI","name":"AgroTools","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22366"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22366","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22366","title":"AgroTools"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22413","domain":"General AI","name":"ReceiptBench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22413"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22413","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22413","title":"ReceiptBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22570","domain":"General AI","name":"VGenST-Bench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22570"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22570","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22570","title":"VGenST-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22643","domain":"General AI","name":"Boiling the Frog","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22643"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22643","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22643","title":"Boiling the Frog"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22645","domain":"General AI","name":"AtelierEval","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22645"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22645","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22645","title":"AtelierEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-22907","domain":"General AI","name":"VideoOdyssey","released":"2026-05-21","url":"https://arxiv.org/abs/2605.22907"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.22907","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.22907","title":"VideoOdyssey"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23055","domain":"General AI","name":"EvalAwareBench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.23055"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23055","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23055","title":"EvalAwareBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23168","domain":"Cybersecurity","name":"PoisonForge","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23168"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23168","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23168","title":"PoisonForge"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23176","domain":"Autonomous Driving","name":"DRIVESPATIAL","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23176"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23176","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23176","title":"DRIVESPATIAL"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23187","domain":"Robotics & Embodied AI","name":"IntentionNav","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23187"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23187","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23187","title":"IntentionNav"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23215","domain":"Code","name":"FastKernels","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23215"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23215","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23215","title":"FastKernels"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23216","domain":"General AI","name":"CaST-Bench","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23216"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23216","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23216","title":"CaST-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23237","domain":"General AI","name":"StereoGenBench","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23237"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23237","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23237","title":"StereoGenBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23543","domain":"Code & Software","name":"JEDI","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23543"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23543","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23543","title":"JEDI"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23629","domain":"Biology & Drug Discovery","name":"DDX-TRACE","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23629"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23629","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23629","title":"DDX-TRACE"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23668","domain":"General AI","name":"NQP-Bench","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23668"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23668","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23668","title":"NQP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-23699","domain":"Cybersecurity","name":"CRONOS","released":"2026-05-22","url":"https://arxiv.org/abs/2605.23699"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.23699","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.23699","title":"CRONOS"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24043","domain":"AI Scientist","name":"ActiveSciBench","released":"2026-05-21","url":"https://arxiv.org/abs/2605.24043"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24043","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24043","title":"ActiveSciBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24045","domain":"Biology & Drug Discovery","name":"A Large-Scale Dataset and Benchmark","released":"2026-05-21","url":"https://arxiv.org/abs/2605.24045"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24045","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24045","title":"A Large-Scale Dataset and Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24110","domain":"General AI","name":"EvoCode-Bench","released":"2026-05-22","url":"https://arxiv.org/abs/2605.24110"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24110","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24110","title":"EvoCode-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24117","domain":"General AI","name":"SkillEvolBench","released":"2026-05-22","url":"https://arxiv.org/abs/2605.24117"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24117","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24117","title":"SkillEvolBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24183","domain":"General AI","name":"AvalancheBench","released":"2026-05-22","url":"https://arxiv.org/abs/2605.24183"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24183","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24183","title":"AvalancheBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24267","domain":"General AI","name":"DRInQ","released":"2026-05-22","url":"https://arxiv.org/abs/2605.24267"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24267","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24267","title":"DRInQ"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24279","domain":"General AI","name":"ContextEcho","released":"2026-05-22","url":"https://arxiv.org/abs/2605.24279"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24279","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24279","title":"ContextEcho"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24305","domain":"General AI","name":"ChaosBench-Logic","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24305"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24305","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24305","title":"ChaosBench-Logic"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24417","domain":"General AI","name":"LLMTabBench","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24417"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24417","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24417","title":"LLMTabBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24456","domain":"General AI","name":"EgoProx","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24456"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24456","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24456","title":"EgoProx"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24503","domain":"General AI","name":"FoodMonitor","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24503"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24503","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24503","title":"FoodMonitor"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24573","domain":"General AI","name":"AstroMind","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24573"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24573","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24573","title":"AstroMind"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24635","domain":"Biology & Drug Discovery","name":"HiMed","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24635"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24635","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24635","title":"HiMed"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24636","domain":"Biology & Drug Discovery","name":"GlobalDentBench","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24636"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24636","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24636","title":"GlobalDentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24652","domain":"General AI","name":"AVBench","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24652"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24652","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24652","title":"AVBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24687","domain":"General AI","name":"HoloFair","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24687"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24687","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24687","title":"HoloFair"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24703","domain":"General AI","name":"TS-Skill","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24703"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24703","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24703","title":"TS-Skill"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24765","domain":"Cybersecurity","name":"CyberMaskQA","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24765"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24765","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24765","title":"CyberMaskQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24777","domain":"General AI","name":"MR-LiDAR","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24777"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24777","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24777","title":"MR-LiDAR"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24782","domain":"General AI","name":"TC-Bench","released":"2026-05-23","url":"https://arxiv.org/abs/2605.24782"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24782","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24782","title":"TC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-24945","domain":"General AI","name":"RealBench","released":"2026-05-24","url":"https://arxiv.org/abs/2605.24945"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.24945","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.24945","title":"RealBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25009","domain":"General AI","name":"ClueAegis-Bench","released":"2026-05-24","url":"https://arxiv.org/abs/2605.25009"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25009","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25009","title":"ClueAegis-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25200","domain":"General AI","name":"GroupTravelBench","released":"2026-05-24","url":"https://arxiv.org/abs/2605.25200"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25200","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25200","title":"GroupTravelBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25240","domain":"General AI","name":"JudgmentBench","released":"2026-05-24","url":"https://arxiv.org/abs/2605.25240"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25240","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25240","title":"JudgmentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25382","domain":"General AI","name":"AuthTrace","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25382"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25382","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25382","title":"AuthTrace"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25388","domain":"Biology & Drug Discovery","name":"ViroBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25388"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25388","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25388","title":"ViroBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25461","domain":"General AI","name":"MetaphorVU-Bench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25461"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25461","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25461","title":"MetaphorVU-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25510","domain":"General AI","name":"KIDBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25510"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25510","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25510","title":"KIDBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25535","domain":"General AI","name":"PerMemBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25535"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25535","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25535","title":"PerMemBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25615","domain":"General AI","name":"UAV-OVO","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25615"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25615","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25615","title":"UAV-OVO"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25626","domain":"General AI","name":"CULTURE-MT","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25626"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25626","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25626","title":"CULTURE-MT"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25707","domain":"Agents","name":"AgentHijack","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25707"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25707","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25707","title":"AgentHijack"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25717","domain":"General AI","name":"FLOATBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25717"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25717","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25717","title":"FLOATBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25758","domain":"General AI","name":"StreamProfileBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25758"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25758","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25758","title":"StreamProfileBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25764","domain":"Biology & Drug Discovery","name":"SpaPath-Bench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25764"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25764","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25764","title":"SpaPath-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25784","domain":"General AI","name":"VertiCue-Bench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25784"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25784","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25784","title":"VertiCue-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-25874","domain":"General AI","name":"WBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.25874"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.25874","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.25874","title":"WBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26036","domain":"General AI","name":"CITYREP","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26036"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26036","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26036","title":"CITYREP"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26068","domain":"General AI","name":"WSADBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26068"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26068","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26068","title":"WSADBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26086","domain":"Agents","name":"Claw-Anything","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26086"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26086","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26086","title":"Claw-Anything"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26087","domain":"Agents","name":"DiscoverPhysics","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26087"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26087","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26087","title":"DiscoverPhysics"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26144","domain":"Code & Software","name":"VISTA (VIsual Spec-To-App Benchmark)","released":"2026-05-22","url":"https://arxiv.org/abs/2605.26144"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26144","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26144","title":"VISTA (VIsual Spec-To-App Benchmark)"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26176","domain":"General AI","name":"PitchBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26176"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26176","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26176","title":"PitchBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26177","domain":"Code & Software","name":"RepoMirage","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26177"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26177","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26177","title":"RepoMirage"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26239","domain":"General AI","name":"Sentinel Challenge","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26239"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26239","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26239","title":"Sentinel Challenge"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26244","domain":"General AI","name":"LongAV-Compass","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26244"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26244","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26244","title":"LongAV-Compass"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26302","domain":"Agents","name":"AgingBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26302"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26302","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26302","title":"AgingBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26320","domain":"General AI","name":"MULTISEISMO","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26320"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26320","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26320","title":"MULTISEISMO"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26321","domain":"Agents","name":"ERP-Bench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26321"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26321","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26321","title":"ERP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26322","domain":"General AI","name":"OmniToM","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26322"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26322","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26322","title":"OmniToM"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26394","domain":"Agents & Tool Use","name":"EnterpriseMem-Bench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26394"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26394","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26394","title":"EnterpriseMem-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26396","domain":"Agents","name":"MM-CreativityBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26396"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26396","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26396","title":"MM-CreativityBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26457","domain":"Code & Software","name":"Verus-SpecBench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26457"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26457","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26457","title":"Verus-SpecBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26485","domain":"Mathematics & Formal Science","name":"OmniInteract","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26485"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26485","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26485","title":"OmniInteract"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26525","domain":"General AI","name":"MSVE-Bench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26525"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26525","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26525","title":"MSVE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26548","domain":"Cybersecurity","name":"SEC-bench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26548"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26548","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26548","title":"SEC-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26641","domain":"General AI","name":"OmniRetriever-Bench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26641"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26641","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26641","title":"OmniRetriever-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26667","domain":"General AI","name":"MemFail","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26667"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26667","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26667","title":"MemFail"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26781","domain":"Materials & Chemistry","name":"LiveK12Bench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26781"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26781","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26781","title":"LiveK12Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26855","domain":"General AI","name":"Receipt Replay OOD","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26855"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26855","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26855","title":"Receipt Replay OOD"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26918","domain":"Factuality","name":"EduVideoBench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26918"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26918","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26918","title":"EduVideoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26978","domain":"General AI","name":"PashtoTTS-Bench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26978"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26978","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26978","title":"PashtoTTS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-26994","domain":"General AI","name":"ChartAct","released":"2026-05-26","url":"https://arxiv.org/abs/2605.26994"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.26994","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.26994","title":"ChartAct"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27015","domain":"General AI","name":"PersLitEval","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27015"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27015","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27015","title":"PersLitEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27074","domain":"General AI","name":"IPIBench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27074"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27074","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27074","title":"IPIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27141","domain":"Agents","name":"VitaBench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27141"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27141","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27141","title":"VitaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27210","domain":"Language & Knowledge","name":"Qiskit QuantumKatas","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27210"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27210","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27210","title":"Qiskit QuantumKatas"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27240","domain":"General AI","name":"ENPMR-Bench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27240"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27240","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27240","title":"ENPMR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27367","domain":"General AI","name":"SpatialBench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27367"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27367","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27367","title":"SpatialBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27472","domain":"Chip Design & EDA","name":"AssertLLM2","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27472"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27472","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27472","title":"AssertLLM2"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27649","domain":"General AI","name":"MTM-Bench","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27649"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27649","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27649","title":"MTM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27759","domain":"Robotics","name":"Colosseum V2","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27759"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27759","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27759","title":"Colosseum V2"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27761","domain":"Agents & Tool Use","name":"AndroidDaily","released":"2026-05-26","url":"https://arxiv.org/abs/2605.27761"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27761","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27761","title":"AndroidDaily"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27805","domain":"General AI","name":"ChildEval","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27805"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27805","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27805","title":"ChildEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27820","domain":"Agents & Tool Use","name":"EgoBench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27820"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27820","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27820","title":"EgoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27851","domain":"General AI","name":"context-flip evaluation","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27851"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27851","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27851","title":"context-flip evaluation"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27886","domain":"Robot manipulation","name":"Tabero","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27886"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27886","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27886","title":"Tabero"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27887","domain":"Finance","name":"PortBench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27887"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27887","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27887","title":"PortBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27896","domain":"Finance","name":"FinBoardBench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27896"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27896","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27896","title":"FinBoardBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27904","domain":"General AI","name":"Dr-CiK","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27904"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27904","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27904","title":"Dr-CiK"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27911","domain":"General AI","name":"SuiChat-CN","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27911"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27911","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27911","title":"SuiChat-CN"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27922","domain":"Agents","name":"Harness-Bench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27922"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27922","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27922","title":"Harness-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27957","domain":"Agents","name":"DisasterBench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27957"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27957","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27957","title":"DisasterBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-27995","domain":"Agents & Tool Use","name":"AsyncTool","released":"2026-05-27","url":"https://arxiv.org/abs/2605.27995"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.27995","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.27995","title":"AsyncTool"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28013","domain":"Cybersecurity","name":"KSAFE-MM","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28013"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28013","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28013","title":"KSAFE-MM"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28035","domain":"General AI","name":"MTAVG-Bench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28035"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28035","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28035","title":"MTAVG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28044","domain":"General AI","name":"FORCEBENCH","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28044"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28044","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28044","title":"FORCEBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28065","domain":"General AI","name":"SpatialBench-Long","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28065"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28065","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28065","title":"SpatialBench-Long"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28073","domain":"General AI","name":"STORYLENSBENCH","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28073"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28073","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28073","title":"STORYLENSBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28091","domain":"General AI","name":"Qwen-Image-Bench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28091"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28091","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28091","title":"Qwen-Image-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28158","domain":"General AI","name":"OR-Space","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28158"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28158","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28158","title":"OR-Space"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28161","domain":"Biology & Drug Discovery","name":"MeniOmni","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28161"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28161","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28161","title":"MeniOmni"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28192","domain":"General AI","name":"MOV-Bench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28192"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28192","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28192","title":"MOV-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28218","domain":"General AI","name":"IFMTBench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28218"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28218","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28218","title":"IFMTBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28237","domain":"General AI","name":"POINav-Bench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28237"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28237","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28237","title":"POINav-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28241","domain":"General AI","name":"PointQ-Bench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28241"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28241","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28241","title":"PointQ-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28257","domain":"CAD","name":"HouseCorr3D","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28257"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28257","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28257","title":"HouseCorr3D"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28315","domain":"Factuality","name":"HardMTBench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28315"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28315","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28315","title":"HardMTBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28375","domain":"Biology & Drug Discovery","name":"PrionNER","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28375"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28375","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28375","title":"PrionNER"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28579","domain":"CAD","name":"MUSE","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28579"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28579","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28579","title":"MUSE"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28683","domain":"Agents","name":"VeriTrip","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28683"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28683","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28683","title":"VeriTrip"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28721","domain":"General AI","name":"LiveBrowseComp","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28721"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28721","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28721","title":"LiveBrowseComp"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28819","domain":"General AI","name":"PEFT-Arena","released":"2026-05-27","url":"https://arxiv.org/abs/2605.28819"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28819","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28819","title":"PEFT-Arena"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-28876","domain":"Agents","name":"LogDx-CI","released":"2026-05-26","url":"https://arxiv.org/abs/2605.28876"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.28876","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.28876","title":"LogDx-CI"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29054","domain":"Code & Software","name":"T2J-Bench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.29054"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29054","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29054","title":"T2J-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29074","domain":"Geometric reasoning","name":"Embodied3DBench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.29074"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29074","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29074","title":"Embodied3DBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29107","domain":"Robot manipulation","name":"GEO-Bench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.29107"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29107","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29107","title":"GEO-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29170","domain":"General AI","name":"UA-Legal-Bench","released":"2026-05-27","url":"https://arxiv.org/abs/2605.29170"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29170","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29170","title":"UA-Legal-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29224","domain":"Cybersecurity","name":"HarmURLBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29224"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29224","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29224","title":"HarmURLBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29277","domain":"Code & Software","name":"Code-QA-Bench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29277"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29277","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29277","title":"Code-QA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29299","domain":"Biology & Drug Discovery","name":"Pocket-Dentist","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29299"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29299","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29299","title":"Pocket-Dentist"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29300","domain":"General AI","name":"MusTBENCH","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29300"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29300","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29300","title":"MusTBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29330","domain":"General AI","name":"EarthShift","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29330"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29330","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29330","title":"EarthShift"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29360","domain":"Language & Knowledge","name":"MiraBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29360"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29360","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29360","title":"MiraBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29427","domain":"Factuality","name":"FinGuard-Bench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29427"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29427","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29427","title":"FinGuard-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29446","domain":"General AI","name":"CrystalXRD-Bench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29446"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29446","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29446","title":"CrystalXRD-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29468","domain":"General AI","name":"SciIntBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29468"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29468","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29468","title":"SciIntBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29523","domain":"Finance","name":"K-FinHallu","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29523"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29523","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29523","title":"K-FinHallu"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29532","domain":"Agents","name":"GUITestScape","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29532"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29532","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29532","title":"GUITestScape"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29579","domain":"General AI","name":"ReactBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29579"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29579","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29579","title":"ReactBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29586","domain":"Finance","name":"FinVerBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29586"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29586","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29586","title":"FinVerBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29653","domain":"Agents & Tool Use","name":"PTCG-Bench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29653"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29653","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29653","title":"PTCG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29710","domain":"Robotics","name":"PhAIL (Physical AI Leaderboard)","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29710"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29710","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29710","title":"PhAIL (Physical AI Leaderboard)"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29738","domain":"Language & Knowledge","name":"Multi-Legal-Bench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29738"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29738","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29738","title":"Multi-Legal-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29833","domain":"Materials & Chemistry","name":"OmniMatBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29833"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29833","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29833","title":"OmniMatBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29893","domain":"Agents","name":"RedundancyBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29893"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29893","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29893","title":"RedundancyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-29933","domain":"General AI","name":"CLUBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.29933"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.29933","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.29933","title":"CLUBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30058","domain":"General AI","name":"HEART-Bench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30058"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30058","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30058","title":"HEART-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30090","domain":"Agents","name":"DirectorBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30090"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30090","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30090","title":"DirectorBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30100","domain":"General AI","name":"Chess-World-Model","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30100"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30100","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30100","title":"Chess-World-Model"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30188","domain":"General AI","name":"CalArena","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30188"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30188","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30188","title":"CalArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30235","domain":"General AI","name":"BullingerDB","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30235"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30235","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30235","title":"BullingerDB"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30241","domain":"General AI","name":"CommunityFact","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30241"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30241","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30241","title":"CommunityFact"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30256","domain":"Cybersecurity","name":"VideoFDB","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30256"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30256","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30256","title":"VideoFDB"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30284","domain":"AI Scientist","name":"ProjectionBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30284"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30284","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30284","title":"ProjectionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30326","domain":"Agents","name":"RoboWits","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30326"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30326","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30326","title":"RoboWits"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30329","domain":"AI Scientist","name":"SoundnessBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30329"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30329","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30329","title":"SoundnessBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30344","domain":"General AI","name":"VisAnomBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30344"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30344","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30344","title":"VisAnomBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30346","domain":"General AI","name":"YoCausal","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30346"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30346","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30346","title":"YoCausal"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30394","domain":"Code","name":"CodeGolf Bench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30394"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30394","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30394","title":"CodeGolf Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30434","domain":"General AI","name":"LongDS-Bench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30434"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30434","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30434","title":"LongDS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30497","domain":"General AI","name":"CanLegalRAGBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30497"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30497","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30497","title":"CanLegalRAGBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30599","domain":"Biology & Drug Discovery","name":"AMNESIA","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30599"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30599","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30599","title":"AMNESIA"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30611","domain":"Agents","name":"CraftBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30611"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30611","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30611","title":"CraftBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30637","domain":"Biology & Drug Discovery","name":"EHRBench","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30637"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30637","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30637","title":"EHRBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30639","domain":"Robotics & Embodied AI","name":"PInVerify","released":"2026-05-28","url":"https://arxiv.org/abs/2605.30639"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30639","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30639","title":"PInVerify"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30727","domain":"General AI","name":"MosaicLeaks","released":"2026-05-29","url":"https://arxiv.org/abs/2605.30727"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30727","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30727","title":"MosaicLeaks"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30900","domain":"General AI","name":"BilliardPhys-Bench","released":"2026-05-29","url":"https://arxiv.org/abs/2605.30900"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30900","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30900","title":"BilliardPhys-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30907","domain":"Finance","name":"BlueFin","released":"2026-05-29","url":"https://arxiv.org/abs/2605.30907"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30907","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30907","title":"BlueFin"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-30931","domain":"Language & Knowledge","name":"MineExplorer","released":"2026-05-29","url":"https://arxiv.org/abs/2605.30931"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.30931","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.30931","title":"MineExplorer"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31031","domain":"General AI","name":"GraphARC","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31031"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31031","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31031","title":"GraphARC"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31042","domain":"General AI","name":"ClawTrojan","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31042"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31042","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31042","title":"ClawTrojan"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31113","domain":"General AI","name":"TSM-Bench","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31113"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31113","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31113","title":"TSM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31196","domain":"Multimodal","name":"TouchSafeBench","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31196"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31196","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31196","title":"TouchSafeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31251","domain":"Multimodal","name":"ERGeoBench","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31251"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31251","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31251","title":"ERGeoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31349","domain":"Cybersecurity","name":"FBHM","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31349"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31349","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31349","title":"FBHM"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31351","domain":"General AI","name":"VIABLE","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31351"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31351","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31351","title":"VIABLE"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31410","domain":"Biology & Drug Discovery","name":"FAM-Bench","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31410"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31410","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31410","title":"FAM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31478","domain":"Code","name":"PowerCodeBench","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31478"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31478","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31478","title":"PowerCodeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31529","domain":"Agents","name":"SVI-Bench","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31529"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31529","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31529","title":"SVI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31557","domain":"General AI","name":"EGOSTREAM","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31557"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31557","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31557","title":"EGOSTREAM"},{"benchmarks":[{"benchmark_id":"claire-radar-2605-31597","domain":"General AI","name":"SOCO","released":"2026-05-29","url":"https://arxiv.org/abs/2605.31597"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2605.31597","source":"claire_radar","source_url":"https://arxiv.org/abs/2605.31597","title":"SOCO"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00123","domain":"Biology & Drug Discovery","name":"CardioLens","released":"2026-05-28","url":"https://arxiv.org/abs/2606.00123"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00123","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00123","title":"CardioLens"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00148","domain":"General AI","name":"StemBind","released":"2026-05-29","url":"https://arxiv.org/abs/2606.00148"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00148","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00148","title":"StemBind"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00152","domain":"General AI","name":"PrivacyPeek","released":"2026-05-29","url":"https://arxiv.org/abs/2606.00152"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00152","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00152","title":"PrivacyPeek"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00154","domain":"Code","name":"WebIGBench","released":"2026-05-29","url":"https://arxiv.org/abs/2606.00154"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00154","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00154","title":"WebIGBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00188","domain":"Reasoning","name":"PaintBench","released":"2026-05-29","url":"https://arxiv.org/abs/2606.00188"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00188","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00188","title":"PaintBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00329","domain":"General AI","name":"Loopzero","released":"2026-05-29","url":"https://arxiv.org/abs/2606.00329"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00329","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00329","title":"Loopzero"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00451","domain":"Biology & Drug Discovery","name":"ProtStructQA","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00451"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00451","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00451","title":"ProtStructQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00477","domain":"Factuality","name":"UniKE","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00477"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00477","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00477","title":"UniKE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00548","domain":"General AI","name":"CAFOSat","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00548"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00548","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00548","title":"CAFOSat"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00622","domain":"Cybersecurity","name":"MM-Snowball","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00622"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00622","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00622","title":"MM-Snowball"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00644","domain":"General AI","name":"ForeSci","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00644"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00644","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00644","title":"ForeSci"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00750","domain":"General AI","name":"I-WebGenBench","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00750"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00750","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00750","title":"I-WebGenBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00773","domain":"Multimodal","name":"SafeVLA-Bench","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00773"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00773","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00773","title":"SafeVLA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00793","domain":"General AI","name":"MBench","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00793"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00793","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00793","title":"MBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00815","domain":"General AI","name":"OmniEEG-Bench","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00815"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00815","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00815","title":"OmniEEG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00832","domain":"General AI","name":"Momento","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00832"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00832","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00832","title":"Momento"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00891","domain":"General AI","name":"MMDG-Bench","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00891"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00891","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00891","title":"MMDG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00925","domain":"General AI","name":"SkillVetBench","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00925"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00925","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00925","title":"SkillVetBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-00931","domain":"General AI","name":"CV-Arena","released":"2026-05-30","url":"https://arxiv.org/abs/2606.00931"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.00931","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.00931","title":"CV-Arena"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01016","domain":"General AI","name":"PolySpeech-100","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01016"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01016","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01016","title":"PolySpeech-100"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01028","domain":"Biology & Drug Discovery","name":"MedGym","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01028"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01028","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01028","title":"MedGym"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01046","domain":"General AI","name":"TravelEval","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01046"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01046","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01046","title":"TravelEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01057","domain":"General AI","name":"3DCodeBench","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01057"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01057","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01057","title":"3DCodeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01132","domain":"General AI","name":"HakushoBench","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01132"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01132","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01132","title":"HakushoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01223","domain":"General AI","name":"RefMem-Bench","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01223"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01223","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01223","title":"RefMem-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01247","domain":"Agents","name":"TVRBench","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01247"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01247","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01247","title":"TVRBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01317","domain":"Agents","name":"SABER","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01317"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01317","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01317","title":"SABER"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01322","domain":"General AI","name":"TukaBench","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01322"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01322","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01322","title":"TukaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01348","domain":"General AI","name":"ChartArena","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01348"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01348","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01348","title":"ChartArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01481","domain":"General AI","name":"SafeGen-Bench","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01481"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01481","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01481","title":"SafeGen-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01498","domain":"General AI","name":"TimeSage-MT","released":"2026-05-31","url":"https://arxiv.org/abs/2606.01498"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01498","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01498","title":"TimeSage-MT"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01552","domain":"General AI","name":"RoleCDE","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01552"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01552","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01552","title":"RoleCDE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01600","domain":"Robot manipulation","name":"RoboTrustBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01600"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01600","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01600","title":"RoboTrustBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01629","domain":"General AI","name":"LongJudgeBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01629"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01629","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01629","title":"LongJudgeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01810","domain":"Planning","name":"Causal-Plan-Bench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01810"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01810","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01810","title":"Causal-Plan-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01815","domain":"Agents","name":"CRAB-Bench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01815"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01815","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01815","title":"CRAB-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01848","domain":"Multimodal","name":"RescueBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01848"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01848","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01848","title":"RescueBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01849","domain":"General AI","name":"ContinuousBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01849"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01849","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01849","title":"ContinuousBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01869","domain":"General AI","name":"WorldCoder-Bench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01869"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01869","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01869","title":"WorldCoder-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01879","domain":"General AI","name":"CultureForest","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01879"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01879","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01879","title":"CultureForest"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01897","domain":"General AI","name":"CASTER-Bench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01897"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01897","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01897","title":"CASTER-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01901","domain":"General AI","name":"The Image Reconstruction Game","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01901"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01901","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01901","title":"The Image Reconstruction Game"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01912","domain":"Agents & Tool Use","name":"SMH-Bench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01912"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01912","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01912","title":"SMH-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01936","domain":"General AI","name":"DocFormBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01936"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01936","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01936","title":"DocFormBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-01961","domain":"Agents","name":"AutoMedBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.01961"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.01961","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.01961","title":"AutoMedBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02060","domain":"Agents","name":"TELBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02060"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02060","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02060","title":"TELBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02240","domain":"General AI","name":"AgentRedBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02240"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02240","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02240","title":"AgentRedBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02246","domain":"Multimodal","name":"Ego-METAS","released":"2026-05-29","url":"https://arxiv.org/abs/2606.02246"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02246","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02246","title":"Ego-METAS"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02277","domain":"Robotics & Embodied AI","name":"RoboSemanticBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02277"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02277","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02277","title":"RoboSemanticBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02287","domain":"General AI","name":"CityTrajBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02287"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02287","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02287","title":"CityTrajBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02380","domain":"General AI","name":"SPADE-Bench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02380"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02380","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02380","title":"SPADE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02404","domain":"Agents","name":"K-BrowseComp","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02404"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02404","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02404","title":"K-BrowseComp"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02443","domain":"General AI","name":"PaSBench-Video","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02443"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02443","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02443","title":"PaSBench-Video"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02449","domain":"Agents & Tool Use","name":"HLL","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02449"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02449","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02449","title":"HLL"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02470","domain":"Agents","name":"MCP-Persona","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02470"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02470","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02470","title":"MCP-Persona"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02482","domain":"Autonomous Driving","name":"X-Stream","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02482"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02482","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02482","title":"X-Stream"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02522","domain":"General AI","name":"Moment-Video","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02522"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02522","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02522","title":"Moment-Video"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02535","domain":"General AI","name":"LL-Bench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02535"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02535","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02535","title":"LL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02540","domain":"General AI","name":"SkillHarm","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02540"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02540","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02540","title":"SkillHarm"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02556","domain":"General AI","name":"HERO'S JOURNEY","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02556"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02556","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02556","title":"HERO'S JOURNEY"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02568","domain":"Agents & Tool Use","name":"ClinEnv","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02568"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02568","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02568","title":"ClinEnv"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02603","domain":"Robustness","name":"COD10K-C","released":"2026-05-23","url":"https://arxiv.org/abs/2606.02603"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02603","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02603","title":"COD10K-C"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02624","domain":"Biology & Drug Discovery","name":"TadA-Bench","released":"2026-05-29","url":"https://arxiv.org/abs/2606.02624"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02624","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02624","title":"TadA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02642","domain":"General AI","name":"SVHalluc","released":"2026-05-31","url":"https://arxiv.org/abs/2606.02642"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02642","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02642","title":"SVHalluc"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02747","domain":"General AI","name":"Plan2Map","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02747"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02747","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02747","title":"Plan2Map"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02798","domain":"General AI","name":"BehaviorBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02798"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02798","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02798","title":"BehaviorBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02852","domain":"General AI","name":"RESCAST-100K","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02852"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02852","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02852","title":"RESCAST-100K"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-02915","domain":"General AI","name":"Any2Poster Bench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.02915"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.02915","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.02915","title":"Any2Poster Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03103","domain":"Agents","name":"DeskCraft","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03103"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03103","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03103","title":"DeskCraft"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03144","domain":"General AI","name":"GTBench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03144"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03144","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03144","title":"GTBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03157","domain":"Biology & Drug Discovery","name":"ClinicalMC","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03157"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03157","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03157","title":"ClinicalMC"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03184","domain":"Finance","name":"FinStressTS","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03184"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03184","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03184","title":"FinStressTS"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03203","domain":"Agents & Tool Use","name":"MedCUA-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03203"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03203","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03203","title":"MedCUA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03273","domain":"General AI","name":"VistaHop","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03273"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03273","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03273","title":"VistaHop"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03301","domain":"General AI","name":"SagaQA","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03301"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03301","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03301","title":"SagaQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03312","domain":"Robotics","name":"RobotValues","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03312"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03312","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03312","title":"RobotValues"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03318","domain":"General AI","name":"RUT-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03318"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03318","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03318","title":"RUT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03348","domain":"General AI","name":"SynCred-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03348"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03348","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03348","title":"SynCred-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03363","domain":"Factuality","name":"EntSQL","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03363"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03363","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03363","title":"EntSQL"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03374","domain":"Robotics & Embodied AI","name":"eMEM-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03374"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03374","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03374","title":"eMEM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03410","domain":"General AI","name":"Enginuity","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03410"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03410","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03410","title":"Enginuity"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03577","domain":"General AI","name":"ReasonMatch-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03577"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03577","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03577","title":"ReasonMatch-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03626","domain":"Code generation","name":"TurtleAI","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03626"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03626","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03626","title":"TurtleAI"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03635","domain":"General AI","name":"VidMsg","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03635"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03635","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03635","title":"VidMsg"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03657","domain":"Agents","name":"NovelAPIBench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03657"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03657","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03657","title":"NovelAPIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03660","domain":"Language & Knowledge","name":"ChemCoTBench-V2","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03660"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03660","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03660","title":"ChemCoTBench-V2"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03829","domain":"Finance","name":"BigFinanceBench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03829"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03829","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03829","title":"BigFinanceBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03858","domain":"Language & Knowledge","name":"PyraMathBench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03858"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03858","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03858","title":"PyraMathBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03890","domain":"Autonomous Driving","name":"OVO-S-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03890"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03890","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03890","title":"OVO-S-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03906","domain":"General AI","name":"scTranslation","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03906"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03906","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03906","title":"scTranslation"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-03918","domain":"Finance","name":"Hedge-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.03918"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.03918","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.03918","title":"Hedge-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04023","domain":"Code & Software","name":"CodegenBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.04023"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04023","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04023","title":"CodegenBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04067","domain":"Biology & Drug Discovery","name":"DelegateCI-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.04067"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04067","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04067","title":"DelegateCI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04098","domain":"Robotics & Embodied AI","name":"EVID-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.04098"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04098","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04098","title":"EVID-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04126","domain":"Agents","name":"HighTide","released":"2026-06-02","url":"https://arxiv.org/abs/2606.04126"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04126","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04126","title":"HighTide"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04184","domain":"General AI","name":"GroupToM-Bench","released":"2026-06-02","url":"https://arxiv.org/abs/2606.04184"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04184","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04184","title":"GroupToM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04262","domain":"Biology & Drug Discovery","name":"DOSEBENCH","released":"2026-06-02","url":"https://arxiv.org/abs/2606.04262"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04262","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04262","title":"DOSEBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04442","domain":"General AI","name":"MemoryDocDataSet","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04442"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04442","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04442","title":"MemoryDocDataSet"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04460","domain":"Cybersecurity","name":"CyberGym-E2E","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04460"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04460","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04460","title":"CyberGym-E2E"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04525","domain":"Biology & Drug Discovery","name":"GENEB","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04525"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04525","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04525","title":"GENEB"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04569","domain":"Robotics & Embodied AI","name":"MineXplore","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04569"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04569","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04569","title":"MineXplore"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04588","domain":"General AI","name":"VCIFBench","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04588"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04588","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04588","title":"VCIFBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04646","domain":"General AI","name":"QO-Bench","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04646"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04646","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04646","title":"QO-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04773","domain":"Multimodal","name":"NextMotionQA","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04773"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04773","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04773","title":"NextMotionQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04806","domain":"General AI","name":"NoRA","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04806"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04806","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04806","title":"NoRA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04867","domain":"Robotics & Embodied AI","name":"AICompanionBench","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04867"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04867","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04867","title":"AICompanionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04874","domain":"Agents","name":"Agent Planning Benchmark","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04874"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04874","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04874","title":"Agent Planning Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-04986","domain":"Multimodal","name":"CalorieBench-80K","released":"2026-06-03","url":"https://arxiv.org/abs/2606.04986"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.04986","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.04986","title":"CalorieBench-80K"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05001","domain":"Agents","name":"TeleSWEBench","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05001"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05001","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05001","title":"TeleSWEBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05058","domain":"CAD","name":"UniCAD","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05058"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05058","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05058","title":"UniCAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05070","domain":"General AI","name":"RIDE","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05070"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05070","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05070","title":"RIDE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05080","domain":"General AI","name":"AutoLab","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05080"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05080","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05080","title":"AutoLab"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05233","domain":"General AI","name":"CUA-HandCrafted","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05233"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05233","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05233","title":"CUA-HandCrafted"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05238","domain":"Code & Software","name":"DeployBench","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05238"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05238","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05238","title":"DeployBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05249","domain":"Code & Software","name":"SWE-InfraBench","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05249"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05249","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05249","title":"SWE-InfraBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05342","domain":"General AI","name":"SentinelBench","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05342"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05342","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05342","title":"SentinelBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05405","domain":"General AI","name":"Agents' Last Exam (ALE)","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05405"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05405","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05405","title":"Agents' Last Exam (ALE)"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05497","domain":"General AI","name":"LEVANTE-bench","released":"2026-06-03","url":"https://arxiv.org/abs/2606.05497"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05497","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05497","title":"LEVANTE-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05531","domain":"General AI","name":"Almieyar-Oryx-BloomBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05531"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05531","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05531","title":"Almieyar-Oryx-BloomBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05563","domain":"General AI","name":"SoCRATES","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05563"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05563","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05563","title":"SoCRATES"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05570","domain":"Language & Knowledge","name":"TensorBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05570"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05570","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05570","title":"TensorBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05574","domain":"Code & Software","name":"SmellBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05574"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05574","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05574","title":"SmellBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05576","domain":"Biology & Drug Discovery","name":"UltraVR","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05576"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05576","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05576","title":"UltraVR"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05622","domain":"Agents & Tool Use","name":"AdaPlanBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05622"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05622","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05622","title":"AdaPlanBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05635","domain":"General AI","name":"TSC-Bench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05635"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05635","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05635","title":"TSC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05661","domain":"General AI","name":"Continual Learning Bench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05661"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05661","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05661","title":"Continual Learning Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05665","domain":"General AI","name":"V2V-Bench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05665"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05665","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05665","title":"V2V-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05744","domain":"General AI","name":"PlanBench-V","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05744"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05744","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05744","title":"PlanBench-V"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05761","domain":"General AI","name":"SubtleMemory","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05761"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05761","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05761","title":"SubtleMemory"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05793","domain":"General AI","name":"CollabBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05793"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05793","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05793","title":"CollabBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05806","domain":"General AI","name":"ToolMaze","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05806"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05806","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05806","title":"ToolMaze"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05874","domain":"General AI","name":"RandomBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05874"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05874","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05874","title":"RandomBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05920","domain":"Code & Software","name":"Asuka-Bench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05920"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05920","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05920","title":"Asuka-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-05949","domain":"General AI","name":"FEPBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.05949"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.05949","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.05949","title":"FEPBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06055","domain":"General AI","name":"HUSH-Bench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06055"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06055","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06055","title":"HUSH-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06099","domain":"Robot manipulation","name":"CogManip","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06099"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06099","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06099","title":"CogManip"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06217","domain":"General AI","name":"DisasterBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06217"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06217","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06217","title":"DisasterBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06294","domain":"General AI","name":"OMTG Bench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06294"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06294","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06294","title":"OMTG Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06322","domain":"Robot manipulation","name":"DragOn","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06322"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06322","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06322","title":"DragOn"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06481","domain":"General AI","name":"OpAI-Bench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06481"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06481","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06481","title":"OpAI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06492","domain":"Code & Software","name":"RepoPeftBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06492"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06492","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06492","title":"RepoPeftBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06538","domain":"General AI","name":"WorldBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06538"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06538","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06538","title":"WorldBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06560","domain":"Agents & Tool Use","name":"MacArena","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06560"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06560","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06560","title":"MacArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06622","domain":"General AI","name":"UnpredictaBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06622"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06622","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06622","title":"UnpredictaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06698","domain":"General AI","name":"RECAP","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06698"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06698","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06698","title":"RECAP"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06717","domain":"Biology & Drug Discovery","name":"ShallowBench","released":"2026-06-04","url":"https://arxiv.org/abs/2606.06717"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06717","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06717","title":"ShallowBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06784","domain":"General AI","name":"SopriBench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.06784"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06784","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06784","title":"SopriBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06881","domain":"General AI","name":"GlucoFM-Bench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.06881"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06881","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06881","title":"GlucoFM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06959","domain":"Cybersecurity","name":"OpenHalDet","released":"2026-06-05","url":"https://arxiv.org/abs/2606.06959"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06959","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06959","title":"OpenHalDet"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-06960","domain":"Agents","name":"FinEvolveBench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.06960"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.06960","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.06960","title":"FinEvolveBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07032","domain":"General AI","name":"ZeroSight","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07032"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07032","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07032","title":"ZeroSight"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07069","domain":"General AI","name":"mmPISA-bench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07069"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07069","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07069","title":"mmPISA-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07131","domain":"Agents","name":"MalSkillBench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07131"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07131","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07131","title":"MalSkillBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07141","domain":"Biology & Drug Discovery","name":"REMEDI","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07141"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07141","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07141","title":"REMEDI"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07146","domain":"General AI","name":"pinn-gym","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07146"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07146","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07146","title":"pinn-gym"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07167","domain":"General AI","name":"UrduMMLU","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07167"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07167","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07167","title":"UrduMMLU"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07229","domain":"General AI","name":"MMAE","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07229"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07229","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07229","title":"MMAE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07297","domain":"Code & Software","name":"SWE-Explore","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07297"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07297","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07297","title":"SWE-Explore"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07300","domain":"General AI","name":"Phun-Bench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07300"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07300","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07300","title":"Phun-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07401","domain":"General AI","name":"RealDocBench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07401"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07401","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07401","title":"RealDocBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07462","domain":"General AI","name":"AARRI-Bench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07462"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07462","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07462","title":"AARRI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07591","domain":"General AI","name":"ResearchClawBench","released":"2026-05-28","url":"https://arxiv.org/abs/2606.07591"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07591","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07591","title":"ResearchClawBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07595","domain":"Agents","name":"VisualLeakBench","released":"2026-05-29","url":"https://arxiv.org/abs/2606.07595"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07595","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07595","title":"VisualLeakBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07641","domain":"General AI","name":"RotOutBench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.07641"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07641","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07641","title":"RotOutBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07643","domain":"General AI","name":"AVI-Bench","released":"2026-06-01","url":"https://arxiv.org/abs/2606.07643"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07643","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07643","title":"AVI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07656","domain":"Language & Knowledge","name":"SC3","released":"2026-06-03","url":"https://arxiv.org/abs/2606.07656"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07656","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07656","title":"SC3"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07661","domain":"General AI","name":"PereStruct benchmark","released":"2026-06-03","url":"https://arxiv.org/abs/2606.07661"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07661","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07661","title":"PereStruct benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07682","domain":"Code","name":"SWE-Marathon","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07682"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07682","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07682","title":"SWE-Marathon"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07706","domain":"General AI","name":"MLingualFC","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07706"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07706","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07706","title":"MLingualFC"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07805","domain":"Agents","name":"MAC-Bench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07805"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07805","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07805","title":"MAC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07861","domain":"General AI","name":"FineSightBench","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07861"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07861","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07861","title":"FineSightBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07872","domain":"General AI","name":"VisualFLIP","released":"2026-06-05","url":"https://arxiv.org/abs/2606.07872"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07872","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07872","title":"VisualFLIP"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-07962","domain":"Cybersecurity","name":"ChronoPhyBench","released":"2026-06-06","url":"https://arxiv.org/abs/2606.07962"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.07962","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.07962","title":"ChronoPhyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08018","domain":"General AI","name":"UniQL","released":"2026-06-06","url":"https://arxiv.org/abs/2606.08018"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08018","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08018","title":"UniQL"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08036","domain":"General AI","name":"GIScholarBench","released":"2026-06-06","url":"https://arxiv.org/abs/2606.08036"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08036","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08036","title":"GIScholarBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08091","domain":"Agents","name":"VideoWeaver","released":"2026-06-06","url":"https://arxiv.org/abs/2606.08091"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08091","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08091","title":"VideoWeaver"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08194","domain":"General AI","name":"GlobeAudio","released":"2026-06-06","url":"https://arxiv.org/abs/2606.08194"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08194","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08194","title":"GlobeAudio"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08278","domain":"Robot manipulation","name":"SIMPLE","released":"2026-06-06","url":"https://arxiv.org/abs/2606.08278"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08278","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08278","title":"SIMPLE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08340","domain":"Agents","name":"alem","released":"2026-06-06","url":"https://arxiv.org/abs/2606.08340"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08340","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08340","title":"alem"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08415","domain":"General AI","name":"CoVEBench","released":"2026-06-07","url":"https://arxiv.org/abs/2606.08415"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08415","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08415","title":"CoVEBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08481","domain":"Agents","name":"PIPE-Cypher","released":"2026-06-07","url":"https://arxiv.org/abs/2606.08481"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08481","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08481","title":"PIPE-Cypher"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08572","domain":"General AI","name":"OmniCap-IF","released":"2026-06-07","url":"https://arxiv.org/abs/2606.08572"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08572","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08572","title":"OmniCap-IF"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08722","domain":"General AI","name":"LilyBench","released":"2026-06-07","url":"https://arxiv.org/abs/2606.08722"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08722","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08722","title":"LilyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08841","domain":"General AI","name":"ZIPBench","released":"2026-06-07","url":"https://arxiv.org/abs/2606.08841"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08841","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08841","title":"ZIPBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08878","domain":"Agents","name":"PerspectiveGap","released":"2026-06-07","url":"https://arxiv.org/abs/2606.08878"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08878","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08878","title":"PerspectiveGap"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08894","domain":"General AI","name":"Distract-Bench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.08894"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08894","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08894","title":"Distract-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08932","domain":"Language & Knowledge","name":"NormBench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.08932"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08932","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08932","title":"NormBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-08976","domain":"Chip Design & EDA","name":"RTL-Bench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.08976"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.08976","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.08976","title":"RTL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09027","domain":"General AI","name":"SafeRun","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09027"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09027","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09027","title":"SafeRun"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09118","domain":"General AI","name":"ComplexConstraints","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09118"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09118","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09118","title":"ComplexConstraints"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09169","domain":"General AI","name":"IMUG-Bench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09169"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09169","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09169","title":"IMUG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09219","domain":"General AI","name":"LAMOST-DET","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09219"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09219","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09219","title":"LAMOST-DET"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09243","domain":"Robotics & Embodied AI","name":"EgoTactile","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09243"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09243","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09243","title":"EgoTactile"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09276","domain":"General AI","name":"ERBench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09276"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09276","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09276","title":"ERBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09278","domain":"General AI","name":"PyGeoX-Bench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09278"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09278","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09278","title":"PyGeoX-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09323","domain":"General AI","name":"TRL-Bench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09323"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09323","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09323","title":"TRL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09376","domain":"General AI","name":"Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09376"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09376","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09376","title":"Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09389","domain":"General AI","name":"LexRubric","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09389"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09389","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09389","title":"LexRubric"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09426","domain":"General AI","name":"WeaveBench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09426"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09426","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09426","title":"WeaveBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09450","domain":"Language & Knowledge","name":"TheoremBench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09450"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09450","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09450","title":"TheoremBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09461","domain":"Biology & Drug Discovery","name":"H2HMem","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09461"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09461","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09461","title":"H2HMem"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09495","domain":"General AI","name":"ContextShift","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09495"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09495","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09495","title":"ContextShift"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09547","domain":"General AI","name":"Ego-MC-Bench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09547"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09547","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09547","title":"Ego-MC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09570","domain":"General AI","name":"UXBench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09570"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09570","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09570","title":"UXBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09578","domain":"General AI","name":"TABVERSE","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09578"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09578","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09578","title":"TABVERSE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09669","domain":"Agents & Tool Use","name":"SpatialWorld","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09669"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09669","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09669","title":"SpatialWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09746","domain":"Autonomous Driving","name":"ST-Bench","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09746"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09746","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09746","title":"ST-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09764","domain":"Agents","name":"iOSWorld","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09764"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09764","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09764","title":"iOSWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09826","domain":"General AI","name":"OmniGameArena","released":"2026-06-08","url":"https://arxiv.org/abs/2606.09826"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09826","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09826","title":"OmniGameArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09890","domain":"General AI","name":"PreAct-Bench","released":"2026-06-03","url":"https://arxiv.org/abs/2606.09890"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09890","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09890","title":"PreAct-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-09908","domain":"General AI","name":"IDP-Bench","released":"2026-06-06","url":"https://arxiv.org/abs/2606.09908"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.09908","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.09908","title":"IDP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10061","domain":"General AI","name":"BenSyc","released":"2026-06-08","url":"https://arxiv.org/abs/2606.10061"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10061","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10061","title":"BenSyc"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10156","domain":"General AI","name":"τ-Rec","released":"2026-06-08","url":"https://arxiv.org/abs/2606.10156"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10156","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10156","title":"τ-Rec"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10254","domain":"Language & Knowledge","name":"RealMath-Eval","released":"2026-06-08","url":"https://arxiv.org/abs/2606.10254"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10254","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10254","title":"RealMath-Eval"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10255","domain":"Software & AI Compute","name":"POPSICLE","released":"2026-06-08","url":"https://arxiv.org/abs/2606.10255"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10255","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10255","title":"POPSICLE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10298","domain":"Factuality","name":"TriState-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.10298"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10298","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10298","title":"TriState-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10382","domain":"Robot manipulation","name":"UMI-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.10382"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10382","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10382","title":"UMI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10388","domain":"Agents","name":"SkillResolve-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.10388"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10388","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10388","title":"SkillResolve-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10394","domain":"Agents","name":"STAGE-Claw","released":"2026-06-09","url":"https://arxiv.org/abs/2606.10394"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10394","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10394","title":"STAGE-Claw"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10460","domain":"General AI","name":"LakeQA","released":"2026-06-09","url":"https://arxiv.org/abs/2606.10460"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10460","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10460","title":"LakeQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10479","domain":"Language & Knowledge","name":"ComBench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.10479"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10479","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10479","title":"ComBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10803","domain":"Agents","name":"PhysTool-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.10803"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10803","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10803","title":"PhysTool-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-10852","domain":"General AI","name":"Janus","released":"2026-06-09","url":"https://arxiv.org/abs/2606.10852"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.10852","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.10852","title":"Janus"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11042","domain":"General AI","name":"Workflow-GYM","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11042"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11042","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11042","title":"Workflow-GYM"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11063","domain":"General AI","name":"CIAware-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11063"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11063","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11063","title":"CIAware-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11070","domain":"General AI","name":"T1-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11070"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11070","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11070","title":"T1-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11105","domain":"General AI","name":"PhantomBench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11105"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11105","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11105","title":"PhantomBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11129","domain":"Robotics & Embodied AI","name":"WorldOlympiad","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11129"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11129","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11129","title":"WorldOlympiad"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11144","domain":"Biology & Drug Discovery","name":"OncoTraj","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11144"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11144","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11144","title":"OncoTraj"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11150","domain":"Biology & Drug Discovery","name":"ABC-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11150"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11150","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11150","title":"ABC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11152","domain":"General AI","name":"P3D-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11152"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11152","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11152","title":"P3D-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11337","domain":"General AI","name":"SciConBench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11337"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11337","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11337","title":"SciConBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11416","domain":"Language & Knowledge","name":"MPC-Patch-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11416"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11416","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11416","title":"MPC-Patch-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11447","domain":"General AI","name":"SocSci-Repro-Bench","released":"2026-06-09","url":"https://arxiv.org/abs/2606.11447"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11447","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11447","title":"SocSci-Repro-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11562","domain":"General AI","name":"GraphInfer-Bench","released":"2026-06-10","url":"https://arxiv.org/abs/2606.11562"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11562","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11562","title":"GraphInfer-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11702","domain":"Biology & Drug Discovery","name":"MedCTA","released":"2026-06-10","url":"https://arxiv.org/abs/2606.11702"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11702","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11702","title":"MedCTA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-11901","domain":"Robot manipulation","name":"DuoBench","released":"2026-06-10","url":"https://arxiv.org/abs/2606.11901"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.11901","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.11901","title":"DuoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12071","domain":"General AI","name":"RQ-Bench","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12071"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12071","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12071","title":"RQ-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12300","domain":"General AI","name":"ExtremeWhenBench","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12300"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12300","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12300","title":"ExtremeWhenBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12344","domain":"Agents","name":"Claw-SWE-Bench","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12344"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12344","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12344","title":"Claw-SWE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12494","domain":"General AI","name":"Net-Ev$^2$-6.5M","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12494"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12494","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12494","title":"Net-Ev$^2$-6.5M"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12581","domain":"General AI","name":"Spreading-Oriented Reduction Benchmark (SORB)","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12581"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12581","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12581","title":"Spreading-Oriented Reduction Benchmark (SORB)"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12594","domain":"Language & Knowledge","name":"MiniF2F-ALF","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12594"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12594","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12594","title":"MiniF2F-ALF"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12608","domain":"Language & Knowledge","name":"Shopping Reasoning Bench","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12608"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12608","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12608","title":"Shopping Reasoning Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12620","domain":"General AI","name":"HybridCodeAuthorship","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12620"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12620","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12620","title":"HybridCodeAuthorship"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12671","domain":"General AI","name":"SalArt-VQA","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12671"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12671","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12671","title":"SalArt-VQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12736","domain":"General AI","name":"SciAgentArena","released":"2026-06-10","url":"https://arxiv.org/abs/2606.12736"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12736","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12736","title":"SciAgentArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12809","domain":"General AI","name":"MLUBench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.12809"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12809","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12809","title":"MLUBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12821","domain":"General AI","name":"GeoNatureAgent Benchmark","released":"2026-06-11","url":"https://arxiv.org/abs/2606.12821"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12821","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12821","title":"GeoNatureAgent Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12864","domain":"Code","name":"UOJ-Bench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.12864"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12864","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12864","title":"UOJ-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12871","domain":"General AI","name":"DailyReport","released":"2026-06-11","url":"https://arxiv.org/abs/2606.12871"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12871","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12871","title":"DailyReport"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-12936","domain":"Robotics","name":"Pipette","released":"2026-06-11","url":"https://arxiv.org/abs/2606.12936"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.12936","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.12936","title":"Pipette"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13020","domain":"General AI","name":"SciR","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13020"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13020","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13020","title":"SciR"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13040","domain":"Multimodal","name":"RoboProcessBench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13040"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13040","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13040","title":"RoboProcessBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13104","domain":"General AI","name":"AuthorityBench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13104"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13104","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13104","title":"AuthorityBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13120","domain":"Factuality","name":"EvoBrowseComp","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13120"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13120","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13120","title":"EvoBrowseComp"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13148","domain":"General AI","name":"TerraBench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13148"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13148","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13148","title":"TerraBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13218","domain":"General AI","name":"SemCog Bench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13218"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13218","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13218","title":"SemCog Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13385","domain":"Agents","name":"StakeBench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13385"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13385","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13385","title":"StakeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13427","domain":"General AI","name":"VietFashion","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13427"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13427","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13427","title":"VietFashion"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13513","domain":"General AI","name":"CloudCons","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13513"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13513","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13513","title":"CloudCons"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13602","domain":"General AI","name":"EpiBench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13602"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13602","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13602","title":"EpiBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13610","domain":"Cybersecurity","name":"FORGE","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13610"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13610","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13610","title":"FORGE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13681","domain":"Agents & Tool Use","name":"EvoArena","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13681"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13681","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13681","title":"EvoArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13706","domain":"Chip Design & EDA","name":"HierSVA-B","released":"2026-06-09","url":"https://arxiv.org/abs/2606.13706"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13706","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13706","title":"HierSVA-B"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13715","domain":"Agents","name":"WorkBench","released":"2026-06-10","url":"https://arxiv.org/abs/2606.13715"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13715","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13715","title":"WorkBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13735","domain":"Chip Design & EDA","name":"VHDLBench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13735"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13735","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13735","title":"VHDLBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13757","domain":"Agents","name":"SEVRA-BENCH","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13757"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13757","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13757","title":"SEVRA-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13782","domain":"Factuality","name":"MA-ProofBench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13782"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13782","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13782","title":"MA-ProofBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13877","domain":"Robot manipulation","name":"ContactWorld","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13877"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13877","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13877","title":"ContactWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13931","domain":"General AI","name":"DLawBench","released":"2026-06-11","url":"https://arxiv.org/abs/2606.13931"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13931","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13931","title":"DLawBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-13994","domain":"Agents","name":"DeCompBench","released":"2026-06-12","url":"https://arxiv.org/abs/2606.13994"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.13994","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.13994","title":"DeCompBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14058","domain":"Autonomous Driving","name":"ReactSim-Bench","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14058"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14058","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14058","title":"ReactSim-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14068","domain":"General AI","name":"GAMA-Bench","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14068"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14068","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14068","title":"GAMA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14240","domain":"General AI","name":"AFFORDANCE20Q","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14240"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14240","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14240","title":"AFFORDANCE20Q"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14383","domain":"General AI","name":"IndustryBench-MIPU","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14383"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14383","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14383","title":"IndustryBench-MIPU"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14397","domain":"Agents","name":"GauntletBench","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14397"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14397","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14397","title":"GauntletBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14571","domain":"Agents","name":"StreamMemBench","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14571"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14571","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14571","title":"StreamMemBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14574","domain":"General AI","name":"SIMMER","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14574"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14574","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14574","title":"SIMMER"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14600","domain":"General AI","name":"LoSoNA","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14600"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14600","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14600","title":"LoSoNA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14697","domain":"Biology & Drug Discovery","name":"ClinHallu","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14697"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14697","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14697","title":"ClinHallu"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14747","domain":"General AI","name":"MMLongEmbed","released":"2026-06-05","url":"https://arxiv.org/abs/2606.14747"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14747","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14747","title":"MMLongEmbed"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-14832","domain":"Agents & Tool Use","name":"PhoneHarness","released":"2026-06-12","url":"https://arxiv.org/abs/2606.14832"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.14832","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.14832","title":"PhoneHarness"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15034","domain":"General AI","name":"OSGuard","released":"2026-06-13","url":"https://arxiv.org/abs/2606.15034"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15034","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15034","title":"OSGuard"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15107","domain":"General AI","name":"IRTS-ToolBench","released":"2026-06-13","url":"https://arxiv.org/abs/2606.15107"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15107","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15107","title":"IRTS-ToolBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15144","domain":"General AI","name":"PACUTE","released":"2026-06-13","url":"https://arxiv.org/abs/2606.15144"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15144","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15144","title":"PACUTE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15191","domain":"General AI","name":"AmchiBias","released":"2026-06-13","url":"https://arxiv.org/abs/2606.15191"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15191","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15191","title":"AmchiBias"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15240","domain":"General AI","name":"EnvShip","released":"2026-06-13","url":"https://arxiv.org/abs/2606.15240"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15240","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15240","title":"EnvShip"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15242","domain":"Agents","name":"SCR-Bench","released":"2026-06-13","url":"https://arxiv.org/abs/2606.15242"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15242","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15242","title":"SCR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15300","domain":"Agents & Tool Use","name":"CODA-BENCH","released":"2026-06-13","url":"https://arxiv.org/abs/2606.15300"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15300","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15300","title":"CODA-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15320","domain":"General AI","name":"CoMET-Bench","released":"2026-06-13","url":"https://arxiv.org/abs/2606.15320"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15320","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15320","title":"CoMET-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15508","domain":"General AI","name":"ToolMenuBench","released":"2026-06-13","url":"https://arxiv.org/abs/2606.15508"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15508","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15508","title":"ToolMenuBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15532","domain":"General AI","name":"EIBench","released":"2026-06-14","url":"https://arxiv.org/abs/2606.15532"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15532","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15532","title":"EIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15684","domain":"Agents","name":"TickingCollabBench","released":"2026-06-14","url":"https://arxiv.org/abs/2606.15684"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15684","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15684","title":"TickingCollabBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15735","domain":"Biology & Drug Discovery","name":"EHRNote-ChatQA","released":"2026-06-14","url":"https://arxiv.org/abs/2606.15735"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15735","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15735","title":"EHRNote-ChatQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15749","domain":"General AI","name":"OmniTraffic","released":"2026-06-14","url":"https://arxiv.org/abs/2606.15749"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15749","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15749","title":"OmniTraffic"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15862","domain":"Language & Knowledge","name":"RetailBench","released":"2026-06-14","url":"https://arxiv.org/abs/2606.15862"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15862","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15862","title":"RetailBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15867","domain":"General AI","name":"CogCanvas","released":"2026-06-14","url":"https://arxiv.org/abs/2606.15867"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15867","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15867","title":"CogCanvas"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15890","domain":"General AI","name":"UrbanWell","released":"2026-06-14","url":"https://arxiv.org/abs/2606.15890"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15890","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15890","title":"UrbanWell"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15899","domain":"Agents","name":"SkillVetBench","released":"2026-06-14","url":"https://arxiv.org/abs/2606.15899"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15899","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15899","title":"SkillVetBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-15949","domain":"Finance","name":"FinBalance","released":"2026-06-14","url":"https://arxiv.org/abs/2606.15949"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.15949","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.15949","title":"FinBalance"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16127","domain":"General AI","name":"AuAu","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16127"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16127","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16127","title":"AuAu"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16151","domain":"General AI","name":"GRACE","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16151"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16151","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16151","title":"GRACE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16173","domain":"General AI","name":"TimeVista","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16173"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16173","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16173","title":"TimeVista"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16175","domain":"Language & Knowledge","name":"PAL-Bench","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16175"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16175","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16175","title":"PAL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16178","domain":"Robot manipulation","name":"ReMemBench","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16178"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16178","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16178","title":"ReMemBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16262","domain":"Code & Software","name":"UXBench","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16262"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16262","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16262","title":"UXBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16613","domain":"Agents","name":"CoffeeBench","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16613"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16613","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16613","title":"CoffeeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16723","domain":"General AI","name":"AgentFairBench","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16723"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16723","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16723","title":"AgentFairBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16748","domain":"General AI","name":"MyPCBench","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16748"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16748","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16748","title":"MyPCBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16753","domain":"General AI","name":"P3B3","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16753"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16753","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16753","title":"P3B3"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16802","domain":"Agents & Tool Use","name":"LabOSBench","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16802"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16802","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16802","title":"LabOSBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16826","domain":"Robot manipulation","name":"ATOM-Bench","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16826"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16826","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16826","title":"ATOM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16863","domain":"General AI","name":"HawkesNest","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16863"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16863","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16863","title":"HawkesNest"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-16953","domain":"General AI","name":"SidewalkBench","released":"2026-06-15","url":"https://arxiv.org/abs/2606.16953"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.16953","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.16953","title":"SidewalkBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17188","domain":"General AI","name":"PuMVR","released":"2026-06-15","url":"https://arxiv.org/abs/2606.17188"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17188","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17188","title":"PuMVR"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17246","domain":"General AI","name":"GeoDisaster","released":"2026-06-15","url":"https://arxiv.org/abs/2606.17246"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17246","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17246","title":"GeoDisaster"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17266","domain":"Language & Knowledge","name":"SkillChain-Gym","released":"2026-06-15","url":"https://arxiv.org/abs/2606.17266"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17266","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17266","title":"SkillChain-Gym"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17328","domain":"Factuality","name":"MemTrace","released":"2026-06-15","url":"https://arxiv.org/abs/2606.17328"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17328","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17328","title":"MemTrace"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17339","domain":"Biology & Drug Discovery","name":"SpeechDx","released":"2026-06-15","url":"https://arxiv.org/abs/2606.17339"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17339","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17339","title":"SpeechDx"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17391","domain":"General AI","name":"NarrativeWorldBench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.17391"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17391","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17391","title":"NarrativeWorldBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17433","domain":"General AI","name":"LADBench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.17433"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17433","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17433","title":"LADBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17459","domain":"Agents","name":"CEO-Bench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.17459"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17459","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17459","title":"CEO-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17639","domain":"Reasoning","name":"ERQA-Plus","released":"2026-06-16","url":"https://arxiv.org/abs/2606.17639"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17639","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17639","title":"ERQA-Plus"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17698","domain":"General AI","name":"EComAgentBench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.17698"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17698","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17698","title":"EComAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17727","domain":"General AI","name":"LongWebBench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.17727"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17727","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17727","title":"LongWebBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17833","domain":"General AI","name":"HumanoidArena","released":"2026-06-16","url":"https://arxiv.org/abs/2606.17833"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17833","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17833","title":"HumanoidArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17861","domain":"General AI","name":"GameCraft-Bench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.17861"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17861","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17861","title":"GameCraft-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-17905","domain":"General AI","name":"ChLogic","released":"2026-06-16","url":"https://arxiv.org/abs/2606.17905"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.17905","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.17905","title":"ChLogic"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18051","domain":"General AI","name":"CompSkillBench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18051"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18051","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18051","title":"CompSkillBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18060","domain":"General AI","name":"PseudoBench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18060"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18060","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18060","title":"PseudoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18097","domain":"Robot manipulation","name":"WireCraft","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18097"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18097","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18097","title":"WireCraft"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18191","domain":"General AI","name":"DRFLOW","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18191"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18191","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18191","title":"DRFLOW"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18239","domain":"Robot manipulation","name":"EBench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18239"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18239","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18239","title":"EBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18338","domain":"Language & Knowledge","name":"ThousandWorlds","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18338"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18338","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18338","title":"ThousandWorlds"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18356","domain":"Agents","name":"SafeClawBench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18356"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18356","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18356","title":"SafeClawBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18539","domain":"General AI","name":"TS-Fault","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18539"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18539","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18539","title":"TS-Fault"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18543","domain":"Language & Knowledge","name":"CEO-Bench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18543"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18543","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18543","title":"CEO-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18557","domain":"Factuality","name":"DeFAb","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18557"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18557","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18557","title":"DeFAb"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18613","domain":"Biology & Drug Discovery","name":"PhysAssistBench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18613"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18613","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18613","title":"PhysAssistBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18640","domain":"Biology & Drug Discovery","name":"MetaboNet-Bench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18640"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18640","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18640","title":"MetaboNet-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18686","domain":"General AI","name":"ForecastBench-Sim","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18686"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18686","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18686","title":"ForecastBench-Sim"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18782","domain":"General AI","name":"RedactionBench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18782"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18782","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18782","title":"RedactionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18829","domain":"Biology & Drug Discovery","name":"GateMem","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18829"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18829","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18829","title":"GateMem"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18847","domain":"Robotics & Embodied AI","name":"WorldLines","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18847"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18847","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18847","title":"WorldLines"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18936","domain":"General AI","name":"SciRisk-Bench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18936"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18936","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18936","title":"SciRisk-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18943","domain":"General AI","name":"Physics-IQ Verified","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18943"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18943","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18943","title":"Physics-IQ Verified"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18952","domain":"General AI","name":"SP-TransientBench","released":"2026-06-16","url":"https://arxiv.org/abs/2606.18952"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18952","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18952","title":"SP-TransientBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-18989","domain":"General AI","name":"G-IdiomAlign","released":"2026-06-17","url":"https://arxiv.org/abs/2606.18989"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.18989","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.18989","title":"G-IdiomAlign"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19096","domain":"General AI","name":"PorTEXTO","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19096"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19096","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19096","title":"PorTEXTO"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19161","domain":"Robotics & Embodied AI","name":"HT-Bench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19161"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19161","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19161","title":"HT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19218","domain":"General AI","name":"RECOM","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19218"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19218","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19218","title":"RECOM"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19245","domain":"Agents","name":"TherapeuticsBench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19245"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19245","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19245","title":"TherapeuticsBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19256","domain":"General AI","name":"X+Slides","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19256"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19256","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19256","title":"X+Slides"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19259","domain":"General AI","name":"TextRich","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19259"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19259","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19259","title":"TextRich"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19338","domain":"General AI","name":"RNG-Bench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19338"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19338","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19338","title":"RNG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19416","domain":"Agents","name":"MortarBench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19416"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19416","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19416","title":"MortarBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19501","domain":"Finance","name":"DeXposure-Bench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19501"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19501","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19501","title":"DeXposure-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19595","domain":"General AI","name":"IHBench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19595"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19595","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19595","title":"IHBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19613","domain":"Code & Software","name":"StaminaBench","released":"2026-06-17","url":"https://arxiv.org/abs/2606.19613"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19613","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19613","title":"StaminaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19727","domain":"General AI","name":"NRITYAM","released":"2026-06-18","url":"https://arxiv.org/abs/2606.19727"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19727","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19727","title":"NRITYAM"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19787","domain":"Agents & Tool Use","name":"ORAgentBench","released":"2026-06-18","url":"https://arxiv.org/abs/2606.19787"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19787","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19787","title":"ORAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19788","domain":"General AI","name":"CombEval","released":"2026-06-18","url":"https://arxiv.org/abs/2606.19788"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19788","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19788","title":"CombEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19830","domain":"Code & Software","name":"JamBench","released":"2026-06-18","url":"https://arxiv.org/abs/2606.19830"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19830","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19830","title":"JamBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19881","domain":"General AI","name":"REDACT","released":"2026-06-18","url":"https://arxiv.org/abs/2606.19881"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19881","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19881","title":"REDACT"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-19988","domain":"Code","name":"SolidityBench","released":"2026-06-18","url":"https://arxiv.org/abs/2606.19988"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.19988","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.19988","title":"SolidityBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20100","domain":"General AI","name":"WeGenBench","released":"2026-06-18","url":"https://arxiv.org/abs/2606.20100"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20100","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20100","title":"WeGenBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20146","domain":"General AI","name":"BIM-Edit","released":"2026-06-18","url":"https://arxiv.org/abs/2606.20146"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20146","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20146","title":"BIM-Edit"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20408","domain":"General AI","name":"NRT-Bench","released":"2026-06-18","url":"https://arxiv.org/abs/2606.20408"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20408","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20408","title":"NRT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20487","domain":"Agents","name":"HeraBench","released":"2026-06-18","url":"https://arxiv.org/abs/2606.20487"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20487","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20487","title":"HeraBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20517","domain":"Code & Software","name":"Multi-LCB","released":"2026-06-18","url":"https://arxiv.org/abs/2606.20517"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20517","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20517","title":"Multi-LCB"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20527","domain":"General AI","name":"StylisticBias","released":"2026-06-18","url":"https://arxiv.org/abs/2606.20527"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20527","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20527","title":"StylisticBias"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20634","domain":"Agents","name":"DEMM-Bench","released":"2026-05-30","url":"https://arxiv.org/abs/2606.20634"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20634","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20634","title":"DEMM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20663","domain":"Biology & Drug Discovery","name":"DrugBench","released":"2026-06-10","url":"https://arxiv.org/abs/2606.20663"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20663","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20663","title":"DrugBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20668","domain":"General AI","name":"BELLS-O","released":"2026-06-12","url":"https://arxiv.org/abs/2606.20668"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20668","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20668","title":"BELLS-O"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20736","domain":"General AI","name":"REKEY","released":"2026-06-17","url":"https://arxiv.org/abs/2606.20736"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20736","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20736","title":"REKEY"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-20950","domain":"Agents","name":"Power Systems Agent Benchmark","released":"2026-06-18","url":"https://arxiv.org/abs/2606.20950"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.20950","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.20950","title":"Power Systems Agent Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21011","domain":"Robotics","name":"R2HandoverSim","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21011"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21011","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21011","title":"R2HandoverSim"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21023","domain":"Finance","name":"MCR-Bench","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21023"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21023","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21023","title":"MCR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21109","domain":"Biology & Drug Discovery","name":"Chem2Gen-Bench","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21109"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21109","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21109","title":"Chem2Gen-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21116","domain":"General AI","name":"ConnectomeBench2","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21116"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21116","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21116","title":"ConnectomeBench2"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21144","domain":"General AI","name":"AdaMem-Bench","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21144"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21144","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21144","title":"AdaMem-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21147","domain":"General AI","name":"AOR-Bench","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21147"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21147","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21147","title":"AOR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21194","domain":"Biology & Drug Discovery","name":"MEDLAYXPLAIN","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21194"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21194","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21194","title":"MEDLAYXPLAIN"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21309","domain":"General AI","name":"WildBox","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21309"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21309","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21309","title":"WildBox"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21517","domain":"Biology & Drug Discovery","name":"MedHal-Loc","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21517"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21517","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21517","title":"MedHal-Loc"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21618","domain":"General AI","name":"CulMind","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21618"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21618","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21618","title":"CulMind"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21657","domain":"General AI","name":"Chehre","released":"2026-06-19","url":"https://arxiv.org/abs/2606.21657"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21657","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21657","title":"Chehre"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21830","domain":"General AI","name":"Mat-Pref","released":"2026-06-20","url":"https://arxiv.org/abs/2606.21830"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21830","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21830","title":"Mat-Pref"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21844","domain":"General AI","name":"Inverse Turing Bench","released":"2026-06-20","url":"https://arxiv.org/abs/2606.21844"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21844","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21844","title":"Inverse Turing Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-21949","domain":"General AI","name":"CapRiCorn-1K","released":"2026-06-20","url":"https://arxiv.org/abs/2606.21949"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.21949","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.21949","title":"CapRiCorn-1K"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22000","domain":"Agents","name":"CFAgentBench","released":"2026-06-20","url":"https://arxiv.org/abs/2606.22000"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22000","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22000","title":"CFAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22061","domain":"Code & Software","name":"NL2Scratch","released":"2026-06-20","url":"https://arxiv.org/abs/2606.22061"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22061","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22061","title":"NL2Scratch"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22329","domain":"Agents","name":"BabelJudge","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22329"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22329","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22329","title":"BabelJudge"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22338","domain":"Robotics","name":"RoboMME-Interference","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22338"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22338","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22338","title":"RoboMME-Interference"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22339","domain":"Robot manipulation","name":"T-IMPACT","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22339"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22339","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22339","title":"T-IMPACT"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22388","domain":"Agents & Tool Use","name":"PlanBench-XL","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22388"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22388","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22388","title":"PlanBench-XL"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22437","domain":"General AI","name":"MMGist","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22437"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22437","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22437","title":"MMGist"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22476","domain":"General AI","name":"CVSBench","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22476"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22476","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22476","title":"CVSBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22557","domain":"Agents","name":"MacAgentBench","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22557"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22557","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22557","title":"MacAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22597","domain":"Logistics & Operations","name":"MapReason-OSM","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22597"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22597","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22597","title":"MapReason-OSM"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22678","domain":"Agents & Tool Use","name":"RigorBench","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22678"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22678","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22678","title":"RigorBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22723","domain":"Language & Knowledge","name":"BLUEX v2","released":"2026-06-21","url":"https://arxiv.org/abs/2606.22723"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22723","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22723","title":"BLUEX v2"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22778","domain":"General AI","name":"HAKARI-Bench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.22778"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22778","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22778","title":"HAKARI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22873","domain":"Biology & Drug Discovery","name":"SingGuard-Bench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.22873"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22873","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22873","title":"SingGuard-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22877","domain":"General AI","name":"DynamicMem","released":"2026-06-22","url":"https://arxiv.org/abs/2606.22877"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22877","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22877","title":"DynamicMem"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22902","domain":"Agents","name":"CodeRouterBench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.22902"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22902","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22902","title":"CodeRouterBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-22977","domain":"General AI","name":"StatABench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.22977"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.22977","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.22977","title":"StatABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23032","domain":"Agents","name":"IPO Finance Agent","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23032"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23032","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23032","title":"IPO Finance Agent"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23061","domain":"General AI","name":"MotionHalluc","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23061"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23061","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23061","title":"MotionHalluc"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23092","domain":"General AI","name":"PIVOTSBench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23092"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23092","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23092","title":"PIVOTSBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23189","domain":"General AI","name":"AgentCIBench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23189"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23189","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23189","title":"AgentCIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23238","domain":"Finance","name":"HOLMES","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23238"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23238","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23238","title":"HOLMES"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23301","domain":"Agents & Tool Use","name":"EHR-Complex","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23301"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23301","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23301","title":"EHR-Complex"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23375","domain":"General AI","name":"TF-RefusalBench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23375"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23375","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23375","title":"TF-RefusalBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23459","domain":"General AI","name":"TriggerBench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23459"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23459","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23459","title":"TriggerBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23643","domain":"General AI","name":"TailorBench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23643"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23643","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23643","title":"TailorBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23654","domain":"Agents","name":"EnterpriseClawBench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23654"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23654","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23654","title":"EnterpriseClawBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23669","domain":"General AI","name":"GeoFidelity-Bench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23669"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23669","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23669","title":"GeoFidelity-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23927","domain":"General AI","name":"RIFT-Bench","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23927"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23927","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23927","title":"RIFT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-23943","domain":"Language & Knowledge","name":"QuechuaTok","released":"2026-06-22","url":"https://arxiv.org/abs/2606.23943"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.23943","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.23943","title":"QuechuaTok"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24026","domain":"General AI","name":"AgenticInterpBench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24026"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24026","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24026","title":"AgenticInterpBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24145","domain":"Biology & Drug Discovery","name":"T2D-Bench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24145"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24145","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24145","title":"T2D-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24155","domain":"Agents","name":"MedBench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24155"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24155","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24155","title":"MedBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24162","domain":"Factuality","name":"BehaviorBench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24162"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24162","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24162","title":"BehaviorBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24200","domain":"Biology & Drug Discovery","name":"MMed-Bench-IR","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24200"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24200","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24200","title":"MMed-Bench-IR"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24235","domain":"Agents","name":"SP-Bench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24235"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24235","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24235","title":"SP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24256","domain":"General AI","name":"Tailor-Bench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24256"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24256","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24256","title":"Tailor-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24422","domain":"General AI","name":"EgoSAT","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24422"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24422","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24422","title":"EgoSAT"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24526","domain":"General AI","name":"AGORA","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24526"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24526","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24526","title":"AGORA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24530","domain":"General AI","name":"NatureBench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24530"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24530","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24530","title":"NatureBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24548","domain":"General AI","name":"Counterfactual-World (CF-World)","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24548"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24548","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24548","title":"Counterfactual-World (CF-World)"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24589","domain":"Cybersecurity","name":"AdversaBench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24589"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24589","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24589","title":"AdversaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24595","domain":"Agents","name":"MEMPROBE","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24595"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24595","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24595","title":"MEMPROBE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24597","domain":"General AI","name":"AgentWorldBench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24597"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24597","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24597","title":"AgentWorldBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24648","domain":"General AI","name":"ParaPairAudioBench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24648"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24648","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24648","title":"ParaPairAudioBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24714","domain":"General AI","name":"CN-NewsTTS Bench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24714"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24714","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24714","title":"CN-NewsTTS Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24829","domain":"General AI","name":"GeoT2V-Bench","released":"2026-06-23","url":"https://arxiv.org/abs/2606.24829"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24829","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24829","title":"GeoT2V-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-24894","domain":"General AI","name":"RWGBench","released":"2026-05-30","url":"https://arxiv.org/abs/2606.24894"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.24894","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.24894","title":"RWGBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25325","domain":"General AI","name":"MEP-Bench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25325"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25325","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25325","title":"MEP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25369","domain":"General AI","name":"Joyo Kanji Yomi Benchmark","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25369"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25369","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25369","title":"Joyo Kanji Yomi Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25402","domain":"Code","name":"LibEvoBench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25402"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25402","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25402","title":"LibEvoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25445","domain":"General AI","name":"C3-Bench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25445"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25445","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25445","title":"C3-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25491","domain":"General AI","name":"HG-Bench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25491"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25491","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25491","title":"HG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25592","domain":"General AI","name":"VVA-Bench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25592"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25592","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25592","title":"VVA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25634","domain":"General AI","name":"SSMNBench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25634"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25634","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25634","title":"SSMNBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25763","domain":"General AI","name":"CaptureGuide-Bench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25763"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25763","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25763","title":"CaptureGuide-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25819","domain":"Agents & Tool Use","name":"ToolBench-X","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25819"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25819","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25819","title":"ToolBench-X"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25877","domain":"General AI","name":"TacVerse","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25877"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25877","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25877","title":"TacVerse"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25915","domain":"Biology & Drug Discovery","name":"FunPiQ","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25915"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25915","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25915","title":"FunPiQ"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-25984","domain":"Finance","name":"InvestPhilBench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.25984"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.25984","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.25984","title":"InvestPhilBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26029","domain":"General AI","name":"TriViewBench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.26029"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26029","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26029","title":"TriViewBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26040","domain":"General AI","name":"LAIT (Literary AI Translation)","released":"2026-06-24","url":"https://arxiv.org/abs/2606.26040"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26040","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26040","title":"LAIT (Literary AI Translation)"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26094","domain":"General AI","name":"RevengeBench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.26094"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26094","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26094","title":"RevengeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26216","domain":"Cybersecurity","name":"CyberChainBench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.26216"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26216","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26216","title":"CyberChainBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26437","domain":"General AI","name":"ConflictBench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.26437"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26437","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26437","title":"ConflictBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26443","domain":"Robot manipulation","name":"WatchAct","released":"2026-06-24","url":"https://arxiv.org/abs/2606.26443"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26443","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26443","title":"WatchAct"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26458","domain":"Factuality","name":"MKG-RAG-Bench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.26458"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26458","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26458","title":"MKG-RAG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26525","domain":"Language & Knowledge","name":"LCS-Bench","released":"2026-06-25","url":"https://arxiv.org/abs/2606.26525"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26525","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26525","title":"LCS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26551","domain":"General AI","name":"PhyEditBench","released":"2026-06-25","url":"https://arxiv.org/abs/2606.26551"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26551","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26551","title":"PhyEditBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26563","domain":"Biology & Drug Discovery","name":"scBench-Long","released":"2026-06-25","url":"https://arxiv.org/abs/2606.26563"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26563","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26563","title":"scBench-Long"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26602","domain":"General AI","name":"DiCoBench","released":"2026-06-25","url":"https://arxiv.org/abs/2606.26602"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26602","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26602","title":"DiCoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26654","domain":"General AI","name":"SocialPersona","released":"2026-06-25","url":"https://arxiv.org/abs/2606.26654"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26654","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26654","title":"SocialPersona"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26793","domain":"Language & Knowledge","name":"ART-SafeBench","released":"2026-06-25","url":"https://arxiv.org/abs/2606.26793"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26793","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26793","title":"ART-SafeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26947","domain":"General AI","name":"OmniRef-Bench","released":"2026-06-25","url":"https://arxiv.org/abs/2606.26947"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26947","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26947","title":"OmniRef-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26968","domain":"General AI","name":"RedVox","released":"2026-06-25","url":"https://arxiv.org/abs/2606.26968"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26968","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26968","title":"RedVox"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-26984","domain":"General AI","name":"Unison","released":"2026-06-25","url":"https://arxiv.org/abs/2606.26984"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.26984","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.26984","title":"Unison"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27047","domain":"Factuality","name":"NuclearQAv2","released":"2026-06-25","url":"https://arxiv.org/abs/2606.27047"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27047","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27047","title":"NuclearQAv2"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27079","domain":"Multimodal","name":"ForesightSafety-VLA","released":"2026-06-25","url":"https://arxiv.org/abs/2606.27079"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27079","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27079","title":"ForesightSafety-VLA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27128","domain":"General AI","name":"FlameVQA","released":"2026-06-25","url":"https://arxiv.org/abs/2606.27128"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27128","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27128","title":"FlameVQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27187","domain":"General AI","name":"HarmVideoBench","released":"2026-06-25","url":"https://arxiv.org/abs/2606.27187"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27187","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27187","title":"HarmVideoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27444","domain":"General AI","name":"SemCityLockeD","released":"2026-06-25","url":"https://arxiv.org/abs/2606.27444"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27444","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27444","title":"SemCityLockeD"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27499","domain":"Agents","name":"DMV-Bench","released":"2026-06-25","url":"https://arxiv.org/abs/2606.27499"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27499","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27499","title":"DMV-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27537","domain":"General AI","name":"MemoBench","released":"2026-06-25","url":"https://arxiv.org/abs/2606.27537"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27537","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27537","title":"MemoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27669","domain":"General AI","name":"DiscoBench","released":"2026-06-26","url":"https://arxiv.org/abs/2606.27669"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27669","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27669","title":"DiscoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27826","domain":"Robotics & Embodied AI","name":"NormAct","released":"2026-06-26","url":"https://arxiv.org/abs/2606.27826"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27826","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27826","title":"NormAct"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27828","domain":"General AI","name":"Video-MME-Logical","released":"2026-06-26","url":"https://arxiv.org/abs/2606.27828"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27828","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27828","title":"Video-MME-Logical"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27876","domain":"General AI","name":"SpatialUAV","released":"2026-06-26","url":"https://arxiv.org/abs/2606.27876"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27876","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27876","title":"SpatialUAV"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-27999","domain":"General AI","name":"HumanMoveVQA","released":"2026-06-26","url":"https://arxiv.org/abs/2606.27999"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.27999","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.27999","title":"HumanMoveVQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-28049","domain":"Multimodal","name":"AirGroundBench","released":"2026-06-26","url":"https://arxiv.org/abs/2606.28049"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.28049","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28049","title":"AirGroundBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-28406","domain":"General AI","name":"SciDraw-Bench","released":"2026-06-24","url":"https://arxiv.org/abs/2606.28406"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.28406","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28406","title":"SciDraw-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-28480","domain":"Code & Software","name":"TUA-Bench","released":"2026-06-26","url":"https://arxiv.org/abs/2606.28480"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.28480","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28480","title":"TUA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-28514","domain":"General AI","name":"GPTNT","released":"2026-06-26","url":"https://arxiv.org/abs/2606.28514"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.28514","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28514","title":"GPTNT"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-28551","domain":"General AI","name":"DataComp for VLMs (DCVLM)","released":"2026-06-26","url":"https://arxiv.org/abs/2606.28551"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.28551","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28551","title":"DataComp for VLMs (DCVLM)"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-28556","domain":"Biology & Drug Discovery","name":"IMCBench","released":"2026-06-26","url":"https://arxiv.org/abs/2606.28556"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.28556","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28556","title":"IMCBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-28593","domain":"Code","name":"Animation2Code","released":"2026-06-26","url":"https://arxiv.org/abs/2606.28593"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.28593","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.28593","title":"Animation2Code"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29247","domain":"Multimodal","name":"SurgVLA-Bench","released":"2026-06-28","url":"https://arxiv.org/abs/2606.29247"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29247","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29247","title":"SurgVLA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29278","domain":"General AI","name":"The Complexity Ceiling Benchmark","released":"2026-06-28","url":"https://arxiv.org/abs/2606.29278"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29278","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29278","title":"The Complexity Ceiling Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29445","domain":"Agents & Tool Use","name":"VG-GUIBench","released":"2026-06-28","url":"https://arxiv.org/abs/2606.29445"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29445","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29445","title":"VG-GUIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29520","domain":"Code & Software","name":"SAKE","released":"2026-06-28","url":"https://arxiv.org/abs/2606.29520"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29520","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29520","title":"SAKE"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29537","domain":"Agents & Tool Use","name":"OSWorld 2.0","released":"2026-06-28","url":"https://arxiv.org/abs/2606.29537"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29537","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29537","title":"OSWorld 2.0"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29573","domain":"General AI","name":"GranFact","released":"2026-06-28","url":"https://arxiv.org/abs/2606.29573"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29573","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29573","title":"GranFact"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29592","domain":"General AI","name":"STEMGym","released":"2026-06-28","url":"https://arxiv.org/abs/2606.29592"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29592","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29592","title":"STEMGym"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29685","domain":"General AI","name":"CAREBench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.29685"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29685","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29685","title":"CAREBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29700","domain":"General AI","name":"NL-PDDL-Bench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.29700"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29700","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29700","title":"NL-PDDL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29887","domain":"General AI","name":"SafePyramid","released":"2026-06-29","url":"https://arxiv.org/abs/2606.29887"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29887","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29887","title":"SafePyramid"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29894","domain":"Information retrieval","name":"SABER-Math","released":"2026-06-29","url":"https://arxiv.org/abs/2606.29894"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29894","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29894","title":"SABER-Math"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29920","domain":"General AI","name":"RuVerBench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.29920"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29920","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29920","title":"RuVerBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29937","domain":"Robotics","name":"REPAIR-Bench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.29937"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29937","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29937","title":"REPAIR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29955","domain":"Code & Software","name":"SpreadsheetBench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.29955"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29955","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29955","title":"SpreadsheetBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-29957","domain":"Agents","name":"SWE-Together","released":"2026-06-29","url":"https://arxiv.org/abs/2606.29957"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.29957","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.29957","title":"SWE-Together"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30026","domain":"General AI","name":"MuseBench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30026"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30026","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30026","title":"MuseBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30124","domain":"General AI","name":"SciIR-Bench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30124"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30124","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30124","title":"SciIR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30182","domain":"Language & Knowledge","name":"MirrorCode","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30182"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30182","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30182","title":"MirrorCode"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30201","domain":"Biology & Drug Discovery","name":"SHOVIR","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30201"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30201","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30201","title":"SHOVIR"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30256","domain":"General AI","name":"EMPATH","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30256"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30256","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30256","title":"EMPATH"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30262","domain":"General AI","name":"OAO-AttackBench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30262"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30262","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30262","title":"OAO-AttackBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30371","domain":"General AI","name":"MaDI-Bench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30371"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30371","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30371","title":"MaDI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30473","domain":"General AI","name":"DevDataBench","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30473"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30473","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30473","title":"DevDataBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30573","domain":"Agents & Tool Use","name":"SWE-INTERACT","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30573"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30573","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30573","title":"SWE-INTERACT"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30702","domain":"Biology & Drug Discovery","name":"NHANES Accelerometry Cardiometabolic Benchmark","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30702"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30702","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30702","title":"NHANES Accelerometry Cardiometabolic Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-30755","domain":"Agents","name":"SafeClawArena","released":"2026-06-29","url":"https://arxiv.org/abs/2606.30755"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.30755","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.30755","title":"SafeClawArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31154","domain":"General AI","name":"PPT-Eval","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31154"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31154","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31154","title":"PPT-Eval"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31174","domain":"General AI","name":"ClawArena-Team","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31174"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31174","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31174","title":"ClawArena-Team"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31179","domain":"Agents & Tool Use","name":"HealthAgentBench","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31179"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31179","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31179","title":"HealthAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31187","domain":"General AI","name":"UCF101-AD","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31187"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31187","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31187","title":"UCF101-AD"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31308","domain":"General AI","name":"InterFLOPBench","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31308"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31308","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31308","title":"InterFLOPBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31435","domain":"General AI","name":"CDR-Bench","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31435"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31435","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31435","title":"CDR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31522","domain":"Agents & Tool Use","name":"FinPersona-Bench","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31522"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31522","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31522","title":"FinPersona-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31672","domain":"General AI","name":"WorldRoamBench","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31672"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31672","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31672","title":"WorldRoamBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31689","domain":"Code & Software","name":"ScratchWorld","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31689"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31689","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31689","title":"ScratchWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31745","domain":"General AI","name":"JL1-CC&QA","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31745"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31745","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31745","title":"JL1-CC&QA"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31767","domain":"Code & Software","name":"JETO-Bench","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31767"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31767","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31767","title":"JETO-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31875","domain":"Autonomous Driving","name":"SENSE-VAD","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31875"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31875","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31875","title":"SENSE-VAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-31966","domain":"Agents","name":"MECoBench","released":"2026-06-30","url":"https://arxiv.org/abs/2606.31966"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.31966","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.31966","title":"MECoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2606-32034","domain":"General AI","name":"QVal","released":"2026-06-30","url":"https://arxiv.org/abs/2606.32034"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2606.32034","source":"claire_radar","source_url":"https://arxiv.org/abs/2606.32034","title":"QVal"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00107","domain":"Code generation","name":"VULBENCH-CPP","released":"2026-06-30","url":"https://arxiv.org/abs/2607.00107"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00107","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00107","title":"VULBENCH-CPP"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00218","domain":"Robotics","name":"EgoSafetyBench","released":"2026-06-30","url":"https://arxiv.org/abs/2607.00218"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00218","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00218","title":"EgoSafetyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00436","domain":"Agents & Tool Use","name":"PHREEQC-MCQ-200","released":"2026-07-01","url":"https://arxiv.org/abs/2607.00436"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00436","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00436","title":"PHREEQC-MCQ-200"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00464","domain":"Biology & Drug Discovery","name":"MolSafeEval","released":"2026-07-01","url":"https://arxiv.org/abs/2607.00464"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00464","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00464","title":"MolSafeEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00491","domain":"Geometric reasoning","name":"MindEdit-Bench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.00491"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00491","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00491","title":"MindEdit-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00547","domain":"Agents","name":"EgoGapBench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.00547"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00547","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00547","title":"EgoGapBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00664","domain":"General AI","name":"YOMI-Bench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.00664"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00664","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00664","title":"YOMI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00700","domain":"Code & Software","name":"LLVM-Bench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.00700"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00700","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00700","title":"LLVM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00711","domain":"Code","name":"ClarifyCodeBench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.00711"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00711","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00711","title":"ClarifyCodeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00724","domain":"General AI","name":"MSQA","released":"2026-07-01","url":"https://arxiv.org/abs/2607.00724"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00724","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00724","title":"MSQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-00726","domain":"General AI","name":"AV-SyncBench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.00726"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.00726","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.00726","title":"AV-SyncBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01071","domain":"Agents","name":"MemSyco-Bench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.01071"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01071","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01071","title":"MemSyco-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01086","domain":"General AI","name":"LongVQUBench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.01086"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01086","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01086","title":"LongVQUBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01117","domain":"General AI","name":"MoHallBench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.01117"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01117","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01117","title":"MoHallBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01152","domain":"General AI","name":"AGC-Bench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.01152"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01152","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01152","title":"AGC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01245","domain":"General AI","name":"Office Comprehension Benchmark","released":"2026-05-29","url":"https://arxiv.org/abs/2607.01245"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01245","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01245","title":"Office Comprehension Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01360","domain":"Code & Software","name":"PAIR-Bench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.01360"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01360","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01360","title":"PAIR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01388","domain":"Finance","name":"RusFinChain","released":"2026-07-01","url":"https://arxiv.org/abs/2607.01388"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01388","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01388","title":"RusFinChain"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01431","domain":"Factuality","name":"IsoSci","released":"2026-07-01","url":"https://arxiv.org/abs/2607.01431"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01431","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01431","title":"IsoSci"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01597","domain":"Code & Software","name":"PatchFuseBench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.01597"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01597","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01597","title":"PatchFuseBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01647","domain":"General AI","name":"AgenticDataBench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.01647"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01647","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01647","title":"AgenticDataBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01667","domain":"General AI","name":"TCA-Bench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.01667"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01667","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01667","title":"TCA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01728","domain":"General AI","name":"WUICC-bench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.01728"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01728","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01728","title":"WUICC-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01751","domain":"Biology & Drug Discovery","name":"MedStreamBench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.01751"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01751","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01751","title":"MedStreamBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01793","domain":"Agents","name":"Vera-Bench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.01793"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01793","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01793","title":"Vera-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01813","domain":"Agents","name":"MMBench-Live","released":"2026-07-02","url":"https://arxiv.org/abs/2607.01813"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01813","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01813","title":"MMBench-Live"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01829","domain":"Factuality","name":"Pre-Flight","released":"2026-07-02","url":"https://arxiv.org/abs/2607.01829"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01829","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01829","title":"Pre-Flight"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-01938","domain":"Geometric reasoning","name":"PhysMani-Bench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.01938"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.01938","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.01938","title":"PhysMani-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02007","domain":"Factuality","name":"EduArt","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02007"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02007","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02007","title":"EduArt"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02047","domain":"General AI","name":"OpenSafeIntent","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02047"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02047","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02047","title":"OpenSafeIntent"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02096","domain":"General AI","name":"LongEgoRefer","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02096"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02096","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02096","title":"LongEgoRefer"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02255","domain":"Agents","name":"AgenticSTS","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02255"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02255","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02255","title":"AgenticSTS"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02269","domain":"General AI","name":"AnyGroundBench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02269"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02269","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02269","title":"AnyGroundBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02271","domain":"General AI","name":"AGVBench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02271"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02271","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02271","title":"AGVBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02294","domain":"Code & Software","name":"UnderSpecBench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02294"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02294","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02294","title":"UnderSpecBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02440","domain":"Agents & Tool Use","name":"EvoPolicyGym","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02440"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02440","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02440","title":"EvoPolicyGym"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02469","domain":"Agents","name":"TestEvo-Bench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02469"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02469","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02469","title":"TestEvo-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02606","domain":"Code & Software","name":"ChainSWE","released":"2026-07-01","url":"https://arxiv.org/abs/2607.02606"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02606","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02606","title":"ChainSWE"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02680","domain":"Multimodal","name":"K9-Bench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02680"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02680","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02680","title":"K9-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02814","domain":"Agents","name":"SovereignNegotiation-Bench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02814"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02814","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02814","title":"SovereignNegotiation-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02825","domain":"Cybersecurity","name":"JavaVulBench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.02825"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02825","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02825","title":"JavaVulBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02897","domain":"General AI","name":"PPE-Bench","released":"2026-07-03","url":"https://arxiv.org/abs/2607.02897"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02897","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02897","title":"PPE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02934","domain":"Materials & Chemistry","name":"MatPhaseBench","released":"2026-07-03","url":"https://arxiv.org/abs/2607.02934"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02934","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02934","title":"MatPhaseBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02956","domain":"General AI","name":"MORE","released":"2026-07-03","url":"https://arxiv.org/abs/2607.02956"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02956","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02956","title":"MORE"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-02991","domain":"General AI","name":"GuideMe","released":"2026-07-03","url":"https://arxiv.org/abs/2607.02991"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.02991","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.02991","title":"GuideMe"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03006","domain":"General AI","name":"POSTERHARNESS","released":"2026-07-03","url":"https://arxiv.org/abs/2607.03006"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03006","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03006","title":"POSTERHARNESS"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03069","domain":"Agents","name":"SafeVid","released":"2026-07-03","url":"https://arxiv.org/abs/2607.03069"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03069","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03069","title":"SafeVid"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03110","domain":"General AI","name":"ExpoMotion","released":"2026-07-03","url":"https://arxiv.org/abs/2607.03110"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03110","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03110","title":"ExpoMotion"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03261","domain":"Chip Design & EDA","name":"OmniLayout","released":"2026-07-03","url":"https://arxiv.org/abs/2607.03261"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03261","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03261","title":"OmniLayout"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03523","domain":"Code & Software","name":"BugSourceBench","released":"2026-07-03","url":"https://arxiv.org/abs/2607.03523"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03523","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03523","title":"BugSourceBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03525","domain":"Code & Software","name":"GameEngineBench","released":"2026-07-03","url":"https://arxiv.org/abs/2607.03525"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03525","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03525","title":"GameEngineBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03562","domain":"General AI","name":"XPlainVerse","released":"2026-07-03","url":"https://arxiv.org/abs/2607.03562"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03562","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03562","title":"XPlainVerse"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03614","domain":"General AI","name":"IDEAL-Bench","released":"2026-07-03","url":"https://arxiv.org/abs/2607.03614"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03614","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03614","title":"IDEAL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03650","domain":"Biology & Drug Discovery","name":"ClinOCR-Bench","released":"2026-07-04","url":"https://arxiv.org/abs/2607.03650"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03650","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03650","title":"ClinOCR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03831","domain":"General AI","name":"ASOB-Bench","released":"2026-07-04","url":"https://arxiv.org/abs/2607.03831"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03831","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03831","title":"ASOB-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03870","domain":"General AI","name":"Single-answer Atomic Long-form Target (SALT)","released":"2026-07-04","url":"https://arxiv.org/abs/2607.03870"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03870","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03870","title":"Single-answer Atomic Long-form Target (SALT)"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03920","domain":"Robotics & Embodied AI","name":"LH-AVLN","released":"2026-07-04","url":"https://arxiv.org/abs/2607.03920"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03920","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03920","title":"LH-AVLN"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03926","domain":"General AI","name":"TabQueryBench","released":"2026-07-04","url":"https://arxiv.org/abs/2607.03926"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03926","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03926","title":"TabQueryBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-03934","domain":"Biology & Drug Discovery","name":"EgoInertia-MI","released":"2026-07-04","url":"https://arxiv.org/abs/2607.03934"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.03934","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.03934","title":"EgoInertia-MI"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04033","domain":"General AI","name":"OmniOpt","released":"2026-07-04","url":"https://arxiv.org/abs/2607.04033"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04033","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04033","title":"OmniOpt"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04234","domain":"Robot manipulation","name":"SoftVTBench","released":"2026-07-05","url":"https://arxiv.org/abs/2607.04234"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04234","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04234","title":"SoftVTBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04281","domain":"General AI","name":"FreshCache-Bench","released":"2026-07-05","url":"https://arxiv.org/abs/2607.04281"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04281","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04281","title":"FreshCache-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04293","domain":"Agents & Tool Use","name":"CausalGame","released":"2026-07-05","url":"https://arxiv.org/abs/2607.04293"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04293","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04293","title":"CausalGame"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04383","domain":"General AI","name":"AEGBench","released":"2026-07-05","url":"https://arxiv.org/abs/2607.04383"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04383","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04383","title":"AEGBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04434","domain":"Robot manipulation","name":"RoboDojo","released":"2026-07-05","url":"https://arxiv.org/abs/2607.04434"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04434","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04434","title":"RoboDojo"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04581","domain":"General AI","name":"MTEB-BR","released":"2026-07-06","url":"https://arxiv.org/abs/2607.04581"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04581","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04581","title":"MTEB-BR"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04610","domain":"Robotics","name":"RoboVista","released":"2026-07-06","url":"https://arxiv.org/abs/2607.04610"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04610","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04610","title":"RoboVista"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04686","domain":"Cybersecurity","name":"ToolFailBench","released":"2026-07-06","url":"https://arxiv.org/abs/2607.04686"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04686","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04686","title":"ToolFailBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-04845","domain":"Language & Knowledge","name":"HamQASBench","released":"2026-07-06","url":"https://arxiv.org/abs/2607.04845"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.04845","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.04845","title":"HamQASBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05155","domain":"Agents","name":"EdgeBench","released":"2026-07-02","url":"https://arxiv.org/abs/2607.05155"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05155","title":"EdgeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05185","domain":"Agents","name":"ClassicLogic","released":"2026-07-06","url":"https://arxiv.org/abs/2607.05185"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05185","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05185","title":"ClassicLogic"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05189","domain":"General AI","name":"WhisperBench","released":"2026-07-06","url":"https://arxiv.org/abs/2607.05189"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05189","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05189","title":"WhisperBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05202","domain":"Agents","name":"EvoAgentBench","released":"2026-07-06","url":"https://arxiv.org/abs/2607.05202"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05202","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05202","title":"EvoAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05264","domain":"General AI","name":"SteelBench","released":"2026-07-06","url":"https://arxiv.org/abs/2607.05264"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05264","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05264","title":"SteelBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05363","domain":"Language & Knowledge","name":"SovereignPA-Bench","released":"2026-07-06","url":"https://arxiv.org/abs/2607.05363"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05363","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05363","title":"SovereignPA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05365","domain":"General AI","name":"SPEARBench","released":"2026-07-06","url":"https://arxiv.org/abs/2607.05365"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05365","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05365","title":"SPEARBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05462","domain":"General AI","name":"BioSecBench-Refusal","released":"2026-07-06","url":"https://arxiv.org/abs/2607.05462"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05462","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05462","title":"BioSecBench-Refusal"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05571","domain":"Code & Software","name":"CSTutorBench","released":"2026-07-06","url":"https://arxiv.org/abs/2607.05571"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05571","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05571","title":"CSTutorBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-05910","domain":"General AI","name":"PolicyShiftBench","released":"2026-07-07","url":"https://arxiv.org/abs/2607.05910"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.05910","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.05910","title":"PolicyShiftBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06008","domain":"General AI","name":"PolyWorkBench","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06008"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06008","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06008","title":"PolyWorkBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06052","domain":"General AI","name":"ThorArena","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06052"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06052","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06052","title":"ThorArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06118","domain":"Agents","name":"WebRetriever","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06118"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06118","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06118","title":"WebRetriever"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06229","domain":"General AI","name":"Spider 2.0-AIFunc","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06229"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06229","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06229","title":"Spider 2.0-AIFunc"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06254","domain":"General AI","name":"VendorBench-100","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06254"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06254","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06254","title":"VendorBench-100"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06306","domain":"Code & Software","name":"UI2App","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06306"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06306","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06306","title":"UI2App"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06411","domain":"Code & Software","name":"RuBench","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06411"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06411","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06411","title":"RuBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06420","domain":"General AI","name":"HoloCount","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06420"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06420","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06420","title":"HoloCount"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06440","domain":"General AI","name":"PIPBench","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06440"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06440","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06440","title":"PIPBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06482","domain":"General AI","name":"DataGovBench","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06482"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06482","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06482","title":"DataGovBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06624","domain":"Agents","name":"AgentLens","released":"2026-07-07","url":"https://arxiv.org/abs/2607.06624"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06624","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06624","title":"AgentLens"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-06929","domain":"General AI","name":"MADB","released":"2026-07-08","url":"https://arxiv.org/abs/2607.06929"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.06929","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.06929","title":"MADB"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-07051","domain":"General AI","name":"OCCUR-Bench","released":"2026-07-08","url":"https://arxiv.org/abs/2607.07051"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.07051","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07051","title":"OCCUR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-07189","domain":"General AI","name":"ImagingBench","released":"2026-07-08","url":"https://arxiv.org/abs/2607.07189"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.07189","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07189","title":"ImagingBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-07343","domain":"General AI","name":"FreightBidBench","released":"2026-07-08","url":"https://arxiv.org/abs/2607.07343"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.07343","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07343","title":"FreightBidBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-07391","domain":"Language & Knowledge","name":"MIRA-Math","released":"2026-07-08","url":"https://arxiv.org/abs/2607.07391"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.07391","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07391","title":"MIRA-Math"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-07469","domain":"General AI","name":"SynthAVE","released":"2026-07-08","url":"https://arxiv.org/abs/2607.07469"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.07469","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07469","title":"SynthAVE"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-07744","domain":"Code & Software","name":"PERFOPT-Bench","released":"2026-07-08","url":"https://arxiv.org/abs/2607.07744"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.07744","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07744","title":"PERFOPT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-07844","domain":"Autonomous Driving","name":"Shift & Drift","released":"2026-07-08","url":"https://arxiv.org/abs/2607.07844"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.07844","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07844","title":"Shift & Drift"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-07946","domain":"Code & Software","name":"DeepSWE","released":"2026-05-18","url":"https://arxiv.org/abs/2607.07946"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.07946","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.07946","title":"DeepSWE"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08093","domain":"General AI","name":"CausalDS","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08093"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08093","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08093","title":"CausalDS"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08112","domain":"General AI","name":"VSRo-200","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08112"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08112","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08112","title":"VSRo-200"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08317","domain":"General AI","name":"Blind-Spots-Bench","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08317"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08317","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08317","title":"Blind-Spots-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08423","domain":"Biology & Drug Discovery","name":"OmniFood-Bench","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08423"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08423","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08423","title":"OmniFood-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08515","domain":"General AI","name":"INCLUDE-BENCH","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08515"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08515","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08515","title":"INCLUDE-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08681","domain":"Cybersecurity","name":"SolarChain-Eval","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08681"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08681","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08681","title":"SolarChain-Eval"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08729","domain":"Software & AI Compute","name":"WaspMOT","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08729"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08729","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08729","title":"WaspMOT"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08751","domain":"Robot manipulation","name":"DexVerse","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08751"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08751","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08751","title":"DexVerse"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08758","domain":"General AI","name":"IdeaGene-Bench","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08758"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08758","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08758","title":"IdeaGene-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08768","domain":"General AI","name":"UniClawBench","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08768"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08768","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08768","title":"UniClawBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08784","domain":"General AI","name":"HERO","released":"2026-06-13","url":"https://arxiv.org/abs/2607.08784"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08784","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08784","title":"HERO"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08964","domain":"Agents & Tool Use","name":"Long-Horizon-Terminal-Bench","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08964"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08964","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08964","title":"Long-Horizon-Terminal-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-08970","domain":"General AI","name":"MultiView-Bench","released":"2026-07-09","url":"https://arxiv.org/abs/2607.08970"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.08970","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.08970","title":"MultiView-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09068","domain":"General AI","name":"OmniMapBench","released":"2026-07-10","url":"https://arxiv.org/abs/2607.09068"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09068","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09068","title":"OmniMapBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09142","domain":"Biology & Drug Discovery","name":"MedRealMM","released":"2026-07-10","url":"https://arxiv.org/abs/2607.09142"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09142","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09142","title":"MedRealMM"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09190","domain":"Robot manipulation","name":"TactiDex","released":"2026-07-10","url":"https://arxiv.org/abs/2607.09190"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09190","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09190","title":"TactiDex"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09322","domain":"Biology & Drug Discovery","name":"LongMedBench","released":"2026-07-10","url":"https://arxiv.org/abs/2607.09322"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09322","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09322","title":"LongMedBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09328","domain":"General AI","name":"WildTrace","released":"2026-07-10","url":"https://arxiv.org/abs/2607.09328"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09328","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09328","title":"WildTrace"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09528","domain":"Finance","name":"TSAI-MetaFraud","released":"2026-07-10","url":"https://arxiv.org/abs/2607.09528"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09528","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09528","title":"TSAI-MetaFraud"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09711","domain":"General AI","name":"EvoClawBench","released":"2026-06-23","url":"https://arxiv.org/abs/2607.09711"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09711","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09711","title":"EvoClawBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09789","domain":"General AI","name":"PHITSBench","released":"2026-07-08","url":"https://arxiv.org/abs/2607.09789"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09789","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09789","title":"PHITSBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09876","domain":"General AI","name":"Prompting-MammAlps","released":"2026-07-10","url":"https://arxiv.org/abs/2607.09876"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09876","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09876","title":"Prompting-MammAlps"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09880","domain":"Biology & Drug Discovery","name":"CLIR-Bench","released":"2026-07-10","url":"https://arxiv.org/abs/2607.09880"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09880","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09880","title":"CLIR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-09996","domain":"General AI","name":"Who&When Pro","released":"2026-07-10","url":"https://arxiv.org/abs/2607.09996"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.09996","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.09996","title":"Who&When Pro"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10059","domain":"Agents","name":"AgentAbstain","released":"2026-07-11","url":"https://arxiv.org/abs/2607.10059"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10059","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10059","title":"AgentAbstain"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10079","domain":"Agents","name":"MAG","released":"2026-07-11","url":"https://arxiv.org/abs/2607.10079"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10079","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10079","title":"MAG"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10180","domain":"Multimodal","name":"ActiveFly-Bench","released":"2026-07-11","url":"https://arxiv.org/abs/2607.10180"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10180","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10180","title":"ActiveFly-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10301","domain":"General AI","name":"ChartSync","released":"2026-07-11","url":"https://arxiv.org/abs/2607.10301"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10301","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10301","title":"ChartSync"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10308","domain":"General AI","name":"VVM-Bench","released":"2026-07-11","url":"https://arxiv.org/abs/2607.10308"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10308","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10308","title":"VVM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10400","domain":"General AI","name":"SynthDocBench","released":"2026-07-11","url":"https://arxiv.org/abs/2607.10400"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10400","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10400","title":"SynthDocBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10428","domain":"General AI","name":"EYT-Bench","released":"2026-07-11","url":"https://arxiv.org/abs/2607.10428"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10428","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10428","title":"EYT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10490","domain":"Agents","name":"NetInjectBench","released":"2026-07-11","url":"https://arxiv.org/abs/2607.10490"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10490","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10490","title":"NetInjectBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10645","domain":"General AI","name":"MafiaScope","released":"2026-07-12","url":"https://arxiv.org/abs/2607.10645"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10645","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10645","title":"MafiaScope"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10768","domain":"Agents","name":"Opti-Agent-Bench","released":"2026-07-12","url":"https://arxiv.org/abs/2607.10768"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10768","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10768","title":"Opti-Agent-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-10789","domain":"Agents","name":"Imaging-101","released":"2026-07-12","url":"https://arxiv.org/abs/2607.10789"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.10789","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.10789","title":"Imaging-101"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11042","domain":"Code","name":"BackendForge","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11042"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11042","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11042","title":"BackendForge"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11074","domain":"General AI","name":"ResearchQA","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11074"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11074","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11074","title":"ResearchQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11079","domain":"AI Scientist","name":"SDABench","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11079"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11079","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11079","title":"SDABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11228","domain":"General AI","name":"DeepBias","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11228"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11228","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11228","title":"DeepBias"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11312","domain":"General AI","name":"SLVMBench","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11312"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11312","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11312","title":"SLVMBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11327","domain":"General AI","name":"TimeConflict","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11327"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11327","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11327","title":"TimeConflict"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11818","domain":"General AI","name":"MM-ToolSandBox","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11818"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11818","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11818","title":"MM-ToolSandBox"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11844","domain":"Cybersecurity","name":"SportMV-Bench","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11844"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11844","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11844","title":"SportMV-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11849","domain":"Language & Knowledge","name":"AdvancedMathBench","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11849"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11849","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11849","title":"AdvancedMathBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-11862","domain":"General AI","name":"ST-Evidence","released":"2026-07-13","url":"https://arxiv.org/abs/2607.11862"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.11862","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.11862","title":"ST-Evidence"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-12076","domain":"General AI","name":"NEEDL-Bench","released":"2026-07-13","url":"https://arxiv.org/abs/2607.12076"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.12076","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12076","title":"NEEDL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-12310","domain":"Biology & Drug Discovery","name":"LakeQuest","released":"2026-07-14","url":"https://arxiv.org/abs/2607.12310"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.12310","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12310","title":"LakeQuest"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-12385","domain":"General AI","name":"PM-Bench","released":"2026-07-14","url":"https://arxiv.org/abs/2607.12385"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.12385","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12385","title":"PM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-12477","domain":"Robotics & Embodied AI","name":"SIS-Bench","released":"2026-07-14","url":"https://arxiv.org/abs/2607.12477"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.12477","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12477","title":"SIS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-12497","domain":"Cybersecurity","name":"TerraLogic","released":"2026-07-14","url":"https://arxiv.org/abs/2607.12497"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.12497","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12497","title":"TerraLogic"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-12523","domain":"General AI","name":"OOD-RL-Bench","released":"2026-07-14","url":"https://arxiv.org/abs/2607.12523"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.12523","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12523","title":"OOD-RL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-12545","domain":"General AI","name":"VanillaBench","released":"2026-07-14","url":"https://arxiv.org/abs/2607.12545"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.12545","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12545","title":"VanillaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-12868","domain":"Software & AI Compute","name":"Deep4ge","released":"2026-07-14","url":"https://arxiv.org/abs/2607.12868"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.12868","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12868","title":"Deep4ge"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-12893","domain":"General AI","name":"MemOps","released":"2026-07-14","url":"https://arxiv.org/abs/2607.12893"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.12893","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.12893","title":"MemOps"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-13034","domain":"General AI","name":"MSE-Bench","released":"2026-07-14","url":"https://arxiv.org/abs/2607.13034"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.13034","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13034","title":"MSE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-13056","domain":"Robotics","name":"HRIBench","released":"2026-07-05","url":"https://arxiv.org/abs/2607.13056"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.13056","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13056","title":"HRIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-13155","domain":"Biology & Drug Discovery","name":"HEDGEHOG","released":"2026-07-14","url":"https://arxiv.org/abs/2607.13155"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.13155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13155","title":"HEDGEHOG"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-13245","domain":"Reasoning","name":"JITOMA-Bench","released":"2026-07-14","url":"https://arxiv.org/abs/2607.13245"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.13245","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13245","title":"JITOMA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-13457","domain":"General AI","name":"Live Gurbani Tracking","released":"2026-07-15","url":"https://arxiv.org/abs/2607.13457"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.13457","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13457","title":"Live Gurbani Tracking"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-13465","domain":"General AI","name":"DevicesWorld","released":"2026-07-15","url":"https://arxiv.org/abs/2607.13465"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.13465","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13465","title":"DevicesWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-13621","domain":"Robotics & Embodied AI","name":"UESF-Bench","released":"2026-07-15","url":"https://arxiv.org/abs/2607.13621"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.13621","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13621","title":"UESF-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-13653","domain":"Robot manipulation","name":"REAL-Bench","released":"2026-07-15","url":"https://arxiv.org/abs/2607.13653"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.13653","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.13653","title":"REAL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14021","domain":"Multimodal","name":"Industrial Dexterity Benchmark","released":"2026-07-15","url":"https://arxiv.org/abs/2607.14021"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14021","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14021","title":"Industrial Dexterity Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14189","domain":"General AI","name":"MultiRef-Compass","released":"2026-07-15","url":"https://arxiv.org/abs/2607.14189"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14189","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14189","title":"MultiRef-Compass"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14202","domain":"General AI","name":"KeyFrame-Compass","released":"2026-07-15","url":"https://arxiv.org/abs/2607.14202"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14202","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14202","title":"KeyFrame-Compass"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14285","domain":"Code & Software","name":"ToolAlignBench","released":"2026-07-15","url":"https://arxiv.org/abs/2607.14285"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14285","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14285","title":"ToolAlignBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14341","domain":"Reasoning","name":"GCA-Bench","released":"2026-07-15","url":"https://arxiv.org/abs/2607.14341"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14341","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14341","title":"GCA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14385","domain":"Biology & Drug Discovery","name":"MamaBench","released":"2026-07-15","url":"https://arxiv.org/abs/2607.14385"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14385","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14385","title":"MamaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14419","domain":"General AI","name":"HyperShadow","released":"2026-07-15","url":"https://arxiv.org/abs/2607.14419"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14419","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14419","title":"HyperShadow"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14541","domain":"Agents","name":"Atrex-Bench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14541"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14541","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14541","title":"Atrex-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14543","domain":"Robotics & Embodied AI","name":"SafeRelBench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14543"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14543","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14543","title":"SafeRelBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14573","domain":"Language & Knowledge","name":"Alipay-PIBench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14573"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14573","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14573","title":"Alipay-PIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14642","domain":"Agents","name":"MCPEvol-Bench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14642"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14642","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14642","title":"MCPEvol-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14660","domain":"General AI","name":"VIABench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14660"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14660","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14660","title":"VIABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14726","domain":"General AI","name":"AE-UAV","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14726"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14726","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14726","title":"AE-UAV"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14770","domain":"Factuality","name":"ChronoQG","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14770"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14770","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14770","title":"ChronoQG"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14846","domain":"General AI","name":"RW-Voice-EQ Bench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14846"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14846","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14846","title":"RW-Voice-EQ Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14865","domain":"Agents","name":"The Energy Society","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14865"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14865","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14865","title":"The Energy Society"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14975","domain":"General AI","name":"CFM-Bench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14975"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14975","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14975","title":"CFM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-14989","domain":"General AI","name":"OmniaBench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.14989"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.14989","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.14989","title":"OmniaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15166","domain":"Biology & Drug Discovery","name":"MedFailBench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.15166"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15166","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15166","title":"MedFailBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15216","domain":"Biology & Drug Discovery","name":"SymbalBench","released":"2026-07-16","url":"https://arxiv.org/abs/2607.15216"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15216","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15216","title":"SymbalBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15418","domain":"General AI","name":"DrawingVQA","released":"2026-07-16","url":"https://arxiv.org/abs/2607.15418"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15418","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15418","title":"DrawingVQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15434","domain":"Agents","name":"Manager Coercion Benchmark","released":"2026-07-16","url":"https://arxiv.org/abs/2607.15434"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15434","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15434","title":"Manager Coercion Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15517","domain":"General AI","name":"SLAPBench","released":"2026-07-17","url":"https://arxiv.org/abs/2607.15517"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15517","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15517","title":"SLAPBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15606","domain":"General AI","name":"Seq2Synth","released":"2026-07-17","url":"https://arxiv.org/abs/2607.15606"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15606","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15606","title":"Seq2Synth"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15641","domain":"Reasoning","name":"IMBench","released":"2026-07-17","url":"https://arxiv.org/abs/2607.15641"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15641","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15641","title":"IMBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15661","domain":"Biology & Drug Discovery","name":"MergeMedBench","released":"2026-07-17","url":"https://arxiv.org/abs/2607.15661"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15661","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15661","title":"MergeMedBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15768","domain":"General AI","name":"ChronoBench","released":"2026-07-17","url":"https://arxiv.org/abs/2607.15768"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15768","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15768","title":"ChronoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15778","domain":"General AI","name":"MEventBench","released":"2026-07-17","url":"https://arxiv.org/abs/2607.15778"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15778","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15778","title":"MEventBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15879","domain":"General AI","name":"DECODEM","released":"2026-07-17","url":"https://arxiv.org/abs/2607.15879"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15879","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15879","title":"DECODEM"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-15899","domain":"General AI","name":"ContinuityBench","released":"2026-07-17","url":"https://arxiv.org/abs/2607.15899"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.15899","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.15899","title":"ContinuityBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-16165","domain":"General AI","name":"ActiveVision","released":"2026-07-17","url":"https://arxiv.org/abs/2607.16165"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.16165","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16165","title":"ActiveVision"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-16229","domain":"Finance","name":"FinBench","released":"2026-06-24","url":"https://arxiv.org/abs/2607.16229"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.16229","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16229","title":"FinBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-16243","domain":"General AI","name":"RobustMAD","released":"2026-06-26","url":"https://arxiv.org/abs/2607.16243"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.16243","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16243","title":"RobustMAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-16311","domain":"General AI","name":"PriVE-Bench","released":"2026-07-14","url":"https://arxiv.org/abs/2607.16311"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.16311","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16311","title":"PriVE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-16609","domain":"Biology & Drug Discovery","name":"OCT-Bench","released":"2026-07-18","url":"https://arxiv.org/abs/2607.16609"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.16609","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16609","title":"OCT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-16610","domain":"Agents","name":"JarvisBench","released":"2026-07-18","url":"https://arxiv.org/abs/2607.16610"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.16610","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16610","title":"JarvisBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-16632","domain":"Chip Design & EDA","name":"CLOSER-Bench","released":"2026-07-18","url":"https://arxiv.org/abs/2607.16632"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.16632","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16632","title":"CLOSER-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-16736","domain":"General AI","name":"RealDESED","released":"2026-07-18","url":"https://arxiv.org/abs/2607.16736"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.16736","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16736","title":"RealDESED"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-16777","domain":"General AI","name":"JOR-Bench","released":"2026-07-18","url":"https://arxiv.org/abs/2607.16777"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.16777","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.16777","title":"JOR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-17701","domain":"Agents","name":"ProEvent","released":"2026-07-20","url":"https://arxiv.org/abs/2607.17701"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.17701","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17701","title":"ProEvent"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-17742","domain":"General AI","name":"TCR-Bench","released":"2026-07-20","url":"https://arxiv.org/abs/2607.17742"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.17742","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17742","title":"TCR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-17745","domain":"General AI","name":"WuYu-EnvLE-Bench","released":"2026-07-20","url":"https://arxiv.org/abs/2607.17745"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.17745","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17745","title":"WuYu-EnvLE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-17765","domain":"Agents & Tool Use","name":"WC2026-Agents","released":"2026-07-20","url":"https://arxiv.org/abs/2607.17765"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.17765","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17765","title":"WC2026-Agents"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-17812","domain":"General AI","name":"ESCUCHA","released":"2026-07-20","url":"https://arxiv.org/abs/2607.17812"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.17812","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17812","title":"ESCUCHA"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-17999","domain":"General AI","name":"ChoroplethMap-Bench","released":"2026-07-20","url":"https://arxiv.org/abs/2607.17999"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.17999","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.17999","title":"ChoroplethMap-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18063","domain":"Agents","name":"Adaptive Adversaries","released":"2026-07-20","url":"https://arxiv.org/abs/2607.18063"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18063","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18063","title":"Adaptive Adversaries"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18066","domain":"General AI","name":"Pancasila-Dilemmas","released":"2026-07-20","url":"https://arxiv.org/abs/2607.18066"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18066","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18066","title":"Pancasila-Dilemmas"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18084","domain":"General AI","name":"WorldCupArena","released":"2026-07-20","url":"https://arxiv.org/abs/2607.18084"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18084","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18084","title":"WorldCupArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18181","domain":"General AI","name":"VEHBench","released":"2026-07-20","url":"https://arxiv.org/abs/2607.18181"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18181","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18181","title":"VEHBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18438","domain":"General AI","name":"Relay-Bench","released":"2026-07-20","url":"https://arxiv.org/abs/2607.18438"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18438","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18438","title":"Relay-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18448","domain":"Biology & Drug Discovery","name":"PathReportEval","released":"2026-07-20","url":"https://arxiv.org/abs/2607.18448"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18448","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18448","title":"PathReportEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18665","domain":"General AI","name":"SciHazard","released":"2026-07-21","url":"https://arxiv.org/abs/2607.18665"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18665","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18665","title":"SciHazard"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18673","domain":"General AI","name":"MissingBench-Verified","released":"2026-07-21","url":"https://arxiv.org/abs/2607.18673"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18673","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18673","title":"MissingBench-Verified"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18789","domain":"General AI","name":"Moving Alphabet","released":"2026-07-21","url":"https://arxiv.org/abs/2607.18789"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18789","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18789","title":"Moving Alphabet"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18867","domain":"Language & Knowledge","name":"HindsightBench","released":"2026-07-21","url":"https://arxiv.org/abs/2607.18867"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18867","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18867","title":"HindsightBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-18999","domain":"Biology & Drug Discovery","name":"MedDDC-Eval","released":"2026-07-21","url":"https://arxiv.org/abs/2607.18999"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.18999","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.18999","title":"MedDDC-Eval"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19056","domain":"General AI","name":"Vector-Bench","released":"2026-07-21","url":"https://arxiv.org/abs/2607.19056"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19056","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19056","title":"Vector-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19108","domain":"General AI","name":"OpenRTAG","released":"2026-07-21","url":"https://arxiv.org/abs/2607.19108"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19108","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19108","title":"OpenRTAG"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19201","domain":"Biology & Drug Discovery","name":"MIRA-Ev","released":"2026-07-21","url":"https://arxiv.org/abs/2607.19201"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19201","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19201","title":"MIRA-Ev"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19235","domain":"General AI","name":"MeetingToM","released":"2026-07-21","url":"https://arxiv.org/abs/2607.19235"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19235","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19235","title":"MeetingToM"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19261","domain":"Biology & Drug Discovery","name":"PathAgentBench","released":"2026-07-21","url":"https://arxiv.org/abs/2607.19261"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19261","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19261","title":"PathAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19262","domain":"Biology & Drug Discovery","name":"BioSecBench-Surveillance","released":"2026-07-21","url":"https://arxiv.org/abs/2607.19262"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19262","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19262","title":"BioSecBench-Surveillance"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19359","domain":"General AI","name":"MemHop","released":"2026-06-01","url":"https://arxiv.org/abs/2607.19359"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19359","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19359","title":"MemHop"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19361","domain":"General AI","name":"CRA-Bench","released":"2026-06-03","url":"https://arxiv.org/abs/2607.19361"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19361","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19361","title":"CRA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19380","domain":"General AI","name":"CruiseBench","released":"2026-07-01","url":"https://arxiv.org/abs/2607.19380"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19380","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19380","title":"CruiseBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19396","domain":"General AI","name":"CrackedPDFs","released":"2026-07-03","url":"https://arxiv.org/abs/2607.19396"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19396","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19396","title":"CrackedPDFs"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19407","domain":"Language & Knowledge","name":"ITPEval","released":"2026-07-07","url":"https://arxiv.org/abs/2607.19407"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19407","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19407","title":"ITPEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19409","domain":"Finance","name":"FORCE-Bench","released":"2026-07-11","url":"https://arxiv.org/abs/2607.19409"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19409","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19409","title":"FORCE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19695","domain":"Robotics","name":"NavVerse","released":"2026-07-22","url":"https://arxiv.org/abs/2607.19695"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19695","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19695","title":"NavVerse"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19772","domain":"General AI","name":"DRGBT-1K","released":"2026-07-22","url":"https://arxiv.org/abs/2607.19772"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19772","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19772","title":"DRGBT-1K"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19834","domain":"General AI","name":"D2VBench","released":"2026-07-22","url":"https://arxiv.org/abs/2607.19834"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19834","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19834","title":"D2VBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19876","domain":"Robotics & Embodied AI","name":"KineBench","released":"2026-07-22","url":"https://arxiv.org/abs/2607.19876"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19876","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19876","title":"KineBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19910","domain":"General AI","name":"MV-Bench","released":"2026-07-22","url":"https://arxiv.org/abs/2607.19910"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19910","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19910","title":"MV-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19911","domain":"Autonomous Driving","name":"LoRFT","released":"2026-07-22","url":"https://arxiv.org/abs/2607.19911"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19911","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19911","title":"LoRFT"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-19951","domain":"General AI","name":"OffNadirLoc","released":"2026-07-22","url":"https://arxiv.org/abs/2607.19951"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.19951","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.19951","title":"OffNadirLoc"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20090","domain":"General AI","name":"SelectBench","released":"2026-07-22","url":"https://arxiv.org/abs/2607.20090"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20090","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20090","title":"SelectBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20121","domain":"Agents","name":"OpenSkillRisk","released":"2026-07-22","url":"https://arxiv.org/abs/2607.20121"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20121","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20121","title":"OpenSkillRisk"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20197","domain":"General AI","name":"RS-RIE-Bench","released":"2026-07-22","url":"https://arxiv.org/abs/2607.20197"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20197","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20197","title":"RS-RIE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20219","domain":"General AI","name":"HalluTruthQA","released":"2026-07-22","url":"https://arxiv.org/abs/2607.20219"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20219","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20219","title":"HalluTruthQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20471","domain":"Language & Knowledge","name":"SDR-Bench","released":"2026-05-23","url":"https://arxiv.org/abs/2607.20471"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20471","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20471","title":"SDR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20482","domain":"Agents & Tool Use","name":"PersonaTrail","released":"2026-05-30","url":"https://arxiv.org/abs/2607.20482"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20482","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20482","title":"PersonaTrail"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20491","domain":"Agents","name":"DFAH-Bench","released":"2026-06-10","url":"https://arxiv.org/abs/2607.20491"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20491","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20491","title":"DFAH-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20492","domain":"General AI","name":"PhantomFill","released":"2026-06-11","url":"https://arxiv.org/abs/2607.20492"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20492","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20492","title":"PhantomFill"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20498","domain":"Factuality","name":"AISE-Bench","released":"2026-06-16","url":"https://arxiv.org/abs/2607.20498"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20498","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20498","title":"AISE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20518","domain":"Agents","name":"CANN Bench","released":"2026-07-08","url":"https://arxiv.org/abs/2607.20518"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20518","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20518","title":"CANN Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20526","domain":"General AI","name":"ConfidenceBench","released":"2026-07-10","url":"https://arxiv.org/abs/2607.20526"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20526","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20526","title":"ConfidenceBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20558","domain":"Language & Knowledge","name":"StabilityBench","released":"2026-07-17","url":"https://arxiv.org/abs/2607.20558"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20558","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20558","title":"StabilityBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20638","domain":"Language & Knowledge","name":"WaveformQA","released":"2026-07-22","url":"https://arxiv.org/abs/2607.20638"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20638","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20638","title":"WaveformQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20645","domain":"Finance","name":"Frontier Financial Judgement","released":"2026-07-22","url":"https://arxiv.org/abs/2607.20645"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20645","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20645","title":"Frontier Financial Judgement"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20764","domain":"General AI","name":"ArbiGraph","released":"2026-07-22","url":"https://arxiv.org/abs/2607.20764"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20764","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20764","title":"ArbiGraph"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20767","domain":"General AI","name":"Rushes","released":"2026-07-22","url":"https://arxiv.org/abs/2607.20767"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20767","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20767","title":"Rushes"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20868","domain":"General AI","name":"ViSTR-Bench","released":"2026-07-23","url":"https://arxiv.org/abs/2607.20868"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20868","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20868","title":"ViSTR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20872","domain":"General AI","name":"LegalCiteTrust","released":"2026-07-23","url":"https://arxiv.org/abs/2607.20872"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20872","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20872","title":"LegalCiteTrust"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20911","domain":"Agents","name":"Tencent WorkBuddy Bench","released":"2026-07-23","url":"https://arxiv.org/abs/2607.20911"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20911","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20911","title":"Tencent WorkBuddy Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20926","domain":"General AI","name":"SciExplore","released":"2026-07-23","url":"https://arxiv.org/abs/2607.20926"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20926","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20926","title":"SciExplore"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-20982","domain":"General AI","name":"GuardianAgentBench","released":"2026-07-23","url":"https://arxiv.org/abs/2607.20982"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.20982","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.20982","title":"GuardianAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21016","domain":"General AI","name":"CultureTalk-ID","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21016"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21016","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21016","title":"CultureTalk-ID"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21050","domain":"General AI","name":"HyperImageNet","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21050"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21050","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21050","title":"HyperImageNet"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21072","domain":"General AI","name":"SpatialGen-Bench","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21072"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21072","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21072","title":"SpatialGen-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21143","domain":"General AI","name":"RegretBench","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21143"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21143","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21143","title":"RegretBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21155","domain":"Factuality","name":"CRAG-MM-Diagnostics","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21155"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21155","title":"CRAG-MM-Diagnostics"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21217","domain":"Agents & Tool Use","name":"ICAE-Bench","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21217"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21217","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21217","title":"ICAE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21306","domain":"General AI","name":"Int-Bench","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21306"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21306","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21306","title":"Int-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21471","domain":"Robotics & Embodied AI","name":"FutureSurf","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21471"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21471","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21471","title":"FutureSurf"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21588","domain":"Robot manipulation","name":"AXIS","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21588"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21588","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21588","title":"AXIS"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21722","domain":"General AI","name":"ConVBench","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21722"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21722","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21722","title":"ConVBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-21799","domain":"General AI","name":"Copyright-Bench","released":"2026-07-23","url":"https://arxiv.org/abs/2607.21799"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.21799","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.21799","title":"Copyright-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22014","domain":"Geometric reasoning","name":"MissionBench","released":"2026-07-24","url":"https://arxiv.org/abs/2607.22014"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22014","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22014","title":"MissionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22015","domain":"Code & Software","name":"SkillAdeqBench","released":"2026-07-24","url":"https://arxiv.org/abs/2607.22015"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22015","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22015","title":"SkillAdeqBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22119","domain":"Agents","name":"DynaBench","released":"2026-07-24","url":"https://arxiv.org/abs/2607.22119"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22119","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22119","title":"DynaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22165","domain":"General AI","name":"DBA-Bench","released":"2026-07-24","url":"https://arxiv.org/abs/2607.22165"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22165","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22165","title":"DBA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22380","domain":"General AI","name":"IR275K","released":"2026-07-24","url":"https://arxiv.org/abs/2607.22380"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22380","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22380","title":"IR275K"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22393","domain":"Agents","name":"SceneActBench","released":"2026-07-24","url":"https://arxiv.org/abs/2607.22393"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22393","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22393","title":"SceneActBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22635","domain":"General AI","name":"CallBench","released":"2026-06-22","url":"https://arxiv.org/abs/2607.22635"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22635","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22635","title":"CallBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22658","domain":"General AI","name":"StanceBench","released":"2026-06-27","url":"https://arxiv.org/abs/2607.22658"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22658","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22658","title":"StanceBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22671","domain":"General AI","name":"AIR-BENCH","released":"2026-07-06","url":"https://arxiv.org/abs/2607.22671"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22671","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22671","title":"AIR-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22689","domain":"Factuality","name":"ParaGUIBench","released":"2026-07-17","url":"https://arxiv.org/abs/2607.22689"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22689","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22689","title":"ParaGUIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-22705","domain":"General AI","name":"EditCLEVR","released":"2026-07-19","url":"https://arxiv.org/abs/2607.22705"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.22705","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.22705","title":"EditCLEVR"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23116","domain":"Language & Knowledge","name":"Poryos2026","released":"2026-07-25","url":"https://arxiv.org/abs/2607.23116"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23116","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23116","title":"Poryos2026"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23123","domain":"General AI","name":"SQBench","released":"2026-07-25","url":"https://arxiv.org/abs/2607.23123"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23123","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23123","title":"SQBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23225","domain":"Chip Design & EDA","name":"ParasGB","released":"2026-07-25","url":"https://arxiv.org/abs/2607.23225"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23225","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23225","title":"ParasGB"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23326","domain":"General AI","name":"ESF-Bench","released":"2026-07-25","url":"https://arxiv.org/abs/2607.23326"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23326","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23326","title":"ESF-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23332","domain":"Finance","name":"AllocBench","released":"2026-07-25","url":"https://arxiv.org/abs/2607.23332"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23332","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23332","title":"AllocBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23425","domain":"Code & Software","name":"TLA+-Bench","released":"2026-07-26","url":"https://arxiv.org/abs/2607.23425"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23425","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23425","title":"TLA+-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23537","domain":"Autonomous Driving","name":"ObsDriveBench","released":"2026-07-26","url":"https://arxiv.org/abs/2607.23537"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23537","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23537","title":"ObsDriveBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23545","domain":"General AI","name":"XIH-Bench","released":"2026-07-26","url":"https://arxiv.org/abs/2607.23545"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23545","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23545","title":"XIH-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23722","domain":"General AI","name":"E-Bench","released":"2026-07-26","url":"https://arxiv.org/abs/2607.23722"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23722","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23722","title":"E-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23740","domain":"General AI","name":"SoMBench","released":"2026-07-26","url":"https://arxiv.org/abs/2607.23740"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23740","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23740","title":"SoMBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23808","domain":"General AI","name":"Indic DiarBench","released":"2026-07-26","url":"https://arxiv.org/abs/2607.23808"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23808","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23808","title":"Indic DiarBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23813","domain":"Finance","name":"Earnings25","released":"2026-07-26","url":"https://arxiv.org/abs/2607.23813"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23813","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23813","title":"Earnings25"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-23870","domain":"General AI","name":"MulRobBench","released":"2026-07-26","url":"https://arxiv.org/abs/2607.23870"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.23870","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.23870","title":"MulRobBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24016","domain":"Robot manipulation","name":"DailyBench","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24016"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24016","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24016","title":"DailyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24101","domain":"General AI","name":"LU-500","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24101"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24101","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24101","title":"LU-500"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24177","domain":"Cybersecurity","name":"EXE-Bench","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24177"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24177","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24177","title":"EXE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24191","domain":"General AI","name":"StanceFlip","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24191"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24191","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24191","title":"StanceFlip"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24241","domain":"General AI","name":"FilmBench","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24241"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24241","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24241","title":"FilmBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24273","domain":"Finance","name":"INS-ActBench","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24273"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24273","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24273","title":"INS-ActBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24471","domain":"General AI","name":"ROUTEBENCH","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24471"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24471","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24471","title":"ROUTEBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24481","domain":"Robot manipulation","name":"ArmnetBench","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24481"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24481","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24481","title":"ArmnetBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24573","domain":"General AI","name":"LLM-SoccerArena","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24573"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24573","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24573","title":"LLM-SoccerArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24663","domain":"Language & Knowledge","name":"APS-Bench","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24663"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24663","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24663","title":"APS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24707","domain":"General AI","name":"ERUnderstand","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24707"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24707","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24707","title":"ERUnderstand"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24810","domain":"General AI","name":"RRS-10K","released":"2026-07-13","url":"https://arxiv.org/abs/2607.24810"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24810","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24810","title":"RRS-10K"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24821","domain":"General AI","name":"AVE-Compass","released":"2026-07-17","url":"https://arxiv.org/abs/2607.24821"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24821","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24821","title":"AVE-Compass"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24878","domain":"General AI","name":"GraphRareBench","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24878"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24878","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24878","title":"GraphRareBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24882","domain":"Agents","name":"Agent Retrieval Bench","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24882"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24882","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24882","title":"Agent Retrieval Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24889","domain":"Agents","name":"GAUGE","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24889"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24889","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24889","title":"GAUGE"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-24957","domain":"General AI","name":"PerceptionBench","released":"2026-07-27","url":"https://arxiv.org/abs/2607.24957"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.24957","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.24957","title":"PerceptionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25218","domain":"Finance","name":"DebtBench","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25218"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25218","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25218","title":"DebtBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25225","domain":"Cybersecurity","name":"SecDrift","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25225"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25225","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25225","title":"SecDrift"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25232","domain":"Biology & Drug Discovery","name":"Neurai-VN Benchmark","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25232"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25232","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25232","title":"Neurai-VN Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25294","domain":"Factuality","name":"CLBench-V","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25294"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25294","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25294","title":"CLBench-V"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25300","domain":"General AI","name":"MEDit-Bench","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25300"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25300","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25300","title":"MEDit-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25392","domain":"General AI","name":"RDVSv2","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25392"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25392","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25392","title":"RDVSv2"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25398","domain":"Biology & Drug Discovery","name":"HANDBOOK.md","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25398"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25398","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25398","title":"HANDBOOK.md"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25485","domain":"Biology & Drug Discovery","name":"PatientAgentBench","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25485"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25485","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25485","title":"PatientAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25518","domain":"Biology & Drug Discovery","name":"AMPBench-MT","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25518"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25518","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25518","title":"AMPBench-MT"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25641","domain":"Factuality","name":"OmniPhys","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25641"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25641","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25641","title":"OmniPhys"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25656","domain":"Agents","name":"OrchBench","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25656"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25656","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25656","title":"OrchBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25765","domain":"Factuality","name":"WorkSurface-Bench","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25765"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25765","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25765","title":"WorkSurface-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25886","domain":"Code & Software","name":"RSIBench-Data","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25886"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25886","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25886","title":"RSIBench-Data"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25911","domain":"General AI","name":"AnnoBench","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25911"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25911","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25911","title":"AnnoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25953","domain":"General AI","name":"Polistemics","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25953"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25953","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25953","title":"Polistemics"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25987","domain":"General AI","name":"IH-Benchmark","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25987"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25987","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25987","title":"IH-Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-25996","domain":"Code & Software","name":"RepoReasoner","released":"2026-07-28","url":"https://arxiv.org/abs/2607.25996"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.25996","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.25996","title":"RepoReasoner"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26041","domain":"General AI","name":"Desktop-Delta Bench","released":"2026-07-28","url":"https://arxiv.org/abs/2607.26041"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26041","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26041","title":"Desktop-Delta Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26072","domain":"General AI","name":"IFCMemoryBench","released":"2026-07-13","url":"https://arxiv.org/abs/2607.26072"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26072","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26072","title":"IFCMemoryBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26155","domain":"Biology & Drug Discovery","name":"ClinLens","released":"2026-07-28","url":"https://arxiv.org/abs/2607.26155"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26155","title":"ClinLens"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26232","domain":"Robot manipulation","name":"BG-REAL","released":"2026-07-28","url":"https://arxiv.org/abs/2607.26232"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26232","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26232","title":"BG-REAL"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26314","domain":"Agents & Tool Use","name":"StealthBench","released":"2026-07-28","url":"https://arxiv.org/abs/2607.26314"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26314","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26314","title":"StealthBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26367","domain":"General AI","name":"StatMechBench-v0","released":"2026-07-29","url":"https://arxiv.org/abs/2607.26367"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26367","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26367","title":"StatMechBench-v0"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26397","domain":"Biology & Drug Discovery","name":"EC-Reason-Bench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.26397"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26397","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26397","title":"EC-Reason-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26451","domain":"Code & Software","name":"ExplainBench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.26451"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26451","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26451","title":"ExplainBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26455","domain":"Factuality","name":"ForgetBench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.26455"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26455","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26455","title":"ForgetBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26465","domain":"General AI","name":"MultivationBench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.26465"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26465","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26465","title":"MultivationBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26518","domain":"General AI","name":"EgoSafe-Bench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.26518"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26518","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26518","title":"EgoSafe-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26791","domain":"Cybersecurity","name":"SecRespond","released":"2026-07-29","url":"https://arxiv.org/abs/2607.26791"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26791","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26791","title":"SecRespond"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26849","domain":"General AI","name":"ToxScreen","released":"2026-07-29","url":"https://arxiv.org/abs/2607.26849"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26849","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26849","title":"ToxScreen"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-26977","domain":"General AI","name":"TREK","released":"2026-07-29","url":"https://arxiv.org/abs/2607.26977"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.26977","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.26977","title":"TREK"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27030","domain":"Cybersecurity","name":"HoF-Bench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27030"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27030","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27030","title":"HoF-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27056","domain":"General AI","name":"Setoka","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27056"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27056","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27056","title":"Setoka"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27080","domain":"Agents","name":"MemSecBench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27080"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27080","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27080","title":"MemSecBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27084","domain":"General AI","name":"SciFigQual-Bench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27084"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27084","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27084","title":"SciFigQual-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27155","domain":"General AI","name":"OmegaUse-OfficeVal","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27155"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27155","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27155","title":"OmegaUse-OfficeVal"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27180","domain":"Multimodal","name":"HumanCLAW-Bench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27180"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27180","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27180","title":"HumanCLAW-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27189","domain":"General AI","name":"APEX-Accounting","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27189"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27189","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27189","title":"APEX-Accounting"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27231","domain":"Agents","name":"KernelGenBench","released":"2026-07-22","url":"https://arxiv.org/abs/2607.27231"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27231","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27231","title":"KernelGenBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27278","domain":"General AI","name":"OVEarth-Bench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27278"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27278","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27278","title":"OVEarth-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27294","domain":"General AI","name":"AgentS4D","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27294"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27294","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27294","title":"AgentS4D"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27353","domain":"General AI","name":"LayerRAG-Bench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27353"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27353","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27353","title":"LayerRAG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27354","domain":"General AI","name":"PAUSE","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27354"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27354","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27354","title":"PAUSE"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27404","domain":"Biology & Drug Discovery","name":"ECG-InterpBench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27404"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27404","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27404","title":"ECG-InterpBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27409","domain":"Code & Software","name":"SWE-NFI","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27409"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27409","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27409","title":"SWE-NFI"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27453","domain":"Agents","name":"VAmoS Bench","released":"2026-07-29","url":"https://arxiv.org/abs/2607.27453"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27453","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27453","title":"VAmoS Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27616","domain":"General AI","name":"MPIE-Bench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27616"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27616","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27616","title":"MPIE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27637","domain":"General AI","name":"MMOOC","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27637"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27637","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27637","title":"MMOOC"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27654","domain":"General AI","name":"MiGUE-Bench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27654"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27654","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27654","title":"MiGUE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27779","domain":"Biology & Drug Discovery","name":"CXR-Retrieve","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27779"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27779","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27779","title":"CXR-Retrieve"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27798","domain":"General AI","name":"MemeBench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27798"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27798","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27798","title":"MemeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27806","domain":"Biology & Drug Discovery","name":"LoMeVQA","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27806"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27806","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27806","title":"LoMeVQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27857","domain":"General AI","name":"EEG-EditBench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27857"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27857","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27857","title":"EEG-EditBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27877","domain":"Agents","name":"MSEval","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27877"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27877","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27877","title":"MSEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27895","domain":"General AI","name":"MMHBench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27895"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27895","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27895","title":"MMHBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-27912","domain":"General AI","name":"IFHierBench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.27912"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.27912","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.27912","title":"IFHierBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28008","domain":"General AI","name":"RepBench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28008"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28008","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28008","title":"RepBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28033","domain":"Agents & Tool Use","name":"DataClawEval","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28033"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28033","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28033","title":"DataClawEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28037","domain":"Agents","name":"ClawTrack","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28037"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28037","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28037","title":"ClawTrack"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28074","domain":"General AI","name":"Echoverse","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28074"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28074","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28074","title":"Echoverse"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28079","domain":"Language & Knowledge","name":"Chem World","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28079"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28079","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28079","title":"Chem World"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28318","domain":"Biology & Drug Discovery","name":"PathView-Bench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28318"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28318","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28318","title":"PathView-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28503","domain":"General AI","name":"InfoOps Bench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28503"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28503","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28503","title":"InfoOps Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28545","domain":"Code & Software","name":"ORCA-bench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28545"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28545","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28545","title":"ORCA-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28645","domain":"General AI","name":"MobileForge","released":"2026-05-29","url":"https://arxiv.org/abs/2607.28645"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28645","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28645","title":"MobileForge"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28692","domain":"Agents","name":"OpenSciToolBench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28692"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28692","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28692","title":"OpenSciToolBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28788","domain":"Biology & Drug Discovery","name":"EarlyDx","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28788"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28788","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28788","title":"EarlyDx"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28880","domain":"General AI","name":"LayoutBench","released":"2026-07-30","url":"https://arxiv.org/abs/2607.28880"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28880","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28880","title":"LayoutBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-28934","domain":"General AI","name":"FairFund-Bench","released":"2026-07-31","url":"https://arxiv.org/abs/2607.28934"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.28934","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.28934","title":"FairFund-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29002","domain":"General AI","name":"MMShopBench","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29002"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29002","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29002","title":"MMShopBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29124","domain":"General AI","name":"SciFigPlag-Bench","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29124"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29124","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29124","title":"SciFigPlag-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29125","domain":"General AI","name":"M3-DuplexBench","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29125"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29125","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29125","title":"M3-DuplexBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29134","domain":"Language & Knowledge","name":"InferQ","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29134"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29134","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29134","title":"InferQ"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29196","domain":"General AI","name":"Hy-MultiTurn","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29196"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29196","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29196","title":"Hy-MultiTurn"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29218","domain":"General AI","name":"MirrorCraft","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29218"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29218","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29218","title":"MirrorCraft"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29577","domain":"General AI","name":"DungeonBench","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29577"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29577","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29577","title":"DungeonBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29602","domain":"General AI","name":"FriendBench","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29602"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29602","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29602","title":"FriendBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29626","domain":"General AI","name":"AgentHPOBench","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29626"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29626","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29626","title":"AgentHPOBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2607-29677","domain":"Factuality","name":"ExtractBench","released":"2026-07-31","url":"https://arxiv.org/abs/2607.29677"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2607.29677","source":"claire_radar","source_url":"https://arxiv.org/abs/2607.29677","title":"ExtractBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00009","domain":"General AI","name":"AgentMemBench","released":"2026-06-16","url":"https://arxiv.org/abs/2608.00009"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00009","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00009","title":"AgentMemBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00012","domain":"General AI","name":"Obshazard-bench","released":"2026-06-24","url":"https://arxiv.org/abs/2608.00012"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00012","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00012","title":"Obshazard-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00018","domain":"General AI","name":"CITBench","released":"2026-06-29","url":"https://arxiv.org/abs/2608.00018"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00018","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00018","title":"CITBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00036","domain":"Biology & Drug Discovery","name":"XL-DocBench","released":"2026-07-21","url":"https://arxiv.org/abs/2608.00036"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00036","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00036","title":"XL-DocBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00068","domain":"General AI","name":"SafeBuild-Bench","released":"2026-07-29","url":"https://arxiv.org/abs/2608.00068"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00068","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00068","title":"SafeBuild-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00107","domain":"General AI","name":"MetaRoute-Bench","released":"2026-07-31","url":"https://arxiv.org/abs/2608.00107"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00107","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00107","title":"MetaRoute-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00267","domain":"Agents","name":"LoopsBench","released":"2026-07-03","url":"https://arxiv.org/abs/2608.00267"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00267","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00267","title":"LoopsBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00355","domain":"Agents","name":"LiveCodeBench","released":"2026-07-31","url":"https://arxiv.org/abs/2608.00355"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00355","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00355","title":"LiveCodeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00548","domain":"General AI","name":"DrawAI-Bench","released":"2026-08-01","url":"https://arxiv.org/abs/2608.00548"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00548","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00548","title":"DrawAI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00610","domain":"Factuality","name":"S2M-Bench","released":"2026-08-01","url":"https://arxiv.org/abs/2608.00610"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00610","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00610","title":"S2M-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00640","domain":"Biology & Drug Discovery","name":"TreeProbe","released":"2026-08-01","url":"https://arxiv.org/abs/2608.00640"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00640","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00640","title":"TreeProbe"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00692","domain":"Code & Software","name":"Vul4Py","released":"2026-08-01","url":"https://arxiv.org/abs/2608.00692"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00692","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00692","title":"Vul4Py"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00739","domain":"Code & Software","name":"EduPluginBench","released":"2026-08-01","url":"https://arxiv.org/abs/2608.00739"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00739","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00739","title":"EduPluginBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00764","domain":"Finance","name":"FinDeepIndicator","released":"2026-08-01","url":"https://arxiv.org/abs/2608.00764"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00764","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00764","title":"FinDeepIndicator"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00805","domain":"Agents & Tool Use","name":"AgentSLABench","released":"2026-08-01","url":"https://arxiv.org/abs/2608.00805"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00805","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00805","title":"AgentSLABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00832","domain":"General AI","name":"AdvPlan-Bench","released":"2026-08-01","url":"https://arxiv.org/abs/2608.00832"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00832","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00832","title":"AdvPlan-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00909","domain":"Finance","name":"FinHardBench","released":"2026-08-02","url":"https://arxiv.org/abs/2608.00909"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00909","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00909","title":"FinHardBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00915","domain":"Language & Knowledge","name":"UpliftBench","released":"2026-08-02","url":"https://arxiv.org/abs/2608.00915"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00915","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00915","title":"UpliftBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-00991","domain":"General AI","name":"SCHEDBench","released":"2026-08-02","url":"https://arxiv.org/abs/2608.00991"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.00991","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.00991","title":"SCHEDBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01094","domain":"Biology & Drug Discovery","name":"Lethe","released":"2026-08-02","url":"https://arxiv.org/abs/2608.01094"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01094","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01094","title":"Lethe"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01189","domain":"Agents","name":"M2ABench","released":"2026-08-02","url":"https://arxiv.org/abs/2608.01189"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01189","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01189","title":"M2ABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01358","domain":"General AI","name":"HopRefusalBench","released":"2026-08-02","url":"https://arxiv.org/abs/2608.01358"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01358","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01358","title":"HopRefusalBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01437","domain":"General AI","name":"FLEX","released":"2026-08-02","url":"https://arxiv.org/abs/2608.01437"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01437","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01437","title":"FLEX"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01451","domain":"Code & Software","name":"Doc2CI","released":"2026-08-02","url":"https://arxiv.org/abs/2608.01451"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01451","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01451","title":"Doc2CI"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01456","domain":"Agents","name":"DunphyBench","released":"2026-08-02","url":"https://arxiv.org/abs/2608.01456"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01456","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01456","title":"DunphyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01575","domain":"General AI","name":"F-ICL","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01575"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01575","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01575","title":"F-ICL"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01645","domain":"General AI","name":"GISAgentBench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01645"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01645","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01645","title":"GISAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01666","domain":"General AI","name":"SciStyleBench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01666"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01666","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01666","title":"SciStyleBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01679","domain":"Agents","name":"AuthMem-Bench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01679"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01679","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01679","title":"AuthMem-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01684","domain":"General AI","name":"GABench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01684"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01684","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01684","title":"GABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01709","domain":"General AI","name":"SPATIALQUERY-1M","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01709"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01709","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01709","title":"SPATIALQUERY-1M"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01792","domain":"General AI","name":"ConfBench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01792"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01792","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01792","title":"ConfBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01942","domain":"General AI","name":"CultureVidBench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01942"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01942","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01942","title":"CultureVidBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01948","domain":"General AI","name":"Event ActivityNet","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01948"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01948","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01948","title":"Event ActivityNet"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-01977","domain":"General AI","name":"SVGEval","released":"2026-08-03","url":"https://arxiv.org/abs/2608.01977"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.01977","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.01977","title":"SVGEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02018","domain":"Software & AI Compute","name":"II-Bench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02018"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02018","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02018","title":"II-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02046","domain":"General AI","name":"CompanionBench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02046"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02046","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02046","title":"CompanionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02171","domain":"General AI","name":"IBA-Bench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02171"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02171","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02171","title":"IBA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02315","domain":"General AI","name":"GEOID-Flood","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02315"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02315","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02315","title":"GEOID-Flood"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02358","domain":"Agents & Tool Use","name":"ScrambleToolBench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02358"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02358","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02358","title":"ScrambleToolBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02372","domain":"General AI","name":"PredAct-Bench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02372"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02372","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02372","title":"PredAct-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02396","domain":"General AI","name":"Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02396"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02396","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02396","title":"Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02520","domain":"Biology & Drug Discovery","name":"MedPRESS","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02520"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02520","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02520","title":"MedPRESS"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02595","domain":"Language & Knowledge","name":"onepot-Bench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02595"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02595","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02595","title":"onepot-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02603","domain":"General AI","name":"WorldExam","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02603"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02603","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02603","title":"WorldExam"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02615","domain":"Biology & Drug Discovery","name":"OncoTriad-QA","released":"2026-05-21","url":"https://arxiv.org/abs/2608.02615"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02615","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02615","title":"OncoTriad-QA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02639","domain":"Code & Software","name":"Instruction Stacking Collapse","released":"2026-07-31","url":"https://arxiv.org/abs/2608.02639"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02639","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02639","title":"Instruction Stacking Collapse"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02642","domain":"Biology & Drug Discovery","name":"MDArena","released":"2026-07-31","url":"https://arxiv.org/abs/2608.02642"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02642","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02642","title":"MDArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02670","domain":"General AI","name":"Boundary-Bench","released":"2026-08-02","url":"https://arxiv.org/abs/2608.02670"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02670","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02670","title":"Boundary-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02684","domain":"Biology & Drug Discovery","name":"SPIKE-Bench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02684"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02684","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02684","title":"SPIKE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02685","domain":"Agents","name":"BulkPR-Bench","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02685"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02685","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02685","title":"BulkPR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-02693","domain":"Code & Software","name":"PRWeaver","released":"2026-08-03","url":"https://arxiv.org/abs/2608.02693"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.02693","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.02693","title":"PRWeaver"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03028","domain":"Biology & Drug Discovery","name":"MedPIC-Bench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03028"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03028","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03028","title":"MedPIC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03070","domain":"Cybersecurity","name":"AI Security Leaderboard","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03070"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03070","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03070","title":"AI Security Leaderboard"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03071","domain":"General AI","name":"ParamBench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03071"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03071","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03071","title":"ParamBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03078","domain":"General AI","name":"LDU-Bench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03078"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03078","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03078","title":"LDU-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03096","domain":"General AI","name":"FakeI2V-Bench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03096"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03096","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03096","title":"FakeI2V-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03206","domain":"Agents","name":"EduClaw-Bench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03206"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03206","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03206","title":"EduClaw-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03259","domain":"Finance","name":"FinVerse","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03259"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03259","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03259","title":"FinVerse"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03358","domain":"General AI","name":"ArtECulture","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03358"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03358","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03358","title":"ArtECulture"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03451","domain":"General AI","name":"DataSpace","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03451"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03451","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03451","title":"DataSpace"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03464","domain":"General AI","name":"ChartAnno","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03464"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03464","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03464","title":"ChartAnno"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03474","domain":"General AI","name":"MT-Web2Code","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03474"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03474","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03474","title":"MT-Web2Code"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03499","domain":"Agents","name":"WeClawArena","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03499"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03499","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03499","title":"WeClawArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03505","domain":"Factuality","name":"ConlangBench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03505"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03505","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03505","title":"ConlangBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03535","domain":"Code","name":"CodeAssay","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03535"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03535","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03535","title":"CodeAssay"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03591","domain":"Cybersecurity","name":"DiagChain","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03591"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03591","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03591","title":"DiagChain"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03700","domain":"General AI","name":"AntiSkillBench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03700"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03700","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03700","title":"AntiSkillBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03708","domain":"General AI","name":"MSP-Bench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03708"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03708","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03708","title":"MSP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03731","domain":"Biology & Drug Discovery","name":"CARE-Bench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03731"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03731","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03731","title":"CARE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03764","domain":"Agents","name":"GDPevo","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03764"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03764","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03764","title":"GDPevo"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03794","domain":"General AI","name":"DBLifeBench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03794"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03794","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03794","title":"DBLifeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03803","domain":"Language & Knowledge","name":"M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03803"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03803","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03803","title":"M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03810","domain":"General AI","name":"VIBE","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03810"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03810","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03810","title":"VIBE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03882","domain":"Factuality","name":"MultiGlobeQA","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03882"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03882","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03882","title":"MultiGlobeQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03884","domain":"General AI","name":"BanglaWild","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03884"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03884","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03884","title":"BanglaWild"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-03983","domain":"Code & Software","name":"SeGaBench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.03983"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.03983","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.03983","title":"SeGaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04003","domain":"Agents","name":"PAST-Bench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.04003"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04003","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04003","title":"PAST-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04009","domain":"Factuality","name":"SocietyBench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.04009"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04009","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04009","title":"SocietyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04077","domain":"Finance","name":"FinProBench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.04077"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04077","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04077","title":"FinProBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04095","domain":"General AI","name":"FinPerMA","released":"2026-08-04","url":"https://arxiv.org/abs/2608.04095"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04095","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04095","title":"FinPerMA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04111","domain":"Mathematics & Formal Science","name":"GEB-Bench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.04111"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04111","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04111","title":"GEB-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04156","domain":"General AI","name":"BrainBench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.04156"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04156","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04156","title":"BrainBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04222","domain":"General AI","name":"TIDE: A Physically Diverse 3D Turbulence Benchmark Dataset for Advancing Scientific Machine Learning","released":"2026-08-04","url":"https://arxiv.org/abs/2608.04222"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04222","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04222","title":"TIDE: A Physically Diverse 3D Turbulence Benchmark Dataset for Advancing Scientific Machine Learning"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04244","domain":"General AI","name":"SIGNPOST-Bench","released":"2026-08-04","url":"https://arxiv.org/abs/2608.04244"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04244","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04244","title":"SIGNPOST-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04302","domain":"General AI","name":"CLIP-CC-Bench","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04302"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04302","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04302","title":"CLIP-CC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04374","domain":"Finance","name":"FinReportBench","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04374"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04374","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04374","title":"FinReportBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04397","domain":"General AI","name":"NOLLI","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04397"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04397","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04397","title":"NOLLI"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04434","domain":"Chip Design & EDA","name":"OmniRouting","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04434"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04434","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04434","title":"OmniRouting"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04514","domain":"Biology & Drug Discovery","name":"RESPClinBench","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04514"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04514","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04514","title":"RESPClinBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04533","domain":"General AI","name":"EgoAfford","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04533"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04533","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04533","title":"EgoAfford"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04549","domain":"General AI","name":"EuroExec","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04549"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04549","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04549","title":"EuroExec"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04570","domain":"General AI","name":"MirageBench","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04570"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04570","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04570","title":"MirageBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04587","domain":"Agents","name":"VA-EvoBench","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04587"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04587","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04587","title":"VA-EvoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04682","domain":"Code & Software","name":"Active-SWE","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04682"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04682","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04682","title":"Active-SWE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04703","domain":"General AI","name":"IslamicTurathBench","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04703"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04703","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04703","title":"IslamicTurathBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04783","domain":"Code & Software","name":"RepoProbe","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04783"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04783","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04783","title":"RepoProbe"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04828","domain":"Agents","name":"Skill-Use","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04828"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04828","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04828","title":"Skill-Use"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-04830","domain":"Agents","name":"ContextWeave","released":"2026-08-05","url":"https://arxiv.org/abs/2608.04830"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.04830","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.04830","title":"ContextWeave"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05049","domain":"General AI","name":"OmniEdit-Bench","released":"2026-08-05","url":"https://arxiv.org/abs/2608.05049"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05049","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05049","title":"OmniEdit-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05138","domain":"General AI","name":"HERA","released":"2026-08-05","url":"https://arxiv.org/abs/2608.05138"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05138","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05138","title":"HERA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05162","domain":"General AI","name":"PoolBench","released":"2026-05-26","url":"https://arxiv.org/abs/2608.05162"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05162","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05162","title":"PoolBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05196","domain":"Biology & Drug Discovery","name":"MS-MLB","released":"2026-08-04","url":"https://arxiv.org/abs/2608.05196"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05196","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05196","title":"MS-MLB"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05212","domain":"General AI","name":"SearchAuditBench","released":"2026-08-05","url":"https://arxiv.org/abs/2608.05212"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05212","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05212","title":"SearchAuditBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05246","domain":"General AI","name":"LUNAR","released":"2026-08-05","url":"https://arxiv.org/abs/2608.05246"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05246","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05246","title":"LUNAR"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05485","domain":"General AI","name":"VideoArgus-Bench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05485"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05485","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05485","title":"VideoArgus-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05519","domain":"General AI","name":"EcoAgent-Bench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05519"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05519","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05519","title":"EcoAgent-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05539","domain":"CAD","name":"OmniMech","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05539"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05539","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05539","title":"OmniMech"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05573","domain":"Agents","name":"SkillTV-Bench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05573"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05573","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05573","title":"SkillTV-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05699","domain":"General AI","name":"TAU-Bench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05699"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05699","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05699","title":"TAU-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05703","domain":"General AI","name":"StreamArena","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05703"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05703","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05703","title":"StreamArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05747","domain":"Robotics & Embodied AI","name":"GST-Bench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05747"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05747","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05747","title":"GST-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05799","domain":"Robotics","name":"XEWorld","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05799"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05799","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05799","title":"XEWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05817","domain":"General AI","name":"M$^3$R-Bench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05817"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05817","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05817","title":"M$^3$R-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05864","domain":"General AI","name":"C-SUITEBENCH","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05864"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05864","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05864","title":"C-SUITEBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-05948","domain":"Robotics & Embodied AI","name":"GAUGE","released":"2026-08-06","url":"https://arxiv.org/abs/2608.05948"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.05948","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.05948","title":"GAUGE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06022","domain":"Biology & Drug Discovery","name":"EpiBench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06022"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06022","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06022","title":"EpiBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06041","domain":"Code & Software","name":"LangChoiceBench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06041"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06041","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06041","title":"LangChoiceBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06108","domain":"Finance","name":"InvestLogicBench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06108"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06108","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06108","title":"InvestLogicBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06144","domain":"Agents","name":"FinEvo-Bench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06144"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06144","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06144","title":"FinEvo-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06301","domain":"General AI","name":"HarnessOpt-Bench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06301"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06301","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06301","title":"HarnessOpt-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06312","domain":"General AI","name":"GB/T-Bench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06312"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06312","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06312","title":"GB/T-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06346","domain":"Agents","name":"TrajErrBench","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06346"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06346","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06346","title":"TrajErrBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06404","domain":"General AI","name":"UAV3DCrop","released":"2026-08-03","url":"https://arxiv.org/abs/2608.06404"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06404","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06404","title":"UAV3DCrop"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06477","domain":"Agents","name":"StepJack","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06477"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06477","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06477","title":"StepJack"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06485","domain":"General AI","name":"BFI-Adapt","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06485"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06485","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06485","title":"BFI-Adapt"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06549","domain":"General AI","name":"TradeVerse","released":"2026-08-06","url":"https://arxiv.org/abs/2608.06549"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06549","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06549","title":"TradeVerse"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06705","domain":"Code & Software","name":"MacroBench","released":"2026-08-07","url":"https://arxiv.org/abs/2608.06705"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06705","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06705","title":"MacroBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06769","domain":"General AI","name":"GraphVerse","released":"2026-08-07","url":"https://arxiv.org/abs/2608.06769"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06769","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06769","title":"GraphVerse"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06772","domain":"General AI","name":"ArchEGraph","released":"2026-08-07","url":"https://arxiv.org/abs/2608.06772"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06772","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06772","title":"ArchEGraph"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06850","domain":"General AI","name":"RegionDet","released":"2026-08-07","url":"https://arxiv.org/abs/2608.06850"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06850","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06850","title":"RegionDet"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06930","domain":"General AI","name":"AVCap-Bench","released":"2026-08-07","url":"https://arxiv.org/abs/2608.06930"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06930","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06930","title":"AVCap-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06931","domain":"AI Scientist","name":"Science Edge Evaluation","released":"2026-08-07","url":"https://arxiv.org/abs/2608.06931"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06931","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06931","title":"Science Edge Evaluation"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06972","domain":"General AI","name":"Generative Embedding Benchmark","released":"2026-08-07","url":"https://arxiv.org/abs/2608.06972"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06972","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06972","title":"Generative Embedding Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-06984","domain":"Agents","name":"HarnessSafe","released":"2026-08-07","url":"https://arxiv.org/abs/2608.06984"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.06984","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.06984","title":"HarnessSafe"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07038","domain":"Code & Software","name":"BinJudgeBench","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07038"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07038","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07038","title":"BinJudgeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07062","domain":"Biology & Drug Discovery","name":"Explanation Stability of Test-Time Adaptation in Computational Pathology: A Large-Scale Benchmark","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07062"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07062","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07062","title":"Explanation Stability of Test-Time Adaptation in Computational Pathology: A Large-Scale Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07370","domain":"General AI","name":"LitTraceQA","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07370"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07370","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07370","title":"LitTraceQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07382","domain":"General AI","name":"SkySeaLand","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07382"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07382","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07382","title":"SkySeaLand"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07400","domain":"Finance","name":"FinRank","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07400"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07400","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07400","title":"FinRank"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07411","domain":"General AI","name":"GeoBenchLLM","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07411"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07411","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07411","title":"GeoBenchLLM"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07417","domain":"General AI","name":"ISYV-Bench","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07417"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07417","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07417","title":"ISYV-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07508","domain":"Agents","name":"JaleesBench","released":"2026-06-27","url":"https://arxiv.org/abs/2608.07508"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07508","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07508","title":"JaleesBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07525","domain":"General AI","name":"UniHall","released":"2026-07-15","url":"https://arxiv.org/abs/2608.07525"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07525","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07525","title":"UniHall"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07529","domain":"Factuality","name":"WuYuEval","released":"2026-07-24","url":"https://arxiv.org/abs/2608.07529"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07529","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07529","title":"WuYuEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07530","domain":"General AI","name":"NL2SHACL-Bench","released":"2026-07-24","url":"https://arxiv.org/abs/2608.07530"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07530","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07530","title":"NL2SHACL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07540","domain":"Factuality","name":"TREAT","released":"2026-07-29","url":"https://arxiv.org/abs/2608.07540"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07540","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07540","title":"TREAT"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07556","domain":"Agents","name":"MasDrift","released":"2026-08-02","url":"https://arxiv.org/abs/2608.07556"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07556","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07556","title":"MasDrift"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07570","domain":"General AI","name":"COMEX","released":"2026-08-04","url":"https://arxiv.org/abs/2608.07570"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07570","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07570","title":"COMEX"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07584","domain":"General AI","name":"ComplexityWorld","released":"2026-08-05","url":"https://arxiv.org/abs/2608.07584"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07584","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07584","title":"ComplexityWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07617","domain":"General AI","name":"TeXFix-Bench","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07617"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07617","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07617","title":"TeXFix-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07640","domain":"General AI","name":"HeatCast","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07640"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07640","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07640","title":"HeatCast"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07641","domain":"General AI","name":"SurveyReview","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07641"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07641","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07641","title":"SurveyReview"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07796","domain":"Biology & Drug Discovery","name":"CliniCARE-Bench","released":"2026-08-07","url":"https://arxiv.org/abs/2608.07796"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07796","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07796","title":"CliniCARE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07835","domain":"General AI","name":"CV-FSS","released":"2026-08-08","url":"https://arxiv.org/abs/2608.07835"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07835","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07835","title":"CV-FSS"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07861","domain":"General AI","name":"VQABench","released":"2026-08-08","url":"https://arxiv.org/abs/2608.07861"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07861","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07861","title":"VQABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07862","domain":"General AI","name":"SurakshaEval","released":"2026-08-08","url":"https://arxiv.org/abs/2608.07862"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07862","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07862","title":"SurakshaEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07899","domain":"Agents","name":"TelemetrySuffBench","released":"2026-08-08","url":"https://arxiv.org/abs/2608.07899"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07899","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07899","title":"TelemetrySuffBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-07958","domain":"Manufacturing & Process Control","name":"LIBAD","released":"2026-08-08","url":"https://arxiv.org/abs/2608.07958"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.07958","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.07958","title":"LIBAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08045","domain":"Agents","name":"Lingjing","released":"2026-08-08","url":"https://arxiv.org/abs/2608.08045"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08045","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08045","title":"Lingjing"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08070","domain":"General AI","name":"SurgWMBench","released":"2026-08-08","url":"https://arxiv.org/abs/2608.08070"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08070","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08070","title":"SurgWMBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08160","domain":"Agents & Tool Use","name":"NCP-Bench","released":"2026-08-08","url":"https://arxiv.org/abs/2608.08160"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08160","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08160","title":"NCP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08264","domain":"General AI","name":"OBLIVION","released":"2026-08-08","url":"https://arxiv.org/abs/2608.08264"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08264","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08264","title":"OBLIVION"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08392","domain":"General AI","name":"CAP","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08392"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08392","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08392","title":"CAP"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08459","domain":"General AI","name":"Doc2DB-Bench","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08459"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08459","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08459","title":"Doc2DB-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08512","domain":"General AI","name":"TIDE","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08512"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08512","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08512","title":"TIDE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08542","domain":"General AI","name":"SkillSafe-Bench","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08542"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08542","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08542","title":"SkillSafe-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08605","domain":"Agents","name":"ForestBench","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08605"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08605","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08605","title":"ForestBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08630","domain":"General AI","name":"LongVLBench","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08630"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08630","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08630","title":"LongVLBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08640","domain":"Agents","name":"SkillReason-Bench","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08640"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08640","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08640","title":"SkillReason-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08691","domain":"Agents","name":"EnergyBridge","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08691"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08691","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08691","title":"EnergyBridge"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08700","domain":"General AI","name":"PluginEval","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08700"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08700","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08700","title":"PluginEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08727","domain":"Biology & Drug Discovery","name":"TomaBench","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08727"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08727","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08727","title":"TomaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08736","domain":"General AI","name":"FitAQA","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08736"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08736","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08736","title":"FitAQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08814","domain":"Robotics & Embodied AI","name":"360CityArena","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08814"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08814","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08814","title":"360CityArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08852","domain":"Factuality","name":"Teaching Monster Challenge","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08852"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08852","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08852","title":"Teaching Monster Challenge"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08964","domain":"Mathematics & Formal Science","name":"Math-Vision Diagrams","released":"2026-08-09","url":"https://arxiv.org/abs/2608.08964"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08964","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08964","title":"Math-Vision Diagrams"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-08994","domain":"Factuality","name":"Guardian Crawler","released":"2026-08-10","url":"https://arxiv.org/abs/2608.08994"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.08994","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.08994","title":"Guardian Crawler"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09024","domain":"Biology & Drug Discovery","name":"EHR2Dial-Triage","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09024"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09024","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09024","title":"EHR2Dial-Triage"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09072","domain":"Code","name":"SWE-RPG","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09072"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09072","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09072","title":"SWE-RPG"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09096","domain":"Agents","name":"Evo-Bench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09096"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09096","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09096","title":"Evo-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09106","domain":"Factuality","name":"LexKairos","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09106"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09106","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09106","title":"LexKairos"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09124","domain":"General AI","name":"ChronoState","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09124"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09124","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09124","title":"ChronoState"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09189","domain":"General AI","name":"EmoSBench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09189"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09189","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09189","title":"EmoSBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09200","domain":"General AI","name":"NBA_Streaming","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09200"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09200","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09200","title":"NBA_Streaming"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09230","domain":"Factuality","name":"SafeSceneReason","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09230"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09230","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09230","title":"SafeSceneReason"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09281","domain":"General AI","name":"MMArch","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09281"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09281","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09281","title":"MMArch"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09282","domain":"Agents & Tool Use","name":"ComboShoppingBench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09282"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09282","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09282","title":"ComboShoppingBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09296","domain":"CAD","name":"CADEngBench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09296"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09296","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09296","title":"CADEngBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09374","domain":"General AI","name":"CircuitReason-1k","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09374"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09374","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09374","title":"CircuitReason-1k"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09412","domain":"General AI","name":"KVDiagnosis","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09412"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09412","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09412","title":"KVDiagnosis"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09476","domain":"General AI","name":"ActBench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09476"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09476","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09476","title":"ActBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09497","domain":"General AI","name":"SwissCrop25","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09497"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09497","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09497","title":"SwissCrop25"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09538","domain":"Language & Knowledge","name":"TCS-BENCH","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09538"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09538","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09538","title":"TCS-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09548","domain":"General AI","name":"ELBench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09548"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09548","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09548","title":"ELBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09573","domain":"General AI","name":"VideoVIBE","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09573"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09573","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09573","title":"VideoVIBE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09593","domain":"Robotics & Embodied AI","name":"MADBench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09593"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09593","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09593","title":"MADBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09638","domain":"General AI","name":"Avalon-ToM-Bench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09638"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09638","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09638","title":"Avalon-ToM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09772","domain":"General AI","name":"PragMatch","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09772"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09772","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09772","title":"PragMatch"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09802","domain":"Code & Software","name":"SWE-Bench ProMax","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09802"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09802","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09802","title":"SWE-Bench ProMax"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09873","domain":"Factuality","name":"Sci-VBench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.09873"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09873","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09873","title":"Sci-VBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-09946","domain":"General AI","name":"HoosierHelp","released":"2026-07-03","url":"https://arxiv.org/abs/2608.09946"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.09946","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.09946","title":"HoosierHelp"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10042","domain":"General AI","name":"UserToolBench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.10042"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10042","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10042","title":"UserToolBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10258","domain":"Biology & Drug Discovery","name":"TAF-MED","released":"2026-08-10","url":"https://arxiv.org/abs/2608.10258"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10258","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10258","title":"TAF-MED"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10317","domain":"General AI","name":"TAR-Bench","released":"2026-08-10","url":"https://arxiv.org/abs/2608.10317"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10317","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10317","title":"TAR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10359","domain":"General AI","name":"VoxSumm","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10359"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10359","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10359","title":"VoxSumm"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10366","domain":"General AI","name":"DSAgentBench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10366"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10366","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10366","title":"DSAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10396","domain":"General AI","name":"FormStruct-Bench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10396"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10396","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10396","title":"FormStruct-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10408","domain":"General AI","name":"VisEditBench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10408"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10408","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10408","title":"VisEditBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10414","domain":"General AI","name":"VialectBench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10414"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10414","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10414","title":"VialectBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10444","domain":"General AI","name":"MPAR-Bench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10444"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10444","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10444","title":"MPAR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10589","domain":"General AI","name":"π-SUB","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10589"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10589","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10589","title":"π-SUB"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10669","domain":"Agents","name":"REDAgentBench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10669"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10669","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10669","title":"REDAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10677","domain":"Finance","name":"Chartography","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10677"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10677","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10677","title":"Chartography"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10679","domain":"General AI","name":"ENTLORE","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10679"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10679","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10679","title":"ENTLORE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10692","domain":"General AI","name":"SPIEval","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10692"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10692","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10692","title":"SPIEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10875","domain":"Agents","name":"VibeLifeBench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10875"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10875","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10875","title":"VibeLifeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10916","domain":"General AI","name":"FaithformBench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10916"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10916","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10916","title":"FaithformBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-10932","domain":"Factuality","name":"CamChoreo","released":"2026-08-11","url":"https://arxiv.org/abs/2608.10932"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.10932","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.10932","title":"CamChoreo"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11002","domain":"General AI","name":"LingT2I","released":"2026-08-11","url":"https://arxiv.org/abs/2608.11002"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11002","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11002","title":"LingT2I"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11074","domain":"Factuality","name":"CapProbe","released":"2026-08-11","url":"https://arxiv.org/abs/2608.11074"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11074","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11074","title":"CapProbe"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11149","domain":"Factuality","name":"PRMU","released":"2026-08-11","url":"https://arxiv.org/abs/2608.11149"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11149","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11149","title":"PRMU"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11154","domain":"Factuality","name":"CriticalSCM-Bench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.11154"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11154","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11154","title":"CriticalSCM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11216","domain":"General AI","name":"AutoWorldModel-Bench","released":"2026-07-20","url":"https://arxiv.org/abs/2608.11216"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11216","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11216","title":"AutoWorldModel-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11232","domain":"Finance","name":"Backtrader-Bench","released":"2026-07-31","url":"https://arxiv.org/abs/2608.11232"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11232","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11232","title":"Backtrader-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11234","domain":"General AI","name":"InfraBench","released":"2026-07-31","url":"https://arxiv.org/abs/2608.11234"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11234","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11234","title":"InfraBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11242","domain":"General AI","name":"COMPINT","released":"2026-07-31","url":"https://arxiv.org/abs/2608.11242"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11242","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11242","title":"COMPINT"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11327","domain":"Finance","name":"ProForma-20Q","released":"2026-08-11","url":"https://arxiv.org/abs/2608.11327"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11327","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11327","title":"ProForma-20Q"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11367","domain":"Biology & Drug Discovery","name":"Gaze-Co","released":"2026-08-11","url":"https://arxiv.org/abs/2608.11367"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11367","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11367","title":"Gaze-Co"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11434","domain":"Agents","name":"MobileJudgeBench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.11434"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11434","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11434","title":"MobileJudgeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11452","domain":"General AI","name":"TangPoetryBench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.11452"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11452","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11452","title":"TangPoetryBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11469","domain":"Cybersecurity","name":"SRE-Bench","released":"2026-08-11","url":"https://arxiv.org/abs/2608.11469"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11469","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11469","title":"SRE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11562","domain":"General AI","name":"S2R-Bench","released":"2026-08-12","url":"https://arxiv.org/abs/2608.11562"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11562","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11562","title":"S2R-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11584","domain":"Factuality","name":"EnterpriseRAG","released":"2026-08-12","url":"https://arxiv.org/abs/2608.11584"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11584","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11584","title":"EnterpriseRAG"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11616","domain":"General AI","name":"MBA-Bench","released":"2026-08-12","url":"https://arxiv.org/abs/2608.11616"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11616","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11616","title":"MBA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11683","domain":"Finance","name":"FrontierFinance","released":"2026-08-12","url":"https://arxiv.org/abs/2608.11683"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11683","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11683","title":"FrontierFinance"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-11941","domain":"General AI","name":"OEIS Open","released":"2026-08-12","url":"https://arxiv.org/abs/2608.11941"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.11941","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.11941","title":"OEIS Open"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12002","domain":"Agents","name":"CTBench","released":"2026-08-12","url":"https://arxiv.org/abs/2608.12002"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12002","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12002","title":"CTBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12004","domain":"Code & Software","name":"RealisticTritonBench","released":"2026-08-12","url":"https://arxiv.org/abs/2608.12004"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12004","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12004","title":"RealisticTritonBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12197","domain":"Robot manipulation","name":"NetlistBench","released":"2026-08-12","url":"https://arxiv.org/abs/2608.12197"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12197","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12197","title":"NetlistBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12262","domain":"General AI","name":"Diagram-MMU","released":"2026-08-12","url":"https://arxiv.org/abs/2608.12262"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12262","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12262","title":"Diagram-MMU"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12282","domain":"General AI","name":"VAKRA","released":"2026-08-12","url":"https://arxiv.org/abs/2608.12282"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12282","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12282","title":"VAKRA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12324","domain":"Biology & Drug Discovery","name":"FMG-Bench","released":"2026-05-29","url":"https://arxiv.org/abs/2608.12324"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12324","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12324","title":"FMG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12342","domain":"Finance","name":"FinED-Bench","released":"2026-06-03","url":"https://arxiv.org/abs/2608.12342"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12342","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12342","title":"FinED-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12345","domain":"General AI","name":"IntegrityBench","released":"2026-06-03","url":"https://arxiv.org/abs/2608.12345"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12345","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12345","title":"IntegrityBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12442","domain":"General AI","name":"MV2","released":"2026-08-12","url":"https://arxiv.org/abs/2608.12442"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12442","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12442","title":"MV2"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12549","domain":"General AI","name":"StrAD","released":"2026-08-12","url":"https://arxiv.org/abs/2608.12549"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12549","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12549","title":"StrAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12593","domain":"General AI","name":"DiG-bench","released":"2026-08-12","url":"https://arxiv.org/abs/2608.12593"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12593","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12593","title":"DiG-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12654","domain":"Agents","name":"SteerBench-Work","released":"2026-08-12","url":"https://arxiv.org/abs/2608.12654"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12654","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12654","title":"SteerBench-Work"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12781","domain":"Factuality","name":"PatternEval","released":"2026-08-13","url":"https://arxiv.org/abs/2608.12781"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12781","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12781","title":"PatternEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12788","domain":"General AI","name":"ARAC-Bench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.12788"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12788","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12788","title":"ARAC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12805","domain":"Biology & Drug Discovery","name":"CoMedBench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.12805"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12805","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12805","title":"CoMedBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12851","domain":"Agents","name":"SkillMisevo-Bench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.12851"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12851","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12851","title":"SkillMisevo-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12860","domain":"Multimodal","name":"HumanoidVLN","released":"2026-08-13","url":"https://arxiv.org/abs/2608.12860"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12860","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12860","title":"HumanoidVLN"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12894","domain":"General AI","name":"BavGround","released":"2026-08-13","url":"https://arxiv.org/abs/2608.12894"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12894","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12894","title":"BavGround"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12904","domain":"Biology & Drug Discovery","name":"HounsBench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.12904"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12904","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12904","title":"HounsBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-12911","domain":"General AI","name":"DocPrivacyBench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.12911"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.12911","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.12911","title":"DocPrivacyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13049","domain":"Robot manipulation","name":"H2R-Bench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13049"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13049","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13049","title":"H2R-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13113","domain":"General AI","name":"EgoMonth","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13113"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13113","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13113","title":"EgoMonth"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13136","domain":"General AI","name":"LigBench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13136"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13136","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13136","title":"LigBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13167","domain":"General AI","name":"TRAPSBench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13167"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13167","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13167","title":"TRAPSBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13210","domain":"General AI","name":"NARU","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13210"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13210","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13210","title":"NARU"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13221","domain":"General AI","name":"TsuGO","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13221"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13221","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13221","title":"TsuGO"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13239","domain":"Multimodal","name":"Intentbench-Prime","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13239"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13239","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13239","title":"Intentbench-Prime"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13267","domain":"General AI","name":"SciFigBench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13267"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13267","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13267","title":"SciFigBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13309","domain":"Biology & Drug Discovery","name":"Time-Aware Multi-View MRI Benchmark","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13309"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13309","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13309","title":"Time-Aware Multi-View MRI Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13344","domain":"General AI","name":"LongEarth-Bench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13344"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13344","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13344","title":"LongEarth-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13441","domain":"Software & AI Compute","name":"Edit2TikZ","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13441"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13441","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13441","title":"Edit2TikZ"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13522","domain":"Language & Knowledge","name":"Vero","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13522"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13522","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13522","title":"Vero"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13547","domain":"General AI","name":"QuoteBench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13547"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13547","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13547","title":"QuoteBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13552","domain":"Agents","name":"PlayWorld","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13552"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13552","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13552","title":"PlayWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13560","domain":"General AI","name":"PosterBench","released":"2026-08-13","url":"https://arxiv.org/abs/2608.13560"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13560","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13560","title":"PosterBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13854","domain":"General AI","name":"HumanEval-X++","released":"2026-08-14","url":"https://arxiv.org/abs/2608.13854"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13854","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13854","title":"HumanEval-X++"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-13990","domain":"General AI","name":"SCOPE-Bench","released":"2026-08-14","url":"https://arxiv.org/abs/2608.13990"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.13990","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.13990","title":"SCOPE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14015","domain":"Agents","name":"MedClawBench","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14015"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14015","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14015","title":"MedClawBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14049","domain":"Robot manipulation","name":"FlatLab","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14049"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14049","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14049","title":"FlatLab"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14075","domain":"General AI","name":"ALD/E-ImageMiner","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14075"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14075","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14075","title":"ALD/E-ImageMiner"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14179","domain":"General AI","name":"mmWave-QA","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14179"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14179","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14179","title":"mmWave-QA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14262","domain":"Biology & Drug Discovery","name":"Endo-C6","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14262"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14262","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14262","title":"Endo-C6"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14270","domain":"General AI","name":"TimeSage-EV","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14270"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14270","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14270","title":"TimeSage-EV"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14284","domain":"Robot manipulation","name":"PRM-as-a-Judge 1.5","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14284"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14284","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14284","title":"PRM-as-a-Judge 1.5"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14320","domain":"General AI","name":"AnchorBench","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14320"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14320","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14320","title":"AnchorBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14329","domain":"Finance","name":"Principle-Bench","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14329"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14329","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14329","title":"Principle-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14379","domain":"Multimodal","name":"ReflexBench","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14379"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14379","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14379","title":"ReflexBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14389","domain":"General AI","name":"GBU-Palm","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14389"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14389","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14389","title":"GBU-Palm"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14391","domain":"General AI","name":"RA-Bench","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14391"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14391","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14391","title":"RA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14441","domain":"Agents","name":"PACE-Bench","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14441"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14441","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14441","title":"PACE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14546","domain":"General AI","name":"CPI-Bench","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14546"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14546","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14546","title":"CPI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14680","domain":"General AI","name":"AGENTCHAOSBENCH","released":"2026-08-04","url":"https://arxiv.org/abs/2608.14680"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14680","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14680","title":"AGENTCHAOSBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14718","domain":"General AI","name":"VideoGAIA","released":"2026-08-12","url":"https://arxiv.org/abs/2608.14718"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14718","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14718","title":"VideoGAIA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14721","domain":"Reasoning","name":"AeroGround","released":"2026-08-12","url":"https://arxiv.org/abs/2608.14721"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14721","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14721","title":"AeroGround"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14741","domain":"General AI","name":"PolyComp","released":"2026-08-13","url":"https://arxiv.org/abs/2608.14741"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14741","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14741","title":"PolyComp"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14863","domain":"Code & Software","name":"DDBench","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14863"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14863","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14863","title":"DDBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-14868","domain":"General AI","name":"HighwayScene","released":"2026-08-14","url":"https://arxiv.org/abs/2608.14868"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.14868","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.14868","title":"HighwayScene"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15064","domain":"General AI","name":"LongDocBench","released":"2026-08-15","url":"https://arxiv.org/abs/2608.15064"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15064","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15064","title":"LongDocBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15092","domain":"Cybersecurity","name":"WeSCE","released":"2026-08-15","url":"https://arxiv.org/abs/2608.15092"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15092","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15092","title":"WeSCE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15127","domain":"General AI","name":"AgentSysBench","released":"2026-08-15","url":"https://arxiv.org/abs/2608.15127"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15127","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15127","title":"AgentSysBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15177","domain":"Finance","name":"FinFraudBench","released":"2026-08-15","url":"https://arxiv.org/abs/2608.15177"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15177","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15177","title":"FinFraudBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15242","domain":"Agents","name":"LongRCA Bench","released":"2026-08-15","url":"https://arxiv.org/abs/2608.15242"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15242","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15242","title":"LongRCA Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15286","domain":"Agents","name":"AgentRelBench","released":"2026-08-15","url":"https://arxiv.org/abs/2608.15286"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15286","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15286","title":"AgentRelBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15404","domain":"General AI","name":"CBX-Bench","released":"2026-08-15","url":"https://arxiv.org/abs/2608.15404"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15404","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15404","title":"CBX-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15410","domain":"Reasoning","name":"FloodReasonBench","released":"2026-08-15","url":"https://arxiv.org/abs/2608.15410"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15410","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15410","title":"FloodReasonBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15425","domain":"General AI","name":"NumerosityVLM","released":"2026-08-15","url":"https://arxiv.org/abs/2608.15425"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15425","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15425","title":"NumerosityVLM"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15535","domain":"Factuality","name":"L3Cube-IndicQuest v2","released":"2026-08-16","url":"https://arxiv.org/abs/2608.15535"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15535","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15535","title":"L3Cube-IndicQuest v2"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15539","domain":"Autonomous Driving","name":"CrossView","released":"2026-08-16","url":"https://arxiv.org/abs/2608.15539"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15539","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15539","title":"CrossView"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15555","domain":"General AI","name":"RigidBench","released":"2026-08-16","url":"https://arxiv.org/abs/2608.15555"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15555","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15555","title":"RigidBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15600","domain":"General AI","name":"VARM-Bench","released":"2026-08-16","url":"https://arxiv.org/abs/2608.15600"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15600","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15600","title":"VARM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15654","domain":"Agents","name":"WSE-bench","released":"2026-08-16","url":"https://arxiv.org/abs/2608.15654"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15654","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15654","title":"WSE-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-15931","domain":"Code & Software","name":"PLSQLBench","released":"2026-08-16","url":"https://arxiv.org/abs/2608.15931"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.15931","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.15931","title":"PLSQLBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16011","domain":"General AI","name":"ReRef-3D","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16011"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16011","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16011","title":"ReRef-3D"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16022","domain":"Code & Software","name":"OpenHarmony Bench","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16022"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16022","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16022","title":"OpenHarmony Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16033","domain":"Code & Software","name":"R3-Bench","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16033"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16033","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16033","title":"R3-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16055","domain":"Agents","name":"Fiducia-bench","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16055"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16055","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16055","title":"Fiducia-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16081","domain":"General AI","name":"SafeGesture","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16081"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16081","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16081","title":"SafeGesture"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16203","domain":"General AI","name":"INSPIRE","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16203"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16203","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16203","title":"INSPIRE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16349","domain":"Agents","name":"AeroCopilotBench","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16349"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16349","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16349","title":"AeroCopilotBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16645","domain":"General AI","name":"Reconstruction","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16645"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16645","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16645","title":"Reconstruction"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16717","domain":"General AI","name":"PersonaShot","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16717"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16717","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16717","title":"PersonaShot"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16765","domain":"General AI","name":"TRACE-Bench","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16765"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16765","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16765","title":"TRACE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-16812","domain":"General AI","name":"ConceptEdit-Bench","released":"2026-08-17","url":"https://arxiv.org/abs/2608.16812"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.16812","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.16812","title":"ConceptEdit-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17110","domain":"General AI","name":"OV3D-Bench","released":"2026-08-17","url":"https://arxiv.org/abs/2608.17110"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17110","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17110","title":"OV3D-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17271","domain":"AI Scientist","name":"ASI-Bench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17271"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17271","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17271","title":"ASI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17299","domain":"Distribution shift","name":"LiveHouse-TS","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17299"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17299","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17299","title":"LiveHouse-TS"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17318","domain":"Multimodal","name":"CondVLN","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17318"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17318","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17318","title":"CondVLN"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17379","domain":"Architecture-specific programming","name":"PTXBench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17379"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17379","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17379","title":"PTXBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17386","domain":"Robot manipulation","name":"ManiGuard-Bench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17386"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17386","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17386","title":"ManiGuard-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17550","domain":"Academic document parsing","name":"CADP-Bench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17550"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17550","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17550","title":"CADP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17566","domain":"General AI","name":"CoinVE-Bench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17566"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17566","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17566","title":"CoinVE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17597","domain":"Agent safety","name":"HarnessRisk","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17597"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17597","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17597","title":"HarnessRisk"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17600","domain":"Embodied manipulation","name":"LIBERO-VIFO","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17600"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17600","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17600","title":"LIBERO-VIFO"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17607","domain":"Biology & Drug Discovery","name":"PathoArgus-Bench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17607"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17607","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17607","title":"PathoArgus-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17659","domain":"Agents","name":"MobileWorldSafety","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17659"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17659","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17659","title":"MobileWorldSafety"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17671","domain":"Code & Software","name":"Porting Benchmark","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17671"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17671","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17671","title":"Porting Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17800","domain":"Agents","name":"StartupBench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17800"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17800","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17800","title":"StartupBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17895","domain":"Bilingual reasoning","name":"BEAR-Bench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17895"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17895","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17895","title":"BEAR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17962","domain":"Contact-rich control","name":"PRISM","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17962"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17962","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17962","title":"PRISM"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-17979","domain":"General AI","name":"AVShift","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17979"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.17979","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.17979","title":"AVShift"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-18034","domain":"General AI","name":"DAS-Bench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.18034"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.18034","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18034","title":"DAS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-18423","domain":"Finance","name":"FM-Bench","released":"2026-08-19","url":"https://arxiv.org/abs/2608.18423"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.18423","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18423","title":"FM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-18534","domain":"Finance","name":"FinRCA-Bench","released":"2026-08-19","url":"https://arxiv.org/abs/2608.18534"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.18534","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18534","title":"FinRCA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-18586","domain":"General AI","name":"OmniHandwritingOCR","released":"2026-08-19","url":"https://arxiv.org/abs/2608.18586"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.18586","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18586","title":"OmniHandwritingOCR"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-18595","domain":"Code & Software","name":"OdinEval","released":"2026-08-19","url":"https://arxiv.org/abs/2608.18595"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.18595","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18595","title":"OdinEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-18607","domain":"General AI","name":"VA-Judger-Bench","released":"2026-08-19","url":"https://arxiv.org/abs/2608.18607"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.18607","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18607","title":"VA-Judger-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-18701","domain":"Robot manipulation","name":"SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation","released":"2026-08-19","url":"https://arxiv.org/abs/2608.18701"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.18701","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18701","title":"SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-18726","domain":"General AI","name":"AtmosCoder-Bench","released":"2026-08-19","url":"https://arxiv.org/abs/2608.18726"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.18726","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18726","title":"AtmosCoder-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-18752","domain":"General AI","name":"GreekBarRetrieval","released":"2026-08-19","url":"https://arxiv.org/abs/2608.18752"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.18752","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.18752","title":"GreekBarRetrieval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-19583","domain":"General AI","name":"VGI-BENCH","released":"2026-08-20","url":"https://arxiv.org/abs/2608.19583"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.19583","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19583","title":"VGI-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-19646","domain":"General AI","name":"PL-NBA","released":"2026-08-20","url":"https://arxiv.org/abs/2608.19646"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.19646","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19646","title":"PL-NBA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-19653","domain":"General AI","name":"DeltaML-Bench","released":"2026-08-20","url":"https://arxiv.org/abs/2608.19653"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.19653","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19653","title":"DeltaML-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-19741","domain":"General AI","name":"Thinkingbox-Bench","released":"2026-08-20","url":"https://arxiv.org/abs/2608.19741"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.19741","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19741","title":"Thinkingbox-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-19746","domain":"General AI","name":"PersonalBench","released":"2026-08-20","url":"https://arxiv.org/abs/2608.19746"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.19746","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19746","title":"PersonalBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-19799","domain":"Code","name":"SWE-bench Science","released":"2026-08-20","url":"https://arxiv.org/abs/2608.19799"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.19799","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19799","title":"SWE-bench Science"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-19901","domain":"Agents","name":"MaliciousSkillBench","released":"2026-08-20","url":"https://arxiv.org/abs/2608.19901"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.19901","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19901","title":"MaliciousSkillBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-19981","domain":"Biology & Drug Discovery","name":"HealMed","released":"2026-08-20","url":"https://arxiv.org/abs/2608.19981"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.19981","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.19981","title":"HealMed"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-20009","domain":"General AI","name":"ExPhy","released":"2026-08-20","url":"https://arxiv.org/abs/2608.20009"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.20009","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20009","title":"ExPhy"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-20106","domain":"Factuality","name":"OenoBench","released":"2026-08-20","url":"https://arxiv.org/abs/2608.20106"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.20106","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20106","title":"OenoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-20107","domain":"General AI","name":"BeyondMasks","released":"2026-08-20","url":"https://arxiv.org/abs/2608.20107"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.20107","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20107","title":"BeyondMasks"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-20153","domain":"Language & Knowledge","name":"FormalTCS","released":"2026-08-20","url":"https://arxiv.org/abs/2608.20153"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.20153","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20153","title":"FormalTCS"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-20204","domain":"General AI","name":"ContractScrub","released":"2026-08-20","url":"https://arxiv.org/abs/2608.20204"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.20204","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20204","title":"ContractScrub"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-20220","domain":"General AI","name":"InsufficiencyBench","released":"2026-08-20","url":"https://arxiv.org/abs/2608.20220"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.20220","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20220","title":"InsufficiencyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-20237","domain":"General AI","name":"RuleMaze","released":"2026-08-20","url":"https://arxiv.org/abs/2608.20237"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.20237","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20237","title":"RuleMaze"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-20312","domain":"General AI","name":"Inter-X++","released":"2026-08-20","url":"https://arxiv.org/abs/2608.20312"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.20312","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20312","title":"Inter-X++"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-20318","domain":"Language & Knowledge","name":"AI4AI-Bench","released":"2026-08-20","url":"https://arxiv.org/abs/2608.20318"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.20318","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.20318","title":"AI4AI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22232","domain":"General AI","name":"MSIBench","released":"2026-08-23","url":"https://arxiv.org/abs/2608.22232"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.22232","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.22232","title":"MSIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22279","domain":"General AI","name":"OVIBench","released":"2026-08-23","url":"https://arxiv.org/abs/2608.22279"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.22279","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.22279","title":"OVIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-22725","domain":"General AI","name":"SEAM-Bench","released":"2026-08-24","url":"https://arxiv.org/abs/2608.22725"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.22725","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.22725","title":"SEAM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23035","domain":"Agents & Tool Use","name":"MobilePA-Bench","released":"2026-08-24","url":"https://arxiv.org/abs/2608.23035"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.23035","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23035","title":"MobilePA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23179","domain":"General AI","name":"NetConfArena","released":"2026-08-24","url":"https://arxiv.org/abs/2608.23179"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.23179","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23179","title":"NetConfArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23200","domain":"Agents","name":"LongWoF-Bench","released":"2026-08-24","url":"https://arxiv.org/abs/2608.23200"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.23200","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23200","title":"LongWoF-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23628","domain":"Agents","name":"AFT-Bench","released":"2026-08-23","url":"https://arxiv.org/abs/2608.23628"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.23628","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23628","title":"AFT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23635","domain":"Agents","name":"ToolRobustBench","released":"2026-08-23","url":"https://arxiv.org/abs/2608.23635"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.23635","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23635","title":"ToolRobustBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23857","domain":"General AI","name":"UHI-Bench","released":"2026-08-24","url":"https://arxiv.org/abs/2608.23857"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.23857","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23857","title":"UHI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23893","domain":"Cybersecurity","name":"RuleShift-Bench","released":"2026-08-24","url":"https://arxiv.org/abs/2608.23893"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.23893","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23893","title":"RuleShift-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23898","domain":"General AI","name":"BenchBench-Protocol","released":"2026-08-24","url":"https://arxiv.org/abs/2608.23898"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.23898","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23898","title":"BenchBench-Protocol"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-23903","domain":"General AI","name":"EvoShift-Bench","released":"2026-08-24","url":"https://arxiv.org/abs/2608.23903"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.23903","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.23903","title":"EvoShift-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24056","domain":"General AI","name":"PhysicsBench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24056"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24056","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24056","title":"PhysicsBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24094","domain":"General AI","name":"SIREN-Bench-v1","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24094"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24094","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24094","title":"SIREN-Bench-v1"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24107","domain":"General AI","name":"MatReplace","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24107"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24107","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24107","title":"MatReplace"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24118","domain":"Biology & Drug Discovery","name":"MC-CXR","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24118"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24118","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24118","title":"MC-CXR"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24145","domain":"General AI","name":"TrustDABench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24145"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24145","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24145","title":"TrustDABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24160","domain":"General AI","name":"D3-Omni","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24160"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24160","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24160","title":"D3-Omni"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24228","domain":"General AI","name":"VTC-Bench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24228"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24228","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24228","title":"VTC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24232","domain":"General AI","name":"TRACE","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24232"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24232","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24232","title":"TRACE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24252","domain":"General AI","name":"SA-Bench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24252"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24252","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24252","title":"SA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24509","domain":"General AI","name":"PeakBench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24509"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24509","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24509","title":"PeakBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24551","domain":"Finance","name":"FraudBench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24551"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24551","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24551","title":"FraudBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24707","domain":"General AI","name":"Lost in Speech","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24707"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24707","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24707","title":"Lost in Speech"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24954","domain":"AI Scientist","name":"AFDBench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24954"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24954","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24954","title":"AFDBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-24979","domain":"General AI","name":"FrontierChallenge","released":"2026-08-25","url":"https://arxiv.org/abs/2608.24979"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.24979","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.24979","title":"FrontierChallenge"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25039","domain":"General AI","name":"LifePlanner","released":"2026-08-25","url":"https://arxiv.org/abs/2608.25039"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25039","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25039","title":"LifePlanner"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25061","domain":"General AI","name":"DataKernelBench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.25061"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25061","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25061","title":"DataKernelBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25071","domain":"Biology & Drug Discovery","name":"HealthBench-Psych","released":"2026-08-25","url":"https://arxiv.org/abs/2608.25071"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25071","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25071","title":"HealthBench-Psych"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25097","domain":"Mathematics & Formal Science","name":"PhysElite","released":"2026-08-25","url":"https://arxiv.org/abs/2608.25097"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25097","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25097","title":"PhysElite"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25158","domain":"Cybersecurity","name":"FuzzingBrain-Bench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.25158"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25158","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25158","title":"FuzzingBrain-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25218","domain":"General AI","name":"TurnBench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.25218"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25218","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25218","title":"TurnBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25286","domain":"General AI","name":"BixBench3","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25286"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25286","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25286","title":"BixBench3"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25325","domain":"Finance","name":"FinRiskAtlas","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25325"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25325","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25325","title":"FinRiskAtlas"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25398","domain":"General AI","name":"OmniPhys","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25398"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25398","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25398","title":"OmniPhys"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25401","domain":"General AI","name":"PIVOT","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25401"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25401","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25401","title":"PIVOT"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25417","domain":"Agents","name":"EASEL","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25417"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25417","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25417","title":"EASEL"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25449","domain":"General AI","name":"MathAdv","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25449"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25449","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25449","title":"MathAdv"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25452","domain":"General AI","name":"VGA-BenchV2","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25452"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25452","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25452","title":"VGA-BenchV2"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25490","domain":"General AI","name":"MMJailBench","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25490"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25490","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25490","title":"MMJailBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25529","domain":"General AI","name":"Video-IFBench","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25529"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25529","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25529","title":"Video-IFBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25655","domain":"General AI","name":"SCALE-QA","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25655"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25655","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25655","title":"SCALE-QA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25939","domain":"Code & Software","name":"XREPOTEST","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25939"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25939","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25939","title":"XREPOTEST"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-25973","domain":"Materials & Chemistry","name":"SciMIF","released":"2026-08-26","url":"https://arxiv.org/abs/2608.25973"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.25973","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.25973","title":"SciMIF"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26105","domain":"General AI","name":"VBVR-Pro","released":"2026-08-26","url":"https://arxiv.org/abs/2608.26105"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26105","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26105","title":"VBVR-Pro"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26204","domain":"Agents & Tool Use","name":"ADeptS-Bench","released":"2026-08-25","url":"https://arxiv.org/abs/2608.26204"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26204","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26204","title":"ADeptS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26382","domain":"Biology & Drug Discovery","name":"VIPER","released":"2026-08-26","url":"https://arxiv.org/abs/2608.26382"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26382","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26382","title":"VIPER"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26433","domain":"General AI","name":"FedCMAPSS","released":"2026-08-26","url":"https://arxiv.org/abs/2608.26433"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26433","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26433","title":"FedCMAPSS"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26517","domain":"General AI","name":"HUG-VIS","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26517"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26517","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26517","title":"HUG-VIS"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26535","domain":"Factuality","name":"Multi2AV-Safety","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26535"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26535","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26535","title":"Multi2AV-Safety"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26546","domain":"Agents","name":"DuMateBench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26546"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26546","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26546","title":"DuMateBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26623","domain":"General AI","name":"AgentJudgeBench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26623"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26623","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26623","title":"AgentJudgeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26757","domain":"Finance","name":"DEEPCHART","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26757"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26757","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26757","title":"DEEPCHART"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26807","domain":"General AI","name":"Behavior2Trip","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26807"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26807","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26807","title":"Behavior2Trip"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26832","domain":"General AI","name":"RuleWeaver","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26832"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26832","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26832","title":"RuleWeaver"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26867","domain":"General AI","name":"BekchiAI-Benchmark","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26867"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26867","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26867","title":"BekchiAI-Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26882","domain":"Agents & Tool Use","name":"PLCBENCH","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26882"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26882","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26882","title":"PLCBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-26947","domain":"Robotics & Embodied AI","name":"4DSynth-Nav","released":"2026-08-27","url":"https://arxiv.org/abs/2608.26947"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.26947","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.26947","title":"4DSynth-Nav"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27021","domain":"General AI","name":"FaulT-Bench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27021"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27021","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27021","title":"FaulT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27154","domain":"General AI","name":"ReViCo","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27154"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27154","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27154","title":"ReViCo"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27169","domain":"General AI","name":"Ancient-Bench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27169"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27169","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27169","title":"Ancient-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27219","domain":"General AI","name":"BALMS","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27219"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27219","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27219","title":"BALMS"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27268","domain":"General AI","name":"BrailleBench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27268"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27268","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27268","title":"BrailleBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27328","domain":"General AI","name":"R2M-Bench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27328"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27328","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27328","title":"R2M-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27334","domain":"Agents","name":"BTS-AgentBench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27334"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27334","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27334","title":"BTS-AgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27345","domain":"General AI","name":"PAWBench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27345"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27345","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27345","title":"PAWBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27371","domain":"Planning","name":"ESRP-Bench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27371"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27371","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27371","title":"ESRP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27391","domain":"Factuality","name":"CorporateBench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27391"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27391","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27391","title":"CorporateBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27394","domain":"General AI","name":"RATIO","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27394"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27394","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27394","title":"RATIO"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27442","domain":"Code & Software","name":"MCR-Bench","released":"2026-08-27","url":"https://arxiv.org/abs/2608.27442"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27442","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27442","title":"MCR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27844","domain":"Factuality","name":"EvoHarmBench","released":"2026-08-28","url":"https://arxiv.org/abs/2608.27844"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27844","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27844","title":"EvoHarmBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-27953","domain":"General AI","name":"WhatIfBench","released":"2026-08-28","url":"https://arxiv.org/abs/2608.27953"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.27953","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.27953","title":"WhatIfBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28009","domain":"General AI","name":"MOSAIC","released":"2026-08-28","url":"https://arxiv.org/abs/2608.28009"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.28009","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28009","title":"MOSAIC"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28062","domain":"General AI","name":"VisTarget-Bench","released":"2026-08-28","url":"https://arxiv.org/abs/2608.28062"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.28062","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28062","title":"VisTarget-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28070","domain":"General AI","name":"Copper Tube Defect Dataset","released":"2026-08-28","url":"https://arxiv.org/abs/2608.28070"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.28070","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28070","title":"Copper Tube Defect Dataset"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28092","domain":"Biology & Drug Discovery","name":"SPAR-Bench","released":"2026-08-28","url":"https://arxiv.org/abs/2608.28092"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.28092","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28092","title":"SPAR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28155","domain":"Factuality","name":"FinExam-10K","released":"2026-08-28","url":"https://arxiv.org/abs/2608.28155"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.28155","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28155","title":"FinExam-10K"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28406","domain":"General AI","name":"MD-VQA","released":"2026-08-28","url":"https://arxiv.org/abs/2608.28406"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.28406","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28406","title":"MD-VQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-28478","domain":"Factuality","name":"ElephantBench","released":"2026-08-28","url":"https://arxiv.org/abs/2608.28478"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.28478","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.28478","title":"ElephantBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29098","domain":"General AI","name":"SafeAtlas-Bench","released":"2026-08-29","url":"https://arxiv.org/abs/2608.29098"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29098","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29098","title":"SafeAtlas-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29128","domain":"General AI","name":"APIFlow-Bench","released":"2026-08-29","url":"https://arxiv.org/abs/2608.29128"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29128","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29128","title":"APIFlow-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29210","domain":"General AI","name":"Imag-Eval","released":"2026-08-29","url":"https://arxiv.org/abs/2608.29210"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29210","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29210","title":"Imag-Eval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29257","domain":"Cybersecurity","name":"PopMCQ","released":"2026-08-29","url":"https://arxiv.org/abs/2608.29257"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29257","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29257","title":"PopMCQ"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29270","domain":"General AI","name":"ShadowBench","released":"2026-08-29","url":"https://arxiv.org/abs/2608.29270"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29270","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29270","title":"ShadowBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29286","domain":"General AI","name":"MMPCBench","released":"2026-08-29","url":"https://arxiv.org/abs/2608.29286"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29286","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29286","title":"MMPCBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29345","domain":"Factuality","name":"BIRD-History","released":"2026-08-29","url":"https://arxiv.org/abs/2608.29345"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29345","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29345","title":"BIRD-History"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29387","domain":"General AI","name":"EvoGenUI-Bench","released":"2026-08-29","url":"https://arxiv.org/abs/2608.29387"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29387","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29387","title":"EvoGenUI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29397","domain":"Agents","name":"AlgoWorlds","released":"2026-08-29","url":"https://arxiv.org/abs/2608.29397"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29397","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29397","title":"AlgoWorlds"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29477","domain":"General AI","name":"MUDDLE","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29477"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29477","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29477","title":"MUDDLE"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29481","domain":"Agents & Tool Use","name":"PitfallBench","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29481"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29481","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29481","title":"PitfallBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29483","domain":"Reasoning","name":"GeoAgent","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29483"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29483","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29483","title":"GeoAgent"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29543","domain":"General AI","name":"TIDE-Bench","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29543"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29543","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29543","title":"TIDE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29577","domain":"General AI","name":"TRINITY","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29577"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29577","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29577","title":"TRINITY"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29607","domain":"General AI","name":"SnapBench","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29607"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29607","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29607","title":"SnapBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29624","domain":"General AI","name":"PrivBench","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29624"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29624","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29624","title":"PrivBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29632","domain":"Code & Software","name":"InteractBench","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29632"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29632","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29632","title":"InteractBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29677","domain":"Biology & Drug Discovery","name":"MEDSEGBENCHMARKER","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29677"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29677","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29677","title":"MEDSEGBENCHMARKER"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29711","domain":"General AI","name":"DVBench","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29711"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29711","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29711","title":"DVBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29793","domain":"General AI","name":"City3D-MultiGen","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29793"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29793","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29793","title":"City3D-MultiGen"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29847","domain":"General AI","name":"ContextBench","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29847"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29847","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29847","title":"ContextBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29852","domain":"General AI","name":"MariSat","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29852"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29852","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29852","title":"MariSat"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-29990","domain":"General AI","name":"Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models","released":"2026-08-30","url":"https://arxiv.org/abs/2608.29990"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.29990","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.29990","title":"Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30065","domain":"General AI","name":"Pak3H","released":"2026-08-30","url":"https://arxiv.org/abs/2608.30065"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30065","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30065","title":"Pak3H"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30068","domain":"General AI","name":"TAKE 85","released":"2026-08-30","url":"https://arxiv.org/abs/2608.30068"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30068","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30068","title":"TAKE 85"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30110","domain":"Finance","name":"LiveMacroEval","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30110"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30110","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30110","title":"LiveMacroEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30241","domain":"General AI","name":"MTPaperBananaBench","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30241"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30241","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30241","title":"MTPaperBananaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30248","domain":"Code","name":"DSEffi-Bench","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30248"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30248","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30248","title":"DSEffi-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30270","domain":"General AI","name":"READI","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30270"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30270","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30270","title":"READI"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30300","domain":"Code & Software","name":"DEPBENCH","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30300"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30300","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30300","title":"DEPBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30451","domain":"Geometric reasoning","name":"SeqAlign3DVG","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30451"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30451","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30451","title":"SeqAlign3DVG"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30508","domain":"General AI","name":"UTILMEM","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30508"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30508","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30508","title":"UTILMEM"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30517","domain":"Language & Knowledge","name":"ScienceArena","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30517"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30517","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30517","title":"ScienceArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30536","domain":"Multimodal","name":"Behavior-Skill","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30536"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30536","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30536","title":"Behavior-Skill"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30584","domain":"General AI","name":"STVG-CompBench","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30584"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30584","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30584","title":"STVG-CompBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30657","domain":"Cybersecurity","name":"InfraOcc","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30657"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30657","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30657","title":"InfraOcc"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30678","domain":"General AI","name":"OCR-MetaReasoning","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30678"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30678","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30678","title":"OCR-MetaReasoning"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30682","domain":"General AI","name":"SemiMat","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30682"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30682","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30682","title":"SemiMat"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30688","domain":"General AI","name":"UFPR-PEs","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30688"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30688","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30688","title":"UFPR-PEs"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30724","domain":"Agents","name":"BAITBENCH","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30724"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30724","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30724","title":"BAITBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30730","domain":"Agents & Tool Use","name":"E-Commerce Bench","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30730"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30730","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30730","title":"E-Commerce Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30768","domain":"General AI","name":"CORAL","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30768"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30768","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30768","title":"CORAL"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30820","domain":"General AI","name":"RealOOB","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30820"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30820","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30820","title":"RealOOB"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30893","domain":"Biology & Drug Discovery","name":"ECGQuest","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30893"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30893","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30893","title":"ECGQuest"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-30903","domain":"General AI","name":"MMDS-Bench","released":"2026-08-31","url":"https://arxiv.org/abs/2608.30903"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.30903","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.30903","title":"MMDS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-31014","domain":"Biology & Drug Discovery","name":"Evidence Package Benchmark","released":"2026-08-31","url":"https://arxiv.org/abs/2608.31014"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.31014","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31014","title":"Evidence Package Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-31016","domain":"Biology & Drug Discovery","name":"OmissionBench","released":"2026-08-31","url":"https://arxiv.org/abs/2608.31016"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.31016","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31016","title":"OmissionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-31022","domain":"General AI","name":"MNIST-PRO","released":"2026-08-31","url":"https://arxiv.org/abs/2608.31022"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.31022","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31022","title":"MNIST-PRO"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-31033","domain":"General AI","name":"Multi4D","released":"2026-08-31","url":"https://arxiv.org/abs/2608.31033"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.31033","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31033","title":"Multi4D"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-31100","domain":"General AI","name":"S3Gym","released":"2026-08-31","url":"https://arxiv.org/abs/2608.31100"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.31100","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31100","title":"S3Gym"},{"benchmarks":[{"benchmark_id":"claire-radar-2608-31111","domain":"General AI","name":"ASPIRE","released":"2026-08-31","url":"https://arxiv.org/abs/2608.31111"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2608.31111","source":"claire_radar","source_url":"https://arxiv.org/abs/2608.31111","title":"ASPIRE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00048","domain":"Agents","name":"GUI-CC","released":"2026-08-30","url":"https://arxiv.org/abs/2609.00048"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00048","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00048","title":"GUI-CC"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00103","domain":"General AI","name":"ECCBench","released":"2026-08-31","url":"https://arxiv.org/abs/2609.00103"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00103","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00103","title":"ECCBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00232","domain":"General AI","name":"VeriOCRBench","released":"2026-08-31","url":"https://arxiv.org/abs/2609.00232"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00232","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00232","title":"VeriOCRBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00384","domain":"Agents","name":"RestoreBench","released":"2026-08-31","url":"https://arxiv.org/abs/2609.00384"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00384","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00384","title":"RestoreBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00474","domain":"Agents","name":"LLAMIA-Bench","released":"2026-08-31","url":"https://arxiv.org/abs/2609.00474"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00474","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00474","title":"LLAMIA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00525","domain":"General AI","name":"GenScale","released":"2026-09-01","url":"https://arxiv.org/abs/2609.00525"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00525","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00525","title":"GenScale"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00641","domain":"General AI","name":"AM-Bench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.00641"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00641","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00641","title":"AM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00646","domain":"General AI","name":"DramaChain Bench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.00646"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00646","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00646","title":"DramaChain Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00787","domain":"Language & Knowledge","name":"StudyBench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.00787"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00787","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00787","title":"StudyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00906","domain":"Robotics & Embodied AI","name":"Peg-in-Bench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.00906"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00906","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00906","title":"Peg-in-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00918","domain":"General AI","name":"RPCBench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.00918"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00918","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00918","title":"RPCBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-00921","domain":"General AI","name":"VIBE-Bench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.00921"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.00921","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.00921","title":"VIBE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01056","domain":"General AI","name":"WorldBench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01056"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01056","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01056","title":"WorldBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01059","domain":"General AI","name":"Dyn-3D","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01059"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01059","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01059","title":"Dyn-3D"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01161","domain":"General AI","name":"CopyShield","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01161"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01161","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01161","title":"CopyShield"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01198","domain":"Finance","name":"FinLifeBench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01198"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01198","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01198","title":"FinLifeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01210","domain":"General AI","name":"C-SafeQA","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01210"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01210","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01210","title":"C-SafeQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01306","domain":"General AI","name":"MeshSplatBench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01306"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01306","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01306","title":"MeshSplatBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01369","domain":"General AI","name":"CAP-Correctness","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01369"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01369","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01369","title":"CAP-Correctness"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01383","domain":"General AI","name":"InSight","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01383"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01383","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01383","title":"InSight"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01437","domain":"Agents","name":"HarnessDev","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01437"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01437","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01437","title":"HarnessDev"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01515","domain":"General AI","name":"TempCloze","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01515"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01515","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01515","title":"TempCloze"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01548","domain":"Cybersecurity","name":"SDARE-Bench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01548"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01548","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01548","title":"SDARE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01552","domain":"General AI","name":"SCILAWS-BENCH","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01552"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01552","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01552","title":"SCILAWS-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01584","domain":"General AI","name":"Unconstrained Vehicle Identification Benchmark","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01584"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01584","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01584","title":"Unconstrained Vehicle Identification Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01600","domain":"Agents","name":"CordisBench","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01600"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01600","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01600","title":"CordisBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01676","domain":"General AI","name":"Sim2Signal","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01676"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01676","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01676","title":"Sim2Signal"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01691","domain":"General AI","name":"FAIRLENS","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01691"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01691","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01691","title":"FAIRLENS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01725","domain":"General AI","name":"LongLSMDC","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01725"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01725","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01725","title":"LongLSMDC"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01772","domain":"General AI","name":"MemeCULT-1K","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01772"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01772","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01772","title":"MemeCULT-1K"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01788","domain":"General AI","name":"VakyArth","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01788"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01788","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01788","title":"VakyArth"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01810","domain":"Factuality","name":"TalkFa","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01810"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01810","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01810","title":"TalkFa"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01899","domain":"General AI","name":"TAPVid-MV","released":"2026-09-01","url":"https://arxiv.org/abs/2609.01899"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01899","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01899","title":"TAPVid-MV"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-01982","domain":"General AI","name":"StatFormBench","released":"2026-09-02","url":"https://arxiv.org/abs/2609.01982"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.01982","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.01982","title":"StatFormBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02012","domain":"Factuality","name":"GeoStore","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02012"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02012","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02012","title":"GeoStore"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02059","domain":"General AI","name":"DocHop","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02059"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02059","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02059","title":"DocHop"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02207","domain":"General AI","name":"LeakageBench","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02207"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02207","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02207","title":"LeakageBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02255","domain":"General AI","name":"T2LSC-Bench","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02255"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02255","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02255","title":"T2LSC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02277","domain":"Factuality","name":"AIB","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02277"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02277","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02277","title":"AIB"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02316","domain":"General AI","name":"Counter-GEO-Bench","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02316"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02316","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02316","title":"Counter-GEO-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02339","domain":"General AI","name":"AGI Maze Prediction Datasets and Benchmark","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02339"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02339","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02339","title":"AGI Maze Prediction Datasets and Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02379","domain":"General AI","name":"MultiGhostBench","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02379"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02379","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02379","title":"MultiGhostBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02459","domain":"General AI","name":"CivBench","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02459"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02459","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02459","title":"CivBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02462","domain":"Autonomous Driving","name":"VIPS","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02462"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02462","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02462","title":"VIPS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02502","domain":"General AI","name":"VMetaphor-Bench","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02502"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02502","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02502","title":"VMetaphor-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02573","domain":"General AI","name":"TIC-Bench","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02573"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02573","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02573","title":"TIC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02683","domain":"General AI","name":"dense500","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02683"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02683","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02683","title":"dense500"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02741","domain":"Cybersecurity","name":"SPADE","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02741"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02741","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02741","title":"SPADE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02864","domain":"General AI","name":"RIG-BENCH","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02864"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02864","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02864","title":"RIG-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-02954","domain":"General AI","name":"LexIssue","released":"2026-09-02","url":"https://arxiv.org/abs/2609.02954"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.02954","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.02954","title":"LexIssue"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03026","domain":"General AI","name":"ObserverBench","released":"2026-09-02","url":"https://arxiv.org/abs/2609.03026"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03026","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03026","title":"ObserverBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03331","domain":"General AI","name":"FPCO-Dialog","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03331"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03331","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03331","title":"FPCO-Dialog"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03370","domain":"General AI","name":"FrameBench","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03370"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03370","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03370","title":"FrameBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03394","domain":"General AI","name":"CHIARO","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03394"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03394","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03394","title":"CHIARO"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03464","domain":"General AI","name":"Mind the Gap: Robustness Risks in PII Detection Systems","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03464"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03464","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03464","title":"Mind the Gap: Robustness Risks in PII Detection Systems"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03467","domain":"General AI","name":"LOCOMO-CONV","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03467"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03467","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03467","title":"LOCOMO-CONV"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03507","domain":"General AI","name":"LongCounsel-8","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03507"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03507","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03507","title":"LongCounsel-8"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03526","domain":"Factuality","name":"CulturalMenuBench","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03526"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03526","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03526","title":"CulturalMenuBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03588","domain":"Agents & Tool Use","name":"KC-Bench","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03588"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03588","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03588","title":"KC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03597","domain":"Multimodal","name":"KhatianDoc","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03597"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03597","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03597","title":"KhatianDoc"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03611","domain":"General AI","name":"FailBench","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03611"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03611","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03611","title":"FailBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03673","domain":"General AI","name":"Statebench","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03673"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03673","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03673","title":"Statebench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03677","domain":"Autonomous Driving","name":"AD-Diff Bench","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03677"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03677","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03677","title":"AD-Diff Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03773","domain":"CAD","name":"RealCADBench","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03773"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03773","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03773","title":"RealCADBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-03804","domain":"General AI","name":"GBA-GCs","released":"2026-09-03","url":"https://arxiv.org/abs/2609.03804"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.03804","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.03804","title":"GBA-GCs"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04075","domain":"Cybersecurity","name":"PatchBench","released":"2026-09-03","url":"https://arxiv.org/abs/2609.04075"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04075","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04075","title":"PatchBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04167","domain":"Code","name":"SWE-Gate","released":"2026-09-03","url":"https://arxiv.org/abs/2609.04167"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04167","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04167","title":"SWE-Gate"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04173","domain":"General AI","name":"Last Translation Benchmark","released":"2026-09-03","url":"https://arxiv.org/abs/2609.04173"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04173","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04173","title":"Last Translation Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04200","domain":"General AI","name":"Principia","released":"2026-09-03","url":"https://arxiv.org/abs/2609.04200"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04200","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04200","title":"Principia"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04598","domain":"Biology & Drug Discovery","name":"PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning","released":"2026-09-04","url":"https://arxiv.org/abs/2609.04598"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04598","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04598","title":"PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04611","domain":"Agents","name":"$\\tau^\\tau$-Bench: An Environment for End-To-End, Realistic Agent Construction","released":"2026-09-04","url":"https://arxiv.org/abs/2609.04611"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04611","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04611","title":"$\\tau^\\tau$-Bench: An Environment for End-To-End, Realistic Agent Construction"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04667","domain":"Finance","name":"ERPBench","released":"2026-09-04","url":"https://arxiv.org/abs/2609.04667"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04667","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04667","title":"ERPBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04706","domain":"Agents","name":"FinalityBench","released":"2026-09-04","url":"https://arxiv.org/abs/2609.04706"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04706","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04706","title":"FinalityBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04720","domain":"General AI","name":"KoNA","released":"2026-09-04","url":"https://arxiv.org/abs/2609.04720"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04720","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04720","title":"KoNA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04842","domain":"Biology & Drug Discovery","name":"MMTClinic","released":"2026-09-04","url":"https://arxiv.org/abs/2609.04842"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04842","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04842","title":"MMTClinic"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04850","domain":"Agents & Tool Use","name":"ElderBench","released":"2026-09-04","url":"https://arxiv.org/abs/2609.04850"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04850","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04850","title":"ElderBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04859","domain":"General AI","name":"MM-IFEval-Pro","released":"2026-09-04","url":"https://arxiv.org/abs/2609.04859"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04859","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04859","title":"MM-IFEval-Pro"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-04867","domain":"General AI","name":"PRISM-Bench","released":"2026-09-04","url":"https://arxiv.org/abs/2609.04867"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.04867","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.04867","title":"PRISM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05043","domain":"General AI","name":"European-IFEval","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05043"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05043","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05043","title":"European-IFEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05079","domain":"AI Scientist","name":"TruthInsightBench","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05079"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05079","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05079","title":"TruthInsightBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05117","domain":"General AI","name":"TIER","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05117"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05117","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05117","title":"TIER"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05141","domain":"General AI","name":"SciDocBench","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05141"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05141","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05141","title":"SciDocBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05171","domain":"Agents","name":"WeBench-MMGenEdit","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05171"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05171","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05171","title":"WeBench-MMGenEdit"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05178","domain":"Robot manipulation","name":"LIBERO-Recover Benchmark","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05178"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05178","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05178","title":"LIBERO-Recover Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05260","domain":"Reasoning","name":"One Word, Different Action","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05260"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05260","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05260","title":"One Word, Different Action"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05324","domain":"Reasoning","name":"RoboSPA","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05324"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05324","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05324","title":"RoboSPA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05395","domain":"General AI","name":"KOPA-Bench","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05395"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05395","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05395","title":"KOPA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05401","domain":"Robotics","name":"ROBORMBENCH","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05401"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05401","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05401","title":"ROBORMBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-05405","domain":"General AI","name":"WearableQA","released":"2026-09-04","url":"https://arxiv.org/abs/2609.05405"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.05405","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.05405","title":"WearableQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06353","domain":"General AI","name":"ChildGaze","released":"2026-09-06","url":"https://arxiv.org/abs/2609.06353"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06353","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06353","title":"ChildGaze"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06368","domain":"General AI","name":"LANTERN","released":"2026-09-06","url":"https://arxiv.org/abs/2609.06368"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06368","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06368","title":"LANTERN"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06527","domain":"General AI","name":"ProcArena","released":"2026-09-06","url":"https://arxiv.org/abs/2609.06527"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06527","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06527","title":"ProcArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06721","domain":"General AI","name":"BuddyVQA","released":"2026-09-06","url":"https://arxiv.org/abs/2609.06721"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06721","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06721","title":"BuddyVQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06771","domain":"General AI","name":"AuthBench","released":"2026-09-06","url":"https://arxiv.org/abs/2609.06771"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06771","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06771","title":"AuthBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06774","domain":"General AI","name":"NEO-Bench","released":"2026-09-06","url":"https://arxiv.org/abs/2609.06774"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06774","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06774","title":"NEO-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06809","domain":"General AI","name":"ZDPShift","released":"2026-09-06","url":"https://arxiv.org/abs/2609.06809"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06809","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06809","title":"ZDPShift"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06966","domain":"General AI","name":"MOLE","released":"2026-09-07","url":"https://arxiv.org/abs/2609.06966"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06966","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06966","title":"MOLE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06972","domain":"Agents","name":"AgentDrift","released":"2026-09-07","url":"https://arxiv.org/abs/2609.06972"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06972","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06972","title":"AgentDrift"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06976","domain":"Biology & Drug Discovery","name":"HealthLoopQA","released":"2026-09-07","url":"https://arxiv.org/abs/2609.06976"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06976","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06976","title":"HealthLoopQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06991","domain":"General AI","name":"AV-SafetyBench","released":"2026-09-07","url":"https://arxiv.org/abs/2609.06991"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06991","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06991","title":"AV-SafetyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-06993","domain":"Code & Software","name":"LatentMD","released":"2026-09-07","url":"https://arxiv.org/abs/2609.06993"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.06993","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.06993","title":"LatentMD"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07047","domain":"Robot manipulation","name":"MEMOBench","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07047"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07047","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07047","title":"MEMOBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07135","domain":"Robotics & Embodied AI","name":"NutriBench-Kitchen","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07135"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07135","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07135","title":"NutriBench-Kitchen"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07151","domain":"Finance","name":"Vishing-Tactics-Bench","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07151"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07151","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07151","title":"Vishing-Tactics-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07441","domain":"Biology & Drug Discovery","name":"TabBench-Bio","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07441"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07441","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07441","title":"TabBench-Bio"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07547","domain":"Autonomous Driving","name":"OmniSmall","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07547"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07547","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07547","title":"OmniSmall"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07601","domain":"Biology & Drug Discovery","name":"ObGynLongBench","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07601"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07601","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07601","title":"ObGynLongBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07603","domain":"Agents & Tool Use","name":"FinCUABuildBench","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07603"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07603","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07603","title":"FinCUABuildBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07712","domain":"Agents & Tool Use","name":"AppSim-Bench","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07712"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07712","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07712","title":"AppSim-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07784","domain":"General AI","name":"xDailyBench","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07784"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07784","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07784","title":"xDailyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-07823","domain":"General AI","name":"SAFIRE","released":"2026-09-07","url":"https://arxiv.org/abs/2609.07823"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.07823","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.07823","title":"SAFIRE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08040","domain":"Language & Knowledge","name":"VEX-Bench","released":"2026-09-07","url":"https://arxiv.org/abs/2609.08040"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08040","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08040","title":"VEX-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08079","domain":"General AI","name":"VI-Bench","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08079"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08079","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08079","title":"VI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08090","domain":"Biology & Drug Discovery","name":"RevalExo","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08090"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08090","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08090","title":"RevalExo"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08117","domain":"General AI","name":"DriveMotion","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08117"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08117","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08117","title":"DriveMotion"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08126","domain":"General AI","name":"SchemeArena","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08126"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08126","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08126","title":"SchemeArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08149","domain":"Code","name":"SWE-Bench Pro Verified","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08149"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08149","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08149","title":"SWE-Bench Pro Verified"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08155","domain":"General AI","name":"SciFigure2Code","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08155"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08155","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08155","title":"SciFigure2Code"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08174","domain":"Biology & Drug Discovery","name":"OntologyBench","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08174"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08174","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08174","title":"OntologyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08209","domain":"Robotics","name":"RoboReel","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08209"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08209","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08209","title":"RoboReel"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08275","domain":"General AI","name":"CutCraft","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08275"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08275","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08275","title":"CutCraft"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08292","domain":"Robotics & Embodied AI","name":"EvoNav-Bench","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08292"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08292","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08292","title":"EvoNav-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08300","domain":"General AI","name":"SPACE","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08300"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08300","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08300","title":"SPACE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08402","domain":"Multimodal","name":"AGOS-Bench","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08402"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08402","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08402","title":"AGOS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08657","domain":"General AI","name":"LayerWiseBench","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08657"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08657","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08657","title":"LayerWiseBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08887","domain":"General AI","name":"Q2D-Web","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08887"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08887","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08887","title":"Q2D-Web"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-08947","domain":"General AI","name":"ONECYL","released":"2026-09-08","url":"https://arxiv.org/abs/2609.08947"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.08947","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.08947","title":"ONECYL"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09090","domain":"General AI","name":"SPINE","released":"2026-09-08","url":"https://arxiv.org/abs/2609.09090"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09090","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09090","title":"SPINE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09113","domain":"Agents & Tool Use","name":"SAEScientist-Bench","released":"2026-09-08","url":"https://arxiv.org/abs/2609.09113"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09113","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09113","title":"SAEScientist-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09264","domain":"Language & Knowledge","name":"StochBench","released":"2026-09-08","url":"https://arxiv.org/abs/2609.09264"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09264","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09264","title":"StochBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09349","domain":"Factuality","name":"SWORD","released":"2026-09-08","url":"https://arxiv.org/abs/2609.09349"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09349","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09349","title":"SWORD"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09396","domain":"Logistics & Operations","name":"VANTAGE-BENCH","released":"2026-09-08","url":"https://arxiv.org/abs/2609.09396"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09396","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09396","title":"VANTAGE-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09404","domain":"General AI","name":"MMPIBench","released":"2026-09-08","url":"https://arxiv.org/abs/2609.09404"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09404","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09404","title":"MMPIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09410","domain":"Factuality","name":"HybridDeepResearch","released":"2026-09-08","url":"https://arxiv.org/abs/2609.09410"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09410","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09410","title":"HybridDeepResearch"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09528","domain":"General AI","name":"MotionBlind","released":"2026-09-08","url":"https://arxiv.org/abs/2609.09528"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09528","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09528","title":"MotionBlind"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09657","domain":"General AI","name":"RESCUE","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09657"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09657","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09657","title":"RESCUE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09664","domain":"General AI","name":"PRAGMA","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09664"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09664","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09664","title":"PRAGMA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09672","domain":"Factuality","name":"SEA-SpeechBench","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09672"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09672","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09672","title":"SEA-SpeechBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09754","domain":"General AI","name":"LexAgentHallu","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09754"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09754","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09754","title":"LexAgentHallu"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09849","domain":"General AI","name":"NetArtifactBench","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09849"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09849","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09849","title":"NetArtifactBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09887","domain":"General AI","name":"JuryBench","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09887"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09887","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09887","title":"JuryBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09889","domain":"General AI","name":"DasanCallDial","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09889"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09889","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09889","title":"DasanCallDial"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09895","domain":"Software & AI Compute","name":"VidHalLoc","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09895"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09895","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09895","title":"VidHalLoc"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09964","domain":"General AI","name":"5-Dialects-BN","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09964"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09964","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09964","title":"5-Dialects-BN"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-09973","domain":"General AI","name":"CapQuiz","released":"2026-09-09","url":"https://arxiv.org/abs/2609.09973"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.09973","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.09973","title":"CapQuiz"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10016","domain":"General AI","name":"MetroLLM-Bench","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10016"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10016","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10016","title":"MetroLLM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10055","domain":"Biology & Drug Discovery","name":"PhenoNormBench","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10055"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10055","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10055","title":"PhenoNormBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10092","domain":"General AI","name":"Research Attention Prediction (RAP)","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10092"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10092","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10092","title":"Research Attention Prediction (RAP)"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10139","domain":"General AI","name":"Urban Driving-Face Scene Gaze (UD-FSG)","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10139"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10139","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10139","title":"Urban Driving-Face Scene Gaze (UD-FSG)"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10153","domain":"General AI","name":"YallaMorph","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10153"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10153","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10153","title":"YallaMorph"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10226","domain":"Language & Knowledge","name":"Φ-Bench","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10226"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10226","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10226","title":"Φ-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10243","domain":"Robot manipulation","name":"FolDeX","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10243"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10243","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10243","title":"FolDeX"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10356","domain":"General AI","name":"SPOT-THE-SHIFT","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10356"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10356","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10356","title":"SPOT-THE-SHIFT"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10394","domain":"General AI","name":"Candor-LR","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10394"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10394","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10394","title":"Candor-LR"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10451","domain":"Agents & Tool Use","name":"JarvisGUI","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10451"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10451","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10451","title":"JarvisGUI"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10539","domain":"General AI","name":"IdeaAMBIG","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10539"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10539","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10539","title":"IdeaAMBIG"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10722","domain":"General AI","name":"CMNIE","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10722"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10722","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10722","title":"CMNIE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-10895","domain":"Multimodal","name":"ReactHuman","released":"2026-09-09","url":"https://arxiv.org/abs/2609.10895"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.10895","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.10895","title":"ReactHuman"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11141","domain":"Agents","name":"DNBENCH","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11141"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11141","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11141","title":"DNBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11172","domain":"Geometric reasoning","name":"EgoGenEval","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11172"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11172","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11172","title":"EgoGenEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11180","domain":"General AI","name":"SemVerBench","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11180"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11180","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11180","title":"SemVerBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11185","domain":"Biology & Drug Discovery","name":"LogiMed-RoB","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11185"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11185","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11185","title":"LogiMed-RoB"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11234","domain":"General AI","name":"NovGauge","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11234"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11234","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11234","title":"NovGauge"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11242","domain":"General AI","name":"VWG-Bench","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11242"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11242","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11242","title":"VWG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11243","domain":"Factuality","name":"Sci-MMR","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11243"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11243","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11243","title":"Sci-MMR"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11318","domain":"General AI","name":"Mr.LHDR","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11318"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11318","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11318","title":"Mr.LHDR"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11399","domain":"General AI","name":"TransClean","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11399"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11399","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11399","title":"TransClean"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11504","domain":"General AI","name":"DeFiFlowBench","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11504"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11504","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11504","title":"DeFiFlowBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11515","domain":"Code & Software","name":"ChurnBench","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11515"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11515","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11515","title":"ChurnBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11601","domain":"General AI","name":"MMGait","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11601"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11601","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11601","title":"MMGait"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11758","domain":"General AI","name":"RAG-Safety-Bench","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11758"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11758","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11758","title":"RAG-Safety-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11877","domain":"General AI","name":"AssayBench-Loop","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11877"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11877","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11877","title":"AssayBench-Loop"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-11900","domain":"General AI","name":"MindTopo","released":"2026-09-10","url":"https://arxiv.org/abs/2609.11900"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.11900","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.11900","title":"MindTopo"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12345","domain":"General AI","name":"ParaRecover","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12345"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12345","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12345","title":"ParaRecover"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12350","domain":"General AI","name":"EgoMaize","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12350"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12350","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12350","title":"EgoMaize"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12404","domain":"General AI","name":"VRL-Bench","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12404"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12404","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12404","title":"VRL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12408","domain":"Logistics & Operations","name":"GSO-Net","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12408"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12408","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12408","title":"GSO-Net"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12412","domain":"General AI","name":"HoliBench","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12412"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12412","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12412","title":"HoliBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12439","domain":"Agents","name":"TraceJudgeBench","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12439"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12439","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12439","title":"TraceJudgeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12446","domain":"General AI","name":"Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12446"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12446","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12446","title":"Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12475","domain":"General AI","name":"ZipBench","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12475"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12475","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12475","title":"ZipBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12509","domain":"Geometric reasoning","name":"ChitraMiti-12.8k","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12509"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12509","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12509","title":"ChitraMiti-12.8k"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12552","domain":"General AI","name":"OV-SGG-Bench","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12552"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12552","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12552","title":"OV-SGG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12606","domain":"Geometric reasoning","name":"GeoVAD-Bench","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12606"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12606","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12606","title":"GeoVAD-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12623","domain":"General AI","name":"SteerBench","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12623"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12623","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12623","title":"SteerBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12658","domain":"General AI","name":"ProactiveBench","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12658"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12658","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12658","title":"ProactiveBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-12808","domain":"General AI","name":"K-Bench","released":"2026-09-11","url":"https://arxiv.org/abs/2609.12808"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.12808","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.12808","title":"K-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-13076","domain":"General AI","name":"Multiparty Bench (MP-Bench)","released":"2026-09-11","url":"https://arxiv.org/abs/2609.13076"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.13076","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.13076","title":"Multiparty Bench (MP-Bench)"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-13082","domain":"Robotics & Embodied AI","name":"Embodied-BenchForge","released":"2026-09-11","url":"https://arxiv.org/abs/2609.13082"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.13082","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.13082","title":"Embodied-BenchForge"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-14302","domain":"Finance","name":"E2A-Bench","released":"2026-09-13","url":"https://arxiv.org/abs/2609.14302"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.14302","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14302","title":"E2A-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-14495","domain":"General AI","name":"MCIQA-2K","released":"2026-09-13","url":"https://arxiv.org/abs/2609.14495"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.14495","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14495","title":"MCIQA-2K"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-14739","domain":"General AI","name":"LegalRewardBench","released":"2026-09-13","url":"https://arxiv.org/abs/2609.14739"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.14739","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14739","title":"LegalRewardBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-14762","domain":"General AI","name":"TriCalRAG","released":"2026-09-13","url":"https://arxiv.org/abs/2609.14762"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.14762","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14762","title":"TriCalRAG"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-14811","domain":"General AI","name":"Crypto Accounting Bench","released":"2026-09-13","url":"https://arxiv.org/abs/2609.14811"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.14811","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14811","title":"Crypto Accounting Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-14857","domain":"General AI","name":"ModularRSI","released":"2026-09-14","url":"https://arxiv.org/abs/2609.14857"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.14857","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14857","title":"ModularRSI"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-14976","domain":"General AI","name":"MemRiskBench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.14976"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.14976","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14976","title":"MemRiskBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-14992","domain":"Language & Knowledge","name":"MTAC-IFBench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.14992"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.14992","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.14992","title":"MTAC-IFBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15017","domain":"General AI","name":"PIDS-Bench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15017"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15017","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15017","title":"PIDS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15028","domain":"General AI","name":"TwinICL","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15028"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15028","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15028","title":"TwinICL"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15087","domain":"General AI","name":"MUSE-Bench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15087"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15087","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15087","title":"MUSE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15128","domain":"General AI","name":"OST-DiagBench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15128"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15128","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15128","title":"OST-DiagBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15455","domain":"Robotics","name":"InterSocialBench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15455"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15455","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15455","title":"InterSocialBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15562","domain":"General AI","name":"PhysForensics-Bench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15562"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15562","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15562","title":"PhysForensics-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15625","domain":"Language & Knowledge","name":"FedLTLib","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15625"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15625","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15625","title":"FedLTLib"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15683","domain":"Agents & Tool Use","name":"V-ICAL Bench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15683"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15683","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15683","title":"V-ICAL Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15726","domain":"Robot manipulation","name":"Bench2Dex","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15726"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15726","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15726","title":"Bench2Dex"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15780","domain":"General AI","name":"MoveBench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15780"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15780","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15780","title":"MoveBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15855","domain":"Biology & Drug Discovery","name":"K-Bench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15855"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15855","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15855","title":"K-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15939","domain":"Cybersecurity","name":"Vulnerability Localization Benchmark","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15939"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15939","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15939","title":"Vulnerability Localization Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-15940","domain":"General AI","name":"LIBERO-CTRL","released":"2026-09-14","url":"https://arxiv.org/abs/2609.15940"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.15940","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.15940","title":"LIBERO-CTRL"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16075","domain":"Geometric reasoning","name":"AssemblyGrid v1","released":"2026-09-13","url":"https://arxiv.org/abs/2609.16075"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16075","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16075","title":"AssemblyGrid v1"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16090","domain":"General AI","name":"MUUNRiver-Bench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.16090"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16090","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16090","title":"MUUNRiver-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16233","domain":"General AI","name":"SceneBench","released":"2026-09-14","url":"https://arxiv.org/abs/2609.16233"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16233","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16233","title":"SceneBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16251","domain":"CAD","name":"CADWorld","released":"2026-09-14","url":"https://arxiv.org/abs/2609.16251"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16251","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16251","title":"CADWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16305","domain":"General AI","name":"BLINDSPOT","released":"2026-09-14","url":"https://arxiv.org/abs/2609.16305"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16305","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16305","title":"BLINDSPOT"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16310","domain":"General AI","name":"Monaco4D","released":"2026-09-14","url":"https://arxiv.org/abs/2609.16310"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16310","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16310","title":"Monaco4D"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16393","domain":"General AI","name":"ParsHate","released":"2026-09-14","url":"https://arxiv.org/abs/2609.16393"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16393","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16393","title":"ParsHate"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16486","domain":"General AI","name":"VPRef","released":"2026-09-15","url":"https://arxiv.org/abs/2609.16486"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16486","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16486","title":"VPRef"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16614","domain":"General AI","name":"RoleBreak","released":"2026-09-15","url":"https://arxiv.org/abs/2609.16614"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16614","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16614","title":"RoleBreak"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16680","domain":"Agents","name":"IPC-Bench","released":"2026-09-15","url":"https://arxiv.org/abs/2609.16680"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16680","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16680","title":"IPC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16816","domain":"Cybersecurity","name":"ImpossibleRubrics","released":"2026-09-15","url":"https://arxiv.org/abs/2609.16816"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16816","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16816","title":"ImpossibleRubrics"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16847","domain":"General AI","name":"REGMB","released":"2026-09-15","url":"https://arxiv.org/abs/2609.16847"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16847","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16847","title":"REGMB"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16900","domain":"General AI","name":"RiskChainBench","released":"2026-09-15","url":"https://arxiv.org/abs/2609.16900"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16900","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16900","title":"RiskChainBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16919","domain":"Autonomous Driving","name":"NeuroSymbEAD","released":"2026-09-15","url":"https://arxiv.org/abs/2609.16919"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16919","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16919","title":"NeuroSymbEAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-16964","domain":"General AI","name":"HUMAID-NER","released":"2026-09-15","url":"https://arxiv.org/abs/2609.16964"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.16964","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.16964","title":"HUMAID-NER"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17106","domain":"Robotics & Embodied AI","name":"BRAVE-6D","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17106"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17106","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17106","title":"BRAVE-6D"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17124","domain":"General AI","name":"LOTUSim-Energy","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17124"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17124","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17124","title":"LOTUSim-Energy"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17128","domain":"General AI","name":"FirmCORe","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17128"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17128","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17128","title":"FirmCORe"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17248","domain":"General AI","name":"Video-HolmesV2","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17248"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17248","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17248","title":"Video-HolmesV2"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17317","domain":"General AI","name":"ASURRE","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17317"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17317","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17317","title":"ASURRE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17439","domain":"General AI","name":"QIQCBench","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17439"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17439","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17439","title":"QIQCBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17458","domain":"General AI","name":"TORQUE","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17458"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17458","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17458","title":"TORQUE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17688","domain":"General AI","name":"CapMem","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17688"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17688","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17688","title":"CapMem"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17745","domain":"Robot manipulation","name":"REVERSAL-BENCH","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17745"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17745","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17745","title":"REVERSAL-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17843","domain":"General AI","name":"RoboVAD","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17843"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17843","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17843","title":"RoboVAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17855","domain":"Biology & Drug Discovery","name":"SNOMED CT Concept Recommendation from Masked Clinical Context","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17855"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17855","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17855","title":"SNOMED CT Concept Recommendation from Masked Clinical Context"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17885","domain":"Finance","name":"ERPBench","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17885"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17885","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17885","title":"ERPBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17910","domain":"General AI","name":"Map2Route","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17910"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17910","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17910","title":"Map2Route"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17930","domain":"Language & Knowledge","name":"Traverse","released":"2026-09-15","url":"https://arxiv.org/abs/2609.17930"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17930","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17930","title":"Traverse"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-17985","domain":"Agents & Tool Use","name":"RideWay","released":"2026-09-16","url":"https://arxiv.org/abs/2609.17985"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.17985","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.17985","title":"RideWay"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18022","domain":"Chip Design & EDA","name":"VeriBugBench","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18022"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18022","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18022","title":"VeriBugBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18104","domain":"General AI","name":"iMINDBench","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18104"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18104","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18104","title":"iMINDBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18123","domain":"Agents","name":"AutoTuneBench","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18123"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18123","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18123","title":"AutoTuneBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18270","domain":"Finance","name":"BENCHCOMPASS","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18270"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18270","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18270","title":"BENCHCOMPASS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18384","domain":"General AI","name":"GYROval","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18384"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18384","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18384","title":"GYROval"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18493","domain":"Factuality","name":"Semantic-ITC","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18493"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18493","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18493","title":"Semantic-ITC"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18549","domain":"General AI","name":"DynoFluxBench","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18549"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18549","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18549","title":"DynoFluxBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18605","domain":"General AI","name":"PACT","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18605"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18605","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18605","title":"PACT"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18680","domain":"General AI","name":"HearInContext","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18680"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18680","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18680","title":"HearInContext"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18805","domain":"Agents & Tool Use","name":"ProgramDistill","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18805"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18805","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18805","title":"ProgramDistill"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18812","domain":"Software & AI Compute","name":"ExecTS-QA","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18812"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18812","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18812","title":"ExecTS-QA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-18908","domain":"Factuality","name":"ECtHR-NPD","released":"2026-09-16","url":"https://arxiv.org/abs/2609.18908"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.18908","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.18908","title":"ECtHR-NPD"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19088","domain":"General AI","name":"MUSE","released":"2026-09-16","url":"https://arxiv.org/abs/2609.19088"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19088","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19088","title":"MUSE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19180","domain":"Language & Knowledge","name":"BioPhys-Bridge","released":"2026-09-15","url":"https://arxiv.org/abs/2609.19180"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19180","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19180","title":"BioPhys-Bridge"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19533","domain":"Robotics","name":"SLAMSqueezeBench","released":"2026-09-17","url":"https://arxiv.org/abs/2609.19533"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19533","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19533","title":"SLAMSqueezeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19554","domain":"Robotics & Embodied AI","name":"VABench","released":"2026-09-17","url":"https://arxiv.org/abs/2609.19554"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19554","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19554","title":"VABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19844","domain":"Chip Design & EDA","name":"SecTB-RTL","released":"2026-09-17","url":"https://arxiv.org/abs/2609.19844"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19844","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19844","title":"SecTB-RTL"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19867","domain":"General AI","name":"CrossUAV","released":"2026-09-17","url":"https://arxiv.org/abs/2609.19867"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19867","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19867","title":"CrossUAV"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19876","domain":"CAD","name":"SlugTrails","released":"2026-09-17","url":"https://arxiv.org/abs/2609.19876"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19876","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19876","title":"SlugTrails"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19879","domain":"General AI","name":"V\\={a}kQA","released":"2026-09-17","url":"https://arxiv.org/abs/2609.19879"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19879","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19879","title":"V\\={a}kQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19883","domain":"General AI","name":"PetriBench","released":"2026-09-17","url":"https://arxiv.org/abs/2609.19883"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19883","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19883","title":"PetriBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19892","domain":"General AI","name":"ClashBench","released":"2026-09-17","url":"https://arxiv.org/abs/2609.19892"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19892","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19892","title":"ClashBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-19916","domain":"General AI","name":"KoNeoBench","released":"2026-09-17","url":"https://arxiv.org/abs/2609.19916"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.19916","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.19916","title":"KoNeoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-20152","domain":"Agents","name":"MTVA-Bench","released":"2026-09-17","url":"https://arxiv.org/abs/2609.20152"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.20152","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20152","title":"MTVA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-20574","domain":"General AI","name":"DocAttriBench","released":"2026-09-17","url":"https://arxiv.org/abs/2609.20574"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.20574","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20574","title":"DocAttriBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-20584","domain":"General AI","name":"SAFARI","released":"2026-09-17","url":"https://arxiv.org/abs/2609.20584"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.20584","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20584","title":"SAFARI"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-20794","domain":"General AI","name":"PosteriorBench","released":"2026-09-17","url":"https://arxiv.org/abs/2609.20794"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.20794","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20794","title":"PosteriorBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-20812","domain":"Code & Software","name":"OverclaimBench","released":"2026-09-17","url":"https://arxiv.org/abs/2609.20812"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.20812","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.20812","title":"OverclaimBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21223","domain":"Multimodal","name":"SafeStage","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21223"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21223","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21223","title":"SafeStage"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21263","domain":"Chip Design & EDA","name":"PlaceReasoner-Bench","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21263"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21263","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21263","title":"PlaceReasoner-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21293","domain":"General AI","name":"GameASG-Bench","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21293"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21293","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21293","title":"GameASG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21344","domain":"General AI","name":"CESBench","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21344"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21344","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21344","title":"CESBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21386","domain":"General AI","name":"AgentVidBench","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21386"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21386","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21386","title":"AgentVidBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21521","domain":"General AI","name":"VidOmni-Bench","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21521"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21521","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21521","title":"VidOmni-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21562","domain":"Code & Software","name":"GameLogicBench","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21562"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21562","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21562","title":"GameLogicBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21637","domain":"General AI","name":"Chinese Competitive Debating Dataset and Benchmark","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21637"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21637","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21637","title":"Chinese Competitive Debating Dataset and Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21722","domain":"General AI","name":"CIBuzzBench","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21722"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21722","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21722","title":"CIBuzzBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-21822","domain":"General AI","name":"Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty","released":"2026-09-18","url":"https://arxiv.org/abs/2609.21822"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.21822","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.21822","title":"Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-22038","domain":"Factuality","name":"QuranicMMLU","released":"2026-09-18","url":"https://arxiv.org/abs/2609.22038"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.22038","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22038","title":"QuranicMMLU"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-22064","domain":"General AI","name":"BrainWideBench","released":"2026-09-18","url":"https://arxiv.org/abs/2609.22064"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.22064","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22064","title":"BrainWideBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-22069","domain":"General AI","name":"OmniVBench","released":"2026-09-18","url":"https://arxiv.org/abs/2609.22069"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.22069","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22069","title":"OmniVBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-22803","domain":"Robotics & Embodied AI","name":"ARCGym","released":"2026-09-19","url":"https://arxiv.org/abs/2609.22803"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.22803","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22803","title":"ARCGym"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-22878","domain":"Code & Software","name":"ISA-Bench","released":"2026-09-19","url":"https://arxiv.org/abs/2609.22878"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.22878","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.22878","title":"ISA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23003","domain":"General AI","name":"M3GA-Wild","released":"2026-09-19","url":"https://arxiv.org/abs/2609.23003"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23003","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23003","title":"M3GA-Wild"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23055","domain":"Factuality","name":"Optimizers for Diffusion Models: A Controlled Benchmark","released":"2026-09-19","url":"https://arxiv.org/abs/2609.23055"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23055","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23055","title":"Optimizers for Diffusion Models: A Controlled Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23064","domain":"General AI","name":"FireWorldBench","released":"2026-09-19","url":"https://arxiv.org/abs/2609.23064"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23064","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23064","title":"FireWorldBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23067","domain":"General AI","name":"LD-RSVIS","released":"2026-09-19","url":"https://arxiv.org/abs/2609.23067"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23067","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23067","title":"LD-RSVIS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23231","domain":"General AI","name":"ChemCLIR-Bench","released":"2026-09-19","url":"https://arxiv.org/abs/2609.23231"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23231","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23231","title":"ChemCLIR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23336","domain":"General AI","name":"MinCU","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23336"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23336","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23336","title":"MinCU"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23345","domain":"General AI","name":"AniPrO-SFD-Bench","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23345"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23345","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23345","title":"AniPrO-SFD-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23363","domain":"Chip Design & EDA","name":"TicTacBench","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23363"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23363","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23363","title":"TicTacBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23407","domain":"Multimodal","name":"OmniEchoBench","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23407"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23407","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23407","title":"OmniEchoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23416","domain":"General AI","name":"MuLA-Bench","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23416"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23416","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23416","title":"MuLA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23417","domain":"Code & Software","name":"Omni2Web","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23417"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23417","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23417","title":"Omni2Web"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23435","domain":"Biology & Drug Discovery","name":"OmicsBench","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23435"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23435","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23435","title":"OmicsBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23490","domain":"Agents","name":"BabelArena","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23490"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23490","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23490","title":"BabelArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23570","domain":"Code & Software","name":"VibeMemBench","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23570"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23570","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23570","title":"VibeMemBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23659","domain":"Finance","name":"ETH-TraceBench","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23659"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23659","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23659","title":"ETH-TraceBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23679","domain":"General AI","name":"mini-CommonForms","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23679"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23679","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23679","title":"mini-CommonForms"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23695","domain":"Language & Knowledge","name":"PhysAI-Bench","released":"2026-09-20","url":"https://arxiv.org/abs/2609.23695"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23695","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23695","title":"PhysAI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-23980","domain":"Agents","name":"MobileCybench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.23980"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.23980","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.23980","title":"MobileCybench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24002","domain":"Agents","name":"FinInteract","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24002"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24002","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24002","title":"FinInteract"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24094","domain":"General AI","name":"WidgetVABench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24094"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24094","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24094","title":"WidgetVABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24124","domain":"Robot manipulation","name":"ActiveArena-Bench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24124"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24124","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24124","title":"ActiveArena-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24161","domain":"General AI","name":"MCP-GRANITE Benchmark","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24161"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24161","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24161","title":"MCP-GRANITE Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24165","domain":"Agents","name":"APEXA-Bench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24165"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24165","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24165","title":"APEXA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24172","domain":"General AI","name":"LegendBench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24172"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24172","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24172","title":"LegendBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24210","domain":"Code","name":"ChartJudgeBench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24210"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24210","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24210","title":"ChartJudgeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24228","domain":"General AI","name":"IMPLICIT-Bench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24228"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24228","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24228","title":"IMPLICIT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24259","domain":"General AI","name":"MemCalib","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24259"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24259","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24259","title":"MemCalib"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24308","domain":"Robotics & Embodied AI","name":"HappyWorld-Bench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24308"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24308","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24308","title":"HappyWorld-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24350","domain":"Robotics & Embodied AI","name":"LIBERO-VPro","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24350"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24350","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24350","title":"LIBERO-VPro"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24444","domain":"Energy & Grid","name":"WPBench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24444"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24444","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24444","title":"WPBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24452","domain":"General AI","name":"LiDAR-Hallu","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24452"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24452","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24452","title":"LiDAR-Hallu"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24555","domain":"General AI","name":"The Endless Exam","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24555"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24555","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24555","title":"The Endless Exam"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24613","domain":"General AI","name":"UK-PRBENCH","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24613"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24613","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24613","title":"UK-PRBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24662","domain":"Agents","name":"DUMA-Bench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24662"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24662","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24662","title":"DUMA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24663","domain":"General AI","name":"EvoPathBench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24663"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24663","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24663","title":"EvoPathBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24778","domain":"Robotics","name":"H2RBench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24778"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24778","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24778","title":"H2RBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24812","domain":"General AI","name":"MSI-Bench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24812"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24812","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24812","title":"MSI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24921","domain":"General AI","name":"BackTrend","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24921"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24921","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24921","title":"BackTrend"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24971","domain":"Agents","name":"DolphinBench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24971"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24971","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24971","title":"DolphinBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-24983","domain":"General AI","name":"Panda-CVL","released":"2026-09-21","url":"https://arxiv.org/abs/2609.24983"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.24983","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.24983","title":"Panda-CVL"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25001","domain":"General AI","name":"GameHorizon-Bench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.25001"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25001","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25001","title":"GameHorizon-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25192","domain":"Finance","name":"FinFIRST","released":"2026-09-21","url":"https://arxiv.org/abs/2609.25192"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25192","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25192","title":"FinFIRST"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25298","domain":"General AI","name":"FineWeb-CLaR","released":"2026-09-21","url":"https://arxiv.org/abs/2609.25298"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25298","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25298","title":"FineWeb-CLaR"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25376","domain":"Multimodal","name":"VLAQuantBench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.25376"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25376","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25376","title":"VLAQuantBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25454","domain":"General AI","name":"CoordBench","released":"2026-09-21","url":"https://arxiv.org/abs/2609.25454"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25454","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25454","title":"CoordBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25467","domain":"Finance","name":"ShowTellArena","released":"2026-09-21","url":"https://arxiv.org/abs/2609.25467"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25467","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25467","title":"ShowTellArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25562","domain":"Robotics","name":"IndustrialVLA-Bench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.25562"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25562","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25562","title":"IndustrialVLA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25569","domain":"General AI","name":"SambaGraph","released":"2026-09-22","url":"https://arxiv.org/abs/2609.25569"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25569","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25569","title":"SambaGraph"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25591","domain":"Cybersecurity","name":"KEX-bench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.25591"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25591","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25591","title":"KEX-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25636","domain":"Robotics & Embodied AI","name":"RoboFollow","released":"2026-09-22","url":"https://arxiv.org/abs/2609.25636"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25636","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25636","title":"RoboFollow"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25689","domain":"Robot manipulation","name":"MotionForge","released":"2026-09-22","url":"https://arxiv.org/abs/2609.25689"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25689","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25689","title":"MotionForge"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25804","domain":"Agents","name":"Taste-Bench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.25804"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25804","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25804","title":"Taste-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25841","domain":"Reasoning","name":"Metric-Bench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.25841"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25841","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25841","title":"Metric-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-25864","domain":"General AI","name":"AV-Remove-Bench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.25864"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.25864","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.25864","title":"AV-Remove-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-26029","domain":"General AI","name":"CQ4OE","released":"2026-09-22","url":"https://arxiv.org/abs/2609.26029"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.26029","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26029","title":"CQ4OE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-26175","domain":"Factuality","name":"EADC","released":"2026-09-22","url":"https://arxiv.org/abs/2609.26175"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.26175","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26175","title":"EADC"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-26292","domain":"Robot manipulation","name":"RoboTwin-Phys","released":"2026-09-22","url":"https://arxiv.org/abs/2609.26292"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.26292","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26292","title":"RoboTwin-Phys"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-26314","domain":"Robotics & Embodied AI","name":"TriWorldBench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.26314"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.26314","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26314","title":"TriWorldBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-26375","domain":"Advertising & Pricing","name":"Ecom-Bench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.26375"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.26375","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26375","title":"Ecom-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-26777","domain":"Language & Knowledge","name":"SWE-Serve","released":"2026-09-22","url":"https://arxiv.org/abs/2609.26777"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.26777","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26777","title":"SWE-Serve"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-26872","domain":"Robotics & Embodied AI","name":"MSK-Bench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.26872"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.26872","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26872","title":"MSK-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-26907","domain":"General AI","name":"MemeCF","released":"2026-09-22","url":"https://arxiv.org/abs/2609.26907"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.26907","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26907","title":"MemeCF"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-26952","domain":"General AI","name":"HG2.9K","released":"2026-09-22","url":"https://arxiv.org/abs/2609.26952"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.26952","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.26952","title":"HG2.9K"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27036","domain":"General AI","name":"PolyBench26","released":"2026-09-22","url":"https://arxiv.org/abs/2609.27036"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27036","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27036","title":"PolyBench26"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27076","domain":"Robotics & Embodied AI","name":"Pro-Bench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.27076"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27076","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27076","title":"Pro-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27086","domain":"General AI","name":"NADI 2026","released":"2026-09-22","url":"https://arxiv.org/abs/2609.27086"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27086","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27086","title":"NADI 2026"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27139","domain":"General AI","name":"SurgHiBench","released":"2026-09-22","url":"https://arxiv.org/abs/2609.27139"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27139","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27139","title":"SurgHiBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27142","domain":"General AI","name":"ROCS","released":"2026-09-22","url":"https://arxiv.org/abs/2609.27142"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27142","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27142","title":"ROCS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27273","domain":"General AI","name":"CAVEAT","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27273"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27273","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27273","title":"CAVEAT"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27275","domain":"General AI","name":"BranchDrive","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27275"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27275","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27275","title":"BranchDrive"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27288","domain":"General AI","name":"PotARCin","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27288"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27288","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27288","title":"PotARCin"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27308","domain":"Robot manipulation","name":"EMBODIEDSWE-BENCH","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27308"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27308","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27308","title":"EMBODIEDSWE-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27349","domain":"Agents","name":"MolDesignBench","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27349"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27349","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27349","title":"MolDesignBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27353","domain":"Agents & Tool Use","name":"WebMRE","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27353"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27353","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27353","title":"WebMRE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27385","domain":"General AI","name":"Forecast Workflow Bench","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27385"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27385","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27385","title":"Forecast Workflow Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27395","domain":"General AI","name":"PRISM-VLM","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27395"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27395","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27395","title":"PRISM-VLM"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27449","domain":"Robot manipulation","name":"X2Real","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27449"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27449","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27449","title":"X2Real"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27461","domain":"General AI","name":"MVCAP-Bench","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27461"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27461","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27461","title":"MVCAP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27510","domain":"General AI","name":"Uncheatable Eval","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27510"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27510","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27510","title":"Uncheatable Eval"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27517","domain":"General AI","name":"NAFBench","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27517"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27517","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27517","title":"NAFBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27571","domain":"Agents & Tool Use","name":"FDE-Bench","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27571"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27571","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27571","title":"FDE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-27603","domain":"General AI","name":"FakeContextBench","released":"2026-09-23","url":"https://arxiv.org/abs/2609.27603"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.27603","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.27603","title":"FakeContextBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-28149","domain":"General AI","name":"EvEMTBench","released":"2026-09-23","url":"https://arxiv.org/abs/2609.28149"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.28149","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28149","title":"EvEMTBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-28236","domain":"Robotics & Embodied AI","name":"EmbodiedMemory-Bench","released":"2026-09-23","url":"https://arxiv.org/abs/2609.28236"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.28236","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28236","title":"EmbodiedMemory-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-28470","domain":"General AI","name":"StudentBench","released":"2026-09-23","url":"https://arxiv.org/abs/2609.28470"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.28470","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28470","title":"StudentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-28610","domain":"General AI","name":"UltraBench 2","released":"2026-09-23","url":"https://arxiv.org/abs/2609.28610"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.28610","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28610","title":"UltraBench 2"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-28851","domain":"Biology & Drug Discovery","name":"VisFlip","released":"2026-09-23","url":"https://arxiv.org/abs/2609.28851"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.28851","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28851","title":"VisFlip"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-28921","domain":"Biology & Drug Discovery","name":"PFArena","released":"2026-09-24","url":"https://arxiv.org/abs/2609.28921"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.28921","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28921","title":"PFArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-28952","domain":"Robotics","name":"RoboRecover","released":"2026-09-24","url":"https://arxiv.org/abs/2609.28952"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.28952","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.28952","title":"RoboRecover"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-29167","domain":"Finance","name":"IndicBankBench","released":"2026-09-24","url":"https://arxiv.org/abs/2609.29167"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.29167","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29167","title":"IndicBankBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-29245","domain":"General AI","name":"CTC-Bench","released":"2026-09-24","url":"https://arxiv.org/abs/2609.29245"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.29245","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29245","title":"CTC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-29429","domain":"General AI","name":"RLCDAlignBench","released":"2026-09-24","url":"https://arxiv.org/abs/2609.29429"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.29429","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29429","title":"RLCDAlignBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-29465","domain":"Language & Knowledge","name":"SWE-Prometheus","released":"2026-09-24","url":"https://arxiv.org/abs/2609.29465"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.29465","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29465","title":"SWE-Prometheus"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-29740","domain":"Biology & Drug Discovery","name":"TopU-LBVS","released":"2026-09-24","url":"https://arxiv.org/abs/2609.29740"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.29740","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29740","title":"TopU-LBVS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-29754","domain":"Code","name":"SWE-PolyVision","released":"2026-09-24","url":"https://arxiv.org/abs/2609.29754"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.29754","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.29754","title":"SWE-PolyVision"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30027","domain":"Biology & Drug Discovery","name":"Synthetic Hospital","released":"2026-09-24","url":"https://arxiv.org/abs/2609.30027"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30027","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30027","title":"Synthetic Hospital"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30030","domain":"General AI","name":"Artificial Societies Benchmark","released":"2026-09-24","url":"https://arxiv.org/abs/2609.30030"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30030","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30030","title":"Artificial Societies Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30094","domain":"General AI","name":"PrivDrift","released":"2026-09-24","url":"https://arxiv.org/abs/2609.30094"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30094","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30094","title":"PrivDrift"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30199","domain":"Factuality","name":"ExplorationBench","released":"2026-09-24","url":"https://arxiv.org/abs/2609.30199"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30199","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30199","title":"ExplorationBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30217","domain":"General AI","name":"EvasionBench","released":"2026-09-24","url":"https://arxiv.org/abs/2609.30217"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30217","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30217","title":"EvasionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30227","domain":"General AI","name":"VeriSpeak","released":"2026-09-24","url":"https://arxiv.org/abs/2609.30227"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30227","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30227","title":"VeriSpeak"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30670","domain":"General AI","name":"TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding","released":"2026-09-25","url":"https://arxiv.org/abs/2609.30670"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30670","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30670","title":"TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30722","domain":"General AI","name":"TrafficImag","released":"2026-09-25","url":"https://arxiv.org/abs/2609.30722"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30722","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30722","title":"TrafficImag"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30749","domain":"General AI","name":"ORCA","released":"2026-09-25","url":"https://arxiv.org/abs/2609.30749"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30749","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30749","title":"ORCA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30928","domain":"Biology & Drug Discovery","name":"UltraG-Bench","released":"2026-09-25","url":"https://arxiv.org/abs/2609.30928"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30928","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30928","title":"UltraG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30952","domain":"Multimodal","name":"MVVBench","released":"2026-09-25","url":"https://arxiv.org/abs/2609.30952"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30952","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30952","title":"MVVBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30979","domain":"General AI","name":"CCRV-Bench","released":"2026-09-25","url":"https://arxiv.org/abs/2609.30979"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30979","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30979","title":"CCRV-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-30981","domain":"General AI","name":"STORM-Bench","released":"2026-09-25","url":"https://arxiv.org/abs/2609.30981"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.30981","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.30981","title":"STORM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-31002","domain":"General AI","name":"ShopRank-Bench","released":"2026-09-25","url":"https://arxiv.org/abs/2609.31002"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.31002","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31002","title":"ShopRank-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-31082","domain":"General AI","name":"SAGE: A sampling-aware global evaluation benchmark for species distribution modeling","released":"2026-09-25","url":"https://arxiv.org/abs/2609.31082"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.31082","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31082","title":"SAGE: A sampling-aware global evaluation benchmark for species distribution modeling"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-31142","domain":"Factuality","name":"JevAdvBench","released":"2026-09-25","url":"https://arxiv.org/abs/2609.31142"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.31142","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31142","title":"JevAdvBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-31245","domain":"General AI","name":"RupeeBias","released":"2026-09-25","url":"https://arxiv.org/abs/2609.31245"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.31245","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31245","title":"RupeeBias"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-31468","domain":"General AI","name":"PriceBench","released":"2026-09-25","url":"https://arxiv.org/abs/2609.31468"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.31468","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31468","title":"PriceBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-31507","domain":"General AI","name":"SatNav","released":"2026-09-25","url":"https://arxiv.org/abs/2609.31507"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.31507","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31507","title":"SatNav"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-31587","domain":"Code & Software","name":"Roundtrip","released":"2026-09-25","url":"https://arxiv.org/abs/2609.31587"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.31587","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.31587","title":"Roundtrip"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32192","domain":"Agents","name":"CoMemBench","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32192"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32192","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32192","title":"CoMemBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32227","domain":"General AI","name":"OptiArena","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32227"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32227","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32227","title":"OptiArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32245","domain":"General AI","name":"AutoPDEBench","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32245"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32245","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32245","title":"AutoPDEBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32313","domain":"Multimodal","name":"MemTransfer","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32313"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32313","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32313","title":"MemTransfer"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32352","domain":"Biology & Drug Discovery","name":"EyeVQA","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32352"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32352","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32352","title":"EyeVQA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32413","domain":"Code","name":"IChart2Code","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32413"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32413","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32413","title":"IChart2Code"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32424","domain":"Agents","name":"CyberClear","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32424"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32424","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32424","title":"CyberClear"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32533","domain":"Advertising & Pricing","name":"LLMAdBench","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32533"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32533","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32533","title":"LLMAdBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32569","domain":"General AI","name":"OmniSmartHome","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32569"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32569","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32569","title":"OmniSmartHome"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32574","domain":"General AI","name":"CUE-Mem","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32574"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32574","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32574","title":"CUE-Mem"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32600","domain":"Agents & Tool Use","name":"CUA-SWE","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32600"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32600","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32600","title":"CUA-SWE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32616","domain":"General AI","name":"CAVE-Bench","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32616"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32616","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32616","title":"CAVE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32662","domain":"Agents","name":"AsynCodeBench","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32662"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32662","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32662","title":"AsynCodeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32734","domain":"General AI","name":"REALIS","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32734"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32734","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32734","title":"REALIS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32740","domain":"Biology & Drug Discovery","name":"AnesTRACE","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32740"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32740","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32740","title":"AnesTRACE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32795","domain":"Agents","name":"AgentHabit","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32795"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32795","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32795","title":"AgentHabit"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32810","domain":"Biology & Drug Discovery","name":"OpenTumorBoard","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32810"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32810","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32810","title":"OpenTumorBoard"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32835","domain":"Finance","name":"FinancialAuditBench","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32835"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32835","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32835","title":"FinancialAuditBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-32915","domain":"Agents","name":"AgentTell","released":"2026-09-26","url":"https://arxiv.org/abs/2609.32915"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.32915","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.32915","title":"AgentTell"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33023","domain":"Agents & Tool Use","name":"SRE-Marathon","released":"2026-09-26","url":"https://arxiv.org/abs/2609.33023"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33023","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33023","title":"SRE-Marathon"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33158","domain":"Biology & Drug Discovery","name":"FOCUS","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33158"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33158","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33158","title":"FOCUS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33233","domain":"Finance","name":"Inspire","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33233"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33233","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33233","title":"Inspire"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33261","domain":"General AI","name":"EngIntervene","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33261"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33261","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33261","title":"EngIntervene"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33286","domain":"General AI","name":"InfoEdit","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33286"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33286","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33286","title":"InfoEdit"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33356","domain":"Chip Design & EDA","name":"Long-Horizon Analog Design Bench","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33356"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33356","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33356","title":"Long-Horizon Analog Design Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33357","domain":"General AI","name":"DISCERN","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33357"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33357","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33357","title":"DISCERN"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33441","domain":"General AI","name":"MIC-Bench","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33441"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33441","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33441","title":"MIC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33521","domain":"Cybersecurity","name":"LLM4Trust","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33521"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33521","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33521","title":"LLM4Trust"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33678","domain":"General AI","name":"SWE-Game","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33678"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33678","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33678","title":"SWE-Game"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33699","domain":"Chip Design & EDA","name":"SpecRead","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33699"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33699","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33699","title":"SpecRead"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33807","domain":"Robot manipulation","name":"CodeActionBench","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33807"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33807","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33807","title":"CodeActionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33893","domain":"General AI","name":"MISHAP-Bench","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33893"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33893","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33893","title":"MISHAP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33905","domain":"General AI","name":"SlopBench","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33905"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33905","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33905","title":"SlopBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-33980","domain":"General AI","name":"DynGraphAgentBench","released":"2026-09-27","url":"https://arxiv.org/abs/2609.33980"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.33980","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.33980","title":"DynGraphAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34047","domain":"General AI","name":"ARCH-B","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34047"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34047","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34047","title":"ARCH-B"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34072","domain":"General AI","name":"PhysFieldBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34072"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34072","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34072","title":"PhysFieldBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34139","domain":"Agents & Tool Use","name":"AnyAppBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34139"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34139","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34139","title":"AnyAppBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34195","domain":"General AI","name":"PainterBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34195"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34195","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34195","title":"PainterBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34210","domain":"Robotics","name":"RLE-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34210"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34210","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34210","title":"RLE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34271","domain":"General AI","name":"GEdit3D-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34271"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34271","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34271","title":"GEdit3D-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34274","domain":"Language & Knowledge","name":"BIABench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34274"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34274","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34274","title":"BIABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34314","domain":"General AI","name":"PlaylistEval","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34314"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34314","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34314","title":"PlaylistEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34428","domain":"General AI","name":"AgentHop","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34428"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34428","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34428","title":"AgentHop"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34455","domain":"General AI","name":"RGDT-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34455"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34455","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34455","title":"RGDT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34469","domain":"Code & Software","name":"OfficeEditBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34469"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34469","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34469","title":"OfficeEditBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34492","domain":"General AI","name":"PowerBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34492"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34492","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34492","title":"PowerBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34510","domain":"Agents","name":"Can AI Make Money in Crypto?","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34510"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34510","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34510","title":"Can AI Make Money in Crypto?"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34547","domain":"General AI","name":"ActionLens","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34547"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34547","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34547","title":"ActionLens"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34661","domain":"Code & Software","name":"Codoku","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34661"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34661","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34661","title":"Codoku"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34687","domain":"Reasoning","name":"VCN-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34687"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34687","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34687","title":"VCN-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34710","domain":"Agents & Tool Use","name":"FromPitch2Board","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34710"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34710","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34710","title":"FromPitch2Board"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34749","domain":"General AI","name":"CoDrive-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34749"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34749","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34749","title":"CoDrive-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34752","domain":"Code & Software","name":"Multi-SWT-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34752"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34752","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34752","title":"Multi-SWT-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34782","domain":"General AI","name":"CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34782"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34782","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34782","title":"CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34790","domain":"Agents","name":"CoSec","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34790"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34790","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34790","title":"CoSec"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34854","domain":"General AI","name":"AUV-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34854"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34854","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34854","title":"AUV-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34913","domain":"Agents","name":"DGF-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34913"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34913","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34913","title":"DGF-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34930","domain":"General AI","name":"PDEU-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34930"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34930","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34930","title":"PDEU-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-34973","domain":"Agents & Tool Use","name":"APEX-Voice","released":"2026-09-28","url":"https://arxiv.org/abs/2609.34973"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.34973","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.34973","title":"APEX-Voice"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35003","domain":"Multimodal","name":"MAIL-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35003"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35003","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35003","title":"MAIL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35025","domain":"General AI","name":"AutoDataBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35025"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35025","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35025","title":"AutoDataBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35026","domain":"General AI","name":"WebPageBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35026"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35026","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35026","title":"WebPageBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35028","domain":"Language & Knowledge","name":"VEX-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35028"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35028","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35028","title":"VEX-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35032","domain":"Reasoning","name":"JRDB-AVR","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35032"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35032","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35032","title":"JRDB-AVR"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35052","domain":"Logistics & Operations","name":"OPIS","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35052"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35052","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35052","title":"OPIS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35113","domain":"General AI","name":"SymbolicArena","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35113"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35113","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35113","title":"SymbolicArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35143","domain":"Agents","name":"Timeline-Bench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35143"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35143","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35143","title":"Timeline-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35312","domain":"Factuality","name":"MemoReason","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35312"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35312","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35312","title":"MemoReason"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35357","domain":"Code & Software","name":"RepoReuse","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35357"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35357","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35357","title":"RepoReuse"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35409","domain":"General AI","name":"AwarenessBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35409"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35409","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35409","title":"AwarenessBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35515","domain":"General AI","name":"MechBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35515"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35515","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35515","title":"MechBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35581","domain":"Language & Knowledge","name":"QC-Stark","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35581"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35581","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35581","title":"QC-Stark"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35596","domain":"Agents","name":"SEABench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35596"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35596","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35596","title":"SEABench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35606","domain":"Language & Knowledge","name":"TCSAlgBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35606"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35606","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35606","title":"TCSAlgBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35639","domain":"General AI","name":"GPUPhysBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35639"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35639","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35639","title":"GPUPhysBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-35641","domain":"General AI","name":"VVRBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.35641"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.35641","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.35641","title":"VVRBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36082","domain":"General AI","name":"GeoOutageBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.36082"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36082","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36082","title":"GeoOutageBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36161","domain":"Code & Software","name":"ReviveBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.36161"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36161","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36161","title":"ReviveBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36218","domain":"General AI","name":"CineSubBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.36218"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36218","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36218","title":"CineSubBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36308","domain":"Factuality","name":"CheatBench","released":"2026-09-28","url":"https://arxiv.org/abs/2609.36308"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36308","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36308","title":"CheatBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36488","domain":"General AI","name":"AdaptArena","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36488"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36488","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36488","title":"AdaptArena"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36518","domain":"Geometric reasoning","name":"LIBERO-MAX","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36518"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36518","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36518","title":"LIBERO-MAX"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36605","domain":"Robotics","name":"RoboChrono","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36605"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36605","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36605","title":"RoboChrono"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36691","domain":"Robot manipulation","name":"Video2Skill","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36691"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36691","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36691","title":"Video2Skill"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36730","domain":"Agents","name":"LibraryDesignBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36730"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36730","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36730","title":"LibraryDesignBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36739","domain":"Agents","name":"Frontier Autolab","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36739"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36739","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36739","title":"Frontier Autolab"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36777","domain":"General AI","name":"Code4Scene","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36777"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36777","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36777","title":"Code4Scene"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36915","domain":"Multimodal","name":"AeroManip-VLA","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36915"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36915","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36915","title":"AeroManip-VLA"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36923","domain":"Information retrieval","name":"InterfereBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36923"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36923","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36923","title":"InterfereBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-36987","domain":"Factuality","name":"CypherTurn","released":"2026-09-29","url":"https://arxiv.org/abs/2609.36987"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.36987","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.36987","title":"CypherTurn"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37053","domain":"Agents & Tool Use","name":"MatToolBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37053"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37053","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37053","title":"MatToolBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37216","domain":"Code & Software","name":"CRJudgeBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37216"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37216","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37216","title":"CRJudgeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37233","domain":"Code & Software","name":"DatalogBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37233"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37233","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37233","title":"DatalogBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37287","domain":"General AI","name":"VISTA-Bench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37287"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37287","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37287","title":"VISTA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37292","domain":"Robot manipulation","name":"BAVO-Bench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37292"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37292","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37292","title":"BAVO-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37317","domain":"General AI","name":"Omni-StoryBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37317"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37317","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37317","title":"Omni-StoryBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37470","domain":"General AI","name":"Probability Contracts","released":"2026-09-27","url":"https://arxiv.org/abs/2609.37470"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37470","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37470","title":"Probability Contracts"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37543","domain":"General AI","name":"RunyaNER","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37543"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37543","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37543","title":"RunyaNER"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37559","domain":"General AI","name":"APM-Bench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37559"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37559","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37559","title":"APM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37576","domain":"General AI","name":"SQUARE-Bench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37576"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37576","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37576","title":"SQUARE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37658","domain":"Finance","name":"EnterpriseBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37658"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37658","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37658","title":"EnterpriseBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37686","domain":"CAD","name":"EngiWorld","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37686"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37686","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37686","title":"EngiWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37709","domain":"General AI","name":"VIF-Bench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37709"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37709","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37709","title":"VIF-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37773","domain":"General AI","name":"OmniVCBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37773"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37773","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37773","title":"OmniVCBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37871","domain":"Autonomous Driving","name":"ExceptionDrive","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37871"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37871","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37871","title":"ExceptionDrive"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-37969","domain":"General AI","name":"Refiner-Bench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.37969"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.37969","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.37969","title":"Refiner-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38043","domain":"Agents","name":"UserProxyBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.38043"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38043","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38043","title":"UserProxyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38137","domain":"General AI","name":"LongHarness Bench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.38137"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38137","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38137","title":"LongHarness Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38269","domain":"Agents","name":"Zero2Repo","released":"2026-09-29","url":"https://arxiv.org/abs/2609.38269"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38269","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38269","title":"Zero2Repo"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38335","domain":"Code & Software","name":"E2E-SWE","released":"2026-09-29","url":"https://arxiv.org/abs/2609.38335"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38335","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38335","title":"E2E-SWE"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38409","domain":"General AI","name":"ArgGYM","released":"2026-09-29","url":"https://arxiv.org/abs/2609.38409"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38409","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38409","title":"ArgGYM"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38480","domain":"Biology & Drug Discovery","name":"KlinikeBench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.38480"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38480","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38480","title":"KlinikeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38523","domain":"Finance","name":"MM-FinEval","released":"2026-09-29","url":"https://arxiv.org/abs/2609.38523"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38523","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38523","title":"MM-FinEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38604","domain":"General AI","name":"Drift-Bench","released":"2026-09-29","url":"https://arxiv.org/abs/2609.38604"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38604","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38604","title":"Drift-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38719","domain":"General AI","name":"FlapKAD","released":"2026-09-30","url":"https://arxiv.org/abs/2609.38719"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38719","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38719","title":"FlapKAD"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38863","domain":"CAD","name":"CircleNest-Bench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.38863"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38863","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38863","title":"CircleNest-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38867","domain":"General AI","name":"Talk2Agent","released":"2026-09-30","url":"https://arxiv.org/abs/2609.38867"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38867","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38867","title":"Talk2Agent"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38948","domain":"Factuality","name":"DrivingBench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.38948"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38948","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38948","title":"DrivingBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-38966","domain":"Robotics & Embodied AI","name":"Video2SwimFish","released":"2026-09-30","url":"https://arxiv.org/abs/2609.38966"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.38966","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.38966","title":"Video2SwimFish"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39146","domain":"Agents","name":"MADBench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39146"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39146","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39146","title":"MADBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39198","domain":"Robot manipulation","name":"DynBench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39198"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39198","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39198","title":"DynBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39284","domain":"Code & Software","name":"EngramBench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39284"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39284","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39284","title":"EngramBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39490","domain":"General AI","name":"OmniReasoningBench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39490"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39490","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39490","title":"OmniReasoningBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39507","domain":"Agents","name":"LIBERO-Agent","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39507"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39507","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39507","title":"LIBERO-Agent"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39533","domain":"Cybersecurity","name":"CATCH","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39533"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39533","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39533","title":"CATCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39568","domain":"Code","name":"VeriCodeBench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39568"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39568","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39568","title":"VeriCodeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39649","domain":"General AI","name":"FANVIDv2","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39649"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39649","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39649","title":"FANVIDv2"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39673","domain":"General AI","name":"NodeGround","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39673"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39673","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39673","title":"NodeGround"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39777","domain":"Agents","name":"GraphMAS","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39777"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39777","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39777","title":"GraphMAS"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39903","domain":"Agents","name":"OSWorld-Science","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39903"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39903","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39903","title":"OSWorld-Science"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-39909","domain":"Code & Software","name":"DoGBench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.39909"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.39909","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.39909","title":"DoGBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-40097","domain":"General AI","name":"AutoDataBench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.40097"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.40097","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.40097","title":"AutoDataBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2609-40325","domain":"Agents & Tool Use","name":"WorldAuditBench","released":"2026-09-30","url":"https://arxiv.org/abs/2609.40325"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2609.40325","source":"claire_radar","source_url":"https://arxiv.org/abs/2609.40325","title":"WorldAuditBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-01092","domain":"Robot manipulation","name":"Ego2Act","released":"2026-10-01","url":"https://arxiv.org/abs/2610.01092"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.01092","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01092","title":"Ego2Act"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-01204","domain":"General AI","name":"DrillBench","released":"2026-10-01","url":"https://arxiv.org/abs/2610.01204"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.01204","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01204","title":"DrillBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-01306","domain":"Finance","name":"DAYJOB","released":"2026-10-01","url":"https://arxiv.org/abs/2610.01306"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.01306","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01306","title":"DAYJOB"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-01497","domain":"Biology & Drug Discovery","name":"OpenMTB-Audit","released":"2026-10-01","url":"https://arxiv.org/abs/2610.01497"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.01497","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01497","title":"OpenMTB-Audit"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-01605","domain":"General AI","name":"Hob-VL","released":"2026-10-01","url":"https://arxiv.org/abs/2610.01605"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.01605","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01605","title":"Hob-VL"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-01778","domain":"General AI","name":"GIFTBench","released":"2026-10-01","url":"https://arxiv.org/abs/2610.01778"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.01778","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.01778","title":"GIFTBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-02089","domain":"Agents","name":"HumanoidToolBench","released":"2026-10-01","url":"https://arxiv.org/abs/2610.02089"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.02089","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02089","title":"HumanoidToolBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-02617","domain":"Agents","name":"WebUIProof","released":"2026-10-02","url":"https://arxiv.org/abs/2610.02617"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.02617","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02617","title":"WebUIProof"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-02715","domain":"Language & Knowledge","name":"Ego2World","released":"2026-10-02","url":"https://arxiv.org/abs/2610.02715"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.02715","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02715","title":"Ego2World"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-02770","domain":"General AI","name":"AptMQL-Bench","released":"2026-10-02","url":"https://arxiv.org/abs/2610.02770"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.02770","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02770","title":"AptMQL-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-02982","domain":"General AI","name":"PLCWorld","released":"2026-10-02","url":"https://arxiv.org/abs/2610.02982"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.02982","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02982","title":"PLCWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-02986","domain":"General AI","name":"OLMo-Detect","released":"2026-10-02","url":"https://arxiv.org/abs/2610.02986"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.02986","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.02986","title":"OLMo-Detect"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03020","domain":"General AI","name":"DyadMem","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03020"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03020","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03020","title":"DyadMem"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03080","domain":"Code & Software","name":"MintEval","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03080"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03080","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03080","title":"MintEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03153","domain":"Agents","name":"EvoRiskBench","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03153"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03153","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03153","title":"EvoRiskBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03226","domain":"General AI","name":"D2K-Bench","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03226"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03226","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03226","title":"D2K-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03248","domain":"Robotics & Embodied AI","name":"EmbPASS","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03248"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03248","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03248","title":"EmbPASS"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03278","domain":"Robot manipulation","name":"DexJoCo-X","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03278"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03278","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03278","title":"DexJoCo-X"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03329","domain":"General AI","name":"SyntaxBench","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03329"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03329","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03329","title":"SyntaxBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03356","domain":"Agents","name":"ReFract","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03356"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03356","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03356","title":"ReFract"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03564","domain":"Agents","name":"FinSkillBench","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03564"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03564","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03564","title":"FinSkillBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-03574","domain":"General AI","name":"HyperBrowseComp","released":"2026-10-02","url":"https://arxiv.org/abs/2610.03574"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.03574","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.03574","title":"HyperBrowseComp"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-04253","domain":"General AI","name":"Spec2Game","released":"2026-10-03","url":"https://arxiv.org/abs/2610.04253"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.04253","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04253","title":"Spec2Game"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-04329","domain":"General AI","name":"CoPE-Bench","released":"2026-10-03","url":"https://arxiv.org/abs/2610.04329"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.04329","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04329","title":"CoPE-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-04375","domain":"General AI","name":"IEC-Bench","released":"2026-10-03","url":"https://arxiv.org/abs/2610.04375"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.04375","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04375","title":"IEC-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-04379","domain":"Agents","name":"AgentPersonaBench","released":"2026-10-03","url":"https://arxiv.org/abs/2610.04379"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.04379","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04379","title":"AgentPersonaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-04506","domain":"General AI","name":"EgoExo-Next","released":"2026-10-03","url":"https://arxiv.org/abs/2610.04506"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.04506","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04506","title":"EgoExo-Next"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-04672","domain":"Agents","name":"MASBench","released":"2026-10-03","url":"https://arxiv.org/abs/2610.04672"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.04672","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04672","title":"MASBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-04691","domain":"Factuality","name":"RAGStress","released":"2026-10-03","url":"https://arxiv.org/abs/2610.04691"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.04691","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04691","title":"RAGStress"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-04800","domain":"Biology & Drug Discovery","name":"MedImageOSWorld","released":"2026-10-03","url":"https://arxiv.org/abs/2610.04800"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.04800","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04800","title":"MedImageOSWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-04927","domain":"Code & Software","name":"MLE-InsightBench","released":"2026-10-04","url":"https://arxiv.org/abs/2610.04927"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.04927","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.04927","title":"MLE-InsightBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05010","domain":"General AI","name":"PortraitAes-Bench","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05010"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05010","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05010","title":"PortraitAes-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05013","domain":"Reasoning","name":"EMBER-Bench","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05013"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05013","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05013","title":"EMBER-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05380","domain":"Chip Design & EDA","name":"VHDL-REPOBENCH","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05380"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05380","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05380","title":"VHDL-REPOBENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05387","domain":"General AI","name":"GNN-CB","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05387"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05387","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05387","title":"GNN-CB"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05398","domain":"General AI","name":"MMPostTrainBench","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05398"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05398","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05398","title":"MMPostTrainBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05414","domain":"Robot manipulation","name":"RMMBench","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05414"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05414","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05414","title":"RMMBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05532","domain":"General AI","name":"DelegationBench","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05532"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05532","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05532","title":"DelegationBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05586","domain":"General AI","name":"AgentDoxx","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05586"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05586","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05586","title":"AgentDoxx"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05590","domain":"Factuality","name":"ColdDDI","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05590"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05590","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05590","title":"ColdDDI"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05622","domain":"Agents & Tool Use","name":"UndoBench","released":"2026-10-04","url":"https://arxiv.org/abs/2610.05622"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05622","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05622","title":"UndoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05715","domain":"General AI","name":"OR-Bench","released":"2026-10-05","url":"https://arxiv.org/abs/2610.05715"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05715","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05715","title":"OR-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05790","domain":"General AI","name":"FairRSFM","released":"2026-10-05","url":"https://arxiv.org/abs/2610.05790"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05790","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05790","title":"FairRSFM"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-05950","domain":"General AI","name":"RMD-Bench","released":"2026-10-05","url":"https://arxiv.org/abs/2610.05950"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.05950","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.05950","title":"RMD-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-06184","domain":"General AI","name":"ACG-Bench","released":"2026-10-05","url":"https://arxiv.org/abs/2610.06184"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.06184","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06184","title":"ACG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-06406","domain":"Agents","name":"AgentMonBench","released":"2026-10-05","url":"https://arxiv.org/abs/2610.06406"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.06406","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06406","title":"AgentMonBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-06450","domain":"General AI","name":"EMG-FM-Bench","released":"2026-10-05","url":"https://arxiv.org/abs/2610.06450"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.06450","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06450","title":"EMG-FM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-06571","domain":"Biology & Drug Discovery","name":"BrainTRACE","released":"2026-10-05","url":"https://arxiv.org/abs/2610.06571"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.06571","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06571","title":"BrainTRACE"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-06587","domain":"Finance","name":"CavaBench","released":"2026-10-05","url":"https://arxiv.org/abs/2610.06587"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.06587","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06587","title":"CavaBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-06748","domain":"Cybersecurity","name":"BazaarBench","released":"2026-10-05","url":"https://arxiv.org/abs/2610.06748"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.06748","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06748","title":"BazaarBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-06824","domain":"General AI","name":"TasteVal","released":"2026-10-05","url":"https://arxiv.org/abs/2610.06824"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.06824","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.06824","title":"TasteVal"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-07338","domain":"General AI","name":"Logbook","released":"2026-10-05","url":"https://arxiv.org/abs/2610.07338"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.07338","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07338","title":"Logbook"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-07423","domain":"General AI","name":"2d-fet-bench","released":"2026-10-05","url":"https://arxiv.org/abs/2610.07423"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.07423","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07423","title":"2d-fet-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-07557","domain":"Code & Software","name":"CheckerBench","released":"2026-10-06","url":"https://arxiv.org/abs/2610.07557"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.07557","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07557","title":"CheckerBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-07614","domain":"Biology & Drug Discovery","name":"BioStudyBench","released":"2026-10-06","url":"https://arxiv.org/abs/2610.07614"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.07614","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07614","title":"BioStudyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-07639","domain":"Agents","name":"HarnessSecurity-Bench","released":"2026-10-06","url":"https://arxiv.org/abs/2610.07639"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.07639","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07639","title":"HarnessSecurity-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-07763","domain":"Agents","name":"ST-Bench","released":"2026-10-06","url":"https://arxiv.org/abs/2610.07763"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.07763","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07763","title":"ST-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-07922","domain":"Robotics & Embodied AI","name":"OpenWAM","released":"2026-10-06","url":"https://arxiv.org/abs/2610.07922"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.07922","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07922","title":"OpenWAM"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-07967","domain":"General AI","name":"DecepEval","released":"2026-10-06","url":"https://arxiv.org/abs/2610.07967"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.07967","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.07967","title":"DecepEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-08076","domain":"Cybersecurity","name":"SpeedrunBench","released":"2026-10-06","url":"https://arxiv.org/abs/2610.08076"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.08076","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08076","title":"SpeedrunBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-08102","domain":"General AI","name":"DSV-Mem","released":"2026-10-06","url":"https://arxiv.org/abs/2610.08102"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.08102","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08102","title":"DSV-Mem"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-08215","domain":"Factuality","name":"Learn2Play Bench","released":"2026-10-06","url":"https://arxiv.org/abs/2610.08215"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.08215","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08215","title":"Learn2Play Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-08571","domain":"General AI","name":"RAG-PIBench","released":"2026-10-06","url":"https://arxiv.org/abs/2610.08571"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.08571","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08571","title":"RAG-PIBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-08662","domain":"Language & Knowledge","name":"ParanoiaEval","released":"2026-10-06","url":"https://arxiv.org/abs/2610.08662"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.08662","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08662","title":"ParanoiaEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-08684","domain":"General AI","name":"RenderBench","released":"2026-10-06","url":"https://arxiv.org/abs/2610.08684"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.08684","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08684","title":"RenderBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-08720","domain":"General AI","name":"WorldSolver","released":"2026-10-06","url":"https://arxiv.org/abs/2610.08720"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.08720","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.08720","title":"WorldSolver"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-09152","domain":"General AI","name":"sk-bench","released":"2026-10-06","url":"https://arxiv.org/abs/2610.09152"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.09152","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09152","title":"sk-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-09294","domain":"Agents","name":"RT-Safe","released":"2026-10-07","url":"https://arxiv.org/abs/2610.09294"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.09294","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09294","title":"RT-Safe"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-09374","domain":"Agents","name":"DUDA-Bench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.09374"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.09374","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09374","title":"DUDA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-09478","domain":"General AI","name":"InstanceBench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.09478"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.09478","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09478","title":"InstanceBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-09550","domain":"General AI","name":"CROSS-Bench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.09550"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.09550","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09550","title":"CROSS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-09671","domain":"General AI","name":"InsClaimBench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.09671"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.09671","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09671","title":"InsClaimBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-09742","domain":"General AI","name":"SoccerNet-FoulRet","released":"2026-10-07","url":"https://arxiv.org/abs/2610.09742"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.09742","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09742","title":"SoccerNet-FoulRet"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-09872","domain":"Agents","name":"LiveMACEBench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.09872"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.09872","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09872","title":"LiveMACEBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-09944","domain":"General AI","name":"AgentTime","released":"2026-10-07","url":"https://arxiv.org/abs/2610.09944"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.09944","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.09944","title":"AgentTime"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10047","domain":"General AI","name":"AdSpark","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10047"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10047","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10047","title":"AdSpark"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10276","domain":"General AI","name":"PatchBench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10276"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10276","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10276","title":"PatchBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10374","domain":"Code","name":"TaoD2C-Bench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10374"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10374","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10374","title":"TaoD2C-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10388","domain":"Robotics & Embodied AI","name":"RoboQuest","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10388"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10388","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10388","title":"RoboQuest"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10409","domain":"Multimodal","name":"RobotWorld","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10409"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10409","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10409","title":"RobotWorld"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10421","domain":"General AI","name":"AirGroundVLN","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10421"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10421","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10421","title":"AirGroundVLN"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10455","domain":"General AI","name":"PHRBench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10455"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10455","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10455","title":"PHRBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10624","domain":"General AI","name":"SAVU-BENCH","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10624"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10624","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10624","title":"SAVU-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10703","domain":"Mobile & Personal Computing","name":"OcuBench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10703"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10703","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10703","title":"OcuBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10942","domain":"Agents","name":"StoreBench","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10942"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10942","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10942","title":"StoreBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-10947","domain":"General AI","name":"RH-Detect","released":"2026-10-07","url":"https://arxiv.org/abs/2610.10947"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.10947","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.10947","title":"RH-Detect"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11050","domain":"General AI","name":"AgentHorizon","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11050"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11050","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11050","title":"AgentHorizon"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11064","domain":"General AI","name":"ModeBench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11064"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11064","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11064","title":"ModeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11129","domain":"General AI","name":"GameCommBench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11129"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11129","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11129","title":"GameCommBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11233","domain":"General AI","name":"MiniVer-V","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11233"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11233","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11233","title":"MiniVer-V"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11312","domain":"Agents","name":"MedBench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11312"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11312","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11312","title":"MedBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11328","domain":"General AI","name":"Mine Odyssey","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11328"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11328","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11328","title":"Mine Odyssey"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11329","domain":"Biology & Drug Discovery","name":"PathLang","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11329"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11329","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11329","title":"PathLang"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11392","domain":"General AI","name":"TypedBench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11392"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11392","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11392","title":"TypedBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11559","domain":"General AI","name":"SWE-Journey","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11559"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11559","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11559","title":"SWE-Journey"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11561","domain":"Agents","name":"Workerville","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11561"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11561","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11561","title":"Workerville"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11618","domain":"Code","name":"PolyCodeEval","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11618"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11618","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11618","title":"PolyCodeEval"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11694","domain":"Language & Knowledge","name":"VenusRX-Bench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11694"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11694","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11694","title":"VenusRX-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-11723","domain":"General AI","name":"MultiWorldBench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.11723"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.11723","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.11723","title":"MultiWorldBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12069","domain":"Robotics & Embodied AI","name":"LIVIN","released":"2026-10-08","url":"https://arxiv.org/abs/2610.12069"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12069","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12069","title":"LIVIN"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12183","domain":"Chip Design & EDA","name":"AgenticBBO-Bench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.12183"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12183","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12183","title":"AgenticBBO-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12190","domain":"AI Scientist","name":"DataSense-Bench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.12190"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12190","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12190","title":"DataSense-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12402","domain":"General AI","name":"SpaceCast-Bench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.12402"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12402","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12402","title":"SpaceCast-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12417","domain":"Multimodal","name":"WOVEN","released":"2026-10-08","url":"https://arxiv.org/abs/2610.12417"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12417","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12417","title":"WOVEN"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12423","domain":"General AI","name":"Pumpire","released":"2026-10-08","url":"https://arxiv.org/abs/2610.12423"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12423","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12423","title":"Pumpire"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12458","domain":"General AI","name":"OmniCapBench","released":"2026-10-08","url":"https://arxiv.org/abs/2610.12458"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12458","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12458","title":"OmniCapBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12823","domain":"Agents","name":"MisVisAgentBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.12823"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12823","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12823","title":"MisVisAgentBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12881","domain":"Language & Knowledge","name":"GeoSpatialBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.12881"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12881","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12881","title":"GeoSpatialBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12920","domain":"Code","name":"SCCGBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.12920"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12920","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12920","title":"SCCGBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12954","domain":"General AI","name":"MathVis","released":"2026-10-09","url":"https://arxiv.org/abs/2610.12954"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12954","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12954","title":"MathVis"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12961","domain":"General AI","name":"MVGraph","released":"2026-10-09","url":"https://arxiv.org/abs/2610.12961"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12961","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12961","title":"MVGraph"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-12975","domain":"Language & Knowledge","name":"SmartSalesBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.12975"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.12975","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.12975","title":"SmartSalesBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13021","domain":"Robotics & Embodied AI","name":"EmbodiedCLBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13021"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13021","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13021","title":"EmbodiedCLBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13022","domain":"Code & Software","name":"ReqClarifyBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13022"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13022","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13022","title":"ReqClarifyBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13116","domain":"Agents","name":"DRYLAB-Bench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13116"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13116","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13116","title":"DRYLAB-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13361","domain":"Agents","name":"AgentSWE","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13361"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13361","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13361","title":"AgentSWE"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13412","domain":"General AI","name":"TextVBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13412"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13412","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13412","title":"TextVBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13435","domain":"Biology & Drug Discovery","name":"PPG-Bench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13435"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13435","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13435","title":"PPG-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13461","domain":"General AI","name":"SqueezeBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13461"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13461","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13461","title":"SqueezeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13466","domain":"CAD","name":"A Real-World CAD Segmentation Benchmark","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13466"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13466","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13466","title":"A Real-World CAD Segmentation Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13534","domain":"Reasoning","name":"MAGMA-Bench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13534"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13534","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13534","title":"MAGMA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13550","domain":"General AI","name":"BASIS-Bench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13550"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13550","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13550","title":"BASIS-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13623","domain":"Agents","name":"APEX-Harm","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13623"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13623","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13623","title":"APEX-Harm"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13626","domain":"Code","name":"SWE-DeceptionBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13626"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13626","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13626","title":"SWE-DeceptionBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-13668","domain":"Agents & Tool Use","name":"CRBugBench","released":"2026-10-09","url":"https://arxiv.org/abs/2610.13668"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.13668","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.13668","title":"CRBugBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-15425","domain":"Robot manipulation","name":"RoboCombine","released":"2026-10-11","url":"https://arxiv.org/abs/2610.15425"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.15425","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15425","title":"RoboCombine"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-15452","domain":"General AI","name":"EmoTrace-Bench","released":"2026-10-11","url":"https://arxiv.org/abs/2610.15452"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.15452","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15452","title":"EmoTrace-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-15562","domain":"General AI","name":"CausalCascade","released":"2026-10-11","url":"https://arxiv.org/abs/2610.15562"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.15562","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15562","title":"CausalCascade"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-15714","domain":"Agents","name":"MedHarnessBench","released":"2026-10-11","url":"https://arxiv.org/abs/2610.15714"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.15714","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15714","title":"MedHarnessBench"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-15722","domain":"Code","name":"PhySimCode","released":"2026-10-11","url":"https://arxiv.org/abs/2610.15722"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.15722","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15722","title":"PhySimCode"},{"benchmarks":[{"benchmark_id":"claire-radar-2610-15732","domain":"General AI","name":"NepPlural","released":"2026-10-11","url":"https://arxiv.org/abs/2610.15732"}],"document_type":"registry_page","id":"claire_radar:https://arxiv.org/abs/2610.15732","source":"claire_radar","source_url":"https://arxiv.org/abs/2610.15732","title":"NepPlural"},{"benchmarks":[{"benchmark_id":"claire-radar-github-0xboyu-company-mental-model-benchmark","domain":"Finance","name":"Company Mental Model Benchmark","released":"2026-08-22","url":"https://github.com/0xboyu/company-mental-model-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/0xboyu/company-mental-model-benchmark","source":"claire_radar","source_url":"https://github.com/0xboyu/company-mental-model-benchmark","title":"Company Mental Model Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-4esv-roman-numeral-harmony-benchmark","domain":"General AI","name":"Roman Numeral Harmony Benchmark","released":"2026-08-22","url":"https://github.com/4esv/roman-numeral-harmony-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/4esv/roman-numeral-harmony-benchmark","source":"claire_radar","source_url":"https://github.com/4esv/roman-numeral-harmony-benchmark","title":"Roman Numeral Harmony Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-aah20-agentic-infrastructure-change-benchmark","domain":"Finance","name":"Agentic Infrastructure ChangeBench","released":"2026-09-05","url":"https://github.com/AAH20/agentic-infrastructure-change-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/AAH20/agentic-infrastructure-change-benchmark","source":"claire_radar","source_url":"https://github.com/AAH20/agentic-infrastructure-change-benchmark","title":"Agentic Infrastructure ChangeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-aah20-ai-agent-infrastructure-benchmark","domain":"Agents","name":"AgentInfraBench","released":"2026-09-05","url":"https://github.com/AAH20/ai-agent-infrastructure-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/AAH20/ai-agent-infrastructure-benchmark","source":"claire_radar","source_url":"https://github.com/AAH20/ai-agent-infrastructure-benchmark","title":"AgentInfraBench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-aiondadotcom-ai-sim-benchmark","domain":"Agents","name":"AI Coding Agent Water Simulation Benchmark","released":"2026-08-21","url":"https://github.com/AiondaDotCom/ai-sim-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/AiondaDotCom/ai-sim-benchmark","source":"claire_radar","source_url":"https://github.com/AiondaDotCom/ai-sim-benchmark","title":"AI Coding Agent Water Simulation Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-akashai7-jev-vs-llm-benchmark","domain":"Language & Knowledge","name":"Jev vs LLM Benchmark","released":"2026-09-23","url":"https://github.com/AkashAi7/jev-vs-llm-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/AkashAi7/jev-vs-llm-benchmark","source":"claire_radar","source_url":"https://github.com/AkashAi7/jev-vs-llm-benchmark","title":"Jev vs LLM Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-andersballegaard-agentic-netdevops-benchmark","domain":"General AI","name":"Agentic NetDevOps Benchmark","released":"2026-09-08","url":"https://github.com/AndersBallegaard/agentic-netdevops-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/AndersBallegaard/agentic-netdevops-benchmark","source":"claire_radar","source_url":"https://github.com/AndersBallegaard/agentic-netdevops-benchmark","title":"Agentic NetDevOps Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-anuragkokate09-asr-model-benchmark","domain":"General AI","name":"ASR Model Benchmark","released":"2026-08-22","url":"https://github.com/Anuragkokate09/asr-model-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Anuragkokate09/asr-model-benchmark","source":"claire_radar","source_url":"https://github.com/Anuragkokate09/asr-model-benchmark","title":"ASR Model Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-artificialanalysis-aa-agentperf-local","domain":"Agents","name":"agentperf-local","released":"2026-09-26","url":"https://github.com/ArtificialAnalysis/aa-agentperf-local"}],"document_type":"registry_page","id":"claire_radar:https://github.com/ArtificialAnalysis/aa-agentperf-local","source":"claire_radar","source_url":"https://github.com/ArtificialAnalysis/aa-agentperf-local","title":"agentperf-local"},{"benchmarks":[{"benchmark_id":"claire-radar-github-buguerito-llm-math-reasoning-benchmark","domain":"Finance","name":"LLM Mathematical Reasoning Evaluation Benchmark","released":"2026-09-23","url":"https://github.com/Buguerito/llm-math-reasoning-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Buguerito/llm-math-reasoning-benchmark","source":"claire_radar","source_url":"https://github.com/Buguerito/llm-math-reasoning-benchmark","title":"LLM Mathematical Reasoning Evaluation Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-coresr-rob2-ai-benchmark","domain":"Safety & Trustworthiness","name":"RoB 2 AI Benchmark","released":"2026-10-12","url":"https://github.com/CoreSR/rob2-ai-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/CoreSR/rob2-ai-benchmark","source":"claire_radar","source_url":"https://github.com/CoreSR/rob2-ai-benchmark","title":"RoB 2 AI Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-enterpilot-ai-gateway-reproducible-benchmark","domain":"Language & Knowledge","name":"AI gateway reproducible benchmark","released":"2026-08-20","url":"https://github.com/ENTERPILOT/ai-gateway-reproducible-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/ENTERPILOT/ai-gateway-reproducible-benchmark","source":"claire_radar","source_url":"https://github.com/ENTERPILOT/ai-gateway-reproducible-benchmark","title":"AI gateway reproducible benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-ecowestern-esac","domain":"Language & Knowledge","name":"EcoWestern Short and Cheap Benchmark","released":"2026-09-22","url":"https://github.com/EcoWestern/ESAC"}],"document_type":"registry_page","id":"claire_radar:https://github.com/EcoWestern/ESAC","source":"claire_radar","source_url":"https://github.com/EcoWestern/ESAC","title":"EcoWestern Short and Cheap Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-ethanning-whatworkedbench","domain":"Language & Knowledge","name":"WhatWorkedBench","released":"2026-09-22","url":"https://github.com/EthanNing/WhatWorkedBench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/EthanNing/WhatWorkedBench","source":"claire_radar","source_url":"https://github.com/EthanNing/WhatWorkedBench","title":"WhatWorkedBench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-flappearlabs-hermes-code-agent-benchmark","domain":"Agents","name":"Hermes Code Agent Benchmark","released":"2026-09-27","url":"https://github.com/FlapPearLabs/hermes-code-agent-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/FlapPearLabs/hermes-code-agent-benchmark","source":"claire_radar","source_url":"https://github.com/FlapPearLabs/hermes-code-agent-benchmark","title":"Hermes Code Agent Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-gohaike-decision-layer-benchmark","domain":"Agents","name":"Decision Layer Benchmark","released":"2026-10-12","url":"https://github.com/GOhaike/decision-layer-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/GOhaike/decision-layer-benchmark","source":"claire_radar","source_url":"https://github.com/GOhaike/decision-layer-benchmark","title":"Decision Layer Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-gorkafs13-euskal-agent-bench","domain":"Agents","name":"euskal-agent-bench","released":"2026-08-20","url":"https://github.com/GorkaFS13/euskal-agent-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/GorkaFS13/euskal-agent-bench","source":"claire_radar","source_url":"https://github.com/GorkaFS13/euskal-agent-bench","title":"euskal-agent-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-inserloft-rockman","domain":"Code","name":"Rockman Benchmark v0.2","released":"2026-09-20","url":"https://github.com/InserLoft/Rockman"}],"document_type":"registry_page","id":"claire_radar:https://github.com/InserLoft/Rockman","source":"claire_radar","source_url":"https://github.com/InserLoft/Rockman","title":"Rockman Benchmark v0.2"},{"benchmarks":[{"benchmark_id":"claire-radar-github-ishant-005-llm-finetuning-benchmark","domain":"Finance","name":"llm-finetuning-benchmark","released":"2026-08-22","url":"https://github.com/Ishant-005/llm-finetuning-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Ishant-005/llm-finetuning-benchmark","source":"claire_radar","source_url":"https://github.com/Ishant-005/llm-finetuning-benchmark","title":"llm-finetuning-benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-json604-silentbug-bench","domain":"Language & Knowledge","name":"silentbug","released":"2026-08-20","url":"https://github.com/Json604/silentbug-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Json604/silentbug-bench","source":"claire_radar","source_url":"https://github.com/Json604/silentbug-bench","title":"silentbug"},{"benchmarks":[{"benchmark_id":"claire-radar-github-majorminorstudio-4b-agent-model-benchmark","domain":"Agents","name":"Small-Model Operational Benchmark","released":"2026-09-19","url":"https://github.com/MAJORminorStudio/4b-agent-model-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/MAJORminorStudio/4b-agent-model-benchmark","source":"claire_radar","source_url":"https://github.com/MAJORminorStudio/4b-agent-model-benchmark","title":"Small-Model Operational Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-majorminorstudio-devin-agent-benchmark","domain":"Agents","name":"Devin + SWE-2 Reasoning Effort Benchmark","released":"2026-09-16","url":"https://github.com/MAJORminorStudio/devin-agent-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/MAJORminorStudio/devin-agent-benchmark","source":"claire_radar","source_url":"https://github.com/MAJORminorStudio/devin-agent-benchmark","title":"Devin + SWE-2 Reasoning Effort Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-mostre-agent-failure-lab","domain":"Agents","name":"Agent Failure Lab","released":"2026-09-18","url":"https://github.com/MOSTRE/agent-failure-lab"}],"document_type":"registry_page","id":"claire_radar:https://github.com/MOSTRE/agent-failure-lab","source":"claire_radar","source_url":"https://github.com/MOSTRE/agent-failure-lab","title":"Agent Failure Lab"},{"benchmarks":[{"benchmark_id":"claire-radar-github-mehkh-analysis-credit-risk-tabpfn-benchmark","domain":"Chip Design & EDA","name":"Credit Risk Benchmark: TabPFN vs. Traditional ML","released":"2026-09-24","url":"https://github.com/MehKh-Analysis/credit-risk-tabpfn-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/MehKh-Analysis/credit-risk-tabpfn-benchmark","source":"claire_radar","source_url":"https://github.com/MehKh-Analysis/credit-risk-tabpfn-benchmark","title":"Credit Risk Benchmark: TabPFN vs. Traditional ML"},{"benchmarks":[{"benchmark_id":"claire-radar-github-mervecaliskann-synthetic-data-benchmark","domain":"Finance","name":"Synthetic Data Generator Benchmark (Credit Risk)","released":"2026-09-07","url":"https://github.com/Mervecaliskann/synthetic-data-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Mervecaliskann/synthetic-data-benchmark","source":"claire_radar","source_url":"https://github.com/Mervecaliskann/synthetic-data-benchmark","title":"Synthetic Data Generator Benchmark (Credit Risk)"},{"benchmarks":[{"benchmark_id":"claire-radar-github-miolonixc-llm-free-benchmark","domain":"General AI","name":"llm-free-benchmark","released":"2026-08-21","url":"https://github.com/Miolonixc/llm-free-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Miolonixc/llm-free-benchmark","source":"claire_radar","source_url":"https://github.com/Miolonixc/llm-free-benchmark","title":"llm-free-benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-netraruntime-trust-me-bro-benchmark","domain":"Language & Knowledge","name":"Trust Me Bro Benchmark","released":"2026-09-27","url":"https://github.com/NetraRuntime/trust-me-bro-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/NetraRuntime/trust-me-bro-benchmark","source":"claire_radar","source_url":"https://github.com/NetraRuntime/trust-me-bro-benchmark","title":"Trust Me Bro Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-parfait18-repo-context-ambiguity","domain":"Language & Knowledge","name":"repo-context-ambiguity","released":"2026-08-23","url":"https://github.com/Parfait18/repo-context-ambiguity"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Parfait18/repo-context-ambiguity","source":"claire_radar","source_url":"https://github.com/Parfait18/repo-context-ambiguity","title":"repo-context-ambiguity"},{"benchmarks":[{"benchmark_id":"claire-radar-github-parth-udawant-distilbert-onnx-quantization-benchmark","domain":"General AI","name":"DistilBERT ONNX Quantization Benchmark","released":"2026-09-06","url":"https://github.com/Parth-Udawant/distilbert-onnx-quantization-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Parth-Udawant/distilbert-onnx-quantization-benchmark","source":"claire_radar","source_url":"https://github.com/Parth-Udawant/distilbert-onnx-quantization-benchmark","title":"DistilBERT ONNX Quantization Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-pesquisadoug-gas-sensor-drift-ml-benchmark","domain":"Language & Knowledge","name":"Gas Sensor Drift ML Benchmark","released":"2026-09-17","url":"https://github.com/PesquisaDoug/gas-sensor-drift-ml-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/PesquisaDoug/gas-sensor-drift-ml-benchmark","source":"claire_radar","source_url":"https://github.com/PesquisaDoug/gas-sensor-drift-ml-benchmark","title":"Gas Sensor Drift ML Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-qrzzzz-pelican-bike-benchmark","domain":"General AI","name":"Pelican Bike Benchmark","released":"2026-09-12","url":"https://github.com/Qrzzzz/pelican-bike-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Qrzzzz/pelican-bike-benchmark","source":"claire_radar","source_url":"https://github.com/Qrzzzz/pelican-bike-benchmark","title":"Pelican Bike Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-rathanpai-variant-pathogenicity-benchmark","domain":"Chip Design & EDA","name":"Quantum-Enhanced Genetic Variant Pathogenicity Prediction","released":"2026-09-16","url":"https://github.com/RathanPai/variant-pathogenicity-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/RathanPai/variant-pathogenicity-benchmark","source":"claire_radar","source_url":"https://github.com/RathanPai/variant-pathogenicity-benchmark","title":"Quantum-Enhanced Genetic Variant Pathogenicity Prediction"},{"benchmarks":[{"benchmark_id":"claire-radar-github-snyderconsulting-delverl","domain":"Language & Knowledge","name":"DelveRL","released":"2026-08-22","url":"https://github.com/SnyderConsulting/DelveRL"}],"document_type":"registry_page","id":"claire_radar:https://github.com/SnyderConsulting/DelveRL","source":"claire_radar","source_url":"https://github.com/SnyderConsulting/DelveRL","title":"DelveRL"},{"benchmarks":[{"benchmark_id":"claire-radar-github-soham0605-benchmarking-mlips","domain":"Language & Knowledge","name":"eos-bench","released":"2026-09-17","url":"https://github.com/Soham0605/Benchmarking_MLIPs"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Soham0605/Benchmarking_MLIPs","source":"claire_radar","source_url":"https://github.com/Soham0605/Benchmarking_MLIPs","title":"eos-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-sver0411-tinyedgebench","domain":"General AI","name":"TinyEdgeBench","released":"2026-09-15","url":"https://github.com/Sver0411/TinyEdgeBench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/Sver0411/TinyEdgeBench","source":"claire_radar","source_url":"https://github.com/Sver0411/TinyEdgeBench","title":"TinyEdgeBench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-uditakhourii-cdaf","domain":"Agents","name":"CDAF — Cached Descriptive Asset Files","released":"2026-08-26","url":"https://github.com/UditAkhourii/cdaf"}],"document_type":"registry_page","id":"claire_radar:https://github.com/UditAkhourii/cdaf","source":"claire_radar","source_url":"https://github.com/UditAkhourii/cdaf","title":"CDAF — Cached Descriptive Asset Files"},{"benchmarks":[{"benchmark_id":"claire-radar-github-vsbdev-little-hermione","domain":"Language & Knowledge","name":"O.W.L. & N.E.W.T. Bench","released":"2026-09-08","url":"https://github.com/VSBDev/little-hermione"}],"document_type":"registry_page","id":"claire_radar:https://github.com/VSBDev/little-hermione","source":"claire_radar","source_url":"https://github.com/VSBDev/little-hermione","title":"O.W.L. & N.E.W.T. Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-abdullahuseyinli-dot-annomi-counselling-dialogue-analysis","domain":"Biology & Drug Discovery","name":"AnnoMI Counselling Dialogue Analysis","released":"2026-08-21","url":"https://github.com/abdullahuseyinli-dot/annomi-counselling-dialogue-analysis"}],"document_type":"registry_page","id":"claire_radar:https://github.com/abdullahuseyinli-dot/annomi-counselling-dialogue-analysis","source":"claire_radar","source_url":"https://github.com/abdullahuseyinli-dot/annomi-counselling-dialogue-analysis","title":"AnnoMI Counselling Dialogue Analysis"},{"benchmarks":[{"benchmark_id":"claire-radar-github-aiast1-rtcbench","domain":"General AI","name":"RTCbench","released":"2026-08-20","url":"https://github.com/aiast1/rtcbench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/aiast1/rtcbench","source":"claire_radar","source_url":"https://github.com/aiast1/rtcbench","title":"RTCbench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-ajbermudezh22-agent-model-bench","domain":"Agents","name":"agent-model-bench","released":"2026-08-21","url":"https://github.com/ajbermudezh22/agent-model-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/ajbermudezh22/agent-model-bench","source":"claire_radar","source_url":"https://github.com/ajbermudezh22/agent-model-bench","title":"agent-model-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-anisha-das-kts-nlp-sentiment-analysis-benchmark","domain":"General AI","name":"NLP Sentiment Analysis Benchmark","released":"2026-09-06","url":"https://github.com/anisha-das-kts/nlp-sentiment-analysis-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/anisha-das-kts/nlp-sentiment-analysis-benchmark","source":"claire_radar","source_url":"https://github.com/anisha-das-kts/nlp-sentiment-analysis-benchmark","title":"NLP Sentiment Analysis Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-ano-kodokushi-rag-doc-qa","domain":"General AI","name":"rag-doc-qa","released":"2026-09-18","url":"https://github.com/ano-kodokushi/rag-doc-qa"}],"document_type":"registry_page","id":"claire_radar:https://github.com/ano-kodokushi/rag-doc-qa","source":"claire_radar","source_url":"https://github.com/ano-kodokushi/rag-doc-qa","title":"rag-doc-qa"},{"benchmarks":[{"benchmark_id":"claire-radar-github-arjunkshah12345-hash-cli-bench","domain":"Agents","name":"CLI-Bench","released":"2026-09-23","url":"https://github.com/arjunkshah12345-hash/cli-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/arjunkshah12345-hash/cli-bench","source":"claire_radar","source_url":"https://github.com/arjunkshah12345-hash/cli-bench","title":"CLI-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-aryanthemscoder-recoveriq","domain":"Finance","name":"recoveriq","released":"2026-08-21","url":"https://github.com/aryanthemscoder/recoveriq"}],"document_type":"registry_page","id":"claire_radar:https://github.com/aryanthemscoder/recoveriq","source":"claire_radar","source_url":"https://github.com/aryanthemscoder/recoveriq","title":"recoveriq"},{"benchmarks":[{"benchmark_id":"claire-radar-github-audn-ai-refusal-benchmark","domain":"Finance","name":"refusal-benchmark","released":"2026-08-22","url":"https://github.com/audn-ai/refusal-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/audn-ai/refusal-benchmark","source":"claire_radar","source_url":"https://github.com/audn-ai/refusal-benchmark","title":"refusal-benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-bagheri365-evallab","domain":"Language & Knowledge","name":"EvalLab","released":"2026-08-22","url":"https://github.com/bagheri365/evallab"}],"document_type":"registry_page","id":"claire_radar:https://github.com/bagheri365/evallab","source":"claire_radar","source_url":"https://github.com/bagheri365/evallab","title":"EvalLab"},{"benchmarks":[{"benchmark_id":"claire-radar-github-basilos-ai-office-agent-benchmark","domain":"Agents","name":"Office Agent Benchmark","released":"2026-09-20","url":"https://github.com/basilos-ai/office-agent-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/basilos-ai/office-agent-benchmark","source":"claire_radar","source_url":"https://github.com/basilos-ai/office-agent-benchmark","title":"Office Agent Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-bedr-ux-amharic-tokenization-benchmark","domain":"General AI","name":"Amharic-Tokenization-Benchmark","released":"2026-09-15","url":"https://github.com/bedr-ux/Amharic-Tokenization-Benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/bedr-ux/Amharic-Tokenization-Benchmark","source":"claire_radar","source_url":"https://github.com/bedr-ux/Amharic-Tokenization-Benchmark","title":"Amharic-Tokenization-Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-bizhanchik-capability-overload","domain":"Agents","name":"Capability Overload Benchmark","released":"2026-09-11","url":"https://github.com/bizhanchik/capability-overload"}],"document_type":"registry_page","id":"claire_radar:https://github.com/bizhanchik/capability-overload","source":"claire_radar","source_url":"https://github.com/bizhanchik/capability-overload","title":"Capability Overload Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-bluebunnyanon-agent-benchmark-2d-maze","domain":"Agents","name":"Agentic Evaluation in 2D Mazes","released":"2026-08-22","url":"https://github.com/bluebunnyanon/agent-benchmark-2d-maze"}],"document_type":"registry_page","id":"claire_radar:https://github.com/bluebunnyanon/agent-benchmark-2d-maze","source":"claire_radar","source_url":"https://github.com/bluebunnyanon/agent-benchmark-2d-maze","title":"Agentic Evaluation in 2D Mazes"},{"benchmarks":[{"benchmark_id":"claire-radar-github-caitlin-leonard-ct-segmentation-benchmark","domain":"Biology & Drug Discovery","name":"Multi-Organ CT Segmentation Benchmark","released":"2026-08-22","url":"https://github.com/caitlin-leonard/ct-segmentation-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/caitlin-leonard/ct-segmentation-benchmark","source":"claire_radar","source_url":"https://github.com/caitlin-leonard/ct-segmentation-benchmark","title":"Multi-Organ CT Segmentation Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-chirindaopensource-from-financial-sentiment-classification-to-return-predictability","domain":"Finance","name":"From Financial Sentiment Classification to Return Predictability","released":"2026-08-22","url":"https://github.com/chirindaopensource/from_financial_sentiment_classification_to_return_predictability"}],"document_type":"registry_page","id":"claire_radar:https://github.com/chirindaopensource/from_financial_sentiment_classification_to_return_predictability","source":"claire_radar","source_url":"https://github.com/chirindaopensource/from_financial_sentiment_classification_to_return_predictability","title":"From Financial Sentiment Classification to Return Predictability"},{"benchmarks":[{"benchmark_id":"claire-radar-github-cl1110goo-afk-oraclephys","domain":"Language & Knowledge","name":"OraclePhys","released":"2026-08-23","url":"https://github.com/cl1110goo-afk/oraclephys"}],"document_type":"registry_page","id":"claire_radar:https://github.com/cl1110goo-afk/oraclephys","source":"claire_radar","source_url":"https://github.com/cl1110goo-afk/oraclephys","title":"OraclePhys"},{"benchmarks":[{"benchmark_id":"claire-radar-github-cmcconnell1-inference-bench","domain":"Language & Knowledge","name":"inference-bench: serving engine benchmarks on Akamai Cloud GPUs","released":"2026-09-27","url":"https://github.com/cmcconnell1/inference-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/cmcconnell1/inference-bench","source":"claire_radar","source_url":"https://github.com/cmcconnell1/inference-bench","title":"inference-bench: serving engine benchmarks on Akamai Cloud GPUs"},{"benchmarks":[{"benchmark_id":"claire-radar-github-cosbi-research-sbmllm-bench","domain":"General AI","name":"SBMLLM-Bench","released":"2026-08-20","url":"https://github.com/cosbi-research/SBMLLM-Bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/cosbi-research/SBMLLM-Bench","source":"claire_radar","source_url":"https://github.com/cosbi-research/SBMLLM-Bench","title":"SBMLLM-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-crispyberry-openbenchmark","domain":"Agents","name":"OpenBenchmark","released":"2026-08-20","url":"https://github.com/crispyberry/OpenBenchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/crispyberry/OpenBenchmark","source":"claire_radar","source_url":"https://github.com/crispyberry/OpenBenchmark","title":"OpenBenchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-datapace-ai-agent-memory-benchmark","domain":"Agents","name":"Agent Memory Benchmark","released":"2026-09-08","url":"https://github.com/datapace-ai/agent-memory-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/datapace-ai/agent-memory-benchmark","source":"claire_radar","source_url":"https://github.com/datapace-ai/agent-memory-benchmark","title":"Agent Memory Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-derrtaderr-strategy-benchmark","domain":"Language & Knowledge","name":"strategy-benchmark","released":"2026-09-07","url":"https://github.com/derrtaderr/strategy-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/derrtaderr/strategy-benchmark","source":"claire_radar","source_url":"https://github.com/derrtaderr/strategy-benchmark","title":"strategy-benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-devissaputra-knowledge-tracing-benchmark","domain":"Factuality","name":"Knowledge Tracing Benchmark on ASSISTments 2009","released":"2026-09-24","url":"https://github.com/devissaputra/knowledge_tracing_benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/devissaputra/knowledge_tracing_benchmark","source":"claire_radar","source_url":"https://github.com/devissaputra/knowledge_tracing_benchmark","title":"Knowledge Tracing Benchmark on ASSISTments 2009"},{"benchmarks":[{"benchmark_id":"claire-radar-github-diovoib-llms-benchmark","domain":"Agents","name":"Local agent benchmark (tools + loop + coding)","released":"2026-09-10","url":"https://github.com/diovoib/llms-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/diovoib/llms-benchmark","source":"claire_radar","source_url":"https://github.com/diovoib/llms-benchmark","title":"Local agent benchmark (tools + loop + coding)"},{"benchmarks":[{"benchmark_id":"claire-radar-github-dkautomation23-agent-hijack-bench","domain":"Agents","name":"agent-hijack-bench","released":"2026-09-20","url":"https://github.com/dkautomation23/agent-hijack-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/dkautomation23/agent-hijack-bench","source":"claire_radar","source_url":"https://github.com/dkautomation23/agent-hijack-bench","title":"agent-hijack-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-drivelineresearch-vast-benchmarking","domain":"General AI","name":"Vast Benchmarking","released":"2026-08-22","url":"https://github.com/drivelineresearch/vast-benchmarking"}],"document_type":"registry_page","id":"claire_radar:https://github.com/drivelineresearch/vast-benchmarking","source":"claire_radar","source_url":"https://github.com/drivelineresearch/vast-benchmarking","title":"Vast Benchmarking"},{"benchmarks":[{"benchmark_id":"claire-radar-github-dustin621408-pixel-token-xray","domain":"Finance","name":"Token X-Ray","released":"2026-09-13","url":"https://github.com/dustin621408-pixel/token-xray"}],"document_type":"registry_page","id":"claire_radar:https://github.com/dustin621408-pixel/token-xray","source":"claire_radar","source_url":"https://github.com/dustin621408-pixel/token-xray","title":"Token X-Ray"},{"benchmarks":[{"benchmark_id":"claire-radar-github-dymoo-code-taste-bench","domain":"General AI","name":"Code Taste Bench","released":"2026-09-22","url":"https://github.com/dymoo/code-taste-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/dymoo/code-taste-bench","source":"claire_radar","source_url":"https://github.com/dymoo/code-taste-bench","title":"Code Taste Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-etsabary-jev-deterministic-benchmark","domain":"Language & Knowledge","name":"JEV-DD-1.0","released":"2026-09-22","url":"https://github.com/etsabary/jev-deterministic-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/etsabary/jev-deterministic-benchmark","source":"claire_radar","source_url":"https://github.com/etsabary/jev-deterministic-benchmark","title":"JEV-DD-1.0"},{"benchmarks":[{"benchmark_id":"claire-radar-github-fahmiwol-migancore","domain":"Geometric reasoning","name":"MiganCore","released":"2026-09-28","url":"https://github.com/fahmiwol/migancore"}],"document_type":"registry_page","id":"claire_radar:https://github.com/fahmiwol/migancore","source":"claire_radar","source_url":"https://github.com/fahmiwol/migancore","title":"MiganCore"},{"benchmarks":[{"benchmark_id":"claire-radar-github-gauthamrohan-financial-sentiment-benchmark","domain":"Finance","name":"financial-sentiment-benchmark","released":"2026-08-21","url":"https://github.com/gauthamRohan/financial-sentiment-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/gauthamRohan/financial-sentiment-benchmark","source":"claire_radar","source_url":"https://github.com/gauthamRohan/financial-sentiment-benchmark","title":"financial-sentiment-benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-gauthamrohan-rag-retrieval-benchmark","domain":"Cybersecurity","name":"rag-retrieval-benchmark","released":"2026-08-23","url":"https://github.com/gauthamRohan/rag-retrieval-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/gauthamRohan/rag-retrieval-benchmark","source":"claire_radar","source_url":"https://github.com/gauthamRohan/rag-retrieval-benchmark","title":"rag-retrieval-benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-gbbarra-llm-summarization-benchmark-ptbr","domain":"Language & Knowledge","name":"LLM Summarization Benchmark — Brazilian Portuguese","released":"2026-08-21","url":"https://github.com/gbbarra/llm-summarization-benchmark-ptbr"}],"document_type":"registry_page","id":"claire_radar:https://github.com/gbbarra/llm-summarization-benchmark-ptbr","source":"claire_radar","source_url":"https://github.com/gbbarra/llm-summarization-benchmark-ptbr","title":"LLM Summarization Benchmark — Brazilian Portuguese"},{"benchmarks":[{"benchmark_id":"claire-radar-github-geodiegodrone-sms-spam-classifier","domain":"Language & Knowledge","name":"sms-spam-classifier","released":"2026-09-22","url":"https://github.com/geodiegodrone/sms-spam-classifier"}],"document_type":"registry_page","id":"claire_radar:https://github.com/geodiegodrone/sms-spam-classifier","source":"claire_radar","source_url":"https://github.com/geodiegodrone/sms-spam-classifier","title":"sms-spam-classifier"},{"benchmarks":[{"benchmark_id":"claire-radar-github-gitayg-agentic-security-benchmark","domain":"Language & Knowledge","name":"Agentic Security Benchmark","released":"2026-09-07","url":"https://github.com/gitayg/agentic-security-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/gitayg/agentic-security-benchmark","source":"claire_radar","source_url":"https://github.com/gitayg/agentic-security-benchmark","title":"Agentic Security Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-greencalculus-greencalculus-benchmark","domain":"General AI","name":"Emission-factor accuracy benchmark","released":"2026-09-10","url":"https://github.com/greencalculus/greencalculus-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/greencalculus/greencalculus-benchmark","source":"claire_radar","source_url":"https://github.com/greencalculus/greencalculus-benchmark","title":"Emission-factor accuracy benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-guptabhishekumar-grit-benchmark","domain":"Robotics","name":"GRIT","released":"2026-08-22","url":"https://github.com/guptabhishekumar/grit-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/guptabhishekumar/grit-benchmark","source":"claire_radar","source_url":"https://github.com/guptabhishekumar/grit-benchmark","title":"GRIT"},{"benchmarks":[{"benchmark_id":"claire-radar-github-hamzawamiqhq-b2b-saas-ai-visibility-benchmark-2026","domain":"Cybersecurity","name":"The B2B SaaS AI Visibility Benchmark 2026","released":"2026-10-13","url":"https://github.com/hamzawamiqhq/b2b-saas-ai-visibility-benchmark-2026"}],"document_type":"registry_page","id":"claire_radar:https://github.com/hamzawamiqhq/b2b-saas-ai-visibility-benchmark-2026","source":"claire_radar","source_url":"https://github.com/hamzawamiqhq/b2b-saas-ai-visibility-benchmark-2026","title":"The B2B SaaS AI Visibility Benchmark 2026"},{"benchmarks":[{"benchmark_id":"claire-radar-github-hemachandra666-silent-misresolution-benchmark","domain":"Biology & Drug Discovery","name":"Silent Misresolution","released":"2026-08-20","url":"https://github.com/hemachandra666/silent-misresolution-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/hemachandra666/silent-misresolution-benchmark","source":"claire_radar","source_url":"https://github.com/hemachandra666/silent-misresolution-benchmark","title":"Silent Misresolution"},{"benchmarks":[{"benchmark_id":"claire-radar-github-hervemottaran-alzheimer-classification-benchmark","domain":"Biology & Drug Discovery","name":"alzheimer-classification-benchmark","released":"2026-08-20","url":"https://github.com/hervemottaran/alzheimer-classification-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/hervemottaran/alzheimer-classification-benchmark","source":"claire_radar","source_url":"https://github.com/hervemottaran/alzheimer-classification-benchmark","title":"alzheimer-classification-benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-humanwill-ai-humanwill-benchmark","domain":"Cybersecurity","name":"HumanWill Benchmark","released":"2026-09-19","url":"https://github.com/humanwill-ai/humanwill-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/humanwill-ai/humanwill-benchmark","source":"claire_radar","source_url":"https://github.com/humanwill-ai/humanwill-benchmark","title":"HumanWill Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-imaginationai31-sys-indian-agent-benchmark","domain":"Agents","name":"Indian Agent Benchmark","released":"2026-09-10","url":"https://github.com/imaginationai31-sys/indian-agent-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/imaginationai31-sys/indian-agent-benchmark","source":"claire_radar","source_url":"https://github.com/imaginationai31-sys/indian-agent-benchmark","title":"Indian Agent Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-jadenguitarman-agent-studio-latency-benchmark","domain":"Agents","name":"Agent Studio latency benchmark","released":"2026-09-23","url":"https://github.com/jadenguitarman/agent-studio-latency-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/jadenguitarman/agent-studio-latency-benchmark","source":"claire_radar","source_url":"https://github.com/jadenguitarman/agent-studio-latency-benchmark","title":"Agent Studio latency benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-james-a-watson-visual-rag-benchmark","domain":"Finance","name":"Visual RAG Benchmark","released":"2026-09-10","url":"https://github.com/james-a-watson/visual-rag-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/james-a-watson/visual-rag-benchmark","source":"claire_radar","source_url":"https://github.com/james-a-watson/visual-rag-benchmark","title":"Visual RAG Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-jbpayton-almanac","domain":"Agents","name":"Almanac","released":"2026-09-25","url":"https://github.com/jbpayton/almanac"}],"document_type":"registry_page","id":"claire_radar:https://github.com/jbpayton/almanac","source":"claire_radar","source_url":"https://github.com/jbpayton/almanac","title":"Almanac"},{"benchmarks":[{"benchmark_id":"claire-radar-github-junru-zhu-failure-transparent-agents","domain":"Language & Knowledge","name":"Failure-Transparent Agents","released":"2026-09-10","url":"https://github.com/junru-zhu/failure-transparent-agents"}],"document_type":"registry_page","id":"claire_radar:https://github.com/junru-zhu/failure-transparent-agents","source":"claire_radar","source_url":"https://github.com/junru-zhu/failure-transparent-agents","title":"Failure-Transparent Agents"},{"benchmarks":[{"benchmark_id":"claire-radar-github-jyotydivya-gpu-accelerated-ml-inference-benchmark","domain":"General AI","name":"GPU-Accelerated ML Inference Benchmark","released":"2026-08-23","url":"https://github.com/jyotydivya/GPU-Accelerated-ML-Inference-Benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/jyotydivya/GPU-Accelerated-ML-Inference-Benchmark","source":"claire_radar","source_url":"https://github.com/jyotydivya/GPU-Accelerated-ML-Inference-Benchmark","title":"GPU-Accelerated ML Inference Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-kadirnar-turkish-llm-benchmark","domain":"Language & Knowledge","name":"Turkish LLM Benchmark","released":"2026-08-22","url":"https://github.com/kadirnar/turkish-llm-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/kadirnar/turkish-llm-benchmark","source":"claire_radar","source_url":"https://github.com/kadirnar/turkish-llm-benchmark","title":"Turkish LLM Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-kadirnar-voice-agent-next","domain":"Agents","name":"voice-agent-next","released":"2026-09-24","url":"https://github.com/kadirnar/voice-agent-next"}],"document_type":"registry_page","id":"claire_radar:https://github.com/kadirnar/voice-agent-next","source":"claire_radar","source_url":"https://github.com/kadirnar/voice-agent-next","title":"voice-agent-next"},{"benchmarks":[{"benchmark_id":"claire-radar-github-kcelestinomaria-overlooked-bench","domain":"General AI","name":"overlooked-bench","released":"2026-09-07","url":"https://github.com/kcelestinomaria/overlooked-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/kcelestinomaria/overlooked-bench","source":"claire_radar","source_url":"https://github.com/kcelestinomaria/overlooked-bench","title":"overlooked-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-kilojoules-election-bias-benchmark","domain":"General AI","name":"Election: issue-contingent bias benchmark","released":"2026-09-26","url":"https://github.com/kilojoules/election-bias-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/kilojoules/election-bias-benchmark","source":"claire_radar","source_url":"https://github.com/kilojoules/election-bias-benchmark","title":"Election: issue-contingent bias benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-kyashrathore-agent-app-benchmark","domain":"Agents","name":"Agent App Benchmark","released":"2026-08-22","url":"https://github.com/kyashrathore/agent-app-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/kyashrathore/agent-app-benchmark","source":"claire_radar","source_url":"https://github.com/kyashrathore/agent-app-benchmark","title":"Agent App Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-liningbest-apitest","domain":"General AI","name":"apitest","released":"2026-09-13","url":"https://github.com/liningbest/apitest"}],"document_type":"registry_page","id":"claire_radar:https://github.com/liningbest/apitest","source":"claire_radar","source_url":"https://github.com/liningbest/apitest","title":"apitest"},{"benchmarks":[{"benchmark_id":"claire-radar-github-luongnv89-dgx-spark-llm-lab","domain":"Language & Knowledge","name":"dgx-spark-llm-lab","released":"2026-08-20","url":"https://github.com/luongnv89/dgx-spark-llm-lab"}],"document_type":"registry_page","id":"claire_radar:https://github.com/luongnv89/dgx-spark-llm-lab","source":"claire_radar","source_url":"https://github.com/luongnv89/dgx-spark-llm-lab","title":"dgx-spark-llm-lab"},{"benchmarks":[{"benchmark_id":"claire-radar-github-martin-beck-agent-systems-benchmark","domain":"Agents","name":"Agent Systems Benchmark (ASB)","released":"2026-09-06","url":"https://github.com/martin-beck/agent-systems-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/martin-beck/agent-systems-benchmark","source":"claire_radar","source_url":"https://github.com/martin-beck/agent-systems-benchmark","title":"Agent Systems Benchmark (ASB)"},{"benchmarks":[{"benchmark_id":"claire-radar-github-master72o-documentvision-multimodal-benchmark","domain":"Biology & Drug Discovery","name":"DocumentVision","released":"2026-10-10","url":"https://github.com/master72o/documentvision-multimodal-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/master72o/documentvision-multimodal-benchmark","source":"claire_radar","source_url":"https://github.com/master72o/documentvision-multimodal-benchmark","title":"DocumentVision"},{"benchmarks":[{"benchmark_id":"claire-radar-github-michailfragkiskos-edgemosaic","domain":"Finance","name":"EdgeMosaic","released":"2026-09-04","url":"https://github.com/michailfragkiskos/EdgeMosaic"}],"document_type":"registry_page","id":"claire_radar:https://github.com/michailfragkiskos/EdgeMosaic","source":"claire_radar","source_url":"https://github.com/michailfragkiskos/EdgeMosaic","title":"EdgeMosaic"},{"benchmarks":[{"benchmark_id":"claire-radar-github-movahedi-ca-strata-bench","domain":"General AI","name":"STRATA-Bench","released":"2026-09-19","url":"https://github.com/movahedi-ca/strata-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/movahedi-ca/strata-bench","source":"claire_radar","source_url":"https://github.com/movahedi-ca/strata-bench","title":"STRATA-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-msavdert-mandate-bench","domain":"Finance","name":"Mandate Bench","released":"2026-08-22","url":"https://github.com/msavdert/mandate-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/msavdert/mandate-bench","source":"claire_radar","source_url":"https://github.com/msavdert/mandate-bench","title":"Mandate Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-mziqudhd92-kaggle-art-benchmark","domain":"Cybersecurity","name":"Attacker-Reachable Sink Triage (ART)","released":"2026-09-24","url":"https://github.com/mziqudhd92/kaggle-art-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/mziqudhd92/kaggle-art-benchmark","source":"claire_radar","source_url":"https://github.com/mziqudhd92/kaggle-art-benchmark","title":"Attacker-Reachable Sink Triage (ART)"},{"benchmarks":[{"benchmark_id":"claire-radar-github-p20030920p-sim2real-algobench","domain":"Robotics","name":"Sim2Real-AlgoBench","released":"2026-09-11","url":"https://github.com/p20030920p/Sim2Real-AlgoBench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/p20030920p/Sim2Real-AlgoBench","source":"claire_radar","source_url":"https://github.com/p20030920p/Sim2Real-AlgoBench","title":"Sim2Real-AlgoBench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-qbalhkim-steam-review-sentiment","domain":"Chip Design & EDA","name":"steam-review-sentiment","released":"2026-09-10","url":"https://github.com/qbalhkim/steam-review-sentiment"}],"document_type":"registry_page","id":"claire_radar:https://github.com/qbalhkim/steam-review-sentiment","source":"claire_radar","source_url":"https://github.com/qbalhkim/steam-review-sentiment","title":"steam-review-sentiment"},{"benchmarks":[{"benchmark_id":"claire-radar-github-rssr25-system-one-bench","domain":"General AI","name":"sys1bench","released":"2026-09-22","url":"https://github.com/rssr25/system-one-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/rssr25/system-one-bench","source":"claire_radar","source_url":"https://github.com/rssr25/system-one-bench","title":"sys1bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-rudycelekli-the-value-engine-benchmark","domain":"Agents & Tool Use","name":"The Value Engine Benchmark (VEB)","released":"2026-08-20","url":"https://github.com/rudycelekli/the-value-engine-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/rudycelekli/the-value-engine-benchmark","source":"claire_radar","source_url":"https://github.com/rudycelekli/the-value-engine-benchmark","title":"The Value Engine Benchmark (VEB)"},{"benchmarks":[{"benchmark_id":"claire-radar-github-sahrasayed3-crypto-clouda-ocr-benchmark","domain":"Language & Knowledge","name":"Clouda OCR Benchmark","released":"2026-09-07","url":"https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark","source":"claire_radar","source_url":"https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark","title":"Clouda OCR Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-sameer-sde-llm-quantization-benchmark","domain":"Language & Knowledge","name":"LLM Quantization Benchmark on Apple Silicon","released":"2026-09-13","url":"https://github.com/sameer-sde/llm-quantization-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/sameer-sde/llm-quantization-benchmark","source":"claire_radar","source_url":"https://github.com/sameer-sde/llm-quantization-benchmark","title":"LLM Quantization Benchmark on Apple Silicon"},{"benchmarks":[{"benchmark_id":"claire-radar-github-senssei-benchrig","domain":"Language & Knowledge","name":"BenchRig","released":"2026-09-18","url":"https://github.com/senssei/benchrig"}],"document_type":"registry_page","id":"claire_radar:https://github.com/senssei/benchrig","source":"claire_radar","source_url":"https://github.com/senssei/benchrig","title":"BenchRig"},{"benchmarks":[{"benchmark_id":"claire-radar-github-sentinelden-agent-injection-bench","domain":"Agents","name":"agent-injection-bench","released":"2026-09-11","url":"https://github.com/sentinelden/agent-injection-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/sentinelden/agent-injection-bench","source":"claire_radar","source_url":"https://github.com/sentinelden/agent-injection-bench","title":"agent-injection-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-shubhraj5575-frontier-agent-benchmark","domain":"Agents","name":"Frontier Agent Benchmark (FAB)","released":"2026-08-22","url":"https://github.com/shubhraj5575/frontier-agent-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/shubhraj5575/frontier-agent-benchmark","source":"claire_radar","source_url":"https://github.com/shubhraj5575/frontier-agent-benchmark","title":"Frontier Agent Benchmark (FAB)"},{"benchmarks":[{"benchmark_id":"claire-radar-github-sipratt-p-coding-agent-bench","domain":"Agents","name":"coding-agent-bench","released":"2026-09-15","url":"https://github.com/sipratt-p/coding-agent-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/sipratt-p/coding-agent-bench","source":"claire_radar","source_url":"https://github.com/sipratt-p/coding-agent-bench","title":"coding-agent-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-srcollins785-samuel-collins-cv-benchmarking","domain":"Software & AI Compute","name":"Samuel_Collins_CV_Benchmarking","released":"2026-09-06","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking"}],"document_type":"registry_page","id":"claire_radar:https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","source":"claire_radar","source_url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","title":"Samuel_Collins_CV_Benchmarking"},{"benchmarks":[{"benchmark_id":"claire-radar-github-starswithboutaina-particle-event-quantum-benchmark","domain":"General AI","name":"Particle Event Classification with Classical and Hybrid Quantum Models: A Reproducible Benchmark","released":"2026-09-08","url":"https://github.com/starswithboutaina/particle-event-quantum-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/starswithboutaina/particle-event-quantum-benchmark","source":"claire_radar","source_url":"https://github.com/starswithboutaina/particle-event-quantum-benchmark","title":"Particle Event Classification with Classical and Hybrid Quantum Models: A Reproducible Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-stateset-nsr-product-use-benchmark","domain":"Agents","name":"StateSet NSR Product Use Benchmark v1","released":"2026-09-26","url":"https://github.com/stateset/nsr-product-use-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/stateset/nsr-product-use-benchmark","source":"claire_radar","source_url":"https://github.com/stateset/nsr-product-use-benchmark","title":"StateSet NSR Product Use Benchmark v1"},{"benchmarks":[{"benchmark_id":"claire-radar-github-supreethbare-decision-model-bench","domain":"General AI","name":"decision-model-bench","released":"2026-09-23","url":"https://github.com/supreethbare/decision-model-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/supreethbare/decision-model-bench","source":"claire_radar","source_url":"https://github.com/supreethbare/decision-model-bench","title":"decision-model-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-suresh4330-edge-model-compression-benchmark","domain":"Factuality","name":"EDGE-BENCH","released":"2026-09-09","url":"https://github.com/suresh4330/edge-model-compression-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/suresh4330/edge-model-compression-benchmark","source":"claire_radar","source_url":"https://github.com/suresh4330/edge-model-compression-benchmark","title":"EDGE-BENCH"},{"benchmarks":[{"benchmark_id":"claire-radar-github-tiago369-cross-engine-humanoid-benchmark","domain":"Language & Knowledge","name":"Cross-Engine Humanoid Benchmark","released":"2026-08-22","url":"https://github.com/tiago369/cross-engine-humanoid-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/tiago369/cross-engine-humanoid-benchmark","source":"claire_radar","source_url":"https://github.com/tiago369/cross-engine-humanoid-benchmark","title":"Cross-Engine Humanoid Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-tim2000s-llm-food-benchmark-2026-09","domain":"Language & Knowledge","name":"Food photograph benchmark, September 2026 rerun","released":"2026-09-21","url":"https://github.com/tim2000s/llm-food-benchmark-2026-09"}],"document_type":"registry_page","id":"claire_radar:https://github.com/tim2000s/llm-food-benchmark-2026-09","source":"claire_radar","source_url":"https://github.com/tim2000s/llm-food-benchmark-2026-09","title":"Food photograph benchmark, September 2026 rerun"},{"benchmarks":[{"benchmark_id":"claire-radar-github-titusblair-agent-arena","domain":"Agents","name":"agent-arena","released":"2026-09-15","url":"https://github.com/titusblair/agent-arena"}],"document_type":"registry_page","id":"claire_radar:https://github.com/titusblair/agent-arena","source":"claire_radar","source_url":"https://github.com/titusblair/agent-arena","title":"agent-arena"},{"benchmarks":[{"benchmark_id":"claire-radar-github-virtuanalytica-virtualv-llm","domain":"Language & Knowledge","name":"VirtualV LLM Testsuite","released":"2026-09-22","url":"https://github.com/virtuanalytica/virtualv_llm"}],"document_type":"registry_page","id":"claire_radar:https://github.com/virtuanalytica/virtualv_llm","source":"claire_radar","source_url":"https://github.com/virtuanalytica/virtualv_llm","title":"VirtualV LLM Testsuite"},{"benchmarks":[{"benchmark_id":"claire-radar-github-vmalguy-rootme-gym","domain":"Language & Knowledge","name":"rootme-gym","released":"2026-09-22","url":"https://github.com/vmalguy/rootme-gym"}],"document_type":"registry_page","id":"claire_radar:https://github.com/vmalguy/rootme-gym","source":"claire_radar","source_url":"https://github.com/vmalguy/rootme-gym","title":"rootme-gym"},{"benchmarks":[{"benchmark_id":"claire-radar-github-vnish-global-vnish-global-operator-routing-benchmark","domain":"General AI","name":"VNISH GLOBAL Multilingual Operator Routing Benchmark","released":"2026-08-22","url":"https://github.com/vnish-global/vnish-global-operator-routing-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/vnish-global/vnish-global-operator-routing-benchmark","source":"claire_radar","source_url":"https://github.com/vnish-global/vnish-global-operator-routing-benchmark","title":"VNISH GLOBAL Multilingual Operator Routing Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-wwewtech-russian-it-community-corpus","domain":"Factuality","name":"Russian IT Community Corpus","released":"2026-08-22","url":"https://github.com/wwewtech/russian-it-community-corpus"}],"document_type":"registry_page","id":"claire_radar:https://github.com/wwewtech/russian-it-community-corpus","source":"claire_radar","source_url":"https://github.com/wwewtech/russian-it-community-corpus","title":"Russian IT Community Corpus"},{"benchmarks":[{"benchmark_id":"claire-radar-github-yanliang-li-hil-safety-benchmark","domain":"Agents","name":"InterveneBench","released":"2026-09-12","url":"https://github.com/yanliang-li/hil-safety-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/yanliang-li/hil-safety-benchmark","source":"claire_radar","source_url":"https://github.com/yanliang-li/hil-safety-benchmark","title":"InterveneBench"},{"benchmarks":[{"benchmark_id":"claire-radar-github-yeyuxin777-baoyan-agent-benchmark","domain":"Agents","name":"Graduate Admissions Agent Benchmark","released":"2026-10-12","url":"https://github.com/yeyuxin777/baoyan-agent-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/yeyuxin777/baoyan-agent-benchmark","source":"claire_radar","source_url":"https://github.com/yeyuxin777/baoyan-agent-benchmark","title":"Graduate Admissions Agent Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-yukevindai-cheme-ml-benchmarks","domain":"General AI","name":"ChemE ML Benchmarks","released":"2026-09-07","url":"https://github.com/yukevindai/cheme-ml-benchmarks"}],"document_type":"registry_page","id":"claire_radar:https://github.com/yukevindai/cheme-ml-benchmarks","source":"claire_radar","source_url":"https://github.com/yukevindai/cheme-ml-benchmarks","title":"ChemE ML Benchmarks"},{"benchmarks":[{"benchmark_id":"claire-radar-github-yunusbaysal-llm-benchmark-panel","domain":"Finance","name":"LLM Benchmark Panel","released":"2026-09-08","url":"https://github.com/yunusbaysal/llm-benchmark-panel"}],"document_type":"registry_page","id":"claire_radar:https://github.com/yunusbaysal/llm-benchmark-panel","source":"claire_radar","source_url":"https://github.com/yunusbaysal/llm-benchmark-panel","title":"LLM Benchmark Panel"},{"benchmarks":[{"benchmark_id":"claire-radar-github-zero-trust-agent-benchmark-zero-trust-agent-benchmark","domain":"Agents","name":"Zero Trust Agent Benchmark","released":"2026-09-26","url":"https://github.com/zero-trust-agent-benchmark/zero-trust-agent-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://github.com/zero-trust-agent-benchmark/zero-trust-agent-benchmark","source":"claire_radar","source_url":"https://github.com/zero-trust-agent-benchmark/zero-trust-agent-benchmark","title":"Zero Trust Agent Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-github-zidaniel123-secagent-bench","domain":"Cybersecurity","name":"secagent-bench","released":"2026-09-22","url":"https://github.com/zidaniel123/secagent-bench"}],"document_type":"registry_page","id":"claire_radar:https://github.com/zidaniel123/secagent-bench","source":"claire_radar","source_url":"https://github.com/zidaniel123/secagent-bench","title":"secagent-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-arcadia-2026-analog-design-bench","domain":"Chip Design & EDA","name":"Analog Design Bench","released":"2026-09-26","url":"https://huggingface.co/datasets/Arcadia-2026/analog-design-bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/Arcadia-2026/analog-design-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/Arcadia-2026/analog-design-bench","title":"Analog Design Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-aymen-lassoued-abacus-bench","domain":"General AI","name":"ABACUS","released":"2026-09-09","url":"https://huggingface.co/datasets/Aymen-Lassoued/ABACUS_Bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/Aymen-Lassoued/ABACUS_Bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/Aymen-Lassoued/ABACUS_Bench","title":"ABACUS"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-eigenformai-groundtruth-dynamic-benchmarking","domain":"General AI","name":"Groundtruth Dynamic Benchmarking","released":"2026-08-22","url":"https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking","source":"claire_radar","source_url":"https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking","title":"Groundtruth Dynamic Benchmarking"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-grobestreet-rcv-bench","domain":"Agents","name":"RCV-Bench","released":"2026-09-28","url":"https://huggingface.co/datasets/GrobeStreet/rcv-bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/GrobeStreet/rcv-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/GrobeStreet/rcv-bench","title":"RCV-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-noeflandre-benchmark-llms-landuse-relevance","domain":"General AI","name":"Land-use relevance benchmark","released":"2026-09-13","url":"https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance","source":"claire_radar","source_url":"https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance","title":"Land-use relevance benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-opentwbench-tw-bench","domain":"General AI","name":"tw-bench","released":"2026-08-22","url":"https://huggingface.co/datasets/OpenTWBench/tw-bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/OpenTWBench/tw-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/OpenTWBench/tw-bench","title":"tw-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-opentwbench-tw-exam-bench","domain":"General AI","name":"tw-exam-bench","released":"2026-08-22","url":"https://huggingface.co/datasets/OpenTWBench/tw-exam-bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/OpenTWBench/tw-exam-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/OpenTWBench/tw-exam-bench","title":"tw-exam-bench"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-oursecurefuture-wps-ai-benchmark-v6","domain":"General AI","name":"WPS AI Benchmark","released":"2026-09-15","url":"https://huggingface.co/datasets/OurSecureFuture/wps-ai-benchmark-v6"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/OurSecureFuture/wps-ai-benchmark-v6","source":"claire_radar","source_url":"https://huggingface.co/datasets/OurSecureFuture/wps-ai-benchmark-v6","title":"WPS AI Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-arca-inc-reg-affairs-bench","domain":"Biology & Drug Discovery","name":"Reg Affairs Bench","released":"2026-10-10","url":"https://huggingface.co/datasets/arca-inc/reg-affairs-bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/arca-inc/reg-affairs-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/arca-inc/reg-affairs-bench","title":"Reg Affairs Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-bryel-labs-musp-bench","domain":"General AI","name":"MuSP-Bench","released":"2026-08-20","url":"https://huggingface.co/datasets/bryel-labs/MuSP-Bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/bryel-labs/MuSP-Bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/bryel-labs/MuSP-Bench","title":"MuSP-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-djalexj-amo-bench-p","domain":"General AI","name":"amo-bench-p","released":"2026-08-20","url":"https://huggingface.co/datasets/djalexj/amo-bench-p"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/djalexj/amo-bench-p","source":"claire_radar","source_url":"https://huggingface.co/datasets/djalexj/amo-bench-p","title":"amo-bench-p"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-lilonghao-mm-contextasr-bench","domain":"General AI","name":"MM-ContextASR Bench","released":"2026-09-15","url":"https://huggingface.co/datasets/lilonghao/MM-ContextASR-Bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/lilonghao/MM-ContextASR-Bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/lilonghao/MM-ContextASR-Bench","title":"MM-ContextASR Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-mondaycake-lifesci-harbor-bench","domain":"Language & Knowledge","name":"LifeSci-Harbor-Bench","released":"2026-09-06","url":"https://huggingface.co/datasets/mondaycake/lifesci-harbor-bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/mondaycake/lifesci-harbor-bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/mondaycake/lifesci-harbor-bench","title":"LifeSci-Harbor-Bench"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-pxzleo-qwen3-8-27b-inference-benchmark-4090","domain":"Language & Knowledge","name":"qwen3.8-27b-inference-benchmark-4090","released":"2026-08-21","url":"https://huggingface.co/datasets/pxzleo/qwen3.8-27b-inference-benchmark-4090"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/pxzleo/qwen3.8-27b-inference-benchmark-4090","source":"claire_radar","source_url":"https://huggingface.co/datasets/pxzleo/qwen3.8-27b-inference-benchmark-4090","title":"qwen3.8-27b-inference-benchmark-4090"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-tryforge-pni-3d-benchmark","domain":"CAD","name":"PNI-3D Multimodal CAD & Geometric Robustness Benchmark","released":"2026-09-10","url":"https://huggingface.co/datasets/tryforge/pni-3d-benchmark"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/tryforge/pni-3d-benchmark","source":"claire_radar","source_url":"https://huggingface.co/datasets/tryforge/pni-3d-benchmark","title":"PNI-3D Multimodal CAD & Geometric Robustness Benchmark"},{"benchmarks":[{"benchmark_id":"claire-radar-huggingface-yulingxi-metric-bench","domain":"General AI","name":"Metric-Bench Test","released":"2026-09-11","url":"https://huggingface.co/datasets/yulingxi/Metric-Bench"}],"document_type":"registry_page","id":"claire_radar:https://huggingface.co/datasets/yulingxi/Metric-Bench","source":"claire_radar","source_url":"https://huggingface.co/datasets/yulingxi/Metric-Bench","title":"Metric-Bench Test"},{"benchmarks":[{"benchmark_id":"claire-radar-jetbrains-kotlin-benchmark-2026","domain":"Code & Software","name":"Kotlin Benchmark","released":"2026-07-08","url":"https://kotlinlang.org/benchmark/"}],"document_type":"registry_page","id":"claire_radar:https://kotlinlang.org/benchmark/","source":"claire_radar","source_url":"https://kotlinlang.org/benchmark/","title":"Kotlin Benchmark"},{"benchmarks":[{"benchmark_id":"llm-stats-aa-briefcase","domain":"productivity","name":"AA-Briefcase","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-briefcase?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aa-briefcase?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-briefcase?top_n=500","title":"AA-Briefcase"},{"benchmarks":[{"benchmark_id":"llm-stats-aa-index","domain":"general","name":"AA-Index","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-index?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aa-index?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-index?top_n=500","title":"AA-Index"},{"benchmarks":[{"benchmark_id":"llm-stats-aa-lcr","domain":"long_context","name":"AA-LCR","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-lcr?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aa-lcr?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-lcr?top_n=500","title":"AA-LCR"},{"benchmarks":[{"benchmark_id":"llm-stats-aa-omniscience-index","domain":"reasoning","name":"AA-Omniscience Index","released":"2025-11-16","url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-omniscience-index?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aa-omniscience-index?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-omniscience-index?top_n=500","title":"AA-Omniscience Index"},{"benchmarks":[{"benchmark_id":"llm-stats-acebench","domain":"reasoning","name":"ACEBench","released":"2025-01-22","url":"https://api.zeroeval.com/leaderboard/benchmarks/acebench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/acebench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/acebench?top_n=500","title":"ACEBench"},{"benchmarks":[{"benchmark_id":"llm-stats-activitynet","domain":"video","name":"ActivityNet","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/activitynet?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/activitynet?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/activitynet?top_n=500","title":"ActivityNet"},{"benchmarks":[{"benchmark_id":"llm-stats-advancedif","domain":"reasoning","name":"AdvancedIF","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/advancedif?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/advancedif?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/advancedif?top_n=500","title":"AdvancedIF"},{"benchmarks":[{"benchmark_id":"llm-stats-aethercode","domain":"reasoning","name":"AetherCode","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aethercode?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aethercode?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aethercode?top_n=500","title":"AetherCode"},{"benchmarks":[{"benchmark_id":"llm-stats-agent-startup-bench","domain":"reasoning","name":"Agent Startup Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/agent-startup-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/agent-startup-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agent-startup-bench?top_n=500","title":"Agent Startup Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-agents-last-exam","domain":"reasoning","name":"Agents' Last Exam","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/agents-last-exam?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/agents-last-exam?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agents-last-exam?top_n=500","title":"Agents' Last Exam"},{"benchmarks":[{"benchmark_id":"llm-stats-agieval","domain":"legal","name":"AGIEval","released":"2023-04-13","url":"https://api.zeroeval.com/leaderboard/benchmarks/agieval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/agieval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/agieval?top_n=500","title":"AGIEval"},{"benchmarks":[{"benchmark_id":"llm-stats-ai2-reasoning-challenge-arc","domain":"reasoning","name":"AI2 Reasoning Challenge (ARC)","released":"2018-03-14","url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2-reasoning-challenge-%28arc%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ai2-reasoning-challenge-%28arc%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2-reasoning-challenge-%28arc%29?top_n=500","title":"AI2 Reasoning Challenge (ARC)"},{"benchmarks":[{"benchmark_id":"llm-stats-ai2d","domain":"multimodal","name":"AI2D","released":"2016-03-24","url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2d?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ai2d?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2d?top_n=500","title":"AI2D"},{"benchmarks":[{"benchmark_id":"llm-stats-aider-polyglot-edit","domain":"general","name":"Aider-Polyglot Edit","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot-edit?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot-edit?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot-edit?top_n=500","title":"Aider-Polyglot Edit"},{"benchmarks":[{"benchmark_id":"llm-stats-aider-polyglot","domain":"general","name":"Aider-Polyglot","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot?top_n=500","title":"Aider-Polyglot"},{"benchmarks":[{"benchmark_id":"llm-stats-aider","domain":"reasoning","name":"Aider","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aider?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aider?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aider?top_n=500","title":"Aider"},{"benchmarks":[{"benchmark_id":"llm-stats-aime-2024","domain":"math","name":"AIME 2024","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2024?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aime-2024?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2024?top_n=500","title":"AIME 2024"},{"benchmarks":[{"benchmark_id":"llm-stats-aime-2025","domain":"math","name":"AIME 2025","released":"2025-02-12","url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2025?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aime-2025?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2025?top_n=500","title":"AIME 2025"},{"benchmarks":[{"benchmark_id":"llm-stats-aime-2026","domain":"math","name":"AIME 2026","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2026?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aime-2026?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2026?top_n=500","title":"AIME 2026"},{"benchmarks":[{"benchmark_id":"llm-stats-aime","domain":"math","name":"AIME","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aime?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aime?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aime?top_n=500","title":"AIME"},{"benchmarks":[{"benchmark_id":"llm-stats-air-bench","domain":"safety","name":"AIR-Bench","released":"2024-07-11","url":"https://api.zeroeval.com/leaderboard/benchmarks/air-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/air-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/air-bench?top_n=500","title":"AIR-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-aitz-em","domain":"multimodal","name":"AITZ_EM","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/aitz-em?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aitz-em?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/aitz-em?top_n=500","title":"AITZ_EM"},{"benchmarks":[{"benchmark_id":"llm-stats-alignbench","domain":"math","name":"AlignBench","released":"2023-11-30","url":"https://api.zeroeval.com/leaderboard/benchmarks/alignbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/alignbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/alignbench?top_n=500","title":"AlignBench"},{"benchmarks":[{"benchmark_id":"llm-stats-alpacaeval-2-0","domain":"reasoning","name":"AlpacaEval 2.0","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/alpacaeval-2.0?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/alpacaeval-2.0?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/alpacaeval-2.0?top_n=500","title":"AlpacaEval 2.0"},{"benchmarks":[{"benchmark_id":"llm-stats-amc-2022-23","domain":"math","name":"AMC_2022_23","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/amc-2022-23?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/amc-2022-23?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/amc-2022-23?top_n=500","title":"AMC_2022_23"},{"benchmarks":[{"benchmark_id":"llm-stats-amo-bench","domain":"math","name":"AMO Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/amo-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/amo-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/amo-bench?top_n=500","title":"AMO Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-android-control-high-em","domain":"multimodal","name":"Android Control High_EM","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-high-em?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/android-control-high-em?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-high-em?top_n=500","title":"Android Control High_EM"},{"benchmarks":[{"benchmark_id":"llm-stats-android-control-low-em","domain":"multimodal","name":"Android Control Low_EM","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-low-em?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/android-control-low-em?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-low-em?top_n=500","title":"Android Control Low_EM"},{"benchmarks":[{"benchmark_id":"llm-stats-androidbench","domain":"agents","name":"AndroidBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/androidbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/androidbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidbench?top_n=500","title":"AndroidBench"},{"benchmarks":[{"benchmark_id":"llm-stats-androidworld-sr","domain":"multimodal","name":"AndroidWorld_SR","released":"2024-05-23","url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld-sr?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/androidworld-sr?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld-sr?top_n=500","title":"AndroidWorld_SR"},{"benchmarks":[{"benchmark_id":"llm-stats-androidworld","domain":"agents","name":"AndroidWorld","released":"2024-05-23","url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/androidworld?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld?top_n=500","title":"AndroidWorld"},{"benchmarks":[{"benchmark_id":"llm-stats-apex-agents","domain":"reasoning","name":"APEX-Agents","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-agents?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/apex-agents?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-agents?top_n=500","title":"APEX-Agents"},{"benchmarks":[{"benchmark_id":"llm-stats-apex-swe","domain":"agents","name":"APEX-SWE","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-swe?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/apex-swe?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-swe?top_n=500","title":"APEX-SWE"},{"benchmarks":[{"benchmark_id":"llm-stats-apex","domain":"reasoning","name":"Apex","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/apex?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/apex?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/apex?top_n=500","title":"Apex"},{"benchmarks":[{"benchmark_id":"llm-stats-api-bank","domain":"reasoning","name":"API-Bank","released":"2023-04-14","url":"https://api.zeroeval.com/leaderboard/benchmarks/api-bank?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/api-bank?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/api-bank?top_n=500","title":"API-Bank"},{"benchmarks":[{"benchmark_id":"llm-stats-arc-agi-3","domain":"reasoning","name":"ARC-AGI-3","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-3?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-3?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-3?top_n=500","title":"ARC-AGI-3"},{"benchmarks":[{"benchmark_id":"llm-stats-arc-agi-v2","domain":"reasoning","name":"ARC-AGI v2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-v2?top_n=500","title":"ARC-AGI v2"},{"benchmarks":[{"benchmark_id":"llm-stats-arc-agi","domain":"reasoning","name":"ARC-AGI","released":"2019-11-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-agi?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi?top_n=500","title":"ARC-AGI"},{"benchmarks":[{"benchmark_id":"llm-stats-arc-c","domain":"reasoning","name":"ARC-C","released":"2018-03-14","url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-c?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-c?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-c?top_n=500","title":"ARC-C"},{"benchmarks":[{"benchmark_id":"llm-stats-arc-e","domain":"reasoning","name":"ARC-E","released":"2018-03-14","url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-e?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-e?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-e?top_n=500","title":"ARC-E"},{"benchmarks":[{"benchmark_id":"llm-stats-arc","domain":"reasoning","name":"Arc","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/arc?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arc?top_n=500","title":"Arc"},{"benchmarks":[{"benchmark_id":"llm-stats-arcagi2","domain":"reasoning","name":"ArcAGI2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/arcagi2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arcagi2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arcagi2?top_n=500","title":"ArcAGI2"},{"benchmarks":[{"benchmark_id":"llm-stats-arena-hard-v2","domain":"reasoning","name":"Arena-Hard v2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arena-hard-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard-v2?top_n=500","title":"Arena-Hard v2"},{"benchmarks":[{"benchmark_id":"llm-stats-arena-hard","domain":"reasoning","name":"Arena Hard","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arena-hard?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard?top_n=500","title":"Arena Hard"},{"benchmarks":[{"benchmark_id":"llm-stats-arkitscenes","domain":"spatial_reasoning","name":"ARKitScenes","released":"2021-11-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/arkitscenes?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arkitscenes?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arkitscenes?top_n=500","title":"ARKitScenes"},{"benchmarks":[{"benchmark_id":"llm-stats-artifacts-bench","domain":"frontend_development","name":"Artifacts Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/artifacts-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/artifacts-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/artifacts-bench?top_n=500","title":"Artifacts Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-artificial-analysis","domain":"general","name":"Artificial Analysis","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/artificial-analysis?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/artificial-analysis?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/artificial-analysis?top_n=500","title":"Artificial Analysis"},{"benchmarks":[{"benchmark_id":"llm-stats-arxivmath","domain":"math","name":"ArXivMath","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/arxivmath?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arxivmath?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/arxivmath?top_n=500","title":"ArXivMath"},{"benchmarks":[{"benchmark_id":"llm-stats-attaq","domain":"safety","name":"AttaQ","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/attaq?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/attaq?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/attaq?top_n=500","title":"AttaQ"},{"benchmarks":[{"benchmark_id":"llm-stats-autologi","domain":"reasoning","name":"AutoLogi","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/autologi?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/autologi?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/autologi?top_n=500","title":"AutoLogi"},{"benchmarks":[{"benchmark_id":"llm-stats-automationbench-aa","domain":"reasoning","name":"AutomationBench-AA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench-aa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/automationbench-aa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench-aa?top_n=500","title":"AutomationBench-AA"},{"benchmarks":[{"benchmark_id":"llm-stats-automationbench","domain":"reasoning","name":"AutomationBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/automationbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench?top_n=500","title":"AutomationBench"},{"benchmarks":[{"benchmark_id":"llm-stats-babyvision","domain":"multimodal","name":"BabyVision","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/babyvision?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/babyvision?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/babyvision?top_n=500","title":"BabyVision"},{"benchmarks":[{"benchmark_id":"llm-stats-bankertoolbench","domain":"finance","name":"BankerToolBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/bankertoolbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bankertoolbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bankertoolbench?top_n=500","title":"BankerToolBench"},{"benchmarks":[{"benchmark_id":"llm-stats-bbh","domain":"math","name":"BBH","released":"2022-10-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/bbh?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bbh?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bbh?top_n=500","title":"BBH"},{"benchmarks":[{"benchmark_id":"llm-stats-bc-vl","domain":"multimodal","name":"BC-VL","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/bc-vl?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bc-vl?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bc-vl?top_n=500","title":"BC-VL"},{"benchmarks":[{"benchmark_id":"llm-stats-beam-128k","domain":"long_context","name":"Beam 128K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/beam-128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/beam-128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/beam-128k?top_n=500","title":"Beam 128K"},{"benchmarks":[{"benchmark_id":"llm-stats-benchcad-with-python-tool","domain":"multimodal","name":"BenchCAD (with Python tool)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad-with-python-tool?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/benchcad-with-python-tool?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad-with-python-tool?top_n=500","title":"BenchCAD (with Python tool)"},{"benchmarks":[{"benchmark_id":"llm-stats-benchcad","domain":"multimodal","name":"BenchCAD","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/benchcad?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad?top_n=500","title":"BenchCAD"},{"benchmarks":[{"benchmark_id":"llm-stats-beyond-aime","domain":"math","name":"Beyond AIME","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/beyond-aime?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/beyond-aime?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/beyond-aime?top_n=500","title":"Beyond AIME"},{"benchmarks":[{"benchmark_id":"llm-stats-bfcl-v2","domain":"reasoning","name":"BFCL v2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v2?top_n=500","title":"BFCL v2"},{"benchmarks":[{"benchmark_id":"llm-stats-bfcl-v3-multiturn","domain":"reasoning","name":"BFCL_v3_MultiTurn","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3-multiturn?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3-multiturn?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3-multiturn?top_n=500","title":"BFCL_v3_MultiTurn"},{"benchmarks":[{"benchmark_id":"llm-stats-bfcl-v3","domain":"reasoning","name":"BFCL-v3","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3?top_n=500","title":"BFCL-v3"},{"benchmarks":[{"benchmark_id":"llm-stats-bfcl-v4","domain":"agents","name":"BFCL-V4","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v4?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v4?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v4?top_n=500","title":"BFCL-V4"},{"benchmarks":[{"benchmark_id":"llm-stats-bfcl","domain":"reasoning","name":"BFCL","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl?top_n=500","title":"BFCL"},{"benchmarks":[{"benchmark_id":"llm-stats-big-bench-audio","domain":"reasoning","name":"Big Bench Audio","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-audio?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-bench-audio?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-audio?top_n=500","title":"Big Bench Audio"},{"benchmarks":[{"benchmark_id":"llm-stats-big-bench-extra-hard","domain":"reasoning","name":"BIG-Bench Extra Hard","released":"2025-02-26","url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-extra-hard?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-bench-extra-hard?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-extra-hard?top_n=500","title":"BIG-Bench Extra Hard"},{"benchmarks":[{"benchmark_id":"llm-stats-big-bench-hard","domain":"math","name":"BIG-Bench Hard","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-hard?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-bench-hard?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-hard?top_n=500","title":"BIG-Bench Hard"},{"benchmarks":[{"benchmark_id":"llm-stats-big-bench","domain":"math","name":"BIG-Bench","released":"2022-06-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench?top_n=500","title":"BIG-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-big-finance-bench","domain":"reasoning","name":"Big Finance Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/big-finance-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-finance-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/big-finance-bench?top_n=500","title":"Big Finance Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-bigcodebench-full","domain":"reasoning","name":"BigCodeBench-Full","released":"2024-06-22","url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-full?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-full?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-full?top_n=500","title":"BigCodeBench-Full"},{"benchmarks":[{"benchmark_id":"llm-stats-bigcodebench-hard","domain":"reasoning","name":"BigCodeBench-Hard","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-hard?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-hard?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-hard?top_n=500","title":"BigCodeBench-Hard"},{"benchmarks":[{"benchmark_id":"llm-stats-bigcodebench","domain":"reasoning","name":"BigCodeBench","released":"2024-06-22","url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench?top_n=500","title":"BigCodeBench"},{"benchmarks":[{"benchmark_id":"llm-stats-biolp-bench","domain":"safety","name":"BioLP-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/biolp-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/biolp-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/biolp-bench?top_n=500","title":"BioLP-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-biomysterybench","domain":"reasoning","name":"BioMysteryBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/biomysterybench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/biomysterybench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/biomysterybench?top_n=500","title":"BioMysteryBench"},{"benchmarks":[{"benchmark_id":"llm-stats-bird-sql-dev","domain":"reasoning","name":"Bird-SQL (dev)","released":"2023-05-04","url":"https://api.zeroeval.com/leaderboard/benchmarks/bird-sql-%28dev%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bird-sql-%28dev%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bird-sql-%28dev%29?top_n=500","title":"Bird-SQL (dev)"},{"benchmarks":[{"benchmark_id":"llm-stats-bixbench","domain":"reasoning","name":"BixBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/bixbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bixbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/bixbench?top_n=500","title":"BixBench"},{"benchmarks":[{"benchmark_id":"llm-stats-blink","domain":"multimodal","name":"BLINK","released":"2024-04-18","url":"https://api.zeroeval.com/leaderboard/benchmarks/blink?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/blink?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/blink?top_n=500","title":"BLINK"},{"benchmarks":[{"benchmark_id":"llm-stats-blueprint-bench-2","domain":"multimodal","name":"Blueprint-Bench 2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/blueprint-bench-2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/blueprint-bench-2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/blueprint-bench-2?top_n=500","title":"Blueprint-Bench 2"},{"benchmarks":[{"benchmark_id":"llm-stats-boolq","domain":"reasoning","name":"BoolQ","released":"2019-05-24","url":"https://api.zeroeval.com/leaderboard/benchmarks/boolq?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/boolq?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/boolq?top_n=500","title":"BoolQ"},{"benchmarks":[{"benchmark_id":"llm-stats-browsecomp-long-128k","domain":"reasoning","name":"BrowseComp Long Context 128k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-128k?top_n=500","title":"BrowseComp Long Context 128k"},{"benchmarks":[{"benchmark_id":"llm-stats-browsecomp-long-256k","domain":"reasoning","name":"BrowseComp Long Context 256k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-256k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-256k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-256k?top_n=500","title":"BrowseComp Long Context 256k"},{"benchmarks":[{"benchmark_id":"llm-stats-browsecomp-vl","domain":"multimodal","name":"BrowseComp-VL","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-vl?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-vl?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-vl?top_n=500","title":"BrowseComp-VL"},{"benchmarks":[{"benchmark_id":"llm-stats-browsecomp-zh","domain":"reasoning","name":"BrowseComp-zh","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-zh?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-zh?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-zh?top_n=500","title":"BrowseComp-zh"},{"benchmarks":[{"benchmark_id":"llm-stats-browsecomp","domain":"reasoning","name":"BrowseComp","released":"2025-04-10","url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp?top_n=500","title":"BrowseComp"},{"benchmarks":[{"benchmark_id":"llm-stats-c-eval","domain":"reasoning","name":"C-Eval","released":"2023-05-15","url":"https://api.zeroeval.com/leaderboard/benchmarks/c-eval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/c-eval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/c-eval?top_n=500","title":"C-Eval"},{"benchmarks":[{"benchmark_id":"llm-stats-capture-the-flag-challenges","domain":"safety","name":"Capture-the-Flag Challenges (Internal)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/capture-the-flag-challenges?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/capture-the-flag-challenges?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/capture-the-flag-challenges?top_n=500","title":"Capture-the-Flag Challenges (Internal)"},{"benchmarks":[{"benchmark_id":"llm-stats-cbnsl","domain":"math","name":"CBNSL","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cbnsl?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cbnsl?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cbnsl?top_n=500","title":"CBNSL"},{"benchmarks":[{"benchmark_id":"llm-stats-cc-bench-v2-backend","domain":"code","name":"CC-Bench-V2 Backend","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-backend?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-backend?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-backend?top_n=500","title":"CC-Bench-V2 Backend"},{"benchmarks":[{"benchmark_id":"llm-stats-cc-bench-v2-frontend","domain":"code","name":"CC-Bench-V2 Frontend","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-frontend?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-frontend?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-frontend?top_n=500","title":"CC-Bench-V2 Frontend"},{"benchmarks":[{"benchmark_id":"llm-stats-cc-bench-v2-repo","domain":"agents","name":"CC-Bench-V2 Repo Exploration","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-repo?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-repo?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-repo?top_n=500","title":"CC-Bench-V2 Repo Exploration"},{"benchmarks":[{"benchmark_id":"llm-stats-cc-ocr","domain":"multimodal","name":"CC-OCR","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-ocr?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cc-ocr?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-ocr?top_n=500","title":"CC-OCR"},{"benchmarks":[{"benchmark_id":"llm-stats-cfeval","domain":"code","name":"CFEval","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cfeval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cfeval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cfeval?top_n=500","title":"CFEval"},{"benchmarks":[{"benchmark_id":"llm-stats-charadessta","domain":"multimodal","name":"CharadesSTA","released":"2017-05-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/charadessta?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/charadessta?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charadessta?top_n=500","title":"CharadesSTA"},{"benchmarks":[{"benchmark_id":"llm-stats-chartmuseum","domain":"multimodal","name":"ChartMuseum","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/chartmuseum?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/chartmuseum?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartmuseum?top_n=500","title":"ChartMuseum"},{"benchmarks":[{"benchmark_id":"llm-stats-chartqa","domain":"multimodal","name":"ChartQA","released":"2022-03-19","url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/chartqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqa?top_n=500","title":"ChartQA"},{"benchmarks":[{"benchmark_id":"llm-stats-chartqapro","domain":"multimodal","name":"ChartQAPro","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqapro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/chartqapro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqapro?top_n=500","title":"ChartQAPro"},{"benchmarks":[{"benchmark_id":"llm-stats-charxiv-d","domain":"multimodal","name":"CharXiv-D","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-d?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/charxiv-d?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-d?top_n=500","title":"CharXiv-D"},{"benchmarks":[{"benchmark_id":"llm-stats-charxiv-r","domain":"multimodal","name":"CharXiv-R","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-r?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/charxiv-r?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-r?top_n=500","title":"CharXiv-R"},{"benchmarks":[{"benchmark_id":"llm-stats-chexpert-cxr","domain":"healthcare","name":"CheXpert CXR","released":"2019-01-21","url":"https://api.zeroeval.com/leaderboard/benchmarks/chexpert-cxr?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/chexpert-cxr?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/chexpert-cxr?top_n=500","title":"CheXpert CXR"},{"benchmarks":[{"benchmark_id":"llm-stats-ci-memories-coverage","domain":"memory","name":"CI Memories Coverage","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-coverage?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-coverage?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-coverage?top_n=500","title":"CI Memories Coverage"},{"benchmarks":[{"benchmark_id":"llm-stats-ci-memories-violation","domain":"memory","name":"CI Memories Violation Rate","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-violation?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-violation?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-violation?top_n=500","title":"CI Memories Violation Rate"},{"benchmarks":[{"benchmark_id":"llm-stats-cl-bench-life","domain":"agents","name":"CL-bench (Life)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench-%28life%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cl-bench-%28life%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench-%28life%29?top_n=500","title":"CL-bench (Life)"},{"benchmarks":[{"benchmark_id":"llm-stats-cl-bench","domain":"agents","name":"CL-bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cl-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench?top_n=500","title":"CL-bench"},{"benchmarks":[{"benchmark_id":"llm-stats-claw-eval-mm","domain":"multimodal","name":"ClawEval-MM","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval-mm?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/claw-eval-mm?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval-mm?top_n=500","title":"ClawEval-MM"},{"benchmarks":[{"benchmark_id":"llm-stats-claw-eval","domain":"agents","name":"Claw-Eval","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/claw-eval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval?top_n=500","title":"Claw-Eval"},{"benchmarks":[{"benchmark_id":"llm-stats-cloningscenarios","domain":"reasoning","name":"CloningScenarios","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cloningscenarios?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cloningscenarios?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cloningscenarios?top_n=500","title":"CloningScenarios"},{"benchmarks":[{"benchmark_id":"llm-stats-cluewsc","domain":"reasoning","name":"CLUEWSC","released":"2020-04-13","url":"https://api.zeroeval.com/leaderboard/benchmarks/cluewsc?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cluewsc?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cluewsc?top_n=500","title":"CLUEWSC"},{"benchmarks":[{"benchmark_id":"llm-stats-cmmlu","domain":"reasoning","name":"CMMLU","released":"2023-06-15","url":"https://api.zeroeval.com/leaderboard/benchmarks/cmmlu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cmmlu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cmmlu?top_n=500","title":"CMMLU"},{"benchmarks":[{"benchmark_id":"llm-stats-cmt-benchmark","domain":"physics","name":"CMT-Benchmark","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cmt-benchmark?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cmt-benchmark?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cmt-benchmark?top_n=500","title":"CMT-Benchmark"},{"benchmarks":[{"benchmark_id":"llm-stats-cnmo-2024","domain":"math","name":"CNMO 2024","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cnmo-2024?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cnmo-2024?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cnmo-2024?top_n=500","title":"CNMO 2024"},{"benchmarks":[{"benchmark_id":"llm-stats-codeforces","domain":"math","name":"CodeForces","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/codeforces?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/codeforces?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/codeforces?top_n=500","title":"CodeForces"},{"benchmarks":[{"benchmark_id":"llm-stats-codegolf-v2-2","domain":"code","name":"Codegolf v2.2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/codegolf-v2.2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/codegolf-v2.2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/codegolf-v2.2?top_n=500","title":"Codegolf v2.2"},{"benchmarks":[{"benchmark_id":"llm-stats-cohere-agentic-question-answering","domain":"question_answering","name":"Cohere Agentic Question Answering","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-agentic-question-answering?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cohere-agentic-question-answering?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-agentic-question-answering?top_n=500","title":"Cohere Agentic Question Answering"},{"benchmarks":[{"benchmark_id":"llm-stats-cohere-data-analysis","domain":"reasoning","name":"Cohere Data Analysis","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-data-analysis?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cohere-data-analysis?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-data-analysis?top_n=500","title":"Cohere Data Analysis"},{"benchmarks":[{"benchmark_id":"llm-stats-cohere-memory-usage-quality","domain":"memory","name":"Cohere Memory Usage Quality","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-memory-usage-quality?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cohere-memory-usage-quality?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-memory-usage-quality?top_n=500","title":"Cohere Memory Usage Quality"},{"benchmarks":[{"benchmark_id":"llm-stats-collie","domain":"reasoning","name":"COLLIE","released":"2023-07-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/collie?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/collie?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/collie?top_n=500","title":"COLLIE"},{"benchmarks":[{"benchmark_id":"llm-stats-common-voice-15","domain":"speech_to_text","name":"Common Voice 15","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/common-voice-15?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/common-voice-15?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/common-voice-15?top_n=500","title":"Common Voice 15"},{"benchmarks":[{"benchmark_id":"llm-stats-commonsenseqa","domain":"reasoning","name":"CommonSenseQA","released":"2018-11-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/commonsenseqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/commonsenseqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/commonsenseqa?top_n=500","title":"CommonSenseQA"},{"benchmarks":[{"benchmark_id":"llm-stats-community-07c9946d-dcf0-4977-a640-a6b1356b4f0b","domain":"general","name":"independence-bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A07c9946d-dcf0-4977-a640-a6b1356b4f0b?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3A07c9946d-dcf0-4977-a640-a6b1356b4f0b?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A07c9946d-dcf0-4977-a640-a6b1356b4f0b?top_n=500","title":"independence-bench"},{"benchmarks":[{"benchmark_id":"llm-stats-community-2256e9c9-b256-4444-b639-7cc3b1855d96","domain":"long_context","name":"nolima","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A2256e9c9-b256-4444-b639-7cc3b1855d96?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3A2256e9c9-b256-4444-b639-7cc3b1855d96?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A2256e9c9-b256-4444-b639-7cc3b1855d96?top_n=500","title":"nolima"},{"benchmarks":[{"benchmark_id":"llm-stats-community-5f95f778-c521-43fa-b80e-6a55465601e3","domain":"other","name":"ael_gate_benchmark_cases_template","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A5f95f778-c521-43fa-b80e-6a55465601e3?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3A5f95f778-c521-43fa-b80e-6a55465601e3?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A5f95f778-c521-43fa-b80e-6a55465601e3?top_n=500","title":"ael_gate_benchmark_cases_template"},{"benchmarks":[{"benchmark_id":"llm-stats-community-64d67847-06bd-423a-923c-c2acfab82281","domain":"other","name":"OptimBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A64d67847-06bd-423a-923c-c2acfab82281?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3A64d67847-06bd-423a-923c-c2acfab82281?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A64d67847-06bd-423a-923c-c2acfab82281?top_n=500","title":"OptimBench"},{"benchmarks":[{"benchmark_id":"llm-stats-community-ed90e889-4678-4fbd-98ab-0e654f4bf35e","domain":"other","name":"atlas","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3Aed90e889-4678-4fbd-98ab-0e654f4bf35e?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3Aed90e889-4678-4fbd-98ab-0e654f4bf35e?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3Aed90e889-4678-4fbd-98ab-0e654f4bf35e?top_n=500","title":"atlas"},{"benchmarks":[{"benchmark_id":"llm-stats-community-fd462fc2-283c-4967-bd7d-b39d7c661807","domain":"other","name":"testing","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3Afd462fc2-283c-4967-bd7d-b39d7c661807?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3Afd462fc2-283c-4967-bd7d-b39d7c661807?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3Afd462fc2-283c-4967-bd7d-b39d7c661807?top_n=500","title":"testing"},{"benchmarks":[{"benchmark_id":"llm-stats-complexfuncbench","domain":"long_context","name":"ComplexFuncBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/complexfuncbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/complexfuncbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/complexfuncbench?top_n=500","title":"ComplexFuncBench"},{"benchmarks":[{"benchmark_id":"llm-stats-contphy","domain":"multimodal","name":"ContPhy","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/contphy?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/contphy?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/contphy?top_n=500","title":"ContPhy"},{"benchmarks":[{"benchmark_id":"llm-stats-corpusqa-1m","domain":"long_context","name":"CorpusQA 1M","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa-1m?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/corpusqa-1m?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa-1m?top_n=500","title":"CorpusQA 1M"},{"benchmarks":[{"benchmark_id":"llm-stats-corpusqa","domain":"long_context","name":"CorpusQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/corpusqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa?top_n=500","title":"CorpusQA"},{"benchmarks":[{"benchmark_id":"llm-stats-countbench","domain":"reasoning","name":"CountBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/countbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/countbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/countbench?top_n=500","title":"CountBench"},{"benchmarks":[{"benchmark_id":"llm-stats-countqa","domain":"multimodal","name":"CountQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/countqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/countqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/countqa?top_n=500","title":"CountQA"},{"benchmarks":[{"benchmark_id":"llm-stats-covost2-en-zh","domain":"speech_to_text","name":"CoVoST2 en-zh","released":"2020-07-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2-en-zh?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/covost2-en-zh?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2-en-zh?top_n=500","title":"CoVoST2 en-zh"},{"benchmarks":[{"benchmark_id":"llm-stats-covost2","domain":"speech_to_text","name":"CoVoST2","released":"2020-07-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/covost2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2?top_n=500","title":"CoVoST2"},{"benchmarks":[{"benchmark_id":"llm-stats-coworkbench","domain":"productivity","name":"CoWorkBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/coworkbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/coworkbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/coworkbench?top_n=500","title":"CoWorkBench"},{"benchmarks":[{"benchmark_id":"llm-stats-crag","domain":"reasoning","name":"CRAG","released":"2024-06-07","url":"https://api.zeroeval.com/leaderboard/benchmarks/crag?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/crag?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crag?top_n=500","title":"CRAG"},{"benchmarks":[{"benchmark_id":"llm-stats-creative-writing-v3","domain":"creativity","name":"Creative Writing v3","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/creative-writing-v3?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/creative-writing-v3?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/creative-writing-v3?top_n=500","title":"Creative Writing v3"},{"benchmarks":[{"benchmark_id":"llm-stats-creativework","domain":"reasoning","name":"CreativeWork","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/creativework?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/creativework?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/creativework?top_n=500","title":"CreativeWork"},{"benchmarks":[{"benchmark_id":"llm-stats-critpt","domain":"math","name":"CritPT","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/critpt?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/critpt?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/critpt?top_n=500","title":"CritPT"},{"benchmarks":[{"benchmark_id":"llm-stats-crossvid","domain":"multimodal","name":"CrossVid","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/crossvid?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/crossvid?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crossvid?top_n=500","title":"CrossVid"},{"benchmarks":[{"benchmark_id":"llm-stats-crperelation","domain":"reasoning","name":"CRPErelation","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/crperelation?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/crperelation?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crperelation?top_n=500","title":"CRPErelation"},{"benchmarks":[{"benchmark_id":"llm-stats-crux-o","domain":"reasoning","name":"CRUX-O","released":"2024-01-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/crux-o?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/crux-o?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/crux-o?top_n=500","title":"CRUX-O"},{"benchmarks":[{"benchmark_id":"llm-stats-cruxeval-input-cot","domain":"reasoning","name":"CRUXEval-Input-CoT","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-input-cot?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-input-cot?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-input-cot?top_n=500","title":"CRUXEval-Input-CoT"},{"benchmarks":[{"benchmark_id":"llm-stats-cruxeval-o","domain":"reasoning","name":"CruxEval-O","released":"2024-01-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-o?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-o?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-o?top_n=500","title":"CruxEval-O"},{"benchmarks":[{"benchmark_id":"llm-stats-cruxeval-output-cot","domain":"reasoning","name":"CRUXEval-Output-CoT","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-output-cot?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-output-cot?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-output-cot?top_n=500","title":"CRUXEval-Output-CoT"},{"benchmarks":[{"benchmark_id":"llm-stats-csimpleqa","domain":"language","name":"CSimpleQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/csimpleqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/csimpleqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/csimpleqa?top_n=500","title":"CSimpleQA"},{"benchmarks":[{"benchmark_id":"llm-stats-cursorbench-3-2","domain":"agents","name":"CursorBench v3.2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cursorbench-3.2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cursorbench-3.2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cursorbench-3.2?top_n=500","title":"CursorBench v3.2"},{"benchmarks":[{"benchmark_id":"llm-stats-cvtg-2k","domain":"image-generation","name":"CVTG-2K","released":"2025-03-30","url":"https://api.zeroeval.com/leaderboard/benchmarks/cvtg-2k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cvtg-2k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cvtg-2k?top_n=500","title":"CVTG-2K"},{"benchmarks":[{"benchmark_id":"llm-stats-cybench","domain":"safety","name":"CyBench","released":"2024-08-15","url":"https://api.zeroeval.com/leaderboard/benchmarks/cybench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cybench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybench?top_n=500","title":"CyBench"},{"benchmarks":[{"benchmark_id":"llm-stats-cybergym","domain":"safety","name":"CyberGym","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cybergym?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cybergym?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybergym?top_n=500","title":"CyberGym"},{"benchmarks":[{"benchmark_id":"llm-stats-cyberseceval-4","domain":"safety","name":"CyberSecEval 4","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cyberseceval-4?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cyberseceval-4?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cyberseceval-4?top_n=500","title":"CyberSecEval 4"},{"benchmarks":[{"benchmark_id":"llm-stats-cybersecurity-ctfs","domain":"safety","name":"Cybersecurity CTFs","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/cybersecurity-ctfs?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cybersecurity-ctfs?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/cybersecurity-ctfs?top_n=500","title":"Cybersecurity CTFs"},{"benchmarks":[{"benchmark_id":"llm-stats-dailyomni","domain":"multimodal","name":"DailyOmni","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/dailyomni?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dailyomni?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dailyomni?top_n=500","title":"DailyOmni"},{"benchmarks":[{"benchmark_id":"llm-stats-deck-bench","domain":"productivity","name":"DECK-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/deck-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deck-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deck-bench?top_n=500","title":"DECK-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-deep-planning","domain":"reasoning","name":"DeepPlanning","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/deep-planning?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deep-planning?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deep-planning?top_n=500","title":"DeepPlanning"},{"benchmarks":[{"benchmark_id":"llm-stats-deepsearchqa","domain":"reasoning","name":"DeepSearchQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/deepsearchqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deepsearchqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepsearchqa?top_n=500","title":"DeepSearchQA"},{"benchmarks":[{"benchmark_id":"llm-stats-deepswe-1-0","domain":"agents","name":"DeepSWE 1.0","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.0?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.0?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.0?top_n=500","title":"DeepSWE 1.0"},{"benchmarks":[{"benchmark_id":"llm-stats-deepswe-1-1","domain":"agents","name":"DeepSWE 1.1","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.1?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.1?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.1?top_n=500","title":"DeepSWE 1.1"},{"benchmarks":[{"benchmark_id":"llm-stats-deepswe","domain":"agents","name":"DeepSWE","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deepswe?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe?top_n=500","title":"DeepSWE"},{"benchmarks":[{"benchmark_id":"llm-stats-dermmcqa","domain":"healthcare","name":"DermMCQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/dermmcqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dermmcqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dermmcqa?top_n=500","title":"DermMCQA"},{"benchmarks":[{"benchmark_id":"llm-stats-design2code","domain":"multimodal","name":"Design2Code","released":"2024-03-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/design2code?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/design2code?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/design2code?top_n=500","title":"Design2Code"},{"benchmarks":[{"benchmark_id":"llm-stats-docvqa","domain":"multimodal","name":"DocVQA","released":"2020-07-01","url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/docvqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqa?top_n=500","title":"DocVQA"},{"benchmarks":[{"benchmark_id":"llm-stats-docvqatest","domain":"multimodal","name":"DocVQAtest","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqatest?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/docvqatest?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqatest?top_n=500","title":"DocVQAtest"},{"benchmarks":[{"benchmark_id":"llm-stats-doubao-multi-turn-bench","domain":"instruction_following","name":"Doubao Multi-Turn Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/doubao-multi-turn-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/doubao-multi-turn-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/doubao-multi-turn-bench?top_n=500","title":"Doubao Multi-Turn Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-draco","domain":"reasoning","name":"DRACO","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/draco?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/draco?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/draco?top_n=500","title":"DRACO"},{"benchmarks":[{"benchmark_id":"llm-stats-drop","domain":"math","name":"DROP","released":"2019-03-01","url":"https://api.zeroeval.com/leaderboard/benchmarks/drop?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/drop?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/drop?top_n=500","title":"DROP"},{"benchmarks":[{"benchmark_id":"llm-stats-ds-arena-code","domain":"reasoning","name":"DS-Arena-Code","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-arena-code?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ds-arena-code?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-arena-code?top_n=500","title":"DS-Arena-Code"},{"benchmarks":[{"benchmark_id":"llm-stats-ds-fim-eval","domain":"general","name":"DS-FIM-Eval","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-fim-eval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ds-fim-eval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-fim-eval?top_n=500","title":"DS-FIM-Eval"},{"benchmarks":[{"benchmark_id":"llm-stats-dsbench-fullstack","domain":"agents","name":"DSBench-FullStack","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-fullstack?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dsbench-fullstack?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-fullstack?top_n=500","title":"DSBench-FullStack"},{"benchmarks":[{"benchmark_id":"llm-stats-dsbench-hard","domain":"agents","name":"DSBench-Hard","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-hard?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dsbench-hard?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-hard?top_n=500","title":"DSBench-Hard"},{"benchmarks":[{"benchmark_id":"llm-stats-dude","domain":"long_context","name":"DUDE","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/dude?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dude?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dude?top_n=500","title":"DUDE"},{"benchmarks":[{"benchmark_id":"llm-stats-dynamath","domain":"math","name":"DynaMath","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/dynamath?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dynamath?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/dynamath?top_n=500","title":"DynaMath"},{"benchmarks":[{"benchmark_id":"llm-stats-eclektic","domain":"reasoning","name":"ECLeKTic","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/eclektic?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/eclektic?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/eclektic?top_n=500","title":"ECLeKTic"},{"benchmarks":[{"benchmark_id":"llm-stats-egoschema","domain":"long_context","name":"EgoSchema","released":"2023-08-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/egoschema?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/egoschema?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/egoschema?top_n=500","title":"EgoSchema"},{"benchmarks":[{"benchmark_id":"llm-stats-embspatialbench","domain":"spatial_reasoning","name":"EmbSpatialBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/embspatialbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/embspatialbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/embspatialbench?top_n=500","title":"EmbSpatialBench"},{"benchmarks":[{"benchmark_id":"llm-stats-emma","domain":"math","name":"EMMA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/emma?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/emma?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/emma?top_n=500","title":"EMMA"},{"benchmarks":[{"benchmark_id":"llm-stats-eq-bench","domain":"reasoning","name":"EQ-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/eq-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/eq-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/eq-bench?top_n=500","title":"EQ-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-erqa","domain":"reasoning","name":"ERQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/erqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/erqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/erqa?top_n=500","title":"ERQA"},{"benchmarks":[{"benchmark_id":"llm-stats-evalplus","domain":"reasoning","name":"EvalPlus","released":"2023-05-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/evalplus?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/evalplus?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/evalplus?top_n=500","title":"EvalPlus"},{"benchmarks":[{"benchmark_id":"llm-stats-exploitbench","domain":"safety","name":"ExploitBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/exploitbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitbench?top_n=500","title":"ExploitBench"},{"benchmarks":[{"benchmark_id":"llm-stats-exploitgym","domain":"safety","name":"ExploitGym","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitgym?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/exploitgym?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitgym?top_n=500","title":"ExploitGym"},{"benchmarks":[{"benchmark_id":"llm-stats-facts-grounding","domain":"reasoning","name":"FACTS Grounding","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/facts-grounding?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/facts-grounding?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/facts-grounding?top_n=500","title":"FACTS Grounding"},{"benchmarks":[{"benchmark_id":"llm-stats-factscore","domain":"reasoning","name":"FActScore","released":"2023-05-23","url":"https://api.zeroeval.com/leaderboard/benchmarks/factscore?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/factscore?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/factscore?top_n=500","title":"FActScore"},{"benchmarks":[{"benchmark_id":"llm-stats-figqa","domain":"safety","name":"FigQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/figqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/figqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/figqa?top_n=500","title":"FigQA"},{"benchmarks":[{"benchmark_id":"llm-stats-finance-agent-v1-1","domain":"reasoning","name":"Finance Agent v1.1","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v1.1?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v1.1?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v1.1?top_n=500","title":"Finance Agent v1.1"},{"benchmarks":[{"benchmark_id":"llm-stats-finance-agent-v2","domain":"reasoning","name":"Finance Agent v2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v2?top_n=500","title":"Finance Agent v2"},{"benchmarks":[{"benchmark_id":"llm-stats-finance-agent","domain":"reasoning","name":"Finance Agent","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finance-agent?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent?top_n=500","title":"Finance Agent"},{"benchmarks":[{"benchmark_id":"llm-stats-finqa","domain":"math","name":"FinQA","released":"2021-09-01","url":"https://api.zeroeval.com/leaderboard/benchmarks/finqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finqa?top_n=500","title":"FinQA"},{"benchmarks":[{"benchmark_id":"llm-stats-finsearchcomp-t2-t3","domain":"reasoning","name":"FinSearchComp T2&T3","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t2-t3?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t2-t3?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t2-t3?top_n=500","title":"FinSearchComp T2&T3"},{"benchmarks":[{"benchmark_id":"llm-stats-finsearchcomp-t3","domain":"reasoning","name":"FinSearchComp-T3","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t3?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t3?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t3?top_n=500","title":"FinSearchComp-T3"},{"benchmarks":[{"benchmark_id":"llm-stats-flame-vlm-code","domain":"multimodal","name":"Flame-VLM-Code","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/flame-vlm-code?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/flame-vlm-code?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/flame-vlm-code?top_n=500","title":"Flame-VLM-Code"},{"benchmarks":[{"benchmark_id":"llm-stats-flenqa","domain":"long_context","name":"FlenQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/flenqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/flenqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/flenqa?top_n=500","title":"FlenQA"},{"benchmarks":[{"benchmark_id":"llm-stats-fleurs","domain":"speech_to_text","name":"FLEURS","released":"2022-05-25","url":"https://api.zeroeval.com/leaderboard/benchmarks/fleurs?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/fleurs?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fleurs?top_n=500","title":"FLEURS"},{"benchmarks":[{"benchmark_id":"llm-stats-frames","domain":"reasoning","name":"FRAMES","released":"2024-09-19","url":"https://api.zeroeval.com/leaderboard/benchmarks/frames?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frames?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frames?top_n=500","title":"FRAMES"},{"benchmarks":[{"benchmark_id":"llm-stats-french-mmlu","domain":"legal","name":"French MMLU","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/french-mmlu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/french-mmlu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/french-mmlu?top_n=500","title":"French MMLU"},{"benchmarks":[{"benchmark_id":"llm-stats-frontier-bench-v0-1","domain":"agents","name":"Frontier-Bench v0.1","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-bench-v0.1?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontier-bench-v0.1?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-bench-v0.1?top_n=500","title":"Frontier-Bench v0.1"},{"benchmarks":[{"benchmark_id":"llm-stats-frontier-science","domain":"reasoning","name":"Frontier Science","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-science?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontier-science?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-science?top_n=500","title":"Frontier Science"},{"benchmarks":[{"benchmark_id":"llm-stats-frontier-swe-impl","domain":"agents","name":"FrontierSWE (Impl.)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-swe-impl?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontier-swe-impl?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-swe-impl?top_n=500","title":"FrontierSWE (Impl.)"},{"benchmarks":[{"benchmark_id":"llm-stats-frontiercode-1-1","domain":"reasoning","name":"FrontierCode 1.1","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode-1.1?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiercode-1.1?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode-1.1?top_n=500","title":"FrontierCode 1.1"},{"benchmarks":[{"benchmark_id":"llm-stats-frontiercode","domain":"reasoning","name":"FrontierCode","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiercode?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode?top_n=500","title":"FrontierCode"},{"benchmarks":[{"benchmark_id":"llm-stats-frontiercs","domain":"reasoning","name":"FrontierCS","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercs?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiercs?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercs?top_n=500","title":"FrontierCS"},{"benchmarks":[{"benchmark_id":"llm-stats-frontiermath-tier-4-v2","domain":"math","name":"FrontierMath Tier 4 (v2)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath-tier-4-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiermath-tier-4-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath-tier-4-v2?top_n=500","title":"FrontierMath Tier 4 (v2)"},{"benchmarks":[{"benchmark_id":"llm-stats-frontiermath","domain":"math","name":"FrontierMath","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiermath?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath?top_n=500","title":"FrontierMath"},{"benchmarks":[{"benchmark_id":"llm-stats-frontierscience-olympiad","domain":"math","name":"FrontierScience Olympiad","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-olympiad?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-olympiad?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-olympiad?top_n=500","title":"FrontierScience Olympiad"},{"benchmarks":[{"benchmark_id":"llm-stats-frontierscience-research","domain":"reasoning","name":"FrontierScience Research","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-research?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-research?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-research?top_n=500","title":"FrontierScience Research"},{"benchmarks":[{"benchmark_id":"llm-stats-frontierswe","domain":"agents","name":"FrontierSWE","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierswe?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontierswe?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierswe?top_n=500","title":"FrontierSWE"},{"benchmarks":[{"benchmark_id":"llm-stats-fullstackbench-en","domain":"reasoning","name":"FullStackBench en","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-en?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-en?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-en?top_n=500","title":"FullStackBench en"},{"benchmarks":[{"benchmark_id":"llm-stats-fullstackbench-zh","domain":"reasoning","name":"FullStackBench zh","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-zh?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-zh?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-zh?top_n=500","title":"FullStackBench zh"},{"benchmarks":[{"benchmark_id":"llm-stats-functionalmath","domain":"math","name":"FunctionalMATH","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/functionalmath?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/functionalmath?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/functionalmath?top_n=500","title":"FunctionalMATH"},{"benchmarks":[{"benchmark_id":"llm-stats-gaia2","domain":"reasoning","name":"GAIA2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/gaia2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gaia2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gaia2?top_n=500","title":"GAIA2"},{"benchmarks":[{"benchmark_id":"llm-stats-gameworld","domain":"multimodal","name":"GameWorld","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/gameworld?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gameworld?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gameworld?top_n=500","title":"GameWorld"},{"benchmarks":[{"benchmark_id":"llm-stats-gdp-pdf","domain":"multimodal","name":"GDP.pdf","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/gdp-pdf?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdp-pdf?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdp-pdf?top_n=500","title":"GDP.pdf"},{"benchmarks":[{"benchmark_id":"llm-stats-gdpval-aa","domain":"legal","name":"GDPval-AA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-aa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdpval-aa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-aa?top_n=500","title":"GDPval-AA"},{"benchmarks":[{"benchmark_id":"llm-stats-gdpval-mm","domain":"multimodal","name":"GDPval-MM","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-mm?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdpval-mm?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-mm?top_n=500","title":"GDPval-MM"},{"benchmarks":[{"benchmark_id":"llm-stats-gdpval-rubrics","domain":"legal","name":"GDPval-Rubrics","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-rubrics?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdpval-rubrics?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-rubrics?top_n=500","title":"GDPval-Rubrics"},{"benchmarks":[{"benchmark_id":"llm-stats-gdpval","domain":"legal","name":"GDPval","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdpval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval?top_n=500","title":"GDPval"},{"benchmarks":[{"benchmark_id":"llm-stats-genebench-pro","domain":"reasoning","name":"GeneBench-Pro","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench-pro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/genebench-pro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench-pro?top_n=500","title":"GeneBench-Pro"},{"benchmarks":[{"benchmark_id":"llm-stats-genebench","domain":"reasoning","name":"GeneBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/genebench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench?top_n=500","title":"GeneBench"},{"benchmarks":[{"benchmark_id":"llm-stats-giantsteps-tempo","domain":"audio","name":"GiantSteps Tempo","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/giantsteps-tempo?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/giantsteps-tempo?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/giantsteps-tempo?top_n=500","title":"GiantSteps Tempo"},{"benchmarks":[{"benchmark_id":"llm-stats-global-mmlu-lite","domain":"reasoning","name":"Global-MMLU-Lite","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu-lite?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu-lite?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu-lite?top_n=500","title":"Global-MMLU-Lite"},{"benchmarks":[{"benchmark_id":"llm-stats-global-mmlu","domain":"reasoning","name":"Global-MMLU","released":"2024-12-04","url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu?top_n=500","title":"Global-MMLU"},{"benchmarks":[{"benchmark_id":"llm-stats-global-piqa","domain":"physics","name":"Global PIQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/global-piqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/global-piqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/global-piqa?top_n=500","title":"Global PIQA"},{"benchmarks":[{"benchmark_id":"llm-stats-gorilla-benchmark-api-bench","domain":"reasoning","name":"Gorilla Benchmark API Bench","released":"2023-05-24","url":"https://api.zeroeval.com/leaderboard/benchmarks/gorilla-benchmark-api-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gorilla-benchmark-api-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gorilla-benchmark-api-bench?top_n=500","title":"Gorilla Benchmark API Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-govreport","domain":"long_context","name":"GovReport","released":"2021-04-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/govreport?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/govreport?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/govreport?top_n=500","title":"GovReport"},{"benchmarks":[{"benchmark_id":"llm-stats-gpqa-biology","domain":"reasoning","name":"GPQA Biology","released":"2023-11-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-biology?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gpqa-biology?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-biology?top_n=500","title":"GPQA Biology"},{"benchmarks":[{"benchmark_id":"llm-stats-gpqa-chemistry","domain":"reasoning","name":"GPQA Chemistry","released":"2023-11-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-chemistry?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gpqa-chemistry?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-chemistry?top_n=500","title":"GPQA Chemistry"},{"benchmarks":[{"benchmark_id":"llm-stats-gpqa-physics","domain":"physics","name":"GPQA Physics","released":"2023-11-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-physics?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gpqa-physics?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-physics?top_n=500","title":"GPQA Physics"},{"benchmarks":[{"benchmark_id":"llm-stats-gpqa","domain":"physics","name":"GPQA","released":"2023-11-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gpqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa?top_n=500","title":"GPQA"},{"benchmarks":[{"benchmark_id":"llm-stats-graphwalks-bfs-128k","domain":"reasoning","name":"Graphwalks BFS <128k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3C128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3C128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3C128k?top_n=500","title":"Graphwalks BFS <128k"},{"benchmarks":[{"benchmark_id":"llm-stats-graphwalks-bfs-128k-2","domain":"long_context","name":"Graphwalks BFS >128k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3E128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3E128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3E128k?top_n=500","title":"Graphwalks BFS >128k"},{"benchmarks":[{"benchmark_id":"llm-stats-graphwalks-bfs-1m","domain":"long_context","name":"Graphwalks BFS 1M","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-1m?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-1m?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-1m?top_n=500","title":"Graphwalks BFS 1M"},{"benchmarks":[{"benchmark_id":"llm-stats-graphwalks-parents-128k","domain":"reasoning","name":"Graphwalks parents <128k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3C128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3C128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3C128k?top_n=500","title":"Graphwalks parents <128k"},{"benchmarks":[{"benchmark_id":"llm-stats-graphwalks-parents-128k-2","domain":"long_context","name":"Graphwalks parents >128k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3E128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3E128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3E128k?top_n=500","title":"Graphwalks parents >128k"},{"benchmarks":[{"benchmark_id":"llm-stats-graphwalks","domain":"long_context","name":"GraphWalks","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks?top_n=500","title":"GraphWalks"},{"benchmarks":[{"benchmark_id":"llm-stats-groundui-1k","domain":"multimodal","name":"GroundUI-1K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/groundui-1k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/groundui-1k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/groundui-1k?top_n=500","title":"GroundUI-1K"},{"benchmarks":[{"benchmark_id":"llm-stats-gsm-8k-cot","domain":"math","name":"GSM-8K (CoT)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm-8k-%28cot%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gsm-8k-%28cot%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm-8k-%28cot%29?top_n=500","title":"GSM-8K (CoT)"},{"benchmarks":[{"benchmark_id":"llm-stats-gsm8k-chat","domain":"math","name":"GSM8K Chat","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k-chat?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gsm8k-chat?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k-chat?top_n=500","title":"GSM8K Chat"},{"benchmarks":[{"benchmark_id":"llm-stats-gsm8k","domain":"math","name":"GSM8k","released":"2021-04-01","url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gsm8k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k?top_n=500","title":"GSM8k"},{"benchmarks":[{"benchmark_id":"llm-stats-hallusion-bench","domain":"reasoning","name":"Hallusion Bench","released":"2023-10-23","url":"https://api.zeroeval.com/leaderboard/benchmarks/hallusion-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hallusion-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hallusion-bench?top_n=500","title":"Hallusion Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-harvey-lab-aa","domain":"knowledge","name":"Harvey LAB-AA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab-aa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab-aa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab-aa?top_n=500","title":"Harvey LAB-AA"},{"benchmarks":[{"benchmark_id":"llm-stats-harvey-lab","domain":"knowledge","name":"Harvey LAB (Vals)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab?top_n=500","title":"Harvey LAB (Vals)"},{"benchmarks":[{"benchmark_id":"llm-stats-healthbench-consensus","domain":"healthcare","name":"HealthBench Consensus","released":"2025-05-12","url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-consensus?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/healthbench-consensus?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-consensus?top_n=500","title":"HealthBench Consensus"},{"benchmarks":[{"benchmark_id":"llm-stats-healthbench-hard","domain":"healthcare","name":"HealthBench Hard","released":"2025-05-12","url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-hard?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/healthbench-hard?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-hard?top_n=500","title":"HealthBench Hard"},{"benchmarks":[{"benchmark_id":"llm-stats-healthbench-professional","domain":"healthcare","name":"HealthBench Professional","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-professional?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/healthbench-professional?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-professional?top_n=500","title":"HealthBench Professional"},{"benchmarks":[{"benchmark_id":"llm-stats-healthbench","domain":"healthcare","name":"HealthBench","released":"2025-05-12","url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/healthbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench?top_n=500","title":"HealthBench"},{"benchmarks":[{"benchmark_id":"llm-stats-hellaswag","domain":"reasoning","name":"HellaSwag","released":"2019-05-19","url":"https://api.zeroeval.com/leaderboard/benchmarks/hellaswag?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hellaswag?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hellaswag?top_n=500","title":"HellaSwag"},{"benchmarks":[{"benchmark_id":"llm-stats-hiddenmath","domain":"math","name":"HiddenMath","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/hiddenmath?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hiddenmath?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hiddenmath?top_n=500","title":"HiddenMath"},{"benchmarks":[{"benchmark_id":"llm-stats-hipho","domain":"multimodal","name":"HiPhO","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/hipho?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hipho?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hipho?top_n=500","title":"HiPhO"},{"benchmarks":[{"benchmark_id":"llm-stats-hle-verified","domain":"reasoning","name":"HLE-Verified","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/hle-verified?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hle-verified?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hle-verified?top_n=500","title":"HLE-Verified"},{"benchmarks":[{"benchmark_id":"llm-stats-hmmt-2025","domain":"math","name":"HMMT 2025","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-2025?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hmmt-2025?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-2025?top_n=500","title":"HMMT 2025"},{"benchmarks":[{"benchmark_id":"llm-stats-hmmt-feb-26","domain":"math","name":"HMMT Feb 26","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-feb-26?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hmmt-feb-26?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-feb-26?top_n=500","title":"HMMT Feb 26"},{"benchmarks":[{"benchmark_id":"llm-stats-hmmt25","domain":"math","name":"HMMT25","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt25?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hmmt25?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt25?top_n=500","title":"HMMT25"},{"benchmarks":[{"benchmark_id":"llm-stats-horizonmath","domain":"math","name":"HorizonMath","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/horizonmath?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/horizonmath?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/horizonmath?top_n=500","title":"HorizonMath"},{"benchmarks":[{"benchmark_id":"llm-stats-hr-bench-4k","domain":"multimodal","name":"HR-Bench (4k)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/hr-bench-4k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hr-bench-4k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hr-bench-4k?top_n=500","title":"HR-Bench (4k)"},{"benchmarks":[{"benchmark_id":"llm-stats-humaneval-2","domain":"reasoning","name":"HumanEval+","released":"2023-05-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval%2B?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval%2B?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval%2B?top_n=500","title":"HumanEval+"},{"benchmarks":[{"benchmark_id":"llm-stats-humaneval-average","domain":"reasoning","name":"HumanEval-Average","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-average?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval-average?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-average?top_n=500","title":"HumanEval-Average"},{"benchmarks":[{"benchmark_id":"llm-stats-humaneval-er","domain":"reasoning","name":"HumanEval-ER","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-er?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval-er?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-er?top_n=500","title":"HumanEval-ER"},{"benchmarks":[{"benchmark_id":"llm-stats-humaneval-mul","domain":"reasoning","name":"HumanEval-Mul","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-mul?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval-mul?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-mul?top_n=500","title":"HumanEval-Mul"},{"benchmarks":[{"benchmark_id":"llm-stats-humaneval-plus","domain":"reasoning","name":"HumanEval Plus","released":"2023-05-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-plus?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval-plus?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-plus?top_n=500","title":"HumanEval Plus"},{"benchmarks":[{"benchmark_id":"llm-stats-humaneval","domain":"reasoning","name":"HumanEval","released":"2021-07-08","url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval?top_n=500","title":"HumanEval"},{"benchmarks":[{"benchmark_id":"llm-stats-humanevalfim-average","domain":"general","name":"HumanEvalFIM-Average","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/humanevalfim-average?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humanevalfim-average?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanevalfim-average?top_n=500","title":"HumanEvalFIM-Average"},{"benchmarks":[{"benchmark_id":"llm-stats-humanity-s-last-exam-no-tools-text-only","domain":"math","name":"Humanity's Last Exam (no tools, text-only)","released":"2025-01-24","url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28no-tools%2C-text-only%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28no-tools%2C-text-only%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28no-tools%2C-text-only%29?top_n=500","title":"Humanity's Last Exam (no tools, text-only)"},{"benchmarks":[{"benchmark_id":"llm-stats-humanity-s-last-exam-with-tools-text-only","domain":"math","name":"Humanity's Last Exam (with tools, text-only)","released":"2025-01-24","url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28with-tools%2C-text-only%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28with-tools%2C-text-only%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28with-tools%2C-text-only%29?top_n=500","title":"Humanity's Last Exam (with tools, text-only)"},{"benchmarks":[{"benchmark_id":"llm-stats-humanity-s-last-exam","domain":"math","name":"Humanity's Last Exam","released":"2025-01-24","url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam?top_n=500","title":"Humanity's Last Exam"},{"benchmarks":[{"benchmark_id":"llm-stats-hypersim","domain":"spatial_reasoning","name":"Hypersim","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/hypersim?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hypersim?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/hypersim?top_n=500","title":"Hypersim"},{"benchmarks":[{"benchmark_id":"llm-stats-if","domain":"structured_output","name":"IF","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/if?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/if?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/if?top_n=500","title":"IF"},{"benchmarks":[{"benchmark_id":"llm-stats-ifbench","domain":"instruction_following","name":"IFBench","released":"2025-07-03","url":"https://api.zeroeval.com/leaderboard/benchmarks/ifbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ifbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ifbench?top_n=500","title":"IFBench"},{"benchmarks":[{"benchmark_id":"llm-stats-ifeval","domain":"structured_output","name":"IFEval","released":"2023-11-14","url":"https://api.zeroeval.com/leaderboard/benchmarks/ifeval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ifeval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ifeval?top_n=500","title":"IFEval"},{"benchmarks":[{"benchmark_id":"llm-stats-image2floorplan","domain":"multimodal","name":"Image2FloorPlan","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/image2floorplan?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/image2floorplan?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/image2floorplan?top_n=500","title":"Image2FloorPlan"},{"benchmarks":[{"benchmark_id":"llm-stats-imagemining","domain":"multimodal","name":"ImageMining","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/imagemining?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/imagemining?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imagemining?top_n=500","title":"ImageMining"},{"benchmarks":[{"benchmark_id":"llm-stats-imo-2025","domain":"math","name":"IMO 2025","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-2025?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/imo-2025?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-2025?top_n=500","title":"IMO 2025"},{"benchmarks":[{"benchmark_id":"llm-stats-imo-answerbench","domain":"math","name":"IMO-AnswerBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-answerbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/imo-answerbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-answerbench?top_n=500","title":"IMO-AnswerBench"},{"benchmarks":[{"benchmark_id":"llm-stats-imoproof-adv","domain":"math","name":"IMOProof-Adv","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/imoproof-adv?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/imoproof-adv?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/imoproof-adv?top_n=500","title":"IMOProof-Adv"},{"benchmarks":[{"benchmark_id":"llm-stats-include","domain":"general","name":"Include","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/include?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/include?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/include?top_n=500","title":"Include"},{"benchmarks":[{"benchmark_id":"llm-stats-infinitebench-en-mc","domain":"long_context","name":"InfiniteBench/En.MC","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.mc?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.mc?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.mc?top_n=500","title":"InfiniteBench/En.MC"},{"benchmarks":[{"benchmark_id":"llm-stats-infinitebench-en-qa","domain":"long_context","name":"InfiniteBench/En.QA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.qa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.qa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.qa?top_n=500","title":"InfiniteBench/En.QA"},{"benchmarks":[{"benchmark_id":"llm-stats-infographicsqa","domain":"multimodal","name":"InfographicsQA","released":"2021-04-26","url":"https://api.zeroeval.com/leaderboard/benchmarks/infographicsqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infographicsqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infographicsqa?top_n=500","title":"InfographicsQA"},{"benchmarks":[{"benchmark_id":"llm-stats-infovqa","domain":"multimodal","name":"InfoVQA","released":"2021-04-26","url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infovqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqa?top_n=500","title":"InfoVQA"},{"benchmarks":[{"benchmark_id":"llm-stats-infovqatest","domain":"multimodal","name":"InfoVQAtest","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqatest?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infovqatest?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqatest?top_n=500","title":"InfoVQAtest"},{"benchmarks":[{"benchmark_id":"llm-stats-instruct-humaneval","domain":"general","name":"Instruct HumanEval","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/instruct-humaneval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/instruct-humaneval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/instruct-humaneval?top_n=500","title":"Instruct HumanEval"},{"benchmarks":[{"benchmark_id":"llm-stats-intergps","domain":"math","name":"InterGPS","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/intergps?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/intergps?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/intergps?top_n=500","title":"InterGPS"},{"benchmarks":[{"benchmark_id":"llm-stats-internal-api-instruction-following-hard","domain":"structured_output","name":"Internal API instruction following (hard)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-api-instruction-following-%28hard%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/internal-api-instruction-following-%28hard%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-api-instruction-following-%28hard%29?top_n=500","title":"Internal API instruction following (hard)"},{"benchmarks":[{"benchmark_id":"llm-stats-internal-research-debugging-evaluation","domain":"reasoning","name":"Internal Research Debugging Evaluation","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-research-debugging-evaluation?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/internal-research-debugging-evaluation?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-research-debugging-evaluation?top_n=500","title":"Internal Research Debugging Evaluation"},{"benchmarks":[{"benchmark_id":"llm-stats-ipho-2025","domain":"physics","name":"IPhO 2025","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ipho-2025?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ipho-2025?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ipho-2025?top_n=500","title":"IPhO 2025"},{"benchmarks":[{"benchmark_id":"llm-stats-job-bench","domain":"productivity","name":"Job Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/job-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/job-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/job-bench?top_n=500","title":"Job Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-kernel-bench-l3","domain":"agents","name":"Kernel Bench L3","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/kernel-bench-l3?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kernel-bench-l3?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernel-bench-l3?top_n=500","title":"Kernel Bench L3"},{"benchmarks":[{"benchmark_id":"llm-stats-kernelbench-hard","domain":"agents","name":"KernelBench Hard","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelbench-hard?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kernelbench-hard?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelbench-hard?top_n=500","title":"KernelBench Hard"},{"benchmarks":[{"benchmark_id":"llm-stats-kernelgen-1p","domain":"agents","name":"KernelGen 1P","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelgen-1p?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kernelgen-1p?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelgen-1p?top_n=500","title":"KernelGen 1P"},{"benchmarks":[{"benchmark_id":"llm-stats-kimi-claw-24-7-bench","domain":"agents","name":"Kimi Claw 24/7 Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-claw-24-7-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kimi-claw-24-7-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-claw-24-7-bench?top_n=500","title":"Kimi Claw 24/7 Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-kimi-code-bench-v2","domain":"agents","name":"Kimi Code Bench v2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-code-bench-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kimi-code-bench-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-code-bench-v2?top_n=500","title":"Kimi Code Bench v2"},{"benchmarks":[{"benchmark_id":"llm-stats-kina","domain":"reasoning","name":"KINA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/kina?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kina?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/kina?top_n=500","title":"KINA"},{"benchmarks":[{"benchmark_id":"llm-stats-labbench2","domain":"reasoning","name":"LABBench2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/labbench2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/labbench2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/labbench2?top_n=500","title":"LABBench2"},{"benchmarks":[{"benchmark_id":"llm-stats-lbpp-v2","domain":"reasoning","name":"LBPP (v2)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/lbpp-%28v2%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lbpp-%28v2%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lbpp-%28v2%29?top_n=500","title":"LBPP (v2)"},{"benchmarks":[{"benchmark_id":"llm-stats-legal-agent-benchmark","domain":"legal","name":"Legal Agent Benchmark","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/legal-agent-benchmark?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/legal-agent-benchmark?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/legal-agent-benchmark?top_n=500","title":"Legal Agent Benchmark"},{"benchmarks":[{"benchmark_id":"llm-stats-lifescibench","domain":"multimodal","name":"LifeSciBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/lifescibench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lifescibench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lifescibench?top_n=500","title":"LifeSciBench"},{"benchmarks":[{"benchmark_id":"llm-stats-lingoqa","domain":"multimodal","name":"LingoQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/lingoqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lingoqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lingoqa?top_n=500","title":"LingoQA"},{"benchmarks":[{"benchmark_id":"llm-stats-livebench-20241125","domain":"math","name":"LiveBench 20241125","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench-20241125?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livebench-20241125?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench-20241125?top_n=500","title":"LiveBench 20241125"},{"benchmarks":[{"benchmark_id":"llm-stats-livebench","domain":"math","name":"LiveBench","released":"2024-06-12","url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livebench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench?top_n=500","title":"LiveBench"},{"benchmarks":[{"benchmark_id":"llm-stats-livecodebench-01-09","domain":"reasoning","name":"LiveCodeBench(01-09)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench%2801-09%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench%2801-09%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench%2801-09%29?top_n=500","title":"LiveCodeBench(01-09)"},{"benchmarks":[{"benchmark_id":"llm-stats-livecodebench-pro","domain":"reasoning","name":"LiveCodeBench Pro","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-pro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-pro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-pro?top_n=500","title":"LiveCodeBench Pro"},{"benchmarks":[{"benchmark_id":"llm-stats-livecodebench-v5-24-12-25-2","domain":"reasoning","name":"LiveCodeBench v5 24.12-25.2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5-24.12-25.2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5-24.12-25.2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5-24.12-25.2?top_n=500","title":"LiveCodeBench v5 24.12-25.2"},{"benchmarks":[{"benchmark_id":"llm-stats-livecodebench-v5","domain":"reasoning","name":"LiveCodeBench v5","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5?top_n=500","title":"LiveCodeBench v5"},{"benchmarks":[{"benchmark_id":"llm-stats-livecodebench-v6","domain":"reasoning","name":"LiveCodeBench v6","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v6?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v6?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v6?top_n=500","title":"LiveCodeBench v6"},{"benchmarks":[{"benchmark_id":"llm-stats-livecodebench","domain":"reasoning","name":"LiveCodeBench","released":"2024-03-12","url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench?top_n=500","title":"LiveCodeBench"},{"benchmarks":[{"benchmark_id":"llm-stats-livemathematicianbench","domain":"math","name":"LiveMathematicianBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/livemathematicianbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livemathematicianbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livemathematicianbench?top_n=500","title":"LiveMathematicianBench"},{"benchmarks":[{"benchmark_id":"llm-stats-livesports-3k","domain":"multimodal","name":"LiveSports-3K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/livesports-3k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livesports-3k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livesports-3k?top_n=500","title":"LiveSports-3K"},{"benchmarks":[{"benchmark_id":"llm-stats-livesqlbench","domain":"agents","name":"LiveSQLBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/livesqlbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livesqlbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/livesqlbench?top_n=500","title":"LiveSQLBench"},{"benchmarks":[{"benchmark_id":"llm-stats-lmarena-text","domain":"reasoning","name":"LMArena Text Leaderboard","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/lmarena-text?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lmarena-text?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lmarena-text?top_n=500","title":"LMArena Text Leaderboard"},{"benchmarks":[{"benchmark_id":"llm-stats-loca-bench-256k","domain":"reasoning","name":"LOCA-Bench (256k)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/loca-bench-256k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/loca-bench-256k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/loca-bench-256k?top_n=500","title":"LOCA-Bench (256k)"},{"benchmarks":[{"benchmark_id":"llm-stats-longbench-v2","domain":"long_context","name":"LongBench v2","released":"2024-12-19","url":"https://api.zeroeval.com/leaderboard/benchmarks/longbench-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longbench-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longbench-v2?top_n=500","title":"LongBench v2"},{"benchmarks":[{"benchmark_id":"llm-stats-longcodebench","domain":"long_context","name":"LongCodeBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/longcodebench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longcodebench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longcodebench?top_n=500","title":"LongCodeBench"},{"benchmarks":[{"benchmark_id":"llm-stats-longfact-concepts","domain":"reasoning","name":"LongFact Concepts","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-concepts?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longfact-concepts?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-concepts?top_n=500","title":"LongFact Concepts"},{"benchmarks":[{"benchmark_id":"llm-stats-longfact-objects","domain":"reasoning","name":"LongFact Objects","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-objects?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longfact-objects?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-objects?top_n=500","title":"LongFact Objects"},{"benchmarks":[{"benchmark_id":"llm-stats-longfact","domain":"factuality","name":"LongFact","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longfact?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact?top_n=500","title":"LongFact"},{"benchmarks":[{"benchmark_id":"llm-stats-longtext-bench","domain":"image-generation","name":"LongText-Bench","released":"2025-07-29","url":"https://api.zeroeval.com/leaderboard/benchmarks/longtext-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longtext-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longtext-bench?top_n=500","title":"LongText-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-longvideobench","domain":"long_context","name":"LongVideoBench","released":"2024-07-22","url":"https://api.zeroeval.com/leaderboard/benchmarks/longvideobench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longvideobench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/longvideobench?top_n=500","title":"LongVideoBench"},{"benchmarks":[{"benchmark_id":"llm-stats-lsat","domain":"legal","name":"LSAT","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/lsat?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lsat?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lsat?top_n=500","title":"LSAT"},{"benchmarks":[{"benchmark_id":"llm-stats-lvbench","domain":"long_context","name":"LVBench","released":"2024-06-12","url":"https://api.zeroeval.com/leaderboard/benchmarks/lvbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lvbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/lvbench?top_n=500","title":"LVBench"},{"benchmarks":[{"benchmark_id":"llm-stats-management-consulting-tasks","domain":"reasoning","name":"Management Consulting Tasks (Internal)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/management-consulting-tasks?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/management-consulting-tasks?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/management-consulting-tasks?top_n=500","title":"Management Consulting Tasks (Internal)"},{"benchmarks":[{"benchmark_id":"llm-stats-mask","domain":"reasoning","name":"MASK","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mask?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mask?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mask?top_n=500","title":"MASK"},{"benchmarks":[{"benchmark_id":"llm-stats-math-cot","domain":"math","name":"MATH (CoT)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/math-%28cot%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/math-%28cot%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math-%28cot%29?top_n=500","title":"MATH (CoT)"},{"benchmarks":[{"benchmark_id":"llm-stats-math-500","domain":"math","name":"MATH-500","released":"2023-05-31","url":"https://api.zeroeval.com/leaderboard/benchmarks/math-500?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/math-500?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math-500?top_n=500","title":"MATH-500"},{"benchmarks":[{"benchmark_id":"llm-stats-math","domain":"math","name":"MATH","released":"2021-11-08","url":"https://api.zeroeval.com/leaderboard/benchmarks/math?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/math?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/math?top_n=500","title":"MATH"},{"benchmarks":[{"benchmark_id":"llm-stats-matharena-apex","domain":"math","name":"MathArena Apex","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/matharena-apex?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/matharena-apex?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/matharena-apex?top_n=500","title":"MathArena Apex"},{"benchmarks":[{"benchmark_id":"llm-stats-mathverse-mini","domain":"math","name":"MathVerse-Mini","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse-mini?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathverse-mini?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse-mini?top_n=500","title":"MathVerse-Mini"},{"benchmarks":[{"benchmark_id":"llm-stats-mathverse","domain":"math","name":"MathVerse","released":"2024-03-21","url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathverse?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse?top_n=500","title":"MathVerse"},{"benchmarks":[{"benchmark_id":"llm-stats-mathvision","domain":"math","name":"MathVision","released":"2024-02-22","url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvision?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathvision?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvision?top_n=500","title":"MathVision"},{"benchmarks":[{"benchmark_id":"llm-stats-mathvista-mini","domain":"math","name":"MathVista-Mini","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista-mini?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathvista-mini?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista-mini?top_n=500","title":"MathVista-Mini"},{"benchmarks":[{"benchmark_id":"llm-stats-mathvista","domain":"math","name":"MathVista","released":"2023-10-03","url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathvista?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista?top_n=500","title":"MathVista"},{"benchmarks":[{"benchmark_id":"llm-stats-maverix","domain":"multimodal","name":"MAVERIX","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/maverix?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/maverix?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/maverix?top_n=500","title":"MAVERIX"},{"benchmarks":[{"benchmark_id":"llm-stats-maxife","domain":"general","name":"MAXIFE","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/maxife?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/maxife?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/maxife?top_n=500","title":"MAXIFE"},{"benchmarks":[{"benchmark_id":"llm-stats-mbpp-2","domain":"reasoning","name":"MBPP+","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp%2B?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp%2B?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp%2B?top_n=500","title":"MBPP+"},{"benchmarks":[{"benchmark_id":"llm-stats-mbpp-base-version","domain":"reasoning","name":"MBPP ++ base version","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-%2B%2B-base-version?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-%2B%2B-base-version?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-%2B%2B-base-version?top_n=500","title":"MBPP ++ base version"},{"benchmarks":[{"benchmark_id":"llm-stats-mbpp-evalplus-base","domain":"reasoning","name":"MBPP EvalPlus (base)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus-%28base%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus-%28base%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus-%28base%29?top_n=500","title":"MBPP EvalPlus (base)"},{"benchmarks":[{"benchmark_id":"llm-stats-mbpp-evalplus","domain":"reasoning","name":"MBPP EvalPlus","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus?top_n=500","title":"MBPP EvalPlus"},{"benchmarks":[{"benchmark_id":"llm-stats-mbpp-pass-1","domain":"reasoning","name":"MBPP pass@1","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-pass%401?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-pass%401?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-pass%401?top_n=500","title":"MBPP pass@1"},{"benchmarks":[{"benchmark_id":"llm-stats-mbpp-plus","domain":"reasoning","name":"MBPP Plus","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-plus?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-plus?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-plus?top_n=500","title":"MBPP Plus"},{"benchmarks":[{"benchmark_id":"llm-stats-mbpp","domain":"reasoning","name":"MBPP","released":"2021-08-16","url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp?top_n=500","title":"MBPP"},{"benchmarks":[{"benchmark_id":"llm-stats-mcp-atlas","domain":"reasoning","name":"MCP Atlas","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-atlas?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mcp-atlas?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-atlas?top_n=500","title":"MCP Atlas"},{"benchmarks":[{"benchmark_id":"llm-stats-mcp-mark","domain":"agents","name":"MCP-Mark","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-mark?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mcp-mark?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-mark?top_n=500","title":"MCP-Mark"},{"benchmarks":[{"benchmark_id":"llm-stats-mcp-universe","domain":"agents","name":"MCP-Universe","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-universe?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mcp-universe?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-universe?top_n=500","title":"MCP-Universe"},{"benchmarks":[{"benchmark_id":"llm-stats-measurebench","domain":"multimodal","name":"MeasureBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/measurebench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/measurebench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/measurebench?top_n=500","title":"MeasureBench"},{"benchmarks":[{"benchmark_id":"llm-stats-medchembench","domain":"reasoning","name":"MedChemBench (Internal)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/medchembench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/medchembench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medchembench?top_n=500","title":"MedChemBench (Internal)"},{"benchmarks":[{"benchmark_id":"llm-stats-medxpertqa-mm","domain":"medical","name":"MedXpertQA-MM","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa-mm?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa-mm?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa-mm?top_n=500","title":"MedXpertQA-MM"},{"benchmarks":[{"benchmark_id":"llm-stats-medxpertqa","domain":"multimodal","name":"MedXpertQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa?top_n=500","title":"MedXpertQA"},{"benchmarks":[{"benchmark_id":"llm-stats-mega-mlqa","domain":"reasoning","name":"MEGA MLQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-mlqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-mlqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-mlqa?top_n=500","title":"MEGA MLQA"},{"benchmarks":[{"benchmark_id":"llm-stats-mega-tydi-qa","domain":"reasoning","name":"MEGA TyDi QA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-tydi-qa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-tydi-qa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-tydi-qa?top_n=500","title":"MEGA TyDi QA"},{"benchmarks":[{"benchmark_id":"llm-stats-mega-udpos","domain":"language","name":"MEGA UDPOS","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-udpos?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-udpos?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-udpos?top_n=500","title":"MEGA UDPOS"},{"benchmarks":[{"benchmark_id":"llm-stats-mega-xcopa","domain":"reasoning","name":"MEGA XCOPA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xcopa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-xcopa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xcopa?top_n=500","title":"MEGA XCOPA"},{"benchmarks":[{"benchmark_id":"llm-stats-mega-xstorycloze","domain":"reasoning","name":"MEGA XStoryCloze","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xstorycloze?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-xstorycloze?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xstorycloze?top_n=500","title":"MEGA XStoryCloze"},{"benchmarks":[{"benchmark_id":"llm-stats-meld","domain":"multimodal","name":"Meld","released":"2018-10-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/meld?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/meld?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/meld?top_n=500","title":"Meld"},{"benchmarks":[{"benchmark_id":"llm-stats-meta-internal-coding-bench","domain":"agents","name":"Meta Internal Coding Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/meta-internal-coding-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/meta-internal-coding-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/meta-internal-coding-bench?top_n=500","title":"Meta Internal Coding Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-mewc","domain":"reasoning","name":"MEWC","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mewc?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mewc?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mewc?top_n=500","title":"MEWC"},{"benchmarks":[{"benchmark_id":"llm-stats-mgsm","domain":"math","name":"MGSM","released":"2022-10-06","url":"https://api.zeroeval.com/leaderboard/benchmarks/mgsm?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mgsm?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mgsm?top_n=500","title":"MGSM"},{"benchmarks":[{"benchmark_id":"llm-stats-miabench","domain":"multimodal","name":"MIABench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/miabench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/miabench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/miabench?top_n=500","title":"MIABench"},{"benchmarks":[{"benchmark_id":"llm-stats-mimic-cxr","domain":"multimodal","name":"MIMIC CXR","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mimic-cxr?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mimic-cxr?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mimic-cxr?top_n=500","title":"MIMIC CXR"},{"benchmarks":[{"benchmark_id":"llm-stats-mimo-coding-bench","domain":"agents","name":"MiMo Coding Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mimo-coding-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mimo-coding-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mimo-coding-bench?top_n=500","title":"MiMo Coding Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-minerva","domain":"multimodal","name":"Minerva","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/minerva?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/minerva?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/minerva?top_n=500","title":"Minerva"},{"benchmarks":[{"benchmark_id":"llm-stats-mle-bench-lite","domain":"agents","name":"MLE-Bench Lite","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench-lite?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mle-bench-lite?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench-lite?top_n=500","title":"MLE-Bench Lite"},{"benchmarks":[{"benchmark_id":"llm-stats-mle-bench","domain":"agents","name":"MLE-Bench","released":"2024-10-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mle-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench?top_n=500","title":"MLE-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-mls-bench-lite","domain":"reasoning","name":"MLS-Bench Lite","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mls-bench-lite?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mls-bench-lite?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mls-bench-lite?top_n=500","title":"MLS-Bench Lite"},{"benchmarks":[{"benchmark_id":"llm-stats-mlvu-m","domain":"general","name":"MLVU-M","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu-m?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mlvu-m?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu-m?top_n=500","title":"MLVU-M"},{"benchmarks":[{"benchmark_id":"llm-stats-mlvu","domain":"long_context","name":"MLVU","released":"2024-06-06","url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mlvu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu?top_n=500","title":"MLVU"},{"benchmarks":[{"benchmark_id":"llm-stats-mm-browsercomp","domain":"multimodal","name":"MM-BrowserComp","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-browsercomp?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-browsercomp?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-browsercomp?top_n=500","title":"MM-BrowserComp"},{"benchmarks":[{"benchmark_id":"llm-stats-mm-clawbench","domain":"agents","name":"MM-ClawBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-clawbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-clawbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-clawbench?top_n=500","title":"MM-ClawBench"},{"benchmarks":[{"benchmark_id":"llm-stats-mm-if-eval","domain":"multimodal","name":"MM IF-Eval","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-if-eval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-if-eval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-if-eval?top_n=500","title":"MM IF-Eval"},{"benchmarks":[{"benchmark_id":"llm-stats-mm-mind2web","domain":"multimodal","name":"MM-Mind2Web","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mind2web?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-mind2web?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mind2web?top_n=500","title":"MM-Mind2Web"},{"benchmarks":[{"benchmark_id":"llm-stats-mm-mt-bench","domain":"multimodal","name":"MM-MT-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mt-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-mt-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mt-bench?top_n=500","title":"MM-MT-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-mmau-music","domain":"multimodal","name":"MMAU Music","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-music?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmau-music?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-music?top_n=500","title":"MMAU Music"},{"benchmarks":[{"benchmark_id":"llm-stats-mmau-sound","domain":"multimodal","name":"MMAU Sound","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-sound?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmau-sound?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-sound?top_n=500","title":"MMAU Sound"},{"benchmarks":[{"benchmark_id":"llm-stats-mmau-speech","domain":"multimodal","name":"MMAU Speech","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-speech?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmau-speech?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-speech?top_n=500","title":"MMAU Speech"},{"benchmarks":[{"benchmark_id":"llm-stats-mmau","domain":"multimodal","name":"MMAU","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmau?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau?top_n=500","title":"MMAU"},{"benchmarks":[{"benchmark_id":"llm-stats-mmbc","domain":"multimodal","name":"MMBC","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbc?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmbc?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbc?top_n=500","title":"MMBC"},{"benchmarks":[{"benchmark_id":"llm-stats-mmbench-v1-1","domain":"multimodal","name":"MMBench-V1.1","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-v1.1?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmbench-v1.1?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-v1.1?top_n=500","title":"MMBench-V1.1"},{"benchmarks":[{"benchmark_id":"llm-stats-mmbench-video","domain":"multimodal","name":"MMBench-Video","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-video?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmbench-video?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-video?top_n=500","title":"MMBench-Video"},{"benchmarks":[{"benchmark_id":"llm-stats-mmbench","domain":"multimodal","name":"MMBench","released":"2023-07-12","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench?top_n=500","title":"MMBench"},{"benchmarks":[{"benchmark_id":"llm-stats-mme-realworld","domain":"multimodal","name":"MME-RealWorld","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mme-realworld?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mme-realworld?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mme-realworld?top_n=500","title":"MME-RealWorld"},{"benchmarks":[{"benchmark_id":"llm-stats-mme","domain":"multimodal","name":"MME","released":"2023-06-23","url":"https://api.zeroeval.com/leaderboard/benchmarks/mme?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mme?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mme?top_n=500","title":"MME"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlongbench-128k","domain":"long_context","name":"MMLongBench-128K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-128k?top_n=500","title":"MMLongBench-128K"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlongbench-doc","domain":"long_context","name":"MMLongBench-Doc","released":"2024-07-01","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-doc?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-doc?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-doc?top_n=500","title":"MMLongBench-Doc"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu-cot","domain":"legal","name":"MMLU (CoT)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-%28cot%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-%28cot%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-%28cot%29?top_n=500","title":"MMLU (CoT)"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu-base","domain":"legal","name":"MMLU-Base","released":"2020-09-07","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-base?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-base?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-base?top_n=500","title":"MMLU-Base"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu-chat","domain":"legal","name":"MMLU Chat","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-chat?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-chat?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-chat?top_n=500","title":"MMLU Chat"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu-french","domain":"legal","name":"MMLU French","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-french?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-french?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-french?top_n=500","title":"MMLU French"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu-pro","domain":"legal","name":"MMLU-Pro","released":"2024-06-03","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-pro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-pro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-pro?top_n=500","title":"MMLU-Pro"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu-prox","domain":"legal","name":"MMLU-ProX","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-prox?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-prox?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-prox?top_n=500","title":"MMLU-ProX"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu-redux-2-0","domain":"math","name":"MMLU-redux-2.0","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux-2.0?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux-2.0?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux-2.0?top_n=500","title":"MMLU-redux-2.0"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu-redux","domain":"math","name":"MMLU-Redux","released":"2024-06-06","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux?top_n=500","title":"MMLU-Redux"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu-stem","domain":"math","name":"MMLU-STEM","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-stem?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-stem?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-stem?top_n=500","title":"MMLU-STEM"},{"benchmarks":[{"benchmark_id":"llm-stats-mmlu","domain":"legal","name":"MMLU","released":"2020-09-07","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu?top_n=500","title":"MMLU"},{"benchmarks":[{"benchmark_id":"llm-stats-mmmlu","domain":"math","name":"MMMLU","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmlu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmlu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmlu?top_n=500","title":"MMMLU"},{"benchmarks":[{"benchmark_id":"llm-stats-mmmu-val","domain":"multimodal","name":"MMMU (val)","released":"2023-11-27","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28val%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28val%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28val%29?top_n=500","title":"MMMU (val)"},{"benchmarks":[{"benchmark_id":"llm-stats-mmmu-validation","domain":"multimodal","name":"MMMU (validation)","released":"2023-11-27","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28validation%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28validation%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28validation%29?top_n=500","title":"MMMU (validation)"},{"benchmarks":[{"benchmark_id":"llm-stats-mmmu-pro-with-tools","domain":"multimodal","name":"MMMU-Pro (with tools)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro-with-tools?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro-with-tools?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro-with-tools?top_n=500","title":"MMMU-Pro (with tools)"},{"benchmarks":[{"benchmark_id":"llm-stats-mmmu-pro","domain":"multimodal","name":"MMMU-Pro","released":"2024-09-04","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro?top_n=500","title":"MMMU-Pro"},{"benchmarks":[{"benchmark_id":"llm-stats-mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu?top_n=500","title":"MMMU"},{"benchmarks":[{"benchmark_id":"llm-stats-mmmuval","domain":"multimodal","name":"MMMUval","released":"2023-11-27","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmuval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmuval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmuval?top_n=500","title":"MMMUval"},{"benchmarks":[{"benchmark_id":"llm-stats-mmsearch-plus","domain":"multimodal","name":"MMSearch-Plus","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch-plus?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmsearch-plus?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch-plus?top_n=500","title":"MMSearch-Plus"},{"benchmarks":[{"benchmark_id":"llm-stats-mmsearch","domain":"multimodal","name":"MMSearch","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmsearch?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch?top_n=500","title":"MMSearch"},{"benchmarks":[{"benchmark_id":"llm-stats-mmsibench","domain":"multimodal","name":"MMSIBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsibench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmsibench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsibench?top_n=500","title":"MMSIBench"},{"benchmarks":[{"benchmark_id":"llm-stats-mmstar","domain":"multimodal","name":"MMStar","released":"2024-04-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmstar?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmstar?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmstar?top_n=500","title":"MMStar"},{"benchmarks":[{"benchmark_id":"llm-stats-mmt-bench","domain":"multimodal","name":"MMT-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmt-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmt-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmt-bench?top_n=500","title":"MMT-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-mmvet","domain":"math","name":"MMVet","released":"2023-08-04","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvet?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmvet?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvet?top_n=500","title":"MMVet"},{"benchmarks":[{"benchmark_id":"llm-stats-mmvetgpt4turbo","domain":"math","name":"MMVetGPT4Turbo","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvetgpt4turbo?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmvetgpt4turbo?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvetgpt4turbo?top_n=500","title":"MMVetGPT4Turbo"},{"benchmarks":[{"benchmark_id":"llm-stats-mmvu","domain":"multimodal","name":"MMVU","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmvu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvu?top_n=500","title":"MMVU"},{"benchmarks":[{"benchmark_id":"llm-stats-mobileminiwob-sr","domain":"multimodal","name":"MobileMiniWob++_SR","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileminiwob%2B%2B-sr?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mobileminiwob%2B%2B-sr?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileminiwob%2B%2B-sr?top_n=500","title":"MobileMiniWob++_SR"},{"benchmarks":[{"benchmark_id":"llm-stats-mobileworld","domain":"multimodal","name":"MobileWorld","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileworld?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mobileworld?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileworld?top_n=500","title":"MobileWorld"},{"benchmarks":[{"benchmark_id":"llm-stats-motionbench","domain":"multimodal","name":"MotionBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/motionbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/motionbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/motionbench?top_n=500","title":"MotionBench"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-128k-2-needle","domain":"long_context","name":"MRCR 128K (2-needle)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%282-needle%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%282-needle%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%282-needle%29?top_n=500","title":"MRCR 128K (2-needle)"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-128k-4-needle","domain":"long_context","name":"MRCR 128K (4-needle)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%284-needle%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%284-needle%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%284-needle%29?top_n=500","title":"MRCR 128K (4-needle)"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-128k-8-needle","domain":"long_context","name":"MRCR 128K (8-needle)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%288-needle%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%288-needle%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%288-needle%29?top_n=500","title":"MRCR 128K (8-needle)"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-1m-pointwise","domain":"long_context","name":"MRCR 1M (pointwise)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m-%28pointwise%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m-%28pointwise%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m-%28pointwise%29?top_n=500","title":"MRCR 1M (pointwise)"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-1m","domain":"long_context","name":"MRCR 1M","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m?top_n=500","title":"MRCR 1M"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-64k-2-needle","domain":"long_context","name":"MRCR 64K (2-needle)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%282-needle%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%282-needle%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%282-needle%29?top_n=500","title":"MRCR 64K (2-needle)"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-64k-4-needle","domain":"long_context","name":"MRCR 64K (4-needle)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%284-needle%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%284-needle%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%284-needle%29?top_n=500","title":"MRCR 64K (4-needle)"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-64k-8-needle","domain":"long_context","name":"MRCR 64K (8-needle)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%288-needle%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%288-needle%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%288-needle%29?top_n=500","title":"MRCR 64K (8-needle)"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-v2-8-needle","domain":"long_context","name":"MRCR v2 (8-needle)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-%288-needle%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-%288-needle%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-%288-needle%29?top_n=500","title":"MRCR v2 (8-needle)"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-v2-8-needle-512k-1m","domain":"long_context","name":"MRCR v2 (8-needle, 512K-1M)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-8-needle-512k-1m?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-8-needle-512k-1m?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-8-needle-512k-1m?top_n=500","title":"MRCR v2 (8-needle, 512K-1M)"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr-v2","domain":"long_context","name":"MRCR v2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2?top_n=500","title":"MRCR v2"},{"benchmarks":[{"benchmark_id":"llm-stats-mrcr","domain":"long_context","name":"MRCR","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr?top_n=500","title":"MRCR"},{"benchmarks":[{"benchmark_id":"llm-stats-msqa","domain":"reasoning","name":"MSQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/msqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/msqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/msqa?top_n=500","title":"MSQA"},{"benchmarks":[{"benchmark_id":"llm-stats-mt-aime-2025","domain":"math","name":"MT-AIME 2025","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-aime-2025?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mt-aime-2025?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-aime-2025?top_n=500","title":"MT-AIME 2025"},{"benchmarks":[{"benchmark_id":"llm-stats-mt-bench","domain":"reasoning","name":"MT-Bench","released":"2023-06-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mt-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-bench?top_n=500","title":"MT-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-mtvqa","domain":"multimodal","name":"MTVQA","released":"2024-05-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/mtvqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mtvqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mtvqa?top_n=500","title":"MTVQA"},{"benchmarks":[{"benchmark_id":"llm-stats-muirbench","domain":"multimodal","name":"MuirBench","released":"2024-06-13","url":"https://api.zeroeval.com/leaderboard/benchmarks/muirbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/muirbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/muirbench?top_n=500","title":"MuirBench"},{"benchmarks":[{"benchmark_id":"llm-stats-multi-if","domain":"reasoning","name":"Multi-IF","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-if?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multi-if?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-if?top_n=500","title":"Multi-IF"},{"benchmarks":[{"benchmark_id":"llm-stats-multi-swe-bench","domain":"reasoning","name":"Multi-SWE-Bench","released":"2025-04-03","url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-swe-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multi-swe-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-swe-bench?top_n=500","title":"Multi-SWE-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-multichallenge","domain":"reasoning","name":"Multi-Challenge","released":"2025-01-29","url":"https://api.zeroeval.com/leaderboard/benchmarks/multichallenge?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multichallenge?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multichallenge?top_n=500","title":"Multi-Challenge"},{"benchmarks":[{"benchmark_id":"llm-stats-multilf","domain":"general","name":"MultiLF","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/multilf?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multilf?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilf?top_n=500","title":"MultiLF"},{"benchmarks":[{"benchmark_id":"llm-stats-multilingual-mgsm-cot","domain":"math","name":"Multilingual MGSM (CoT)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mgsm-%28cot%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mgsm-%28cot%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mgsm-%28cot%29?top_n=500","title":"Multilingual MGSM (CoT)"},{"benchmarks":[{"benchmark_id":"llm-stats-multilingual-mmlu","domain":"reasoning","name":"Multilingual MMLU","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mmlu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mmlu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mmlu?top_n=500","title":"Multilingual MMLU"},{"benchmarks":[{"benchmark_id":"llm-stats-multipl-e-humaneval","domain":"language","name":"Multipl-E HumanEval","released":"2022-08-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-humaneval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-humaneval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-humaneval?top_n=500","title":"Multipl-E HumanEval"},{"benchmarks":[{"benchmark_id":"llm-stats-multipl-e-mbpp","domain":"reasoning","name":"Multipl-E MBPP","released":"2022-08-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-mbpp?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-mbpp?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-mbpp?top_n=500","title":"Multipl-E MBPP"},{"benchmarks":[{"benchmark_id":"llm-stats-multipl-e","domain":"language","name":"MultiPL-E","released":"2022-08-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multipl-e?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e?top_n=500","title":"MultiPL-E"},{"benchmarks":[{"benchmark_id":"llm-stats-musiccaps","domain":"multimodal","name":"MusicCaps","released":"2023-01-26","url":"https://api.zeroeval.com/leaderboard/benchmarks/musiccaps?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/musiccaps?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/musiccaps?top_n=500","title":"MusicCaps"},{"benchmarks":[{"benchmark_id":"llm-stats-musr","domain":"reasoning","name":"MuSR","released":"2023-10-24","url":"https://api.zeroeval.com/leaderboard/benchmarks/musr?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/musr?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/musr?top_n=500","title":"MuSR"},{"benchmarks":[{"benchmark_id":"llm-stats-mvbench","domain":"multimodal","name":"MVBench","released":"2023-11-28","url":"https://api.zeroeval.com/leaderboard/benchmarks/mvbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mvbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/mvbench?top_n=500","title":"MVBench"},{"benchmarks":[{"benchmark_id":"llm-stats-nanogpt","domain":"agents","name":"NanoGPT","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nanogpt?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nanogpt?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nanogpt?top_n=500","title":"NanoGPT"},{"benchmarks":[{"benchmark_id":"llm-stats-natural-questions","domain":"reasoning","name":"Natural Questions","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/natural-questions?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/natural-questions?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/natural-questions?top_n=500","title":"Natural Questions"},{"benchmarks":[{"benchmark_id":"llm-stats-natural2code","domain":"reasoning","name":"Natural2Code","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/natural2code?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/natural2code?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/natural2code?top_n=500","title":"Natural2Code"},{"benchmarks":[{"benchmark_id":"llm-stats-nexus","domain":"general","name":"Nexus","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nexus?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nexus?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nexus?top_n=500","title":"Nexus"},{"benchmarks":[{"benchmark_id":"llm-stats-nih-multi-needle","domain":"long_context","name":"NIH/Multi-needle","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nih-multi-needle?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nih-multi-needle?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nih-multi-needle?top_n=500","title":"NIH/Multi-needle"},{"benchmarks":[{"benchmark_id":"llm-stats-nl2repo","domain":"agents","name":"NL2Repo","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nl2repo?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nl2repo?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nl2repo?top_n=500","title":"NL2Repo"},{"benchmarks":[{"benchmark_id":"llm-stats-nmos","domain":"general","name":"NMOS","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nmos?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nmos?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nmos?top_n=500","title":"NMOS"},{"benchmarks":[{"benchmark_id":"llm-stats-nolima-128k","domain":"long_context","name":"NoLiMa 128K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nolima-128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-128k?top_n=500","title":"NoLiMa 128K"},{"benchmarks":[{"benchmark_id":"llm-stats-nolima-32k","domain":"long_context","name":"NoLiMa 32K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-32k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nolima-32k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-32k?top_n=500","title":"NoLiMa 32K"},{"benchmarks":[{"benchmark_id":"llm-stats-nolima-64k","domain":"long_context","name":"NoLiMa 64K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-64k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nolima-64k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-64k?top_n=500","title":"NoLiMa 64K"},{"benchmarks":[{"benchmark_id":"llm-stats-nova-63","domain":"general","name":"NOVA-63","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nova-63?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nova-63?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nova-63?top_n=500","title":"NOVA-63"},{"benchmarks":[{"benchmark_id":"llm-stats-nq","domain":"reasoning","name":"NQ","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/nq?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nq?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nq?top_n=500","title":"NQ"},{"benchmarks":[{"benchmark_id":"llm-stats-nuscene","domain":"multimodal","name":"Nuscene","released":"2019-03-26","url":"https://api.zeroeval.com/leaderboard/benchmarks/nuscene?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nuscene?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/nuscene?top_n=500","title":"Nuscene"},{"benchmarks":[{"benchmark_id":"llm-stats-objectron","domain":"3d","name":"Objectron","released":"2020-12-18","url":"https://api.zeroeval.com/leaderboard/benchmarks/objectron?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/objectron?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/objectron?top_n=500","title":"Objectron"},{"benchmarks":[{"benchmark_id":"llm-stats-ocrbench-v2-en","domain":"image_to_text","name":"OCRBench-V2 (en)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28en%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28en%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28en%29?top_n=500","title":"OCRBench-V2 (en)"},{"benchmarks":[{"benchmark_id":"llm-stats-ocrbench-v2-zh","domain":"image_to_text","name":"OCRBench-V2 (zh)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28zh%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28zh%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28zh%29?top_n=500","title":"OCRBench-V2 (zh)"},{"benchmarks":[{"benchmark_id":"llm-stats-ocrbench-v2","domain":"image_to_text","name":"OCRBench_V2","released":"2024-12-31","url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2?top_n=500","title":"OCRBench_V2"},{"benchmarks":[{"benchmark_id":"llm-stats-ocrbench","domain":"image_to_text","name":"OCRBench","released":"2024-01-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ocrbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench?top_n=500","title":"OCRBench"},{"benchmarks":[{"benchmark_id":"llm-stats-octocodingbench","domain":"code","name":"OctoCodingBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/octocodingbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/octocodingbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/octocodingbench?top_n=500","title":"OctoCodingBench"},{"benchmarks":[{"benchmark_id":"llm-stats-odinw","domain":"vision","name":"ODinW","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/odinw?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/odinw?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/odinw?top_n=500","title":"ODinW"},{"benchmarks":[{"benchmark_id":"llm-stats-officeqa-pro","domain":"reasoning","name":"OfficeQA Pro","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/officeqa-pro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/officeqa-pro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/officeqa-pro?top_n=500","title":"OfficeQA Pro"},{"benchmarks":[{"benchmark_id":"llm-stats-ojbench-cpp","domain":"reasoning","name":"OJBench (C++)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench-cpp?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ojbench-cpp?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench-cpp?top_n=500","title":"OJBench (C++)"},{"benchmarks":[{"benchmark_id":"llm-stats-ojbench","domain":"reasoning","name":"OJBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ojbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench?top_n=500","title":"OJBench"},{"benchmarks":[{"benchmark_id":"llm-stats-olympiadbench","domain":"math","name":"OlympiadBench","released":"2024-02-21","url":"https://api.zeroeval.com/leaderboard/benchmarks/olympiadbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/olympiadbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/olympiadbench?top_n=500","title":"OlympiadBench"},{"benchmarks":[{"benchmark_id":"llm-stats-omnibench-music","domain":"multimodal","name":"OmniBench Music","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench-music?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnibench-music?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench-music?top_n=500","title":"OmniBench Music"},{"benchmarks":[{"benchmark_id":"llm-stats-omnibench","domain":"multimodal","name":"OmniBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnibench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench?top_n=500","title":"OmniBench"},{"benchmarks":[{"benchmark_id":"llm-stats-omnidocbench-1-5","domain":"multimodal","name":"OmniDocBench 1.5","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench-1.5?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench-1.5?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench-1.5?top_n=500","title":"OmniDocBench 1.5"},{"benchmarks":[{"benchmark_id":"llm-stats-omnidocbench","domain":"multimodal","name":"OmniDocBench","released":"2024-12-10","url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench?top_n=500","title":"OmniDocBench"},{"benchmarks":[{"benchmark_id":"llm-stats-omnigaia","domain":"multimodal","name":"OmniGAIA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/omnigaia?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnigaia?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnigaia?top_n=500","title":"OmniGAIA"},{"benchmarks":[{"benchmark_id":"llm-stats-omnimath","domain":"math","name":"OmniMath","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/omnimath?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnimath?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omnimath?top_n=500","title":"OmniMath"},{"benchmarks":[{"benchmark_id":"llm-stats-omniscience-non-hallucination-rate","domain":"reasoning","name":"OmniScience (non-hallucination rate)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience-non-hallucination-rate?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omniscience-non-hallucination-rate?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience-non-hallucination-rate?top_n=500","title":"OmniScience (non-hallucination rate)"},{"benchmarks":[{"benchmark_id":"llm-stats-omniscience","domain":"reasoning","name":"OmniScience","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omniscience?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience?top_n=500","title":"OmniScience"},{"benchmarks":[{"benchmark_id":"llm-stats-onemillion-bench","domain":"long_context","name":"OneMillion Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/onemillion-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/onemillion-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/onemillion-bench?top_n=500","title":"OneMillion Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-open-rewrite","domain":"language","name":"Open-rewrite","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/open-rewrite?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/open-rewrite?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/open-rewrite?top_n=500","title":"Open-rewrite"},{"benchmarks":[{"benchmark_id":"llm-stats-openai-connectors","domain":"agents","name":"Connectors","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-connectors?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-connectors?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-connectors?top_n=500","title":"Connectors"},{"benchmarks":[{"benchmark_id":"llm-stats-openai-mmlu","domain":"legal","name":"OpenAI MMLU","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mmlu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-mmlu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mmlu?top_n=500","title":"OpenAI MMLU"},{"benchmarks":[{"benchmark_id":"llm-stats-openai-mrcr-2-needle-128k","domain":"long_context","name":"OpenAI-MRCR: 2 needle 128k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-128k?top_n=500","title":"OpenAI-MRCR: 2 needle 128k"},{"benchmarks":[{"benchmark_id":"llm-stats-openai-mrcr-2-needle-1m","domain":"long_context","name":"OpenAI-MRCR: 2 needle 1M","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-1m?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-1m?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-1m?top_n=500","title":"OpenAI-MRCR: 2 needle 1M"},{"benchmarks":[{"benchmark_id":"llm-stats-openai-mrcr-2-needle-256k","domain":"long_context","name":"OpenAI-MRCR: 2 needle 256k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-256k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-256k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-256k?top_n=500","title":"OpenAI-MRCR: 2 needle 256k"},{"benchmarks":[{"benchmark_id":"llm-stats-openai-search-function-calling","domain":"agents","name":"Search and Function-Calling","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-search-function-calling?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-search-function-calling?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-search-function-calling?top_n=500","title":"Search and Function-Calling"},{"benchmarks":[{"benchmark_id":"llm-stats-openbookqa","domain":"reasoning","name":"OpenBookQA","released":"2018-09-08","url":"https://api.zeroeval.com/leaderboard/benchmarks/openbookqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openbookqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openbookqa?top_n=500","title":"OpenBookQA"},{"benchmarks":[{"benchmark_id":"llm-stats-openrca","domain":"reasoning","name":"OpenRCA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/openrca?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openrca?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/openrca?top_n=500","title":"OpenRCA"},{"benchmarks":[{"benchmark_id":"llm-stats-osworld-2-0","domain":"multimodal","name":"OSWorld 2.0","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-2.0?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-2.0?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-2.0?top_n=500","title":"OSWorld 2.0"},{"benchmarks":[{"benchmark_id":"llm-stats-osworld-extended","domain":"multimodal","name":"OSWorld Extended","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-extended?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-extended?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-extended?top_n=500","title":"OSWorld Extended"},{"benchmarks":[{"benchmark_id":"llm-stats-osworld-g","domain":"multimodal","name":"OSWorld-G","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-g?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-g?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-g?top_n=500","title":"OSWorld-G"},{"benchmarks":[{"benchmark_id":"llm-stats-osworld-screenshot-only","domain":"multimodal","name":"OSWorld Screenshot-only","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-screenshot-only?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-screenshot-only?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-screenshot-only?top_n=500","title":"OSWorld Screenshot-only"},{"benchmarks":[{"benchmark_id":"llm-stats-osworld-verified","domain":"multimodal","name":"OSWorld-Verified","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-verified?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-verified?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-verified?top_n=500","title":"OSWorld-Verified"},{"benchmarks":[{"benchmark_id":"llm-stats-osworld","domain":"multimodal","name":"OSWorld","released":"2024-04-11","url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld?top_n=500","title":"OSWorld"},{"benchmarks":[{"benchmark_id":"llm-stats-ovbench","domain":"multimodal","name":"OVBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ovbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ovbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ovbench?top_n=500","title":"OVBench"},{"benchmarks":[{"benchmark_id":"llm-stats-ovobench","domain":"multimodal","name":"OVOBench","released":"2025-01-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/ovobench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ovobench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ovobench?top_n=500","title":"OVOBench"},{"benchmarks":[{"benchmark_id":"llm-stats-paperbench","domain":"reasoning","name":"PaperBench","released":"2025-04-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/paperbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/paperbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/paperbench?top_n=500","title":"PaperBench"},{"benchmarks":[{"benchmark_id":"llm-stats-pathmcqa","domain":"multimodal","name":"PathMCQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/pathmcqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pathmcqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pathmcqa?top_n=500","title":"PathMCQA"},{"benchmarks":[{"benchmark_id":"llm-stats-perceptionbench","domain":"multimodal","name":"PerceptionBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptionbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/perceptionbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptionbench?top_n=500","title":"PerceptionBench"},{"benchmarks":[{"benchmark_id":"llm-stats-perceptiontest","domain":"multimodal","name":"PerceptionTest","released":"2023-05-23","url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptiontest?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/perceptiontest?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptiontest?top_n=500","title":"PerceptionTest"},{"benchmarks":[{"benchmark_id":"llm-stats-phibench","domain":"math","name":"PhiBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/phibench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/phibench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/phibench?top_n=500","title":"PhiBench"},{"benchmarks":[{"benchmark_id":"llm-stats-phybench","domain":"physics","name":"PHYBench","released":"2025-04-22","url":"https://api.zeroeval.com/leaderboard/benchmarks/phybench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/phybench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/phybench?top_n=500","title":"PHYBench"},{"benchmarks":[{"benchmark_id":"llm-stats-physicsfinals","domain":"math","name":"PhysicsFinals","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/physicsfinals?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/physicsfinals?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/physicsfinals?top_n=500","title":"PhysicsFinals"},{"benchmarks":[{"benchmark_id":"llm-stats-pinchbench","domain":"agents","name":"PinchBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/pinchbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pinchbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pinchbench?top_n=500","title":"PinchBench"},{"benchmarks":[{"benchmark_id":"llm-stats-piqa","domain":"physics","name":"PIQA","released":"2019-11-26","url":"https://api.zeroeval.com/leaderboard/benchmarks/piqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/piqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/piqa?top_n=500","title":"PIQA"},{"benchmarks":[{"benchmark_id":"llm-stats-plawbench","domain":"legal","name":"PLawBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/plawbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/plawbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/plawbench?top_n=500","title":"PLawBench"},{"benchmarks":[{"benchmark_id":"llm-stats-pmc-vqa","domain":"multimodal","name":"PMC-VQA","released":"2023-05-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/pmc-vqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pmc-vqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pmc-vqa?top_n=500","title":"PMC-VQA"},{"benchmarks":[{"benchmark_id":"llm-stats-pointgrounding","domain":"multimodal","name":"PointGrounding","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/pointgrounding?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pointgrounding?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pointgrounding?top_n=500","title":"PointGrounding"},{"benchmarks":[{"benchmark_id":"llm-stats-polymath-en","domain":"math","name":"PolyMath-en","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath-en?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/polymath-en?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath-en?top_n=500","title":"PolyMath-en"},{"benchmarks":[{"benchmark_id":"llm-stats-polymath","domain":"math","name":"PolyMATH","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/polymath?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath?top_n=500","title":"PolyMATH"},{"benchmarks":[{"benchmark_id":"llm-stats-pope","domain":"multimodal","name":"POPE","released":"2023-05-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/pope?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pope?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/pope?top_n=500","title":"POPE"},{"benchmarks":[{"benchmark_id":"llm-stats-popqa","domain":"reasoning","name":"PopQA","released":"2022-12-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/popqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/popqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/popqa?top_n=500","title":"PopQA"},{"benchmarks":[{"benchmark_id":"llm-stats-posttrainbench-lite","domain":"reasoning","name":"PostTrainBench Lite","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench-lite?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench-lite?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench-lite?top_n=500","title":"PostTrainBench Lite"},{"benchmarks":[{"benchmark_id":"llm-stats-posttrainbench","domain":"reasoning","name":"PostTrainBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench?top_n=500","title":"PostTrainBench"},{"benchmarks":[{"benchmark_id":"llm-stats-prbench-finance","domain":"reasoning","name":"PRBench-Finance","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-finance?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/prbench-finance?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-finance?top_n=500","title":"PRBench-Finance"},{"benchmarks":[{"benchmark_id":"llm-stats-prbench-legal","domain":"legal","name":"PRBench-Legal","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-legal?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/prbench-legal?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-legal?top_n=500","title":"PRBench-Legal"},{"benchmarks":[{"benchmark_id":"llm-stats-presentbench","domain":"reasoning","name":"PresentBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/presentbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/presentbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/presentbench?top_n=500","title":"PresentBench"},{"benchmarks":[{"benchmark_id":"llm-stats-profbench","domain":"reasoning","name":"ProfBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/profbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/profbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/profbench?top_n=500","title":"ProfBench"},{"benchmarks":[{"benchmark_id":"llm-stats-program-bench","domain":"agents","name":"Program Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/program-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/program-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/program-bench?top_n=500","title":"Program Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-protocolqa","domain":"safety","name":"ProtocolQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/protocolqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/protocolqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/protocolqa?top_n=500","title":"ProtocolQA"},{"benchmarks":[{"benchmark_id":"llm-stats-qasper","domain":"long_context","name":"Qasper","released":"2021-05-07","url":"https://api.zeroeval.com/leaderboard/benchmarks/qasper?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qasper?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qasper?top_n=500","title":"Qasper"},{"benchmarks":[{"benchmark_id":"llm-stats-qmsum","domain":"long_context","name":"QMSum","released":"2021-04-13","url":"https://api.zeroeval.com/leaderboard/benchmarks/qmsum?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qmsum?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qmsum?top_n=500","title":"QMSum"},{"benchmarks":[{"benchmark_id":"llm-stats-qvhighlights","domain":"multimodal","name":"QVHighlights","released":"2021-07-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/qvhighlights?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qvhighlights?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qvhighlights?top_n=500","title":"QVHighlights"},{"benchmarks":[{"benchmark_id":"llm-stats-qwen-qoder-bench","domain":"agents","name":"QwenQoderBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-qoder-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwen-qoder-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-qoder-bench?top_n=500","title":"QwenQoderBench"},{"benchmarks":[{"benchmark_id":"llm-stats-qwen-react-bench","domain":"multimodal","name":"QwenReactBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-react-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwen-react-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-react-bench?top_n=500","title":"QwenReactBench"},{"benchmarks":[{"benchmark_id":"llm-stats-qwen-svg","domain":"multimodal","name":"QwenSVG","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-svg?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwen-svg?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-svg?top_n=500","title":"QwenSVG"},{"benchmarks":[{"benchmark_id":"llm-stats-qwen-swe-bench","domain":"agents","name":"QwenSWEBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-swe-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwen-swe-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-swe-bench?top_n=500","title":"QwenSWEBench"},{"benchmarks":[{"benchmark_id":"llm-stats-qwenclawbench","domain":"agents","name":"QwenClawBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenclawbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwenclawbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenclawbench?top_n=500","title":"QwenClawBench"},{"benchmarks":[{"benchmark_id":"llm-stats-qwenwebbench","domain":"multimodal","name":"QwenWebBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenwebbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwenwebbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenwebbench?top_n=500","title":"QwenWebBench"},{"benchmarks":[{"benchmark_id":"llm-stats-qwenworldbench","domain":"reasoning","name":"QwenWorldBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenworldbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwenworldbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenworldbench?top_n=500","title":"QwenWorldBench"},{"benchmarks":[{"benchmark_id":"llm-stats-realkie-fcc","domain":"multimodal","name":"RealKIE-FCC","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/realkie-fcc?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/realkie-fcc?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/realkie-fcc?top_n=500","title":"RealKIE-FCC"},{"benchmarks":[{"benchmark_id":"llm-stats-realworldqa","domain":"spatial_reasoning","name":"RealWorldQA","released":"2024-04-12","url":"https://api.zeroeval.com/leaderboard/benchmarks/realworldqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/realworldqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/realworldqa?top_n=500","title":"RealWorldQA"},{"benchmarks":[{"benchmark_id":"llm-stats-recreationbench","domain":"multimodal","name":"RecreationBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/recreationbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/recreationbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/recreationbench?top_n=500","title":"RecreationBench"},{"benchmarks":[{"benchmark_id":"llm-stats-refcoco-avg","domain":"spatial_reasoning","name":"RefCOCO-avg","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/refcoco-avg?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/refcoco-avg?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refcoco-avg?top_n=500","title":"RefCOCO-avg"},{"benchmarks":[{"benchmark_id":"llm-stats-refcocog","domain":"multimodal","name":"RefCOCOg","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/refcocog?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/refcocog?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refcocog?top_n=500","title":"RefCOCOg"},{"benchmarks":[{"benchmark_id":"llm-stats-refspatialbench","domain":"spatial_reasoning","name":"RefSpatialBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/refspatialbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/refspatialbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/refspatialbench?top_n=500","title":"RefSpatialBench"},{"benchmarks":[{"benchmark_id":"llm-stats-repo-env","domain":"agents","name":"Repo Env","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/repo-env?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/repo-env?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repo-env?top_n=500","title":"Repo Env"},{"benchmarks":[{"benchmark_id":"llm-stats-repobench","domain":"reasoning","name":"RepoBench","released":"2023-06-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/repobench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/repobench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repobench?top_n=500","title":"RepoBench"},{"benchmarks":[{"benchmark_id":"llm-stats-repoqa","domain":"long_context","name":"RepoQA","released":"2024-06-10","url":"https://api.zeroeval.com/leaderboard/benchmarks/repoqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/repoqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/repoqa?top_n=500","title":"RepoQA"},{"benchmarks":[{"benchmark_id":"llm-stats-researchclawbench","domain":"research","name":"ResearchClawBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/researchclawbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/researchclawbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/researchclawbench?top_n=500","title":"ResearchClawBench"},{"benchmarks":[{"benchmark_id":"llm-stats-robospatialhome","domain":"robotics","name":"RoboSpatialHome","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/robospatialhome?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/robospatialhome?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/robospatialhome?top_n=500","title":"RoboSpatialHome"},{"benchmarks":[{"benchmark_id":"llm-stats-robust-if","domain":"reasoning","name":"Robust IF","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/robust-if?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/robust-if?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/robust-if?top_n=500","title":"Robust IF"},{"benchmarks":[{"benchmark_id":"llm-stats-rsi-index","domain":"reasoning","name":"RSI Index","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/rsi-index?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/rsi-index?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/rsi-index?top_n=500","title":"RSI Index"},{"benchmarks":[{"benchmark_id":"llm-stats-ruler-1000k","domain":"long_context","name":"RULER 1000K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-1000k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-1000k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-1000k?top_n=500","title":"RULER 1000K"},{"benchmarks":[{"benchmark_id":"llm-stats-ruler-128k","domain":"long_context","name":"RULER 128k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-128k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-128k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-128k?top_n=500","title":"RULER 128k"},{"benchmarks":[{"benchmark_id":"llm-stats-ruler-2048k","domain":"long_context","name":"RULER 2048K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-2048k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-2048k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-2048k?top_n=500","title":"RULER 2048K"},{"benchmarks":[{"benchmark_id":"llm-stats-ruler-512k","domain":"long_context","name":"RULER 512K","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-512k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-512k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-512k?top_n=500","title":"RULER 512K"},{"benchmarks":[{"benchmark_id":"llm-stats-ruler-64k","domain":"long_context","name":"RULER 64k","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-64k?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-64k?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-64k?top_n=500","title":"RULER 64k"},{"benchmarks":[{"benchmark_id":"llm-stats-ruler","domain":"long_context","name":"RULER","released":"2024-04-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler?top_n=500","title":"RULER"},{"benchmarks":[{"benchmark_id":"llm-stats-sat-math","domain":"math","name":"SAT Math","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/sat-math?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sat-math?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sat-math?top_n=500","title":"SAT Math"},{"benchmarks":[{"benchmark_id":"llm-stats-scicode","domain":"math","name":"SciCode","released":"2024-07-18","url":"https://api.zeroeval.com/leaderboard/benchmarks/scicode?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/scicode?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scicode?top_n=500","title":"SciCode"},{"benchmarks":[{"benchmark_id":"llm-stats-scienceqa-visual","domain":"multimodal","name":"ScienceQA Visual","released":"2022-09-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa-visual?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/scienceqa-visual?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa-visual?top_n=500","title":"ScienceQA Visual"},{"benchmarks":[{"benchmark_id":"llm-stats-scienceqa","domain":"math","name":"ScienceQA","released":"2022-09-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/scienceqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa?top_n=500","title":"ScienceQA"},{"benchmarks":[{"benchmark_id":"llm-stats-screenspot-pro","domain":"multimodal","name":"ScreenSpot Pro","released":"2025-04-04","url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot-pro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/screenspot-pro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot-pro?top_n=500","title":"ScreenSpot Pro"},{"benchmarks":[{"benchmark_id":"llm-stats-screenspot","domain":"multimodal","name":"ScreenSpot","released":"2024-01-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/screenspot?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot?top_n=500","title":"ScreenSpot"},{"benchmarks":[{"benchmark_id":"llm-stats-seal-0","domain":"reasoning","name":"Seal-0","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/seal-0?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/seal-0?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seal-0?top_n=500","title":"Seal-0"},{"benchmarks":[{"benchmark_id":"llm-stats-sec-bench-pro","domain":"safety","name":"SEC-bench Pro","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/sec-bench-pro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sec-bench-pro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sec-bench-pro?top_n=500","title":"SEC-bench Pro"},{"benchmarks":[{"benchmark_id":"llm-stats-seccodebench","domain":"code","name":"SecCodeBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/seccodebench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/seccodebench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seccodebench?top_n=500","title":"SecCodeBench"},{"benchmarks":[{"benchmark_id":"llm-stats-seedclawbench","domain":"agents","name":"SeedClawBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/seedclawbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/seedclawbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/seedclawbench?top_n=500","title":"SeedClawBench"},{"benchmarks":[{"benchmark_id":"llm-stats-sifo-multiturn","domain":"structured_output","name":"SIFO-Multiturn","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo-multiturn?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sifo-multiturn?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo-multiturn?top_n=500","title":"SIFO-Multiturn"},{"benchmarks":[{"benchmark_id":"llm-stats-sifo","domain":"structured_output","name":"SIFO","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sifo?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo?top_n=500","title":"SIFO"},{"benchmarks":[{"benchmark_id":"llm-stats-simpleqa-verified","domain":"reasoning","name":"SimpleQA Verified","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa-verified?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/simpleqa-verified?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa-verified?top_n=500","title":"SimpleQA Verified"},{"benchmarks":[{"benchmark_id":"llm-stats-simpleqa","domain":"reasoning","name":"SimpleQA","released":"2024-10-30","url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/simpleqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa?top_n=500","title":"SimpleQA"},{"benchmarks":[{"benchmark_id":"llm-stats-simplevqa","domain":"multimodal","name":"SimpleVQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/simplevqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/simplevqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/simplevqa?top_n=500","title":"SimpleVQA"},{"benchmarks":[{"benchmark_id":"llm-stats-siren-agentdojo-attack-success","domain":"safety","name":"Siren AgentDojo Attack Success Rate","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-attack-success?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-attack-success?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-attack-success?top_n=500","title":"Siren AgentDojo Attack Success Rate"},{"benchmarks":[{"benchmark_id":"llm-stats-siren-agentdojo-utility","domain":"safety","name":"Siren AgentDojo Utility","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-utility?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-utility?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-utility?top_n=500","title":"Siren AgentDojo Utility"},{"benchmarks":[{"benchmark_id":"llm-stats-skillsbench","domain":"agents","name":"SkillsBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/skillsbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/skillsbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/skillsbench?top_n=500","title":"SkillsBench"},{"benchmarks":[{"benchmark_id":"llm-stats-slakevqa","domain":"multimodal","name":"SlakeVQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/slakevqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/slakevqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/slakevqa?top_n=500","title":"SlakeVQA"},{"benchmarks":[{"benchmark_id":"llm-stats-social-iqa","domain":"psychology","name":"Social IQa","released":"2019-04-22","url":"https://api.zeroeval.com/leaderboard/benchmarks/social-iqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/social-iqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/social-iqa?top_n=500","title":"Social IQa"},{"benchmarks":[{"benchmark_id":"llm-stats-spider","domain":"reasoning","name":"Spider","released":"2018-09-24","url":"https://api.zeroeval.com/leaderboard/benchmarks/spider?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/spider?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spider?top_n=500","title":"Spider"},{"benchmarks":[{"benchmark_id":"llm-stats-spreadsheetbench-2","domain":"productivity","name":"SpreadsheetBench 2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-2?top_n=500","title":"SpreadsheetBench 2"},{"benchmarks":[{"benchmark_id":"llm-stats-spreadsheetbench-v1","domain":"productivity","name":"SpreadSheetBench-v1","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-v1?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-v1?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-v1?top_n=500","title":"SpreadSheetBench-v1"},{"benchmarks":[{"benchmark_id":"llm-stats-squality","domain":"long_context","name":"SQuALITY","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/squality?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/squality?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/squality?top_n=500","title":"SQuALITY"},{"benchmarks":[{"benchmark_id":"llm-stats-stem","domain":"math","name":"STEM","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/stem?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/stem?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/stem?top_n=500","title":"STEM"},{"benchmarks":[{"benchmark_id":"llm-stats-summscreenfd","domain":"long_context","name":"SummScreenFD","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/summscreenfd?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/summscreenfd?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/summscreenfd?top_n=500","title":"SummScreenFD"},{"benchmarks":[{"benchmark_id":"llm-stats-sunrgbd","domain":"spatial_reasoning","name":"SUNRGBD","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/sunrgbd?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sunrgbd?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/sunrgbd?top_n=500","title":"SUNRGBD"},{"benchmarks":[{"benchmark_id":"llm-stats-superchem","domain":"reasoning","name":"SuperChem","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/superchem?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/superchem?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/superchem?top_n=500","title":"SuperChem"},{"benchmarks":[{"benchmark_id":"llm-stats-superglue","domain":"reasoning","name":"SuperGLUE","released":"2019-05-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/superglue?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/superglue?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/superglue?top_n=500","title":"SuperGLUE"},{"benchmarks":[{"benchmark_id":"llm-stats-supergpqa","domain":"legal","name":"SuperGPQA","released":"2025-02-20","url":"https://api.zeroeval.com/leaderboard/benchmarks/supergpqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/supergpqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/supergpqa?top_n=500","title":"SuperGPQA"},{"benchmarks":[{"benchmark_id":"llm-stats-surds","domain":"multimodal","name":"SURDS","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/surds?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/surds?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/surds?top_n=500","title":"SURDS"},{"benchmarks":[{"benchmark_id":"llm-stats-svg-bench","domain":"multimodal","name":"SVG-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/svg-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/svg-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/svg-bench?top_n=500","title":"SVG-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-atlas-codebase-qna","domain":"agents","name":"SWE Atlas - Codebase QnA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-codebase-qna?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-codebase-qna?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-codebase-qna?top_n=500","title":"SWE Atlas - Codebase QnA"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-atlas-test-writing","domain":"agents","name":"SWE Atlas - Test Writing","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-test-writing?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-test-writing?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-test-writing?top_n=500","title":"SWE Atlas - Test Writing"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-atlas","domain":"agents","name":"SWE-Atlas","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas?top_n=500","title":"SWE-Atlas"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-bench-multilingual","domain":"reasoning","name":"SWE-bench Multilingual","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multilingual?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multilingual?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multilingual?top_n=500","title":"SWE-bench Multilingual"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-bench-multimodal","domain":"multimodal","name":"SWE-Bench Multimodal","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multimodal?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multimodal?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multimodal?top_n=500","title":"SWE-Bench Multimodal"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-bench-pro","domain":"reasoning","name":"SWE-Bench Pro","released":"2025-09-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-pro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-pro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-pro?top_n=500","title":"SWE-Bench Pro"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-bench-verified-agentic-coding","domain":"reasoning","name":"SWE-bench Verified (Agentic Coding)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentic-coding%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentic-coding%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentic-coding%29?top_n=500","title":"SWE-bench Verified (Agentic Coding)"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-bench-verified-agentless","domain":"reasoning","name":"SWE-bench Verified (Agentless)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentless%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentless%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentless%29?top_n=500","title":"SWE-bench Verified (Agentless)"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-bench-verified-multiple-attempts","domain":"reasoning","name":"SWE-bench Verified (Multiple Attempts)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28multiple-attempts%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28multiple-attempts%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28multiple-attempts%29?top_n=500","title":"SWE-bench Verified (Multiple Attempts)"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-bench-verified","domain":"reasoning","name":"SWE-Bench Verified","released":"2024-08-13","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified?top_n=500","title":"SWE-Bench Verified"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-fficiency","domain":"agents","name":"SWE-fficiency","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-fficiency?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-fficiency?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-fficiency?top_n=500","title":"SWE-fficiency"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-lancer-ic-diamond-subset","domain":"reasoning","name":"SWE-Lancer (IC-Diamond subset)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer-%28ic-diamond-subset%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer-%28ic-diamond-subset%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer-%28ic-diamond-subset%29?top_n=500","title":"SWE-Lancer (IC-Diamond subset)"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-lancer","domain":"reasoning","name":"SWE-Lancer","released":"2025-02-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer?top_n=500","title":"SWE-Lancer"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-marathon","domain":"agents","name":"SWE-Marathon","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-marathon?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-marathon?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-marathon?top_n=500","title":"SWE-Marathon"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-mm","domain":"multimodal","name":"SWE-MM","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-mm?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-mm?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-mm?top_n=500","title":"SWE-MM"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-perf","domain":"code","name":"SWE-Perf","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-perf?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-perf?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-perf?top_n=500","title":"SWE-Perf"},{"benchmarks":[{"benchmark_id":"llm-stats-swe-review","domain":"code","name":"SWE-Review","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-review?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-review?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-review?top_n=500","title":"SWE-Review"},{"benchmarks":[{"benchmark_id":"llm-stats-swt-bench","domain":"code","name":"SWT-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/swt-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swt-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/swt-bench?top_n=500","title":"SWT-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-t2-bench","domain":"reasoning","name":"t2-bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/t2-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/t2-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/t2-bench?top_n=500","title":"t2-bench"},{"benchmarks":[{"benchmark_id":"llm-stats-tau-bench-airline","domain":"reasoning","name":"TAU-bench Airline","released":"2024-06-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-airline?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-airline?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-airline?top_n=500","title":"TAU-bench Airline"},{"benchmarks":[{"benchmark_id":"llm-stats-tau-bench-retail","domain":"reasoning","name":"TAU-bench Retail","released":"2024-06-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-retail?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-retail?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-retail?top_n=500","title":"TAU-bench Retail"},{"benchmarks":[{"benchmark_id":"llm-stats-tau-bench","domain":"reasoning","name":"Tau-bench","released":"2024-06-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench?top_n=500","title":"Tau-bench"},{"benchmarks":[{"benchmark_id":"llm-stats-tau2-airline","domain":"reasoning","name":"Tau2 Airline","released":"2025-06-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-airline?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau2-airline?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-airline?top_n=500","title":"Tau2 Airline"},{"benchmarks":[{"benchmark_id":"llm-stats-tau2-retail","domain":"reasoning","name":"Tau2 Retail","released":"2025-06-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-retail?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau2-retail?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-retail?top_n=500","title":"Tau2 Retail"},{"benchmarks":[{"benchmark_id":"llm-stats-tau2-telecom","domain":"reasoning","name":"Tau2 Telecom","released":"2025-06-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-telecom?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau2-telecom?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-telecom?top_n=500","title":"Tau2 Telecom"},{"benchmarks":[{"benchmark_id":"llm-stats-tau3-airline","domain":"reasoning","name":"Tau3 Airline","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-airline?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-airline?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-airline?top_n=500","title":"Tau3 Airline"},{"benchmarks":[{"benchmark_id":"llm-stats-tau3-banking","domain":"reasoning","name":"Tau3 Banking","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-banking?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-banking?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-banking?top_n=500","title":"Tau3 Banking"},{"benchmarks":[{"benchmark_id":"llm-stats-tau3-bench","domain":"reasoning","name":"TAU3-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-bench?top_n=500","title":"TAU3-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-tau3-retail","domain":"reasoning","name":"Tau3 Retail","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-retail?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-retail?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-retail?top_n=500","title":"Tau3 Retail"},{"benchmarks":[{"benchmark_id":"llm-stats-tau3-telecom","domain":"reasoning","name":"Tau3 Telecom","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-telecom?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-telecom?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-telecom?top_n=500","title":"Tau3 Telecom"},{"benchmarks":[{"benchmark_id":"llm-stats-tempcompass","domain":"multimodal","name":"TempCompass","released":"2024-03-01","url":"https://api.zeroeval.com/leaderboard/benchmarks/tempcompass?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tempcompass?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tempcompass?top_n=500","title":"TempCompass"},{"benchmarks":[{"benchmark_id":"llm-stats-terminal-bench-2-1","domain":"reasoning","name":"Terminal-Bench 2.1","released":"2026-05-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2.1?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2.1?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2.1?top_n=500","title":"Terminal-Bench 2.1"},{"benchmarks":[{"benchmark_id":"llm-stats-terminal-bench-2","domain":"reasoning","name":"Terminal-Bench 2.0","released":"2025-09-25","url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2?top_n=500","title":"Terminal-Bench 2.0"},{"benchmarks":[{"benchmark_id":"llm-stats-terminal-bench-3-0","domain":"reasoning","name":"Terminal-Bench 3.0","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-3.0?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-3.0?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-3.0?top_n=500","title":"Terminal-Bench 3.0"},{"benchmarks":[{"benchmark_id":"llm-stats-terminal-bench-hard","domain":"reasoning","name":"Terminal-Bench Hard","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-hard?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-hard?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-hard?top_n=500","title":"Terminal-Bench Hard"},{"benchmarks":[{"benchmark_id":"llm-stats-terminal-bench","domain":"reasoning","name":"Terminal-Bench","released":"2025-01-17","url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench?top_n=500","title":"Terminal-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-terminus","domain":"reasoning","name":"Terminus","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/terminus?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminus?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/terminus?top_n=500","title":"Terminus"},{"benchmarks":[{"benchmark_id":"llm-stats-textvqa","domain":"multimodal","name":"TextVQA","released":"2019-04-18","url":"https://api.zeroeval.com/leaderboard/benchmarks/textvqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/textvqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/textvqa?top_n=500","title":"TextVQA"},{"benchmarks":[{"benchmark_id":"llm-stats-theoremqa","domain":"math","name":"TheoremQA","released":"2023-05-21","url":"https://api.zeroeval.com/leaderboard/benchmarks/theoremqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/theoremqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/theoremqa?top_n=500","title":"TheoremQA"},{"benchmarks":[{"benchmark_id":"llm-stats-tir-bench","domain":"multimodal","name":"TIR-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/tir-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tir-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tir-bench?top_n=500","title":"TIR-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-tldr9-test","domain":"summarization","name":"TLDR9+ (test)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/tldr9%2B-%28test%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tldr9%2B-%28test%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tldr9%2B-%28test%29?top_n=500","title":"TLDR9+ (test)"},{"benchmarks":[{"benchmark_id":"llm-stats-tomato","domain":"multimodal","name":"TOMATO","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/tomato?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tomato?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tomato?top_n=500","title":"TOMATO"},{"benchmarks":[{"benchmark_id":"llm-stats-toolathlon","domain":"reasoning","name":"Toolathlon","released":"2025-10-29","url":"https://api.zeroeval.com/leaderboard/benchmarks/toolathlon?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/toolathlon?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/toolathlon?top_n=500","title":"Toolathlon"},{"benchmarks":[{"benchmark_id":"llm-stats-trae-code-gen","domain":"agents","name":"Trae Code Gen","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-code-gen?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/trae-code-gen?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-code-gen?top_n=500","title":"Trae Code Gen"},{"benchmarks":[{"benchmark_id":"llm-stats-trae-error-fix","domain":"agents","name":"Trae Error Fix","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-error-fix?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/trae-error-fix?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-error-fix?top_n=500","title":"Trae Error Fix"},{"benchmarks":[{"benchmark_id":"llm-stats-translation-en-set1-comet22","domain":"language","name":"Translation en→Set1 COMET22","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-comet22?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-comet22?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-comet22?top_n=500","title":"Translation en→Set1 COMET22"},{"benchmarks":[{"benchmark_id":"llm-stats-translation-en-set1-spbleu","domain":"language","name":"Translation en→Set1 spBleu","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-spbleu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-spbleu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-spbleu?top_n=500","title":"Translation en→Set1 spBleu"},{"benchmarks":[{"benchmark_id":"llm-stats-translation-set1-en-comet22","domain":"language","name":"Translation Set1→en COMET22","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-comet22?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-comet22?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-comet22?top_n=500","title":"Translation Set1→en COMET22"},{"benchmarks":[{"benchmark_id":"llm-stats-translation-set1-en-spbleu","domain":"language","name":"Translation Set1→en spBleu","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-spbleu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-spbleu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-spbleu?top_n=500","title":"Translation Set1→en spBleu"},{"benchmarks":[{"benchmark_id":"llm-stats-treebench","domain":"multimodal","name":"TreeBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/treebench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/treebench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/treebench?top_n=500","title":"TreeBench"},{"benchmarks":[{"benchmark_id":"llm-stats-triviaqa","domain":"reasoning","name":"TriviaQA","released":"2017-05-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/triviaqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/triviaqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/triviaqa?top_n=500","title":"TriviaQA"},{"benchmarks":[{"benchmark_id":"llm-stats-truthfulqa","domain":"legal","name":"TruthfulQA","released":"2021-09-08","url":"https://api.zeroeval.com/leaderboard/benchmarks/truthfulqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/truthfulqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/truthfulqa?top_n=500","title":"TruthfulQA"},{"benchmarks":[{"benchmark_id":"llm-stats-tvbench","domain":"multimodal","name":"TVBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/tvbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tvbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tvbench?top_n=500","title":"TVBench"},{"benchmarks":[{"benchmark_id":"llm-stats-tydiqa","domain":"reasoning","name":"TydiQA","released":"2020-03-10","url":"https://api.zeroeval.com/leaderboard/benchmarks/tydiqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tydiqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/tydiqa?top_n=500","title":"TydiQA"},{"benchmarks":[{"benchmark_id":"llm-stats-uniform-bar-exam","domain":"legal","name":"Uniform Bar Exam","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/uniform-bar-exam?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/uniform-bar-exam?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/uniform-bar-exam?top_n=500","title":"Uniform Bar Exam"},{"benchmarks":[{"benchmark_id":"llm-stats-usamo-2026","domain":"math","name":"USAMO 2026","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo-2026?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/usamo-2026?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo-2026?top_n=500","title":"USAMO 2026"},{"benchmarks":[{"benchmark_id":"llm-stats-usamo25","domain":"math","name":"USAMO25","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo25?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/usamo25?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo25?top_n=500","title":"USAMO25"},{"benchmarks":[{"benchmark_id":"llm-stats-v-star","domain":"multimodal","name":"V*","released":"2023-12-21","url":"https://api.zeroeval.com/leaderboard/benchmarks/v-star?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/v-star?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/v-star?top_n=500","title":"V*"},{"benchmarks":[{"benchmark_id":"llm-stats-vatex","domain":"multimodal","name":"VATEX","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vatex?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vatex?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vatex?top_n=500","title":"VATEX"},{"benchmarks":[{"benchmark_id":"llm-stats-vcr-en-easy","domain":"reasoning","name":"VCR_en_easy","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vcr-en-easy?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vcr-en-easy?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vcr-en-easy?top_n=500","title":"VCR_en_easy"},{"benchmarks":[{"benchmark_id":"llm-stats-vct","domain":"safety","name":"Virology Capabilities Test","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vct?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vct?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vct?top_n=500","title":"Virology Capabilities Test"},{"benchmarks":[{"benchmark_id":"llm-stats-vending-bench-2","domain":"reasoning","name":"Vending-Bench 2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vending-bench-2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vending-bench-2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vending-bench-2?top_n=500","title":"Vending-Bench 2"},{"benchmarks":[{"benchmark_id":"llm-stats-vibe-android","domain":"code","name":"VIBE Android","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-android?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-android?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-android?top_n=500","title":"VIBE Android"},{"benchmarks":[{"benchmark_id":"llm-stats-vibe-backend","domain":"code","name":"VIBE Backend","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-backend?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-backend?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-backend?top_n=500","title":"VIBE Backend"},{"benchmarks":[{"benchmark_id":"llm-stats-vibe-eval","domain":"multimodal","name":"Vibe-Eval","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-eval?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-eval?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-eval?top_n=500","title":"Vibe-Eval"},{"benchmarks":[{"benchmark_id":"llm-stats-vibe-ios","domain":"code","name":"VIBE iOS","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-ios?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-ios?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-ios?top_n=500","title":"VIBE iOS"},{"benchmarks":[{"benchmark_id":"llm-stats-vibe-pro","domain":"agents","name":"VIBE-Pro","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-pro?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-pro?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-pro?top_n=500","title":"VIBE-Pro"},{"benchmarks":[{"benchmark_id":"llm-stats-vibe-simulation","domain":"code","name":"VIBE Simulation","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-simulation?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-simulation?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-simulation?top_n=500","title":"VIBE Simulation"},{"benchmarks":[{"benchmark_id":"llm-stats-vibe-v2","domain":"agents","name":"VIBE-V2","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-v2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-v2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-v2?top_n=500","title":"VIBE-V2"},{"benchmarks":[{"benchmark_id":"llm-stats-vibe-web","domain":"code","name":"VIBE Web","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-web?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-web?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-web?top_n=500","title":"VIBE Web"},{"benchmarks":[{"benchmark_id":"llm-stats-vibe","domain":"code","name":"VIBE","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe?top_n=500","title":"VIBE"},{"benchmarks":[{"benchmark_id":"llm-stats-video-mme-long-no-subtitles","domain":"multimodal","name":"Video-MME (long, no subtitles)","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme-%28long%2C-no-subtitles%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/video-mme-%28long%2C-no-subtitles%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme-%28long%2C-no-subtitles%29?top_n=500","title":"Video-MME (long, no subtitles)"},{"benchmarks":[{"benchmark_id":"llm-stats-video-mme","domain":"multimodal","name":"Video-MME","released":"2024-05-31","url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/video-mme?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme?top_n=500","title":"Video-MME"},{"benchmarks":[{"benchmark_id":"llm-stats-videoholmes","domain":"multimodal","name":"VideoHolmes","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/videoholmes?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videoholmes?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videoholmes?top_n=500","title":"VideoHolmes"},{"benchmarks":[{"benchmark_id":"llm-stats-videomme-w-o-sub","domain":"multimodal","name":"VideoMME w/o sub.","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-o-sub.?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-o-sub.?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-o-sub.?top_n=500","title":"VideoMME w/o sub."},{"benchmarks":[{"benchmark_id":"llm-stats-videomme-w-sub","domain":"multimodal","name":"VideoMME w sub.","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-sub.?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-sub.?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-sub.?top_n=500","title":"VideoMME w sub."},{"benchmarks":[{"benchmark_id":"llm-stats-videommmu","domain":"multimodal","name":"VideoMMMU","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/videommmu?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videommmu?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videommmu?top_n=500","title":"VideoMMMU"},{"benchmarks":[{"benchmark_id":"llm-stats-videosimpleqa","domain":"multimodal","name":"VideoSimpleQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/videosimpleqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videosimpleqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/videosimpleqa?top_n=500","title":"VideoSimpleQA"},{"benchmarks":[{"benchmark_id":"llm-stats-visfactor","domain":"multimodal","name":"VisFactor","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/visfactor?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/visfactor?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visfactor?top_n=500","title":"VisFactor"},{"benchmarks":[{"benchmark_id":"llm-stats-vision2web","domain":"multimodal","name":"Vision2Web","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vision2web?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vision2web?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vision2web?top_n=500","title":"Vision2Web"},{"benchmarks":[{"benchmark_id":"llm-stats-visualwebbench","domain":"multimodal","name":"VisualWebBench","released":"2024-04-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/visualwebbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/visualwebbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visualwebbench?top_n=500","title":"VisualWebBench"},{"benchmarks":[{"benchmark_id":"llm-stats-visulogic","domain":"multimodal","name":"VisuLogic","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/visulogic?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/visulogic?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/visulogic?top_n=500","title":"VisuLogic"},{"benchmarks":[{"benchmark_id":"llm-stats-vita-bench","domain":"reasoning","name":"VITA-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vita-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vita-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vita-bench?top_n=500","title":"VITA-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-vladbench","domain":"multimodal","name":"VLADBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vladbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vladbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vladbench?top_n=500","title":"VLADBench"},{"benchmarks":[{"benchmark_id":"llm-stats-vlmsarebiased","domain":"multimodal","name":"VLMsAreBiased","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsarebiased?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vlmsarebiased?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsarebiased?top_n=500","title":"VLMsAreBiased"},{"benchmarks":[{"benchmark_id":"llm-stats-vlmsareblind","domain":"multimodal","name":"VLMsAreBlind","released":"2024-07-09","url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsareblind?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vlmsareblind?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsareblind?top_n=500","title":"VLMsAreBlind"},{"benchmarks":[{"benchmark_id":"llm-stats-vocalsound","domain":"audio","name":"VocalSound","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vocalsound?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vocalsound?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vocalsound?top_n=500","title":"VocalSound"},{"benchmarks":[{"benchmark_id":"llm-stats-voicebench-avg","domain":"reasoning","name":"VoiceBench Avg","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/voicebench-avg?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/voicebench-avg?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/voicebench-avg?top_n=500","title":"VoiceBench Avg"},{"benchmarks":[{"benchmark_id":"llm-stats-vqa-rad","domain":"multimodal","name":"VQA-Rad","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/vqa-rad?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vqa-rad?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqa-rad?top_n=500","title":"VQA-Rad"},{"benchmarks":[{"benchmark_id":"llm-stats-vqav2-test","domain":"multimodal","name":"VQAv2 (test)","released":"2016-12-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28test%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28test%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28test%29?top_n=500","title":"VQAv2 (test)"},{"benchmarks":[{"benchmark_id":"llm-stats-vqav2-val","domain":"multimodal","name":"VQAv2 (val)","released":"2016-12-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28val%29?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28val%29?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28val%29?top_n=500","title":"VQAv2 (val)"},{"benchmarks":[{"benchmark_id":"llm-stats-vqav2","domain":"multimodal","name":"VQAv2","released":"2016-12-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vqav2?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2?top_n=500","title":"VQAv2"},{"benchmarks":[{"benchmark_id":"llm-stats-we-math","domain":"math","name":"We-Math","released":"2024-07-01","url":"https://api.zeroeval.com/leaderboard/benchmarks/we-math?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/we-math?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/we-math?top_n=500","title":"We-Math"},{"benchmarks":[{"benchmark_id":"llm-stats-web-bench","domain":"agents","name":"Web Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/web-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/web-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/web-bench?top_n=500","title":"Web Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-webarena-verified","domain":"multimodal","name":"WebArena-Verified","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/webarena-verified?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/webarena-verified?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webarena-verified?top_n=500","title":"WebArena-Verified"},{"benchmarks":[{"benchmark_id":"llm-stats-webdev-arena","domain":"reasoning","name":"WebDev Arena","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/webdev-arena?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/webdev-arena?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webdev-arena?top_n=500","title":"WebDev Arena"},{"benchmarks":[{"benchmark_id":"llm-stats-webvoyager","domain":"agents","name":"WebVoyager","released":"2024-01-25","url":"https://api.zeroeval.com/leaderboard/benchmarks/webvoyager?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/webvoyager?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/webvoyager?top_n=500","title":"WebVoyager"},{"benchmarks":[{"benchmark_id":"llm-stats-widesearch","domain":"reasoning","name":"WideSearch","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/widesearch?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/widesearch?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/widesearch?top_n=500","title":"WideSearch"},{"benchmarks":[{"benchmark_id":"llm-stats-wild-bench","domain":"reasoning","name":"Wild Bench","released":"2024-06-07","url":"https://api.zeroeval.com/leaderboard/benchmarks/wild-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wild-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wild-bench?top_n=500","title":"Wild Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-wildclawbench","domain":"agents","name":"WildClawBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/wildclawbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wildclawbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wildclawbench?top_n=500","title":"WildClawBench"},{"benchmarks":[{"benchmark_id":"llm-stats-winogrande","domain":"reasoning","name":"Winogrande","released":"2019-11-21","url":"https://api.zeroeval.com/leaderboard/benchmarks/winogrande?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/winogrande?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/winogrande?top_n=500","title":"Winogrande"},{"benchmarks":[{"benchmark_id":"llm-stats-wmdp","domain":"safety","name":"WMDP","released":"2024-03-05","url":"https://api.zeroeval.com/leaderboard/benchmarks/wmdp?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wmdp?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmdp?top_n=500","title":"WMDP"},{"benchmarks":[{"benchmark_id":"llm-stats-wmt23","domain":"language","name":"WMT23","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt23?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wmt23?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt23?top_n=500","title":"WMT23"},{"benchmarks":[{"benchmark_id":"llm-stats-wmt24","domain":"language","name":"WMT24++","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt24%2B%2B?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wmt24%2B%2B?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt24%2B%2B?top_n=500","title":"WMT24++"},{"benchmarks":[{"benchmark_id":"llm-stats-workspace-bench","domain":"reasoning","name":"Workspace Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/workspace-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/workspace-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/workspace-bench?top_n=500","title":"Workspace Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-worldbench","domain":"multimodal","name":"WorldBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/worldbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/worldbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/worldbench?top_n=500","title":"WorldBench"},{"benchmarks":[{"benchmark_id":"llm-stats-worldvqa","domain":"multimodal","name":"WorldVQA","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/worldvqa?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/worldvqa?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/worldvqa?top_n=500","title":"WorldVQA"},{"benchmarks":[{"benchmark_id":"llm-stats-writingbench","domain":"legal","name":"WritingBench","released":"2025-03-07","url":"https://api.zeroeval.com/leaderboard/benchmarks/writingbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/writingbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/writingbench?top_n=500","title":"WritingBench"},{"benchmarks":[{"benchmark_id":"llm-stats-xdailybench","domain":"reasoning","name":"xDailyBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/xdailybench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/xdailybench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xdailybench?top_n=500","title":"xDailyBench"},{"benchmarks":[{"benchmark_id":"llm-stats-xlsum-english","domain":"summarization","name":"XLSum English","released":"2021-06-25","url":"https://api.zeroeval.com/leaderboard/benchmarks/xlsum-english?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/xlsum-english?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xlsum-english?top_n=500","title":"XLSum English"},{"benchmarks":[{"benchmark_id":"llm-stats-xstest","domain":"safety","name":"XSTest","released":"2023-08-02","url":"https://api.zeroeval.com/leaderboard/benchmarks/xstest?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/xstest?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/xstest?top_n=500","title":"XSTest"},{"benchmarks":[{"benchmark_id":"llm-stats-yc-bench","domain":"finance","name":"YC-Bench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/yc-bench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/yc-bench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/yc-bench?top_n=500","title":"YC-Bench"},{"benchmarks":[{"benchmark_id":"llm-stats-zclawbench","domain":"agents","name":"ZClawBench","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/zclawbench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/zclawbench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zclawbench?top_n=500","title":"ZClawBench"},{"benchmarks":[{"benchmark_id":"llm-stats-zebralogic","domain":"reasoning","name":"ZebraLogic","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/zebralogic?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/zebralogic?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zebralogic?top_n=500","title":"ZebraLogic"},{"benchmarks":[{"benchmark_id":"llm-stats-zerobench-sub","domain":"multimodal","name":"ZEROBench-Sub","released":null,"url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench-sub?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/zerobench-sub?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench-sub?top_n=500","title":"ZEROBench-Sub"},{"benchmarks":[{"benchmark_id":"llm-stats-zerobench","domain":"multimodal","name":"ZEROBench","released":"2025-02-13","url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench?top_n=500"}],"document_type":"registry_page","id":"llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/zerobench?top_n=500","source":"llm_stats","source_url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench?top_n=500","title":"ZEROBench"},{"benchmarks":[{"benchmark_id":"agieval","domain":"knowledge","name":"AGIEval","released":"2023-04-13","url":"https://github.com/ruixiangcui/AGIEval"},{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"bfcl","domain":"tool_use","name":"BFCL","released":"2024-02-26","url":"https://gorilla.cs.berkeley.edu/leaderboard.html"},{"benchmark_id":"drop","domain":"reasoning","name":"DROP","released":"2019-03-01","url":"https://allenai.org/data/drop"},{"benchmark_id":"gpqa","domain":"science","name":"GPQA (full)","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"gsm8k","domain":"math","name":"GSM8K","released":"2021-10-27","url":"https://github.com/openai/grade-school-math"},{"benchmark_id":"humaneval","domain":"coding","name":"HumanEval","released":"2021-07-07","url":"https://github.com/openai/human-eval"},{"benchmark_id":"ifbench","domain":"instruction_following","name":"IFBench","released":"2025-07-03","url":"https://arxiv.org/abs/2507.02833"},{"benchmark_id":"ifeval","domain":"instruction_following","name":"IFEval","released":"2023-11-14","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"mbpp","domain":"coding","name":"MBPP","released":"2021-08-16","url":"https://github.com/google-research/google-research/tree/master/mbpp"},{"benchmark_id":"mmlu","domain":"knowledge","name":"MMLU","released":"2020-09-07","url":"https://github.com/hendrycks/test"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"}],"document_type":"release_post","id":"model_reports:ai2_olmo_3","model_name":"Olmo 3 (7B and 32B)","organization":"Ai2","published":"2025-11-20","retrieved_at":"2026-08-15","source":"model_reports","source_id":"ai2_olmo_3","source_url":"https://allenai.org/blog/olmo3","title":"Olmo 3 (7B and 32B)"},{"benchmarks":[{"benchmark_id":"ai4ai_bench","domain":"agents","name":"AI4AI-Bench","released":"2026-08-20","url":"https://lab.einsia.ai/ai4ai/"}],"document_type":"benchmark_leaderboard","id":"model_reports:ai4ai_bench_paper_v1_table3","model_name":null,"organization":"Navers Lab / Einsia.AI","published":"2026-08-20","retrieved_at":"2026-10-27","source":"model_reports","source_id":"ai4ai_bench_paper_v1_table3","source_url":"https://arxiv.org/html/2608.20318v1#S3.T3","title":"AI4AI-Bench paper v1 — mean mapped scores, Table 3"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"ifeval","domain":"instruction_following","name":"IFEval","released":"2023-11-14","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"tau_bench","domain":"tool_use","name":"tau-bench","released":"2024-06-17","url":"https://github.com/sierra-research/tau-bench"}],"document_type":"system_card","id":"model_reports:anthropic_claude_3_7_sonnet","model_name":"Claude 3.7 Sonnet","organization":"Anthropic","published":"2025-02-24","retrieved_at":"2026-08-02","source":"model_reports","source_id":"anthropic_claude_3_7_sonnet","source_url":"https://www.anthropic.com/news/claude-3-7-sonnet","title":"Claude 3.7 Sonnet"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"mmmlu","domain":"multilingual","name":"MMMLU","released":"2024-09-24","url":"https://huggingface.co/datasets/openai/MMMLU"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"},{"benchmark_id":"mrcr","domain":"long_context","name":"MRCR","released":"2025-04-14","url":"https://huggingface.co/datasets/openai/mrcr"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"tau_bench","domain":"tool_use","name":"tau-bench","released":"2024-06-17","url":"https://github.com/sierra-research/tau-bench"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"}],"document_type":"system_card","id":"model_reports:anthropic_claude_4_system_card","model_name":"Claude Opus 4 and Sonnet 4","organization":"Anthropic","published":"2025-05-22","retrieved_at":"2026-08-02","source":"model_reports","source_id":"anthropic_claude_4_system_card","source_url":"https://www.anthropic.com/news/claude-4","title":"Claude Opus 4 and Sonnet 4"},{"benchmarks":[{"benchmark_id":"automationbench","domain":"agent","name":"AutomationBench","released":"2026-03-10","url":"https://github.com/zapier/automation-bench"},{"benchmark_id":"biomysterybench","domain":"biology","name":"BioMysteryBench","released":"2026-04-29","url":"https://huggingface.co/datasets/Anthropic/BioMysteryBench-full"},{"benchmark_id":"blueprint_bench_2","domain":"spatial_reasoning","name":"Blueprint-Bench 2","released":"2026-05-04","url":"https://andonlabs.com/evals/blueprint-bench-2"},{"benchmark_id":"cursor_bench","domain":"coding_agent","name":"CursorBench","released":"2025-10-29","url":"https://cursor.com/blog/cursor-bench"},{"benchmark_id":"exploitbench","domain":"security","name":"ExploitBench","released":"2026-05-13","url":"https://exploitbench.ai/"},{"benchmark_id":"frontiercode","domain":"coding_agent","name":"FrontierCode","released":"2026-06-08","url":"https://cognition.com/blog/frontier-code"},{"benchmark_id":"gdp_pdf","domain":"multimodal","name":"GDP.pdf","released":"2026-04-14","url":"https://surgehq.ai/blog/gdp-pdf-can-100b-ai-models-master-the-documents-that-run-the-world"},{"benchmark_id":"gdpval","domain":"professional","name":"GDPval","released":"2025-09-25","url":"https://openai.com/index/gdpval/"},{"benchmark_id":"healthbench_professional","domain":"health","name":"HealthBench Professional","released":"2026-04-22","url":"https://cdn.openai.com/dd128428-0184-4e25-b155-3a7686c7d744/HealthBench-Professional.pdf"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"legal_agent_benchmark","domain":"legal","name":"Legal Agent Benchmark","released":"2026-05-06","url":"https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark"},{"benchmark_id":"osworld","domain":"computer_use","name":"OSWorld","released":"2024-04-11","url":"https://os-world.github.io/"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"vibench","domain":"coding_agent","name":"ViBench","released":"2026-05-26","url":"https://vibench.ai/"}],"document_type":"release_post","id":"model_reports:anthropic_claude_fable_5_mythos_5","model_name":"Claude Fable 5 and Mythos 5","organization":"Anthropic","published":"2026-06-09","retrieved_at":"2026-08-02","source":"model_reports","source_id":"anthropic_claude_fable_5_mythos_5","source_url":"https://www.anthropic.com/news/claude-fable-5-mythos-5","title":"Claude Fable 5 and Mythos 5"},{"benchmarks":[{"benchmark_id":"arc_agi_3","domain":"reasoning","name":"ARC-AGI-3","released":"2026-01-15","url":"https://arcprize.org/arc-agi/3/"},{"benchmark_id":"automationbench","domain":"agent","name":"AutomationBench","released":"2026-03-10","url":"https://github.com/zapier/automation-bench"},{"benchmark_id":"cursor_bench","domain":"coding_agent","name":"CursorBench","released":"2025-10-29","url":"https://cursor.com/blog/cursor-bench"},{"benchmark_id":"deepsearchqa","domain":"agent","name":"DeepSearchQA","released":"2026-02-10","url":"https://huggingface.co/datasets/PokeeAI/DeepSearchQA"},{"benchmark_id":"gdpval","domain":"professional","name":"GDPval","released":"2025-09-25","url":"https://openai.com/index/gdpval/"},{"benchmark_id":"osworld","domain":"computer_use","name":"OSWorld","released":"2024-04-11","url":"https://os-world.github.io/"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"}],"document_type":"system_card","id":"model_reports:anthropic_claude_opus_5_system_card","model_name":"Claude Opus 5","organization":"Anthropic","published":"2026-07-24","retrieved_at":"2026-08-02","source":"model_reports","source_id":"anthropic_claude_opus_5_system_card","source_url":"https://www.anthropic.com/news/claude-opus-5","title":"Claude Opus 5"},{"benchmarks":[{"benchmark_id":"asi_bench","domain":"science","name":"ASI-Bench","released":"2026-08-18","url":"https://arxiv.org/abs/2608.17271"}],"document_type":"benchmark_leaderboard","id":"model_reports:asi_bench_results_table","model_name":null,"organization":"Apexin AI","published":null,"retrieved_at":"2026-09-18","source":"model_reports","source_id":"asi_bench_results_table","source_url":"https://arxiv.org/html/2608.17271v1","title":"ASI-Bench results table"},{"benchmarks":[{"benchmark_id":"blenderworld","domain":"multimodal","name":"BlenderWorld","released":"2026-10-14","url":"https://blenderworld.github.io/"}],"document_type":"benchmark_leaderboard","id":"model_reports:blenderworld_v1_results_table","model_name":null,"organization":"BlenderWorld authors","published":"2026-10-14","retrieved_at":"2026-10-25","source":"model_reports","source_id":"blenderworld_v1_results_table","source_url":"https://arxiv.org/html/2610.23986v1#S5.T2","title":"BlenderWorld paper v1 · Table 2"},{"benchmarks":[{"benchmark_id":"brain_to_text_24","domain":"science","name":"Brain-to-Text Benchmark '24","released":"2023-06-01","url":"https://eval.ai/web/challenges/challenge-page/2099/overview"}],"document_type":"benchmark_leaderboard","id":"model_reports:brain_to_text_24_evalai_leaderboard","model_name":null,"organization":"Stanford Neural Prosthetics Translational Laboratory","published":null,"retrieved_at":"2026-09-19","source":"model_reports","source_id":"brain_to_text_24_evalai_leaderboard","source_url":"https://eval.ai/web/challenges/challenge-page/2099/overview","title":"Brain-to-Text Benchmark '24 leaderboard"},{"benchmarks":[{"benchmark_id":"brain_to_text_25","domain":"science","name":"Brain-to-Text '25","released":"2025-07-07","url":"https://www.kaggle.com/competitions/brain-to-text-25"}],"document_type":"benchmark_leaderboard","id":"model_reports:brain_to_text_25_kaggle_leaderboard","model_name":null,"organization":"UC Davis Neuroprosthetics Lab","published":null,"retrieved_at":"2026-09-19","source":"model_reports","source_id":"brain_to_text_25_kaggle_leaderboard","source_url":"https://www.kaggle.com/competitions/brain-to-text-25/leaderboard","title":"Brain-to-Text '25 private leaderboard (2026-09-19 public snapshot)"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"arc_agi","domain":"reasoning","name":"ARC-AGI","released":"2019-11-05","url":"https://arcprize.org/"},{"benchmark_id":"babe","domain":"reasoning","name":"BABE","released":"2026-02-05","url":"https://arxiv.org/abs/2602.05857"},{"benchmark_id":"beyond_aime","domain":"math","name":"BeyondAIME","released":null,"url":"https://huggingface.co/datasets/ByteDance-Seed/BeyondAIME"},{"benchmark_id":"codeforces","domain":"coding","name":"Codeforces","released":"2010-02-19","url":"https://codeforces.com/"},{"benchmark_id":"collie","domain":"reasoning","name":"COLLIE","released":"2023-07-17","url":"https://collie-benchmark.github.io/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"healthbench","domain":"health","name":"HealthBench","released":"2025-05-12","url":"https://openai.com/index/healthbench/"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"hmmt","domain":"math","name":"HMMT","released":"2025-02-15","url":"https://www.hmmt.org/"},{"benchmark_id":"imo_answer_bench","domain":"math","name":"IMOAnswerBench","released":"2025-09-18","url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench"},{"benchmark_id":"inverse_ifeval","domain":"instruction_following","name":"Inverse IFEval","released":"2025-09-04","url":"https://arxiv.org/abs/2509.04292"},{"benchmark_id":"korbench","domain":"reasoning","name":"KORBench","released":"2024-10-09","url":"https://kor-bench.github.io/"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"lpfqa","domain":"knowledge","name":"LPFQA","released":null,"url":"https://huggingface.co/datasets/m-a-p/LPFQA"},{"benchmark_id":"mars_bench","domain":"reasoning","name":"MARS-Bench","released":null,"url":"https://syuchin.github.io/MARS-Bench/"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"multi_challenge","domain":"reasoning","name":"MultiChallenge","released":"2025-01-29","url":"https://arxiv.org/abs/2501.17399"},{"benchmark_id":"multi_swe_bench","domain":"coding_agent","name":"Multi-SWE-Bench","released":"2025-04-03","url":"https://github.com/multi-swe-bench/multi-swe-bench/"},{"benchmark_id":"phybench","domain":"science","name":"PhyBench","released":"2025-04-22","url":"https://github.com/phybench-official/phybench"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"},{"benchmark_id":"super_gpqa","domain":"knowledge","name":"SuperGPQA","released":"2025-02-20","url":"https://arxiv.org/abs/2502.14739"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"swe_lancer","domain":"coding_agent","name":"SWE-Lancer","released":"2025-02-17","url":"https://github.com/openai/frontier-evals/tree/main/project/swelancer"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"zerobench","domain":"multimodal","name":"ZeroBench","released":"2025-02-13","url":"https://github.com/jonathan-roberts1/zerobench"}],"document_type":"model_card","id":"model_reports:bytedance_seed2_0_model_card","model_name":"Seed2.0 (Pro / Lite / Mini)","organization":"ByteDance","published":"2026-02-14","retrieved_at":"2026-09-11","source":"model_reports","source_id":"bytedance_seed2_0_model_card","source_url":"https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model%20Card.pdf","title":"Seed2.0 (Pro / Lite / Mini)"},{"benchmarks":[{"benchmark_id":"aider_polyglot","domain":"coding","name":"Aider Polyglot","released":"2024-12-21","url":"https://aider.chat/docs/leaderboards/"},{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"arena_hard","domain":"human_preference","name":"Arena-Hard","released":"2024-04-19","url":"https://github.com/lmarena/arena-hard-auto"},{"benchmark_id":"drop","domain":"reasoning","name":"DROP","released":"2019-03-01","url":"https://allenai.org/data/drop"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"ifeval","domain":"instruction_following","name":"IFEval","released":"2023-11-14","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mmlu","domain":"knowledge","name":"MMLU","released":"2020-09-07","url":"https://github.com/hendrycks/test"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"mmlu_redux","domain":"knowledge","name":"MMLU-Redux","released":"2024-06-06","url":"https://github.com/aryopg/mmlu-redux"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"}],"document_type":"technical_report","id":"model_reports:deepseek_r1_report","model_name":"DeepSeek-R1","organization":"DeepSeek","published":"2025-01-22","retrieved_at":"2026-08-02","source":"model_reports","source_id":"deepseek_r1_report","source_url":"https://arxiv.org/abs/2501.12948","title":"DeepSeek-R1"},{"benchmarks":[{"benchmark_id":"agieval","domain":"knowledge","name":"AGIEval","released":"2023-04-13","url":"https://github.com/ruixiangcui/AGIEval"},{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"arena_hard","domain":"human_preference","name":"Arena-Hard","released":"2024-04-19","url":"https://github.com/lmarena/arena-hard-auto"},{"benchmark_id":"drop","domain":"reasoning","name":"DROP","released":"2019-03-01","url":"https://allenai.org/data/drop"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"gsm8k","domain":"math","name":"GSM8K","released":"2021-10-27","url":"https://github.com/openai/grade-school-math"},{"benchmark_id":"humaneval","domain":"coding","name":"HumanEval","released":"2021-07-07","url":"https://github.com/openai/human-eval"},{"benchmark_id":"ifeval","domain":"instruction_following","name":"IFEval","released":"2023-11-14","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mbpp","domain":"coding","name":"MBPP","released":"2021-08-16","url":"https://github.com/google-research/google-research/tree/master/mbpp"},{"benchmark_id":"mmlu","domain":"knowledge","name":"MMLU","released":"2020-09-07","url":"https://github.com/hendrycks/test"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"mmlu_redux","domain":"knowledge","name":"MMLU-Redux","released":"2024-06-06","url":"https://github.com/aryopg/mmlu-redux"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"}],"document_type":"technical_report","id":"model_reports:deepseek_v3_report","model_name":"DeepSeek-V3","organization":"DeepSeek","published":"2024-12-27","retrieved_at":"2026-08-02","source":"model_reports","source_id":"deepseek_v3_report","source_url":"https://arxiv.org/abs/2412.19437","title":"DeepSeek-V3"},{"benchmarks":[{"benchmark_id":"agents_last_exam","domain":"coding_agent","name":"Agents' Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"automationbench","domain":"agent","name":"AutomationBench","released":"2026-03-10","url":"https://github.com/zapier/automation-bench"},{"benchmark_id":"babyvision","domain":"multimodal","name":"BabyVision","released":"2025-12-01","url":"https://github.com/babyvision/babyvision"},{"benchmark_id":"chartography","domain":"vision","name":"Chartography","released":"2025-06-01","url":"https://github.com/Chartography/Chartography"},{"benchmark_id":"codeforces","domain":"coding","name":"Codeforces","released":"2010-02-19","url":"https://codeforces.com/"},{"benchmark_id":"cybergym","domain":"security","name":"CyberGym","released":null,"url":"https://openreview.net/forum?id=2YvbLQEdYt"},{"benchmark_id":"deepswe","domain":"coding_agent","name":"DeepSWE","released":"2026-05-26","url":"https://deepswe.datacurve.ai/blog/deepswe"},{"benchmark_id":"exploitgym","domain":"security","name":"ExploitGym","released":null,"url":"https://github.com/sunblaze-ucb/exploitgym"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"matharena_apex_2025","domain":"math","name":"MathArena Apex 2025","released":null,"url":"https://matharena.ai/apex"},{"benchmark_id":"nl2repo","domain":"coding_agent","name":"NL2Repo","released":"2025-06-15","url":"https://github.com/nl2repo/nl2repo"},{"benchmark_id":"programbench","domain":"coding","name":"ProgramBench","released":"2026-05-05","url":"https://arxiv.org/abs/2605.03546"},{"benchmark_id":"sec_bench_pro","domain":"security","name":"SEC-Bench Pro","released":null,"url":"https://github.com/sec-bench/sec-bench"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"zerobench","domain":"multimodal","name":"ZeroBench","released":"2025-02-13","url":"https://github.com/jonathan-roberts1/zerobench"}],"document_type":"technical_report","id":"model_reports:deepseek_v4_1_flash_technical_report","model_name":"DeepSeek-V4.1-Flash","organization":"DeepSeek","published":"2026-09-10","retrieved_at":"2026-09-11","source":"model_reports","source_id":"deepseek_v4_1_flash_technical_report","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf","title":"DeepSeek-V4.1-Flash"},{"benchmarks":[{"benchmark_id":"agieval","domain":"knowledge","name":"AGIEval","released":"2023-04-13","url":"https://github.com/ruixiangcui/AGIEval"},{"benchmark_id":"apex_agents","domain":"agent","name":"APEX-Agents","released":"2026-01-20","url":"https://arxiv.org/abs/2601.14242"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"codeforces","domain":"coding","name":"Codeforces","released":"2010-02-19","url":"https://codeforces.com/"},{"benchmark_id":"drop","domain":"reasoning","name":"DROP","released":"2019-03-01","url":"https://allenai.org/data/drop"},{"benchmark_id":"gdpval","domain":"professional","name":"GDPval","released":"2025-09-25","url":"https://openai.com/index/gdpval/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"gsm8k","domain":"math","name":"GSM8K","released":"2021-10-27","url":"https://github.com/openai/grade-school-math"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"hmmt","domain":"math","name":"HMMT","released":"2025-02-15","url":"https://www.hmmt.org/"},{"benchmark_id":"humaneval","domain":"coding","name":"HumanEval","released":"2021-07-07","url":"https://github.com/openai/human-eval"},{"benchmark_id":"imo_answer_bench","domain":"math","name":"IMOAnswerBench","released":"2025-09-18","url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"longbench","domain":"long_context","name":"LongBench","released":"2023-08-28","url":"https://github.com/THUDM/LongBench"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mcp_atlas","domain":"tool_use","name":"MCP Atlas","released":"2025-11-18","url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},{"benchmark_id":"mmlu","domain":"knowledge","name":"MMLU","released":"2020-09-07","url":"https://github.com/hendrycks/test"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"mmlu_redux","domain":"knowledge","name":"MMLU-Redux","released":"2024-06-06","url":"https://github.com/aryopg/mmlu-redux"},{"benchmark_id":"mmmlu","domain":"multilingual","name":"MMMLU","released":"2024-09-24","url":"https://huggingface.co/datasets/openai/MMMLU"},{"benchmark_id":"mrcr","domain":"long_context","name":"MRCR","released":"2025-04-14","url":"https://huggingface.co/datasets/openai/mrcr"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"},{"benchmark_id":"super_gpqa","domain":"knowledge","name":"SuperGPQA","released":"2025-02-20","url":"https://arxiv.org/abs/2502.14739"},{"benchmark_id":"swe_bench_multilingual","domain":"coding_agent","name":"SWE-bench Multilingual","released":"2025-04-22","url":"https://www.swebench.com/multilingual.html"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"tool_decathlon","domain":"tool_use","name":"Tool Decathlon","released":"2025-10-28","url":"https://toolathlon.xyz/"}],"document_type":"model_card","id":"model_reports:deepseek_v4_model_card","model_name":"DeepSeek-V4-Pro","organization":"DeepSeek","published":"2026-04-22","retrieved_at":"2026-08-02","source":"model_reports","source_id":"deepseek_v4_model_card","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","title":"DeepSeek-V4-Pro"},{"benchmarks":[{"benchmark_id":"automationbench","domain":"agent","name":"AutomationBench","released":"2026-03-10","url":"https://github.com/zapier/automation-bench"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"tool_decathlon","domain":"tool_use","name":"Tool Decathlon","released":"2025-10-28","url":"https://toolathlon.xyz/"}],"document_type":"model_card","id":"model_reports:deepseek_v4_pro_0813_model_card","model_name":"DeepSeek-V4-Pro-0813","organization":"DeepSeek","published":"2026-08-13","retrieved_at":"2026-08-15","source":"model_reports","source_id":"deepseek_v4_pro_0813_model_card","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813","title":"DeepSeek-V4-Pro-0813"},{"benchmarks":[{"benchmark_id":"apex_agents","domain":"agent","name":"APEX-Agents","released":"2026-01-20","url":"https://arxiv.org/abs/2601.14242"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"codeforces","domain":"coding","name":"Codeforces","released":"2010-02-19","url":"https://codeforces.com/"},{"benchmark_id":"gdpval","domain":"professional","name":"GDPval","released":"2025-09-25","url":"https://openai.com/index/gdpval/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"hmmt","domain":"math","name":"HMMT","released":"2025-02-15","url":"https://www.hmmt.org/"},{"benchmark_id":"imo_answer_bench","domain":"math","name":"IMOAnswerBench","released":"2025-09-18","url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"longbench","domain":"long_context","name":"LongBench","released":"2023-08-28","url":"https://github.com/THUDM/LongBench"},{"benchmark_id":"mcp_atlas","domain":"tool_use","name":"MCP Atlas","released":"2025-11-18","url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"mrcr","domain":"long_context","name":"MRCR","released":"2025-04-14","url":"https://huggingface.co/datasets/openai/mrcr"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"},{"benchmark_id":"swe_bench_multilingual","domain":"coding_agent","name":"SWE-bench Multilingual","released":"2025-04-22","url":"https://www.swebench.com/multilingual.html"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"tool_decathlon","domain":"tool_use","name":"Tool Decathlon","released":"2025-10-28","url":"https://toolathlon.xyz/"}],"document_type":"technical_report","id":"model_reports:deepseek_v4_technical_report","model_name":"DeepSeek-V4","organization":"DeepSeek","published":"2026-06-25","retrieved_at":"2026-08-02","source":"model_reports","source_id":"deepseek_v4_technical_report","source_url":"https://arxiv.org/abs/2606.19348","title":"DeepSeek-V4"},{"benchmarks":[{"benchmark_id":"frontier_engineering","domain":"agents","name":"Frontier-Engineering","released":"2026-04-14","url":"https://lab.einsia.ai/frontier-eng/"}],"document_type":"benchmark_leaderboard","id":"model_reports:einsia_frontier_engineering_leaderboard_2026_09_15","model_name":null,"organization":"Navers Lab / Einsia","published":null,"retrieved_at":"2026-10-27","source":"model_reports","source_id":"einsia_frontier_engineering_leaderboard_2026_09_15","source_url":"https://lab.einsia.ai/frontier-eng/leaderboard/","title":"Frontier-Engineering corrected Medal leaderboard · scores dated 2026-09-15"},{"benchmarks":[{"benchmark_id":"quant_arena","domain":"finance","name":"Quant-Arena","released":"2026-10-21","url":"https://lab.einsia.ai/quant-arena/"}],"document_type":"benchmark_leaderboard","id":"model_reports:einsia_quant_arena_v1_results","model_name":null,"organization":"Navers Lab / Einsia AI","published":"2026-10-21","retrieved_at":"2026-10-27","source":"model_reports","source_id":"einsia_quant_arena_v1_results","source_url":"https://arxiv.org/html/2610.30113v1","title":"Quant-Arena paper v1 · configuration results and proprietary reference"},{"benchmarks":[{"benchmark_id":"swe_refactor_bench","domain":"coding","name":"SWE Refactor Bench","released":"2026-08-24","url":"https://lab.einsia.ai/swe-refactor-bench/"}],"document_type":"benchmark_leaderboard","id":"model_reports:einsia_swe_refactor_bench_v1_table3","model_name":null,"organization":"Navers Lab / Einsia","published":"2026-08-24","retrieved_at":"2026-10-27","source":"model_reports","source_id":"einsia_swe_refactor_bench_v1_table3","source_url":"https://arxiv.org/html/2608.23564v1#S3.T3","title":"SWE Refactor Bench paper v1 · Table 3"},{"benchmarks":[{"benchmark_id":"frontier_challenge","domain":"scientific_agent","name":"FrontierChallenge","released":"2026-08-25","url":"https://apodexai.github.io/FrontierAgent/benchmarks/FrontierChallenge/"}],"document_type":"benchmark_leaderboard","id":"model_reports:frontier_challenge_leaderboard_2026_08_25","model_name":null,"organization":"ApodexAI","published":null,"retrieved_at":"2026-09-02","source":"model_reports","source_id":"frontier_challenge_leaderboard_2026_08_25","source_url":"https://apodexai.github.io/FrontierAgent/benchmarks/FrontierChallenge/","title":"FrontierChallenge leaderboard"},{"benchmarks":[{"benchmark_id":"chartqa","domain":"multimodal","name":"ChartQA","released":"2022-03-19","url":"https://github.com/vis-nlp/ChartQA"},{"benchmark_id":"docvqa","domain":"multimodal","name":"DocVQA","released":"2020-07-01","url":"https://www.docvqa.org/"},{"benchmark_id":"drop","domain":"reasoning","name":"DROP","released":"2019-03-01","url":"https://allenai.org/data/drop"},{"benchmark_id":"gsm8k","domain":"math","name":"GSM8K","released":"2021-10-27","url":"https://github.com/openai/grade-school-math"},{"benchmark_id":"humaneval","domain":"coding","name":"HumanEval","released":"2021-07-07","url":"https://github.com/openai/human-eval"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mathvista","domain":"multimodal","name":"MathVista","released":"2023-10-03","url":"https://mathvista.github.io/"},{"benchmark_id":"mmlu","domain":"knowledge","name":"MMLU","released":"2020-09-07","url":"https://github.com/hendrycks/test"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"}],"document_type":"technical_report","id":"model_reports:google_gemini_1_5_report","model_name":"Gemini 1.5","organization":"Google","published":"2024-03-08","retrieved_at":"2026-08-02","source":"model_reports","source_id":"google_gemini_1_5_report","source_url":"https://arxiv.org/abs/2403.05530","title":"Gemini 1.5"},{"benchmarks":[{"benchmark_id":"aider_polyglot","domain":"coding","name":"Aider Polyglot","released":"2024-12-21","url":"https://aider.chat/docs/leaderboards/"},{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"arc_agi","domain":"reasoning","name":"ARC-AGI","released":"2019-11-05","url":"https://arcprize.org/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"mathvista","domain":"multimodal","name":"MathVista","released":"2023-10-03","url":"https://mathvista.github.io/"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"},{"benchmark_id":"mrcr","domain":"long_context","name":"MRCR","released":"2025-04-14","url":"https://huggingface.co/datasets/openai/mrcr"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"video_mme","domain":"multimodal","name":"Video-MME","released":"2024-05-31","url":"https://video-mme.github.io/"}],"document_type":"technical_report","id":"model_reports:google_gemini_2_5_report","model_name":"Gemini 2.5 Pro","organization":"Google","published":"2025-06-17","retrieved_at":"2026-08-02","source":"model_reports","source_id":"google_gemini_2_5_report","source_url":"https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf","title":"Gemini 2.5 Pro"},{"benchmarks":[{"benchmark_id":"apex_agents","domain":"agent","name":"APEX-Agents","released":"2026-01-20","url":"https://arxiv.org/abs/2601.14242"},{"benchmark_id":"arc_agi_2","domain":"reasoning","name":"ARC-AGI-2","released":"2025-03-24","url":"https://arcprize.org/arc-agi/2/"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"livecodebench_pro","domain":"coding","name":"LiveCodeBench Pro","released":"2025-06-13","url":"https://livecodebenchpro.com/"},{"benchmark_id":"mcp_atlas","domain":"tool_use","name":"MCP Atlas","released":"2025-11-18","url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},{"benchmark_id":"mmmlu","domain":"multilingual","name":"MMMLU","released":"2024-09-24","url":"https://huggingface.co/datasets/openai/MMMLU"},{"benchmark_id":"mmmu_pro","domain":"multimodal","name":"MMMU-Pro","released":"2024-09-04","url":"https://arxiv.org/abs/2409.02813"},{"benchmark_id":"mrcr","domain":"long_context","name":"MRCR","released":"2025-04-14","url":"https://huggingface.co/datasets/openai/mrcr"},{"benchmark_id":"scicode","domain":"science","name":"SciCode","released":"2024-07-18","url":"https://scicode-bench.github.io/"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"tau2_bench","domain":"tool_use","name":"tau2-bench","released":"2025-06-09","url":"https://arxiv.org/abs/2506.07982"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"}],"document_type":"model_card","id":"model_reports:google_gemini_3_1_pro_model_card","model_name":"Gemini 3.1 Pro","organization":"Google","published":"2026-02-19","retrieved_at":"2026-08-02","source":"model_reports","source_id":"google_gemini_3_1_pro_model_card","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","title":"Gemini 3.1 Pro"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"bigbench_extra_hard","domain":"reasoning","name":"BIG-Bench Extra Hard","released":"2025-02-26","url":"https://arxiv.org/abs/2502.19187"},{"benchmark_id":"codeforces","domain":"coding","name":"Codeforces","released":"2010-02-19","url":"https://codeforces.com/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"mathvision","domain":"multimodal","name":"MathVision","released":"2024-02-22","url":"https://mathllm.github.io/mathvision/"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"mmmlu","domain":"multilingual","name":"MMMLU","released":"2024-09-24","url":"https://huggingface.co/datasets/openai/MMMLU"},{"benchmark_id":"mmmu_pro","domain":"multimodal","name":"MMMU-Pro","released":"2024-09-04","url":"https://arxiv.org/abs/2409.02813"},{"benchmark_id":"mrcr","domain":"long_context","name":"MRCR","released":"2025-04-14","url":"https://huggingface.co/datasets/openai/mrcr"},{"benchmark_id":"omnidocbench","domain":"multimodal","name":"OmniDocBench","released":"2024-12-10","url":"https://github.com/opendatalab/OmniDocBench"},{"benchmark_id":"tau2_bench","domain":"tool_use","name":"tau2-bench","released":"2025-06-09","url":"https://arxiv.org/abs/2506.07982"}],"document_type":"model_card","id":"model_reports:google_gemma_4_model_card","model_name":"Gemma 4 (31B)","organization":"Google","published":"2026-03-11","retrieved_at":"2026-08-02","source":"model_reports","source_id":"google_gemma_4_model_card","source_url":"https://huggingface.co/google/gemma-4-31B-it","title":"Gemma 4 (31B)"},{"benchmarks":[{"benchmark_id":"jointavbench","domain":"multimodal","name":"JointAVBench","released":"2025-12-14","url":"https://arxiv.org/abs/2512.12772"}],"document_type":"benchmark_leaderboard","id":"model_reports:jointavbench_v2_results_table","model_name":null,"organization":"Renmin University of China","published":null,"retrieved_at":"2026-09-18","source":"model_reports","source_id":"jointavbench_v2_results_table","source_url":"https://arxiv.org/html/2512.12772v2","title":"JointAVBench v2 results table"},{"benchmarks":[{"benchmark_id":"agieval","domain":"knowledge","name":"AGIEval","released":"2023-04-13","url":"https://github.com/ruixiangcui/AGIEval"},{"benchmark_id":"arena_hard","domain":"human_preference","name":"Arena-Hard","released":"2024-04-19","url":"https://github.com/lmarena/arena-hard-auto"},{"benchmark_id":"bfcl","domain":"tool_use","name":"BFCL","released":"2024-02-26","url":"https://gorilla.cs.berkeley.edu/leaderboard.html"},{"benchmark_id":"drop","domain":"reasoning","name":"DROP","released":"2019-03-01","url":"https://allenai.org/data/drop"},{"benchmark_id":"gpqa","domain":"science","name":"GPQA (full)","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"gsm8k","domain":"math","name":"GSM8K","released":"2021-10-27","url":"https://github.com/openai/grade-school-math"},{"benchmark_id":"humaneval","domain":"coding","name":"HumanEval","released":"2021-07-07","url":"https://github.com/openai/human-eval"},{"benchmark_id":"ifeval","domain":"instruction_following","name":"IFEval","released":"2023-11-14","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mbpp","domain":"coding","name":"MBPP","released":"2021-08-16","url":"https://github.com/google-research/google-research/tree/master/mbpp"},{"benchmark_id":"mmlu","domain":"knowledge","name":"MMLU","released":"2020-09-07","url":"https://github.com/hendrycks/test"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"}],"document_type":"model_card","id":"model_reports:meta_llama_3_1","model_name":"Llama 3.1 405B","organization":"Meta","published":"2024-07-23","retrieved_at":"2026-08-02","source":"model_reports","source_id":"meta_llama_3_1","source_url":"https://ai.meta.com/blog/meta-llama-3-1/","title":"Llama 3.1 405B"},{"benchmarks":[{"benchmark_id":"chartqa","domain":"multimodal","name":"ChartQA","released":"2022-03-19","url":"https://github.com/vis-nlp/ChartQA"},{"benchmark_id":"chatbot_arena","domain":"human_preference","name":"Chatbot Arena","released":"2023-05-03","url":"https://lmarena.ai/"},{"benchmark_id":"docvqa","domain":"multimodal","name":"DocVQA","released":"2020-07-01","url":"https://www.docvqa.org/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"mathvista","domain":"multimodal","name":"MathVista","released":"2023-10-03","url":"https://mathvista.github.io/"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"mmmlu","domain":"multilingual","name":"MMMLU","released":"2024-09-24","url":"https://huggingface.co/datasets/openai/MMMLU"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"},{"benchmark_id":"mtob","domain":"long_context","name":"MTOB","released":"2023-09-28","url":"https://arxiv.org/abs/2309.16575"}],"document_type":"model_card","id":"model_reports:meta_llama_4","model_name":"Llama 4","organization":"Meta","published":"2025-04-05","retrieved_at":"2026-08-02","source":"model_reports","source_id":"meta_llama_4","source_url":"https://ai.meta.com/blog/llama-4-multimodal-intelligence/","title":"Llama 4"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"}],"document_type":"model_card","id":"model_reports:microsoft_phi_4_mini_reasoning_model_card","model_name":"Phi-4-mini-reasoning (3.8B)","organization":"Microsoft","published":"2025-04-30","retrieved_at":"2026-10-10","source":"model_reports","source_id":"microsoft_phi_4_mini_reasoning_model_card","source_url":"https://huggingface.co/microsoft/Phi-4-mini-reasoning/blob/661e72224c678e7360429603fd898712ee0c74b9/README.md","title":"Phi-4-mini-reasoning (3.8B)"},{"benchmarks":[{"benchmark_id":"gsm8k","domain":"math","name":"GSM8K","released":"2021-10-27","url":"https://github.com/openai/grade-school-math"},{"benchmark_id":"humaneval","domain":"coding","name":"HumanEval","released":"2021-07-07","url":"https://github.com/openai/human-eval"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mbpp","domain":"coding","name":"MBPP","released":"2021-08-16","url":"https://github.com/google-research/google-research/tree/master/mbpp"},{"benchmark_id":"mmlu","domain":"knowledge","name":"MMLU","released":"2020-09-07","url":"https://github.com/hendrycks/test"}],"document_type":"release_post","id":"model_reports:mistral_large_2","model_name":"Mistral Large 2","organization":"Mistral","published":"2024-07-24","retrieved_at":"2026-08-02","source":"model_reports","source_id":"mistral_large_2","source_url":"https://mistral.ai/news/mistral-large-2407/","title":"Mistral Large 2"},{"benchmarks":[{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"mmmlu","domain":"multilingual","name":"MMMLU","released":"2024-09-24","url":"https://huggingface.co/datasets/openai/MMMLU"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"}],"document_type":"model_card","id":"model_reports:mistral_large_3","model_name":"Mistral Large 3 (675B)","organization":"Mistral","published":"2025-11-28","retrieved_at":"2026-08-02","source":"model_reports","source_id":"mistral_large_3","source_url":"https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512","title":"Mistral Large 3 (675B)"},{"benchmarks":[{"benchmark_id":"aider_polyglot","domain":"coding","name":"Aider Polyglot","released":"2024-12-21","url":"https://aider.chat/docs/leaderboards/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"humaneval","domain":"coding","name":"HumanEval","released":"2021-07-07","url":"https://github.com/openai/human-eval"},{"benchmark_id":"ifeval","domain":"instruction_following","name":"IFEval","released":"2023-11-14","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"}],"document_type":"release_post","id":"model_reports:mistral_medium_3","model_name":"Mistral Medium 3","organization":"Mistral","published":"2025-05-07","retrieved_at":"2026-08-02","source":"model_reports","source_id":"mistral_medium_3","source_url":"https://mistral.ai/news/mistral-medium-3","title":"Mistral Medium 3"},{"benchmarks":[{"benchmark_id":"aa_lcr","domain":"long_context","name":"AA-LCR","released":"2025-09-16","url":"https://artificialanalysis.ai/evaluations/aa-lcr"},{"benchmark_id":"apex_agents","domain":"agent","name":"APEX-Agents","released":"2026-01-20","url":"https://arxiv.org/abs/2601.14242"},{"benchmark_id":"automationbench","domain":"agent","name":"AutomationBench","released":"2026-03-10","url":"https://github.com/zapier/automation-bench"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"critpt","domain":"science","name":"CritPt","released":"2025-09-30","url":"https://arxiv.org/abs/2509.26574"},{"benchmark_id":"deepsearchqa","domain":"agent","name":"DeepSearchQA","released":"2026-02-10","url":"https://huggingface.co/datasets/PokeeAI/DeepSearchQA"},{"benchmark_id":"gdpval","domain":"professional","name":"GDPval","released":"2025-09-25","url":"https://openai.com/index/gdpval/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"mathvision","domain":"multimodal","name":"MathVision","released":"2024-02-22","url":"https://mathllm.github.io/mathvision/"},{"benchmark_id":"mcp_atlas","domain":"tool_use","name":"MCP Atlas","released":"2025-11-18","url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},{"benchmark_id":"mcp_mark","domain":"tool_use","name":"MCPMark","released":"2025-09-30","url":"https://mcpmark.ai/"},{"benchmark_id":"mmmu_pro","domain":"multimodal","name":"MMMU-Pro","released":"2024-09-04","url":"https://arxiv.org/abs/2409.02813"},{"benchmark_id":"omnidocbench","domain":"multimodal","name":"OmniDocBench","released":"2024-12-10","url":"https://github.com/opendatalab/OmniDocBench"},{"benchmark_id":"osworld","domain":"computer_use","name":"OSWorld","released":"2024-04-11","url":"https://os-world.github.io/"},{"benchmark_id":"posttrainbench","domain":"ai_research","name":"PostTrainBench","released":"2026-01-20","url":"https://posttrainbench.com/"},{"benchmark_id":"scicode","domain":"science","name":"SciCode","released":"2024-07-18","url":"https://scicode-bench.github.io/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"tool_decathlon","domain":"tool_use","name":"Tool Decathlon","released":"2025-10-28","url":"https://toolathlon.xyz/"},{"benchmark_id":"video_mme","domain":"multimodal","name":"Video-MME","released":"2024-05-31","url":"https://video-mme.github.io/"}],"document_type":"model_card","id":"model_reports:moonshot_kimi_k3_model_card","model_name":"Kimi K3","organization":"Moonshot AI","published":"2026-06-13","retrieved_at":"2026-08-02","source":"model_reports","source_id":"moonshot_kimi_k3_model_card","source_url":"https://huggingface.co/moonshotai/Kimi-K3","title":"Kimi K3"},{"benchmarks":[{"benchmark_id":"omnivideobench","domain":"multimodal","name":"OmniVideoBench","released":"2025-10-12","url":"https://arxiv.org/abs/2510.10689"}],"document_type":"benchmark_leaderboard","id":"model_reports:omnivideobench_v3_results_table","model_name":null,"organization":"NJU-LINK (Nanjing University)","published":null,"retrieved_at":"2026-09-18","source":"model_reports","source_id":"omnivideobench_v3_results_table","source_url":"https://arxiv.org/html/2510.10689v3","title":"OmniVideoBench v3 results table"},{"benchmarks":[{"benchmark_id":"aider_polyglot","domain":"coding","name":"Aider Polyglot","released":"2024-12-21","url":"https://aider.chat/docs/leaderboards/"},{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"ifeval","domain":"instruction_following","name":"IFEval","released":"2023-11-14","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"benchmark_id":"mmlu","domain":"knowledge","name":"MMLU","released":"2020-09-07","url":"https://github.com/hendrycks/test"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"},{"benchmark_id":"mrcr","domain":"long_context","name":"MRCR","released":"2025-04-14","url":"https://huggingface.co/datasets/openai/mrcr"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"video_mme","domain":"multimodal","name":"Video-MME","released":"2024-05-31","url":"https://video-mme.github.io/"}],"document_type":"release_post","id":"model_reports:openai_gpt_4_1","model_name":"GPT-4.1","organization":"OpenAI","published":"2025-04-14","retrieved_at":"2026-08-02","source":"model_reports","source_id":"openai_gpt_4_1","source_url":"https://openai.com/index/gpt-4-1/","title":"GPT-4.1"},{"benchmarks":[{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"}],"document_type":"release_post","id":"model_reports:openai_gpt_5_6_release","model_name":"GPT-5.6","organization":"OpenAI","published":"2026-06-26","retrieved_at":"2026-08-02","source":"model_reports","source_id":"openai_gpt_5_6_release","source_url":"https://openai.com/index/gpt-5-6/","title":"GPT-5.6"},{"benchmarks":[{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"cvebench","domain":"security","name":"CVE-Bench","released":"2025-03-21","url":"https://github.com/uiuc-kang-lab/cve-bench"},{"benchmark_id":"exploitbench","domain":"security","name":"ExploitBench","released":"2026-05-13","url":"https://exploitbench.ai/"},{"benchmark_id":"healthbench","domain":"health","name":"HealthBench","released":"2025-05-12","url":"https://openai.com/index/healthbench/"},{"benchmark_id":"mle_bench","domain":"ai_research","name":"MLE-bench","released":"2024-10-09","url":"https://openai.com/index/mle-bench/"},{"benchmark_id":"posttrainbench","domain":"ai_research","name":"PostTrainBench","released":"2026-01-20","url":"https://posttrainbench.com/"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"}],"document_type":"system_card","id":"model_reports:openai_gpt_5_6_system_card","model_name":"GPT-5.6 (Sol, Terra, Luna)","organization":"OpenAI","published":"2026-07-09","retrieved_at":"2026-08-02","source":"model_reports","source_id":"openai_gpt_5_6_system_card","source_url":"https://deploymentsafety.openai.com/gpt-5-6-preview","title":"GPT-5.6 (Sol, Terra, Luna)"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"healthbench","domain":"health","name":"HealthBench","released":"2025-05-12","url":"https://openai.com/index/healthbench/"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"mle_bench","domain":"ai_research","name":"MLE-bench","released":"2024-10-09","url":"https://openai.com/index/mle-bench/"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"},{"benchmark_id":"mrcr","domain":"long_context","name":"MRCR","released":"2025-04-14","url":"https://huggingface.co/datasets/openai/mrcr"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"tau_bench","domain":"tool_use","name":"tau-bench","released":"2024-06-17","url":"https://github.com/sierra-research/tau-bench"}],"document_type":"system_card","id":"model_reports:openai_gpt_5_system_card","model_name":"GPT-5","organization":"OpenAI","published":"2025-08-07","retrieved_at":"2026-08-02","source":"model_reports","source_id":"openai_gpt_5_system_card","source_url":"https://openai.com/index/gpt-5-system-card/","title":"GPT-5"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"arc_agi","domain":"reasoning","name":"ARC-AGI","released":"2019-11-05","url":"https://arcprize.org/"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"},{"benchmark_id":"simpleqa","domain":"factuality","name":"SimpleQA","released":"2024-10-30","url":"https://openai.com/index/introducing-simpleqa/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"}],"document_type":"system_card","id":"model_reports:openai_o3_o4_mini_system_card","model_name":"o3 and o4-mini","organization":"OpenAI","published":"2025-04-16","retrieved_at":"2026-08-02","source":"model_reports","source_id":"openai_o3_o4_mini_system_card","source_url":"https://openai.com/index/o3-o4-mini-system-card/","title":"o3 and o4-mini"},{"benchmarks":[{"benchmark_id":"pptbench","domain":"multimodal","name":"PPTBench (Einsia)","released":"2026-08-31","url":"https://lab.einsia.ai/pptbench/"}],"document_type":"benchmark_leaderboard","id":"model_reports:pptbench_v2_results_table","model_name":null,"organization":"Navers Lab / Einsia.AI","published":"2026-09-28","retrieved_at":"2026-10-27","source":"model_reports","source_id":"pptbench_v2_results_table","source_url":"https://arxiv.org/html/2609.29718v2#S3.T1","title":"PPTBench paper v2 · Table 1"},{"benchmarks":[{"benchmark_id":"gsm8k","domain":"math","name":"GSM8K","released":"2021-10-27","url":"https://github.com/openai/grade-school-math"},{"benchmark_id":"humaneval","domain":"coding","name":"HumanEval","released":"2021-07-07","url":"https://github.com/openai/human-eval"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mbpp","domain":"coding","name":"MBPP","released":"2021-08-16","url":"https://github.com/google-research/google-research/tree/master/mbpp"},{"benchmark_id":"mmlu","domain":"knowledge","name":"MMLU","released":"2020-09-07","url":"https://github.com/hendrycks/test"}],"document_type":"technical_report","id":"model_reports:qwen2_5_coder_report","model_name":"Qwen2.5-Coder","organization":"Qwen","published":"2024-09-18","retrieved_at":"2026-08-02","source":"model_reports","source_id":"qwen2_5_coder_report","source_url":"https://arxiv.org/abs/2409.12186","title":"Qwen2.5-Coder"},{"benchmarks":[{"benchmark_id":"aa_lcr","domain":"long_context","name":"AA-LCR","released":"2025-09-16","url":"https://artificialanalysis.ai/evaluations/aa-lcr"},{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"bfcl","domain":"tool_use","name":"BFCL","released":"2024-02-26","url":"https://gorilla.cs.berkeley.edu/leaderboard.html"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"browsecomp_zh","domain":"agent","name":"BrowseComp-ZH","released":"2025-04-27","url":"https://arxiv.org/abs/2504.19314"},{"benchmark_id":"gpqa","domain":"science","name":"GPQA (full)","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"hmmt","domain":"math","name":"HMMT","released":"2025-02-15","url":"https://www.hmmt.org/"},{"benchmark_id":"ifbench","domain":"instruction_following","name":"IFBench","released":"2025-07-03","url":"https://arxiv.org/abs/2507.02833"},{"benchmark_id":"ifeval","domain":"instruction_following","name":"IFEval","released":"2023-11-14","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"benchmark_id":"imo_answer_bench","domain":"math","name":"IMOAnswerBench","released":"2025-09-18","url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"longbench","domain":"long_context","name":"LongBench","released":"2023-08-28","url":"https://github.com/THUDM/LongBench"},{"benchmark_id":"mathvision","domain":"multimodal","name":"MathVision","released":"2024-02-22","url":"https://mathllm.github.io/mathvision/"},{"benchmark_id":"mathvista","domain":"multimodal","name":"MathVista","released":"2023-10-03","url":"https://mathvista.github.io/"},{"benchmark_id":"mcp_mark","domain":"tool_use","name":"MCPMark","released":"2025-09-30","url":"https://mcpmark.ai/"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"mmlu_redux","domain":"knowledge","name":"MMLU-Redux","released":"2024-06-06","url":"https://github.com/aryopg/mmlu-redux"},{"benchmark_id":"mmmlu","domain":"multilingual","name":"MMMLU","released":"2024-09-24","url":"https://huggingface.co/datasets/openai/MMMLU"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"},{"benchmark_id":"mmmu_pro","domain":"multimodal","name":"MMMU-Pro","released":"2024-09-04","url":"https://arxiv.org/abs/2409.02813"},{"benchmark_id":"multichallenge","domain":"instruction_following","name":"MultiChallenge","released":"2025-01-29","url":"https://arxiv.org/abs/2501.17399"},{"benchmark_id":"omnidocbench","domain":"multimodal","name":"OmniDocBench","released":"2024-12-10","url":"https://github.com/opendatalab/OmniDocBench"},{"benchmark_id":"osworld","domain":"computer_use","name":"OSWorld","released":"2024-04-11","url":"https://os-world.github.io/"},{"benchmark_id":"seccodebench","domain":"security","name":"SecCodeBench","released":"2025-10-14","url":"https://github.com/alibaba/SecCodeBench"},{"benchmark_id":"super_gpqa","domain":"knowledge","name":"SuperGPQA","released":"2025-02-20","url":"https://arxiv.org/abs/2502.14739"},{"benchmark_id":"swe_bench_multilingual","domain":"coding_agent","name":"SWE-bench Multilingual","released":"2025-04-22","url":"https://www.swebench.com/multilingual.html"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"tau2_bench","domain":"tool_use","name":"tau2-bench","released":"2025-06-09","url":"https://arxiv.org/abs/2506.07982"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"tool_decathlon","domain":"tool_use","name":"Tool Decathlon","released":"2025-10-28","url":"https://toolathlon.xyz/"},{"benchmark_id":"video_mme","domain":"multimodal","name":"Video-MME","released":"2024-05-31","url":"https://video-mme.github.io/"},{"benchmark_id":"widesearch","domain":"agent","name":"WideSearch","released":"2025-08-11","url":"https://widesearch-seed.github.io/"}],"document_type":"model_card","id":"model_reports:qwen3_5_model_card","model_name":"Qwen3.5-397B-A17B","organization":"Qwen","published":"2026-02-16","retrieved_at":"2026-08-02","source":"model_reports","source_id":"qwen3_5_model_card","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","title":"Qwen3.5-397B-A17B"},{"benchmarks":[{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"ifbench","domain":"instruction_following","name":"IFBench","released":"2025-07-03","url":"https://arxiv.org/abs/2507.02833"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"mathvision","domain":"multimodal","name":"MathVision","released":"2024-02-22","url":"https://mathllm.github.io/mathvision/"},{"benchmark_id":"omnidocbench","domain":"multimodal","name":"OmniDocBench","released":"2024-12-10","url":"https://github.com/opendatalab/OmniDocBench"},{"benchmark_id":"osworld","domain":"computer_use","name":"OSWorld","released":"2024-04-11","url":"https://os-world.github.io/"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"}],"document_type":"model_card","id":"model_reports:qwen3_8_model_card","model_name":"Qwen3.8-27B","organization":"Qwen","published":"2026-08-05","retrieved_at":"2026-08-15","source":"model_reports","source_id":"qwen3_8_model_card","source_url":"https://ollama.com/library/qwen3.8","title":"Qwen3.8-27B"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"arena_hard","domain":"human_preference","name":"Arena-Hard","released":"2024-04-19","url":"https://github.com/lmarena/arena-hard-auto"},{"benchmark_id":"bfcl","domain":"tool_use","name":"BFCL","released":"2024-02-26","url":"https://gorilla.cs.berkeley.edu/leaderboard.html"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"humaneval","domain":"coding","name":"HumanEval","released":"2021-07-07","url":"https://github.com/openai/human-eval"},{"benchmark_id":"ifeval","domain":"instruction_following","name":"IFEval","released":"2023-11-14","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"benchmark_id":"livebench","domain":"general","name":"LiveBench","released":"2024-06-06","url":"https://livebench.ai/"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"math_500","domain":"math","name":"MATH-500","released":"2021-03-05","url":"https://github.com/openai/prm800k"},{"benchmark_id":"mmlu_pro","domain":"knowledge","name":"MMLU-Pro","released":"2024-06-03","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"benchmark_id":"mmlu_redux","domain":"knowledge","name":"MMLU-Redux","released":"2024-06-06","url":"https://github.com/aryopg/mmlu-redux"}],"document_type":"technical_report","id":"model_reports:qwen3_technical_report","model_name":"Qwen3","organization":"Qwen","published":"2025-05-14","retrieved_at":"2026-08-02","source":"model_reports","source_id":"qwen3_technical_report","source_url":"https://arxiv.org/abs/2505.09388","title":"Qwen3"},{"benchmarks":[{"benchmark_id":"swe_bench_science","domain":"coding_agent","name":"SWE-bench Science","released":"2026-08-20","url":"https://arxiv.org/abs/2608.19799"}],"document_type":"benchmark_leaderboard","id":"model_reports:swe_bench_science_results_table","model_name":null,"organization":"OpenMOSS (Fudan University)","published":null,"retrieved_at":"2026-09-18","source":"model_reports","source_id":"swe_bench_science_results_table","source_url":"https://arxiv.org/html/2608.19799v2","title":"SWE-bench Science results table"},{"benchmarks":[{"benchmark_id":"agents_last_exam","domain":"coding_agent","name":"Agents' Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"apex_agents","domain":"agent","name":"APEX-Agents","released":"2026-01-20","url":"https://arxiv.org/abs/2601.14242"},{"benchmark_id":"arxivmath","domain":"math","name":"ArXivMath","released":null,"url":"https://matharena.ai/arxivmath"},{"benchmark_id":"automationbench","domain":"agent","name":"AutomationBench","released":"2026-03-10","url":"https://github.com/zapier/automation-bench"},{"benchmark_id":"bankertoolbench","domain":"professional","name":"BankerToolBench","released":null,"url":"https://github.com/Handshake-AI-Research/bankertoolbench"},{"benchmark_id":"biomysterybench","domain":"biology","name":"BioMysteryBench","released":"2026-04-29","url":"https://huggingface.co/datasets/Anthropic/BioMysteryBench-full"},{"benchmark_id":"brokenarxiv","domain":"math","name":"BrokenArXiv","released":null,"url":"https://matharena.ai/brokenarxiv"},{"benchmark_id":"critpt","domain":"science","name":"CritPt","released":"2025-09-30","url":"https://arxiv.org/abs/2509.26574"},{"benchmark_id":"cybergym","domain":"security","name":"CyberGym","released":null,"url":"https://openreview.net/forum?id=2YvbLQEdYt"},{"benchmark_id":"deepswe","domain":"coding_agent","name":"DeepSWE","released":"2026-05-26","url":"https://deepswe.datacurve.ai/blog/deepswe"},{"benchmark_id":"draco","domain":"long_context","name":"DRACO","released":null,"url":"https://huggingface.co/datasets/perplexity-ai/draco"},{"benchmark_id":"gdpval","domain":"professional","name":"GDPval","released":"2025-09-25","url":"https://openai.com/index/gdpval/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"harbor_index","domain":"coding_agent","name":"Harbor-Index","released":null,"url":"https://github.com/harbor-framework/harbor-index"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"horizonmath","domain":"math","name":"HorizonMath","released":null,"url":"https://github.com/ewang26/HorizonMath"},{"benchmark_id":"jobbench","domain":"professional","name":"JobBench","released":"2026-05-25","url":"https://arxiv.org/abs/2605.26329"},{"benchmark_id":"matharena_apex_2025","domain":"math","name":"MathArena Apex 2025","released":null,"url":"https://matharena.ai/apex"},{"benchmark_id":"mcp_atlas","domain":"tool_use","name":"MCP Atlas","released":"2025-11-18","url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},{"benchmark_id":"nl2repo","domain":"coding_agent","name":"NL2Repo","released":"2025-06-15","url":"https://github.com/nl2repo/nl2repo"},{"benchmark_id":"office_qa_pro","domain":"vision","name":"OfficeQA Pro","released":"2025-06-01","url":"https://github.com/OfficeQA/OfficeQA-Pro"},{"benchmark_id":"onemillionbench","domain":"long_context","name":"OneMillionBench","released":null,"url":"https://github.com/humanlaya/OneMillion-Bench"},{"benchmark_id":"posttrainbench","domain":"ai_research","name":"PostTrainBench","released":"2026-01-20","url":"https://posttrainbench.com/"},{"benchmark_id":"programbench","domain":"coding","name":"ProgramBench","released":"2026-05-05","url":"https://arxiv.org/abs/2605.03546"},{"benchmark_id":"skillsbench","domain":"agent","name":"SkillsBench","released":"2026-02-13","url":"https://skillsbench.ai/"},{"benchmark_id":"superchem","domain":"science","name":"SUPERChem","released":null,"url":"https://github.com/catalystforyou/SUPERChem_eval"},{"benchmark_id":"swe_atlas","domain":"coding_agent","name":"SWE Atlas","released":"2026-05-08","url":"https://github.com/scaleapi/SWE-Atlas"},{"benchmark_id":"swe_bench_multilingual","domain":"coding_agent","name":"SWE-bench Multilingual","released":"2025-04-22","url":"https://www.swebench.com/multilingual.html"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"swe_marathon","domain":"coding_agent","name":"SWE-Marathon","released":null,"url":"https://github.com/abundant-ai/swe-marathon"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"toolathlon_verified","domain":"tool_use","name":"Toolathlon Verified","released":"2025-11-01","url":"https://github.com/toolathlon/toolathlon"},{"benchmark_id":"widesearch","domain":"agent","name":"WideSearch","released":"2025-08-11","url":"https://widesearch-seed.github.io/"},{"benchmark_id":"workspacebench","domain":"professional","name":"WorkspaceBench","released":"2026-05-05","url":"https://arxiv.org/abs/2605.03596"}],"document_type":"technical_report","id":"model_reports:tencent_hy4_preview","model_name":"Hy4 preview","organization":"Tencent","published":"2026-08-28","retrieved_at":"2026-08-31","source":"model_reports","source_id":"tencent_hy4_preview","source_url":"https://hy.tencent.ai/research/hy4-preview","title":"Hy4 preview"},{"benchmarks":[{"benchmark_id":"world_models_last_exam_physics","domain":"multimodal","name":"World Models’ Last Exam in Physics","released":"2026-10-06","url":"https://lab.einsia.ai/phys-last-exam/"}],"document_type":"benchmark_leaderboard","id":"model_reports:world_models_last_exam_physics_v1_results_table","model_name":null,"organization":"Navers Lab / Einsia.AI","published":"2026-10-06","retrieved_at":"2026-10-27","source":"model_reports","source_id":"world_models_last_exam_physics_v1_results_table","source_url":"https://arxiv.org/html/2610.08791v1#S3.T2","title":"World Models’ Last Exam in Physics paper v1 · Table 2"},{"benchmarks":[{"benchmark_id":"arc_agi_2","domain":"reasoning","name":"ARC-AGI-2","released":"2025-03-24","url":"https://arcprize.org/arc-agi/2/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"}],"document_type":"release_post","id":"model_reports:xai_grok_4_5","model_name":"Grok 4.5","organization":"xAI","published":"2026-07-08","retrieved_at":"2026-08-02","source":"model_reports","source_id":"xai_grok_4_5","source_url":"https://x.ai/news/grok-4-5","title":"Grok 4.5"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"arc_agi","domain":"reasoning","name":"ARC-AGI","released":"2019-11-05","url":"https://arcprize.org/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"livecodebench","domain":"coding","name":"LiveCodeBench","released":"2024-03-12","url":"https://livecodebench.github.io/"},{"benchmark_id":"mmmu","domain":"multimodal","name":"MMMU","released":"2023-11-27","url":"https://mmmu-benchmark.github.io/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"}],"document_type":"model_card","id":"model_reports:xai_grok_4_model_card","model_name":"Grok 4","organization":"xAI","published":"2025-07-09","retrieved_at":"2026-08-02","source":"model_reports","source_id":"xai_grok_4_model_card","source_url":"https://x.ai/news/grok-4","title":"Grok 4"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"hmmt","domain":"math","name":"HMMT","released":"2025-02-15","url":"https://www.hmmt.org/"},{"benchmark_id":"imo_answer_bench","domain":"math","name":"IMOAnswerBench","released":"2025-09-18","url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench"},{"benchmark_id":"mcp_atlas","domain":"tool_use","name":"MCP Atlas","released":"2025-11-18","url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"tau2_bench","domain":"tool_use","name":"tau2-bench","released":"2025-06-09","url":"https://arxiv.org/abs/2506.07982"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"tool_decathlon","domain":"tool_use","name":"Tool Decathlon","released":"2025-10-28","url":"https://toolathlon.xyz/"},{"benchmark_id":"vending_bench","domain":"agent","name":"Vending Bench","released":"2025-02-18","url":"https://andonlabs.com/evals/vending-bench-2"}],"document_type":"model_card","id":"model_reports:zai_glm_5_1_model_card","model_name":"GLM-5.1","organization":"Z.ai","published":"2026-04-03","retrieved_at":"2026-08-15","source":"model_reports","source_id":"zai_glm_5_1_model_card","source_url":"https://huggingface.co/zai-org/GLM-5.1","title":"GLM-5.1"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"critpt","domain":"science","name":"CritPt","released":"2025-09-30","url":"https://arxiv.org/abs/2509.26574"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"hmmt","domain":"math","name":"HMMT","released":"2025-02-15","url":"https://www.hmmt.org/"},{"benchmark_id":"imo_answer_bench","domain":"math","name":"IMOAnswerBench","released":"2025-09-18","url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench"},{"benchmark_id":"mcp_atlas","domain":"tool_use","name":"MCP Atlas","released":"2025-11-18","url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},{"benchmark_id":"posttrainbench","domain":"ai_research","name":"PostTrainBench","released":"2026-01-20","url":"https://posttrainbench.com/"},{"benchmark_id":"swe_bench_pro","domain":"coding_agent","name":"SWE-bench Pro","released":"2025-09-23","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"tool_decathlon","domain":"tool_use","name":"Tool Decathlon","released":"2025-10-28","url":"https://toolathlon.xyz/"}],"document_type":"model_card","id":"model_reports:zai_glm_5_2_model_card","model_name":"GLM-5.2","organization":"Z.ai","published":"2026-06-16","retrieved_at":"2026-08-15","source":"model_reports","source_id":"zai_glm_5_2_model_card","source_url":"https://huggingface.co/zai-org/GLM-5.2","title":"GLM-5.2"},{"benchmarks":[{"benchmark_id":"agents_last_exam","domain":"coding_agent","name":"Agents' Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"automationbench","domain":"agent","name":"AutomationBench","released":"2026-03-10","url":"https://github.com/zapier/automation-bench"},{"benchmark_id":"babyvision","domain":"multimodal","name":"BabyVision","released":"2025-12-01","url":"https://github.com/babyvision/babyvision"},{"benchmark_id":"chartography","domain":"vision","name":"Chartography","released":"2025-06-01","url":"https://github.com/Chartography/Chartography"},{"benchmark_id":"charxiv_reasoning","domain":"vision","name":"CharXiv Reasoning","released":"2025-06-01","url":"https://github.com/CharXiv/CharXiv"},{"benchmark_id":"deepswe","domain":"coding_agent","name":"DeepSWE","released":"2026-05-26","url":"https://deepswe.datacurve.ai/blog/deepswe"},{"benchmark_id":"gdpval","domain":"professional","name":"GDPval","released":"2025-09-25","url":"https://openai.com/index/gdpval/"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"mmvu","domain":"vision","name":"MMVU","released":"2025-06-01","url":"https://github.com/MMVU/MMVU"},{"benchmark_id":"mvbench","domain":"vision","name":"MVbench","released":"2025-06-01","url":"https://github.com/MVBench/MVBench"},{"benchmark_id":"nl2repo","domain":"coding_agent","name":"NL2Repo","released":"2025-06-15","url":"https://github.com/nl2repo/nl2repo"},{"benchmark_id":"office_qa_pro","domain":"vision","name":"OfficeQA Pro","released":"2025-06-01","url":"https://github.com/OfficeQA/OfficeQA-Pro"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"toolathlon_verified","domain":"tool_use","name":"Toolathlon Verified","released":"2025-11-01","url":"https://github.com/toolathlon/toolathlon"}],"document_type":"model_card","id":"model_reports:zai_glm_5_3_flash_model_card","model_name":"GLM-5.3-Flash","organization":"Z.ai","published":"2026-08-27","retrieved_at":"2026-08-27","source":"model_reports","source_id":"zai_glm_5_3_flash_model_card","source_url":"https://z.ai/blog/glm-5.3-flash","title":"GLM-5.3-Flash"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"browsecomp_zh","domain":"agent","name":"BrowseComp-ZH","released":"2025-04-27","url":"https://arxiv.org/abs/2504.19314"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"hmmt","domain":"math","name":"HMMT","released":"2025-02-15","url":"https://www.hmmt.org/"},{"benchmark_id":"imo_answer_bench","domain":"math","name":"IMOAnswerBench","released":"2025-09-18","url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench"},{"benchmark_id":"mcp_atlas","domain":"tool_use","name":"MCP Atlas","released":"2025-11-18","url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},{"benchmark_id":"swe_bench_multilingual","domain":"coding_agent","name":"SWE-bench Multilingual","released":"2025-04-22","url":"https://www.swebench.com/multilingual.html"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"tau2_bench","domain":"tool_use","name":"tau2-bench","released":"2025-06-09","url":"https://arxiv.org/abs/2506.07982"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"},{"benchmark_id":"tool_decathlon","domain":"tool_use","name":"Tool Decathlon","released":"2025-10-28","url":"https://toolathlon.xyz/"},{"benchmark_id":"vending_bench","domain":"agent","name":"Vending Bench","released":"2025-02-18","url":"https://andonlabs.com/evals/vending-bench-2"}],"document_type":"model_card","id":"model_reports:zai_glm_5_model_card","model_name":"GLM-5","organization":"Z.ai","published":"2026-02-11","retrieved_at":"2026-08-02","source":"model_reports","source_id":"zai_glm_5_model_card","source_url":"https://huggingface.co/zai-org/GLM-5","title":"GLM-5"},{"benchmarks":[{"benchmark_id":"aime","domain":"math","name":"AIME","released":"2024-02-01","url":"https://maa.org/maa-invitational-competitions/"},{"benchmark_id":"browsecomp","domain":"agent","name":"BrowseComp","released":"2025-04-10","url":"https://openai.com/index/browsecomp/"},{"benchmark_id":"gpqa_diamond","domain":"science","name":"GPQA Diamond","released":"2023-11-20","url":"https://arxiv.org/abs/2311.12022"},{"benchmark_id":"hle","domain":"reasoning","name":"Humanity's Last Exam","released":"2025-01-23","url":"https://lastexam.ai/"},{"benchmark_id":"swe_bench_verified","domain":"coding_agent","name":"SWE-bench Verified","released":"2024-08-13","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"benchmark_id":"tau2_bench","domain":"tool_use","name":"tau2-bench","released":"2025-06-09","url":"https://arxiv.org/abs/2506.07982"},{"benchmark_id":"terminal_bench","domain":"agent","name":"Terminal-Bench","released":"2025-05-19","url":"https://www.tbench.ai/"}],"document_type":"technical_report","id":"model_reports:zai_glm_5_paper","model_name":"GLM-5","organization":"Z.ai","published":"2026-02-11","retrieved_at":"2026-08-02","source":"model_reports","source_id":"zai_glm_5_paper","source_url":"https://huggingface.co/papers/2602.15763","title":"GLM-5"},{"benchmarks":[{"benchmark_id":"opencompass-1580-a-bench","domain":"多模态","name":"A-Bench","released":"2024-06-05","url":"https://hub.opencompass.org.cn/dataset-detail/A-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/A-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/A-Bench","title":"A-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1367-a-okvqa","domain":"多模态","name":"A-OKVQA","released":"2022-06-03","url":"https://hub.opencompass.org.cn/dataset-detail/A-OKVQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/A-OKVQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/A-OKVQA","title":"A-OKVQA"},{"benchmarks":[{"benchmark_id":"opencompass-2452-aecbench","domain":"科学","name":"AECBench","released":"2026-04-10","url":"https://hub.opencompass.org.cn/dataset-detail/AECBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AECBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AECBench","title":"AECBench"},{"benchmarks":[{"benchmark_id":"opencompass-506-afqmc","domain":"语言","name":"AFQMC","released":"2020-04-13","url":"https://hub.opencompass.org.cn/dataset-detail/AFQMC"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AFQMC","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AFQMC","title":"AFQMC"},{"benchmarks":[{"benchmark_id":"opencompass-497-agieval","domain":"学科","name":"AGIEval","released":"2023-09-18","url":"https://hub.opencompass.org.cn/dataset-detail/AGIEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AGIEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AGIEval","title":"AGIEval"},{"benchmarks":[{"benchmark_id":"opencompass-2396-aidabench","domain":"多模态","name":"AIDABench","released":"2026-02-14","url":"https://hub.opencompass.org.cn/dataset-detail/AIDABench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AIDABench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AIDABench","title":"AIDABench"},{"benchmarks":[{"benchmark_id":"opencompass-1069-air-bench","domain":"知识","name":"AIR-Bench","released":"2024-02-12","url":"https://hub.opencompass.org.cn/dataset-detail/AIR-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AIR-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AIR-Bench","title":"AIR-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1557-airbench-2024","domain":"安全","name":"AIRBench-2024","released":"2024-08-05","url":"https://hub.opencompass.org.cn/dataset-detail/AIRBench-2024"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AIRBench-2024","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AIRBench-2024","title":"AIRBench-2024"},{"benchmarks":[{"benchmark_id":"opencompass-1995-airtbench","domain":"智能体","name":"AIRTBench","released":"2025-06-17","url":"https://hub.opencompass.org.cn/dataset-detail/AIRTBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AIRTBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AIRTBench","title":"AIRTBench"},{"benchmarks":[{"benchmark_id":"opencompass-1969-ale-bench","domain":"代码","name":"ALE-Bench","released":"2025-06-10","url":"https://hub.opencompass.org.cn/dataset-detail/ALE-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ALE-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ALE-Bench","title":"ALE-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1280-ambrosia","domain":"推理","name":"AMBROSIA","released":"2024-06-27","url":"https://hub.opencompass.org.cn/dataset-detail/AMBROSIA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AMBROSIA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AMBROSIA","title":"AMBROSIA"},{"benchmarks":[{"benchmark_id":"opencompass-1966-amsbench","domain":"多模态","name":"AMSbench","released":"2025-06-21","url":"https://hub.opencompass.org.cn/dataset-detail/AMSbench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AMSbench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AMSbench","title":"AMSbench"},{"benchmarks":[{"benchmark_id":"opencompass-1093-apps","domain":"代码","name":"APPS","released":"2021-11-08","url":"https://hub.opencompass.org.cn/dataset-detail/APPS"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/APPS","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/APPS","title":"APPS"},{"benchmarks":[{"benchmark_id":"opencompass-1116-aqua-rat","domain":"数学","name":"AQUA-RAT","released":"2017-10-23","url":"https://hub.opencompass.org.cn/dataset-detail/AQUA-RAT"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AQUA-RAT","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AQUA-RAT","title":"AQUA-RAT"},{"benchmarks":[{"benchmark_id":"opencompass-502-arc-c","domain":"学科","name":"ARC-c","released":"2018-03-14","url":"https://hub.opencompass.org.cn/dataset-detail/ARC-c"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ARC-c","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ARC-c","title":"ARC-c"},{"benchmarks":[{"benchmark_id":"opencompass-503-arc-e","domain":"学科","name":"ARC-e","released":"2018-03-14","url":"https://hub.opencompass.org.cn/dataset-detail/ARC-e"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ARC-e","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ARC-e","title":"ARC-e"},{"benchmarks":[{"benchmark_id":"opencompass-1114-asdiv","domain":"数学","name":"ASDiv","released":"2020-07-05","url":"https://hub.opencompass.org.cn/dataset-detail/ASDiv"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ASDiv","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ASDiv","title":"ASDiv"},{"benchmarks":[{"benchmark_id":"opencompass-1396-av-odyssey-bench","domain":"多模态","name":"AV-Odyssey-Bench","released":"2024-12-03","url":"https://hub.opencompass.org.cn/dataset-detail/AV-Odyssey-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AV-Odyssey-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AV-Odyssey-Bench","title":"AV-Odyssey-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-526-ax-b","domain":"推理","name":"AX-b","released":"2019-05-02","url":"https://hub.opencompass.org.cn/dataset-detail/AX-b"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AX-b","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AX-b","title":"AX-b"},{"benchmarks":[{"benchmark_id":"opencompass-527-ax-g","domain":"推理","name":"AX-g","released":"2019-07-12","url":"https://hub.opencompass.org.cn/dataset-detail/AX-g"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AX-g","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AX-g","title":"AX-g"},{"benchmarks":[{"benchmark_id":"opencompass-2084-axbench","domain":"理解","name":"AXBENCH","released":"2025-01-28","url":"https://hub.opencompass.org.cn/dataset-detail/AXBENCH"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AXBENCH","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AXBENCH","title":"AXBENCH"},{"benchmarks":[{"benchmark_id":"opencompass-1148-abspyramid","domain":"知识","name":"AbsPyramid","released":"2024-06-16","url":"https://hub.opencompass.org.cn/dataset-detail/AbsPyramid"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AbsPyramid","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AbsPyramid","title":"AbsPyramid"},{"benchmarks":[{"benchmark_id":"opencompass-1319-actionatlas","domain":"多模态","name":"ActionAtlas","released":"2024-10-08","url":"https://hub.opencompass.org.cn/dataset-detail/ActionAtlas"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ActionAtlas","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ActionAtlas","title":"ActionAtlas"},{"benchmarks":[{"benchmark_id":"opencompass-1155-ada-leval","domain":"长文本","name":"Ada-LEval","released":"2024-06-16","url":"https://hub.opencompass.org.cn/dataset-detail/Ada-LEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Ada-LEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Ada-LEval","title":"Ada-LEval"},{"benchmarks":[{"benchmark_id":"opencompass-1753-agmmu","domain":"多模态","name":"AgMMU","released":"2025-04-14","url":"https://hub.opencompass.org.cn/dataset-detail/AgMMU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgMMU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgMMU","title":"AgMMU"},{"benchmarks":[{"benchmark_id":"opencompass-1242-agentboard","domain":"智能体","name":"AgentBoard","released":"2024-06-24","url":"https://hub.opencompass.org.cn/dataset-detail/AgentBoard"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgentBoard","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgentBoard","title":"AgentBoard"},{"benchmarks":[{"benchmark_id":"opencompass-1351-agentharm","domain":"安全","name":"AgentHarm","released":"2024-10-11","url":"https://hub.opencompass.org.cn/dataset-detail/AgentHarm"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgentHarm","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgentHarm","title":"AgentHarm"},{"benchmarks":[{"benchmark_id":"opencompass-2061-agenthazard","domain":"多模态","name":"AgentHazard","released":"2025-07-16","url":"https://hub.opencompass.org.cn/dataset-detail/AgentHazard"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgentHazard","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgentHazard","title":"AgentHazard"},{"benchmarks":[{"benchmark_id":"opencompass-1778-agentrewardbench","domain":"推理","name":"AgentRewardBench","released":"2025-04-11","url":"https://hub.opencompass.org.cn/dataset-detail/AgentRewardBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgentRewardBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AgentRewardBench","title":"AgentRewardBench"},{"benchmarks":[{"benchmark_id":"opencompass-1075-alignbench","domain":"理解","name":"AlignBench","released":"2024-08-25","url":"https://hub.opencompass.org.cn/dataset-detail/AlignBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AlignBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AlignBench","title":"AlignBench"},{"benchmarks":[{"benchmark_id":"opencompass-1876-aneumo","domain":"多模态","name":"Aneumo","released":"2025-05-19","url":"https://hub.opencompass.org.cn/dataset-detail/Aneumo"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Aneumo","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Aneumo","title":"Aneumo"},{"benchmarks":[{"benchmark_id":"opencompass-2075-arena-hard-auto","domain":"安全","name":"Arena-Hard-Auto","released":"2024-04-19","url":"https://hub.opencompass.org.cn/dataset-detail/Arena-Hard-Auto"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Arena-Hard-Auto","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Arena-Hard-Auto","title":"Arena-Hard-Auto"},{"benchmarks":[{"benchmark_id":"opencompass-2370-argusinspection","domain":"多模态","name":"ArgusInspection","released":"2025-10-27","url":"https://hub.opencompass.org.cn/dataset-detail/ArgusInspection"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ArgusInspection","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ArgusInspection","title":"ArgusInspection"},{"benchmarks":[{"benchmark_id":"opencompass-2052-artifactsbench","domain":"多模态","name":"ArtifactsBench","released":"2025-07-01","url":"https://hub.opencompass.org.cn/dataset-detail/ArtifactsBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ArtifactsBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ArtifactsBench","title":"ArtifactsBench"},{"benchmarks":[{"benchmark_id":"opencompass-1991-assetopsbench","domain":"智能体","name":"AssetOpsBench","released":"2025-06-04","url":"https://hub.opencompass.org.cn/dataset-detail/AssetOpsBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AssetOpsBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AssetOpsBench","title":"AssetOpsBench"},{"benchmarks":[{"benchmark_id":"opencompass-1835-audiojailbreak","domain":"多模态","name":"AudioJailbreak","released":"2025-05-24","url":"https://hub.opencompass.org.cn/dataset-detail/AudioJailbreak"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AudioJailbreak","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AudioJailbreak","title":"AudioJailbreak"},{"benchmarks":[{"benchmark_id":"opencompass-1908-audiotrust","domain":"多模态","name":"AudioTrust","released":"2025-06-06","url":"https://hub.opencompass.org.cn/dataset-detail/AudioTrust"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AudioTrust","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AudioTrust","title":"AudioTrust"},{"benchmarks":[{"benchmark_id":"opencompass-2078-autoadvexbench","domain":"安全","name":"AutoAdvExBench","released":"2025-03-03","url":"https://hub.opencompass.org.cn/dataset-detail/AutoAdvExBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AutoAdvExBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/AutoAdvExBench","title":"AutoAdvExBench"},{"benchmarks":[{"benchmark_id":"opencompass-1266-babilong","domain":"长文本","name":"BABILong","released":"2024-06-14","url":"https://hub.opencompass.org.cn/dataset-detail/BABILong"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BABILong","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BABILong","title":"BABILong"},{"benchmarks":[{"benchmark_id":"opencompass-539-bbh","domain":"推理","name":"BBH","released":"2022-10-17","url":"https://hub.opencompass.org.cn/dataset-detail/BBH"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BBH","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BBH","title":"BBH"},{"benchmarks":[{"benchmark_id":"opencompass-1365-blink","domain":"多模态","name":"BLINK","released":"2024-04-28","url":"https://hub.opencompass.org.cn/dataset-detail/BLINK"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BLINK","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BLINK","title":"BLINK"},{"benchmarks":[{"benchmark_id":"opencompass-1571-bright","domain":"强推理","name":"BRIGHT","released":"2024-10-24","url":"https://hub.opencompass.org.cn/dataset-detail/BRIGHT"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BRIGHT","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BRIGHT","title":"BRIGHT"},{"benchmarks":[{"benchmark_id":"opencompass-1146-bust","domain":"理解","name":"BUST","released":"2024-06-16","url":"https://hub.opencompass.org.cn/dataset-detail/BUST"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BUST","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BUST","title":"BUST"},{"benchmarks":[{"benchmark_id":"opencompass-1086-belebele","domain":"语言","name":"Belebele","released":"2024-07-25","url":"https://hub.opencompass.org.cn/dataset-detail/Belebele"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Belebele","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Belebele","title":"Belebele"},{"benchmarks":[{"benchmark_id":"opencompass-1562-benchmax","domain":"语言","name":"BenchMAX","released":"2025-02-11","url":"https://hub.opencompass.org.cn/dataset-detail/BenchMAX"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BenchMAX","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BenchMAX","title":"BenchMAX"},{"benchmarks":[{"benchmark_id":"opencompass-1253-bigcodebench","domain":"强推理","name":"BigCodeBench","released":"2024-06-22","url":"https://hub.opencompass.org.cn/dataset-detail/BigCodeBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BigCodeBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BigCodeBench","title":"BigCodeBench"},{"benchmarks":[{"benchmark_id":"opencompass-1680-bigobench","domain":"代码","name":"BigOBench","released":"2025-03-19","url":"https://hub.opencompass.org.cn/dataset-detail/BigOBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BigOBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BigOBench","title":"BigOBench"},{"benchmarks":[{"benchmark_id":"opencompass-510-boolq","domain":"知识","name":"BoolQ","released":"2019-05-24","url":"https://hub.opencompass.org.cn/dataset-detail/BoolQ"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BoolQ","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/BoolQ","title":"BoolQ"},{"benchmarks":[{"benchmark_id":"opencompass-1983-bytemorph","domain":"创作","name":"ByteMorph","released":"2025-06-03","url":"https://hub.opencompass.org.cn/dataset-detail/ByteMorph"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ByteMorph","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ByteMorph","title":"ByteMorph"},{"benchmarks":[{"benchmark_id":"opencompass-496-c-eval","domain":"学科","name":"C-Eval","released":"2023-05-15","url":"https://hub.opencompass.org.cn/dataset-detail/C-Eval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/C-Eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/C-Eval","title":"C-Eval"},{"benchmarks":[{"benchmark_id":"opencompass-1780-c-faith","domain":"语言","name":"C-FAITH","released":"2025-04-14","url":"https://hub.opencompass.org.cn/dataset-detail/C-FAITH"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/C-FAITH","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/C-FAITH","title":"C-FAITH"},{"benchmarks":[{"benchmark_id":"opencompass-514-c3","domain":"理解","name":"C3","released":"2019-04-21","url":"https://hub.opencompass.org.cn/dataset-detail/C3"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/C3","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/C3","title":"C3"},{"benchmarks":[{"benchmark_id":"opencompass-1552-ceb","domain":"其他","name":"CEB","released":"2024-07-03","url":"https://hub.opencompass.org.cn/dataset-detail/CEB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CEB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CEB","title":"CEB"},{"benchmarks":[{"benchmark_id":"opencompass-1079-cflue","domain":"知识","name":"CFLUE","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/CFLUE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CFLUE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CFLUE","title":"CFLUE"},{"benchmarks":[{"benchmark_id":"opencompass-1907-cfinbench","domain":"知识","name":"CFinBench","released":"2024-10-01","url":"https://hub.opencompass.org.cn/dataset-detail/CFinBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CFinBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CFinBench","title":"CFinBench"},{"benchmarks":[{"benchmark_id":"opencompass-1513-cg-bench","domain":"多模态","name":"CG-Bench","released":"2024-12-16","url":"https://hub.opencompass.org.cn/dataset-detail/CG-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CG-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CG-Bench","title":"CG-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1141-charm","domain":"推理","name":"CHARM","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/CHARM"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CHARM","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CHARM","title":"CHARM"},{"benchmarks":[{"benchmark_id":"opencompass-1534-chase-code","domain":"代码","name":"CHASE-Code","released":"2025-02-14","url":"https://hub.opencompass.org.cn/dataset-detail/CHASE-Code"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CHASE-Code","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CHASE-Code","title":"CHASE-Code"},{"benchmarks":[{"benchmark_id":"opencompass-505-chid","domain":"语言","name":"CHID","released":"2019-06-04","url":"https://hub.opencompass.org.cn/dataset-detail/CHID"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CHID","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CHID","title":"CHID"},{"benchmarks":[{"benchmark_id":"opencompass-1834-clever","domain":"代码","name":"CLEVER","released":"2025-05-20","url":"https://hub.opencompass.org.cn/dataset-detail/CLEVER"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CLEVER","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CLEVER","title":"CLEVER"},{"benchmarks":[{"benchmark_id":"opencompass-1137-cmb","domain":"其他","name":"CMB","released":"2024-04-04","url":"https://hub.opencompass.org.cn/dataset-detail/CMB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CMB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CMB","title":"CMB"},{"benchmarks":[{"benchmark_id":"opencompass-499-cmmlu","domain":"学科","name":"CMMLU","released":"2023-06-15","url":"https://hub.opencompass.org.cn/dataset-detail/CMMLU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CMMLU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CMMLU","title":"CMMLU"},{"benchmarks":[{"benchmark_id":"opencompass-524-cmnli","domain":"推理","name":"CMNLI","released":"2020-04-13","url":"https://hub.opencompass.org.cn/dataset-detail/CMNLI"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CMNLI","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CMNLI","title":"CMNLI"},{"benchmarks":[{"benchmark_id":"opencompass-529-copa","domain":"推理","name":"COPA","released":null,"url":"https://hub.opencompass.org.cn/dataset-detail/COPA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/COPA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/COPA","title":"COPA"},{"benchmarks":[{"benchmark_id":"opencompass-1622-coral","domain":"其他","name":"CORAL","released":"2024-10-30","url":"https://hub.opencompass.org.cn/dataset-detail/CORAL"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CORAL","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CORAL","title":"CORAL"},{"benchmarks":[{"benchmark_id":"opencompass-1972-cpret","domain":"代码","name":"CPRet","released":"2025-05-19","url":"https://hub.opencompass.org.cn/dataset-detail/CPRet"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CPRet","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CPRet","title":"CPRet"},{"benchmarks":[{"benchmark_id":"opencompass-1626-crag","domain":"其他","name":"CRAG","released":"2024-06-07","url":"https://hub.opencompass.org.cn/dataset-detail/CRAG"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CRAG","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CRAG","title":"CRAG"},{"benchmarks":[{"benchmark_id":"opencompass-2046-crew-wildfire","domain":"智能体","name":"CREW-WILDFIRE","released":"2025-07-07","url":"https://hub.opencompass.org.cn/dataset-detail/CREW-WILDFIRE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CREW-WILDFIRE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CREW-WILDFIRE","title":"CREW-WILDFIRE"},{"benchmarks":[{"benchmark_id":"opencompass-1500-crpe","domain":"多模态","name":"CRPE","released":"2024-02-29","url":"https://hub.opencompass.org.cn/dataset-detail/CRPE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CRPE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CRPE","title":"CRPE"},{"benchmarks":[{"benchmark_id":"opencompass-924-cs-bench","domain":"推理","name":"CS-Bench","released":"2024-06-12","url":"https://hub.opencompass.org.cn/dataset-detail/CS-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CS-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CS-Bench","title":"CS-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1219-cs-eval","domain":"学科","name":"CS-Eval","released":"2024-05-31","url":"https://hub.opencompass.org.cn/dataset-detail/CS-Eval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CS-Eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CS-Eval","title":"CS-Eval"},{"benchmarks":[{"benchmark_id":"opencompass-519-csl","domain":"理解","name":"CSL","released":"2022-09-12","url":"https://hub.opencompass.org.cn/dataset-detail/CSL"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CSL","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CSL","title":"CSL"},{"benchmarks":[{"benchmark_id":"opencompass-1833-csts","domain":"其他","name":"CSTS","released":"2025-05-20","url":"https://hub.opencompass.org.cn/dataset-detail/CSTS"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CSTS","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CSTS","title":"CSTS"},{"benchmarks":[{"benchmark_id":"opencompass-1268-ctibench","domain":"安全","name":"CTIBench","released":"2024-06-11","url":"https://hub.opencompass.org.cn/dataset-detail/CTIBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CTIBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CTIBench","title":"CTIBench"},{"benchmarks":[{"benchmark_id":"opencompass-1985-cvdp","domain":"智能体","name":"CVDP","released":"2025-06-17","url":"https://hub.opencompass.org.cn/dataset-detail/CVDP"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CVDP","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CVDP","title":"CVDP"},{"benchmarks":[{"benchmark_id":"opencompass-1239-cvqa","domain":"多模态","name":"CVQA","released":"2024-06-10","url":"https://hub.opencompass.org.cn/dataset-detail/CVQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CVQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CVQA","title":"CVQA"},{"benchmarks":[{"benchmark_id":"opencompass-1052-calm","domain":"推理","name":"CaLM","released":"2024-05-01","url":"https://hub.opencompass.org.cn/dataset-detail/CaLM"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CaLM","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CaLM","title":"CaLM"},{"benchmarks":[{"benchmark_id":"opencompass-1975-causalvqa","domain":"多模态","name":"CausalVQA","released":"2025-06-11","url":"https://hub.opencompass.org.cn/dataset-detail/CausalVQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CausalVQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CausalVQA","title":"CausalVQA"},{"benchmarks":[{"benchmark_id":"opencompass-692-chembench","domain":"知识","name":"ChemBench","released":"2024-02-15","url":"https://hub.opencompass.org.cn/dataset-detail/ChemBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ChemBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ChemBench","title":"ChemBench"},{"benchmarks":[{"benchmark_id":"opencompass-1278-chronomagic-bench","domain":"创作","name":"ChronoMagic-Bench","released":"2024-06-26","url":"https://hub.opencompass.org.cn/dataset-detail/ChronoMagic-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ChronoMagic-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ChronoMagic-Bench","title":"ChronoMagic-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1849-cleanpatrick","domain":"理解","name":"CleanPatrick","released":"2025-05-16","url":"https://hub.opencompass.org.cn/dataset-detail/CleanPatrick"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CleanPatrick","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CleanPatrick","title":"CleanPatrick"},{"benchmarks":[{"benchmark_id":"opencompass-1954-climateviz","domain":"多模态","name":"ClimateViz","released":"2025-06-11","url":"https://hub.opencompass.org.cn/dataset-detail/ClimateViz"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ClimateViz","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ClimateViz","title":"ClimateViz"},{"benchmarks":[{"benchmark_id":"opencompass-1546-codecriticbench","domain":"代码","name":"CodeCriticBench","released":"2025-02-23","url":"https://hub.opencompass.org.cn/dataset-detail/CodeCriticBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CodeCriticBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CodeCriticBench","title":"CodeCriticBench"},{"benchmarks":[{"benchmark_id":"opencompass-1624-codeelo","domain":"代码","name":"CodeElo","released":"2025-01-03","url":"https://hub.opencompass.org.cn/dataset-detail/CodeElo"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CodeElo","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CodeElo","title":"CodeElo"},{"benchmarks":[{"benchmark_id":"opencompass-1582-codemmlu","domain":"代码","name":"CodeMMLU","released":"2024-06-22","url":"https://hub.opencompass.org.cn/dataset-detail/CodeMMLU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CodeMMLU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CodeMMLU","title":"CodeMMLU"},{"benchmarks":[{"benchmark_id":"opencompass-1250-collie","domain":"强推理","name":"Collie","released":"2023-07-17","url":"https://hub.opencompass.org.cn/dataset-detail/Collie"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Collie","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Collie","title":"Collie"},{"benchmarks":[{"benchmark_id":"opencompass-1777-colorbench","domain":"多模态","name":"ColorBench","released":"2025-04-10","url":"https://hub.opencompass.org.cn/dataset-detail/ColorBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ColorBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ColorBench","title":"ColorBench"},{"benchmarks":[{"benchmark_id":"opencompass-1943-combibench","domain":"数学","name":"CombiBench","released":"2025-04-28","url":"https://hub.opencompass.org.cn/dataset-detail/CombiBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CombiBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CombiBench","title":"CombiBench"},{"benchmarks":[{"benchmark_id":"opencompass-511-commonsenseqa","domain":"知识","name":"CommonSenseQA","released":"2018-11-02","url":"https://hub.opencompass.org.cn/dataset-detail/CommonSenseQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CommonSenseQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CommonSenseQA","title":"CommonSenseQA"},{"benchmarks":[{"benchmark_id":"opencompass-1336-compbench","domain":"多模态","name":"CompBench","released":"2024-07-23","url":"https://hub.opencompass.org.cn/dataset-detail/CompBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CompBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CompBench","title":"CompBench"},{"benchmarks":[{"benchmark_id":"opencompass-1679-contextualjudgebench","domain":"理解","name":"ContextualJudgeBench","released":"2025-03-19","url":"https://hub.opencompass.org.cn/dataset-detail/ContextualJudgeBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ContextualJudgeBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ContextualJudgeBench","title":"ContextualJudgeBench"},{"benchmarks":[{"benchmark_id":"opencompass-1269-convbench","domain":"推理","name":"ConvBench","released":"2024-03-29","url":"https://hub.opencompass.org.cn/dataset-detail/ConvBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ConvBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ConvBench","title":"ConvBench"},{"benchmarks":[{"benchmark_id":"opencompass-1643-creation-mmbench","domain":"多模态","name":"Creation-MMBench","released":"2025-03-18","url":"https://hub.opencompass.org.cn/dataset-detail/Creation-MMBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Creation-MMBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Creation-MMBench","title":"Creation-MMBench"},{"benchmarks":[{"benchmark_id":"opencompass-568-criticbench","domain":"理解","name":"CriticBench","released":"2024-02-22","url":"https://hub.opencompass.org.cn/dataset-detail/CriticBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CriticBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CriticBench","title":"CriticBench"},{"benchmarks":[{"benchmark_id":"opencompass-1739-crosswordbench","domain":"多模态","name":"CrossWordBench","released":"2025-03-30","url":"https://hub.opencompass.org.cn/dataset-detail/CrossWordBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CrossWordBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CrossWordBench","title":"CrossWordBench"},{"benchmarks":[{"benchmark_id":"opencompass-1123-crows-pairs","domain":"安全","name":"Crows-Pairs","released":"2020-09-30","url":"https://hub.opencompass.org.cn/dataset-detail/Crows-Pairs"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Crows-Pairs","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Crows-Pairs","title":"Crows-Pairs"},{"benchmarks":[{"benchmark_id":"opencompass-1349-cyberseceval","domain":"安全","name":"CyberSecEval","released":"2023-12-07","url":"https://hub.opencompass.org.cn/dataset-detail/CyberSecEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CyberSecEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/CyberSecEval","title":"CyberSecEval"},{"benchmarks":[{"benchmark_id":"opencompass-2030-dabstep","domain":"智能体","name":"DABstep","released":"2025-06-30","url":"https://hub.opencompass.org.cn/dataset-detail/DABstep"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DABstep","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DABstep","title":"DABstep"},{"benchmarks":[{"benchmark_id":"opencompass-2032-dice-bench","domain":"智能体","name":"DICE-BENCH","released":"2025-06-28","url":"https://hub.opencompass.org.cn/dataset-detail/DICE-BENCH"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DICE-BENCH","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DICE-BENCH","title":"DICE-BENCH"},{"benchmarks":[{"benchmark_id":"opencompass-1648-dme","domain":"多模态","name":"DME","released":"2024-10-11","url":"https://hub.opencompass.org.cn/dataset-detail/DME"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DME","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DME","title":"DME"},{"benchmarks":[{"benchmark_id":"opencompass-1731-dove","domain":"其他","name":"DOVE","released":"2025-03-04","url":"https://hub.opencompass.org.cn/dataset-detail/DOVE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DOVE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DOVE","title":"DOVE"},{"benchmarks":[{"benchmark_id":"opencompass-2045-dragon","domain":"知识","name":"DRAGON","released":"2025-07-08","url":"https://hub.opencompass.org.cn/dataset-detail/DRAGON"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DRAGON","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DRAGON","title":"DRAGON"},{"benchmarks":[{"benchmark_id":"opencompass-536-drop","domain":"推理","name":"DROP","released":"2019-03-01","url":"https://hub.opencompass.org.cn/dataset-detail/DROP"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DROP","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DROP","title":"DROP"},{"benchmarks":[{"benchmark_id":"opencompass-544-ds-1000","domain":"代码","name":"DS-1000","released":"2022-11-18","url":"https://hub.opencompass.org.cn/dataset-detail/DS-1000"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DS-1000","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DS-1000","title":"DS-1000"},{"benchmarks":[{"benchmark_id":"opencompass-1078-debugbench","domain":"代码","name":"DebugBench","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/DebugBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DebugBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DebugBench","title":"DebugBench"},{"benchmarks":[{"benchmark_id":"opencompass-1987-deepresearchbench","domain":"智能体","name":"DeepResearchBench","released":"2025-06-13","url":"https://hub.opencompass.org.cn/dataset-detail/DeepResearchBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DeepResearchBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DeepResearchBench","title":"DeepResearchBench"},{"benchmarks":[{"benchmark_id":"opencompass-1605-deepfake-eval-2024","domain":"其他","name":"Deepfake-Eval-2024","released":"2025-03-04","url":"https://hub.opencompass.org.cn/dataset-detail/Deepfake-Eval-2024"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Deepfake-Eval-2024","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Deepfake-Eval-2024","title":"Deepfake-Eval-2024"},{"benchmarks":[{"benchmark_id":"opencompass-1971-dycodeeval","domain":"强推理","name":"DyCodeEval","released":"2025-06-23","url":"https://hub.opencompass.org.cn/dataset-detail/DyCodeEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DyCodeEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DyCodeEval","title":"DyCodeEval"},{"benchmarks":[{"benchmark_id":"opencompass-1374-dynamath","domain":"多模态","name":"DynaMath","released":"2024-10-29","url":"https://hub.opencompass.org.cn/dataset-detail/DynaMath"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DynaMath","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/DynaMath","title":"DynaMath"},{"benchmarks":[{"benchmark_id":"opencompass-1080-e-eval","domain":"学科","name":"E-EVAL","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/E-EVAL"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/E-EVAL","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/E-EVAL","title":"E-EVAL"},{"benchmarks":[{"benchmark_id":"opencompass-1327-ehrnoteqa","domain":"知识","name":"EHRNoteQA","released":"2024-02-25","url":"https://hub.opencompass.org.cn/dataset-detail/EHRNoteQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EHRNoteQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EHRNoteQA","title":"EHRNoteQA"},{"benchmarks":[{"benchmark_id":"opencompass-2571-elbench","domain":"推理","name":"ELBench","released":"2026-06-23","url":"https://hub.opencompass.org.cn/dataset-detail/ELBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ELBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ELBench","title":"ELBench"},{"benchmarks":[{"benchmark_id":"opencompass-1640-emma","domain":"多模态","name":"EMMA","released":"2025-01-09","url":"https://hub.opencompass.org.cn/dataset-detail/EMMA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EMMA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EMMA","title":"EMMA"},{"benchmarks":[{"benchmark_id":"opencompass-522-eprstmt","domain":"理解","name":"EPRSTMT","released":"2021-07-15","url":"https://hub.opencompass.org.cn/dataset-detail/EPRSTMT"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EPRSTMT","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EPRSTMT","title":"EPRSTMT"},{"benchmarks":[{"benchmark_id":"opencompass-1892-er-reason","domain":"推理","name":"ER-Reason","released":"2025-05-28","url":"https://hub.opencompass.org.cn/dataset-detail/ER-Reason"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ER-Reason","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ER-Reason","title":"ER-Reason"},{"benchmarks":[{"benchmark_id":"opencompass-1843-ewmbench","domain":"多模态","name":"EWMBench","released":"2025-05-16","url":"https://hub.opencompass.org.cn/dataset-detail/EWMBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EWMBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EWMBench","title":"EWMBench"},{"benchmarks":[{"benchmark_id":"opencompass-1970-editinspector","domain":"创作","name":"EditInspector","released":"2025-06-11","url":"https://hub.opencompass.org.cn/dataset-detail/EditInspector"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EditInspector","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EditInspector","title":"EditInspector"},{"benchmarks":[{"benchmark_id":"opencompass-1604-egonormia","domain":"学科","name":"EgoNormia","released":"2025-02-27","url":"https://hub.opencompass.org.cn/dataset-detail/EgoNormia"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EgoNormia","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EgoNormia","title":"EgoNormia"},{"benchmarks":[{"benchmark_id":"opencompass-1243-embodiedagentinterface","domain":"推理","name":"EmbodiedAgentInterface","released":"2024-06-24","url":"https://hub.opencompass.org.cn/dataset-detail/EmbodiedAgentInterface"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EmbodiedAgentInterface","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EmbodiedAgentInterface","title":"EmbodiedAgentInterface"},{"benchmarks":[{"benchmark_id":"opencompass-1564-embodiedbench","domain":"多模态","name":"EmbodiedBench","released":"2025-02-13","url":"https://hub.opencompass.org.cn/dataset-detail/EmbodiedBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EmbodiedBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/EmbodiedBench","title":"EmbodiedBench"},{"benchmarks":[{"benchmark_id":"opencompass-1733-feabench","domain":"智能体","name":"FEABench","released":"2025-04-08","url":"https://hub.opencompass.org.cn/dataset-detail/FEABench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FEABench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FEABench","title":"FEABench"},{"benchmarks":[{"benchmark_id":"opencompass-1324-flub","domain":"推理","name":"FLUB","released":"2024-02-16","url":"https://hub.opencompass.org.cn/dataset-detail/FLUB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FLUB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FLUB","title":"FLUB"},{"benchmarks":[{"benchmark_id":"opencompass-1145-freb-tqa","domain":"理解","name":"FREB-TQA","released":"2024-06-16","url":"https://hub.opencompass.org.cn/dataset-detail/FREB-TQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FREB-TQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FREB-TQA","title":"FREB-TQA"},{"benchmarks":[{"benchmark_id":"opencompass-1980-falsereject","domain":"推理","name":"FalseReject","released":"2025-05-12","url":"https://hub.opencompass.org.cn/dataset-detail/FalseReject"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FalseReject","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FalseReject","title":"FalseReject"},{"benchmarks":[{"benchmark_id":"opencompass-1618-fedmabench","domain":"智能体","name":"FedMABench","released":"2025-03-07","url":"https://hub.opencompass.org.cn/dataset-detail/FedMABench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FedMABench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FedMABench","title":"FedMABench"},{"benchmarks":[{"benchmark_id":"opencompass-895-fin-eva","domain":"知识","name":"Fin-Eva","released":"2023-12-21","url":"https://hub.opencompass.org.cn/dataset-detail/Fin-Eva"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Fin-Eva","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Fin-Eva","title":"Fin-Eva"},{"benchmarks":[{"benchmark_id":"opencompass-945-flames","domain":"安全","name":"Flames","released":"2024-03-13","url":"https://hub.opencompass.org.cn/dataset-detail/Flames"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Flames","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Flames","title":"Flames"},{"benchmarks":[{"benchmark_id":"opencompass-509-flores","domain":"语言","name":"Flores","released":"2021-06-06","url":"https://hub.opencompass.org.cn/dataset-detail/Flores"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Flores","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Flores","title":"Flores"},{"benchmarks":[{"benchmark_id":"opencompass-1671-forensics-bench","domain":"多模态","name":"Forensics-bench","released":"2025-03-24","url":"https://hub.opencompass.org.cn/dataset-detail/Forensics-bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Forensics-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Forensics-bench","title":"Forensics-bench"},{"benchmarks":[{"benchmark_id":"opencompass-1737-fortisavqa","domain":"多模态","name":"FortisAVQA","released":"2025-04-02","url":"https://hub.opencompass.org.cn/dataset-detail/FortisAVQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FortisAVQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/FortisAVQA","title":"FortisAVQA"},{"benchmarks":[{"benchmark_id":"opencompass-1599-gaia","domain":"智能体","name":"GAIA","released":"2023-11-21","url":"https://hub.opencompass.org.cn/dataset-detail/GAIA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GAIA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GAIA","title":"GAIA"},{"benchmarks":[{"benchmark_id":"opencompass-500-gaokao-bench","domain":"学科","name":"GAOKAO-Bench","released":"2023-05-21","url":"https://hub.opencompass.org.cn/dataset-detail/GAOKAO-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GAOKAO-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GAOKAO-Bench","title":"GAOKAO-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1083-gaokao-mm","domain":"学科","name":"GAOKAO-MM","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/GAOKAO-MM"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GAOKAO-MM","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GAOKAO-MM","title":"GAOKAO-MM"},{"benchmarks":[{"benchmark_id":"opencompass-2574-gauge","domain":"其他","name":"GAUGE","released":"2026-08-12","url":"https://hub.opencompass.org.cn/dataset-detail/GAUGE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GAUGE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GAUGE","title":"GAUGE"},{"benchmarks":[{"benchmark_id":"opencompass-1927-gmaimmbench","domain":"多模态","name":"GMAIMMBench","released":"2024-08-31","url":"https://hub.opencompass.org.cn/dataset-detail/GMAIMMBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GMAIMMBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GMAIMMBench","title":"GMAIMMBench"},{"benchmarks":[{"benchmark_id":"opencompass-1135-gpqa","domain":"知识","name":"GPQA","released":"2023-11-20","url":"https://hub.opencompass.org.cn/dataset-detail/GPQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GPQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GPQA","title":"GPQA"},{"benchmarks":[{"benchmark_id":"opencompass-1740-gpt-imgeval","domain":"多模态","name":"GPT-ImgEval","released":"2025-04-03","url":"https://hub.opencompass.org.cn/dataset-detail/GPT-ImgEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GPT-ImgEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GPT-ImgEval","title":"GPT-ImgEval"},{"benchmarks":[{"benchmark_id":"opencompass-1272-gsm1k","domain":"数学","name":"GSM1k","released":"2024-05-01","url":"https://hub.opencompass.org.cn/dataset-detail/GSM1k"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GSM1k","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GSM1k","title":"GSM1k"},{"benchmarks":[{"benchmark_id":"opencompass-535-gsm8k","domain":"数学","name":"GSM8K","released":"2021-04-01","url":"https://hub.opencompass.org.cn/dataset-detail/GSM8K"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GSM8K","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GSM8K","title":"GSM8K"},{"benchmarks":[{"benchmark_id":"opencompass-2233-gsm8k-v","domain":"多模态","name":"GSM8K-V","released":"2025-09-28","url":"https://hub.opencompass.org.cn/dataset-detail/GSM8K-V"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GSM8K-V","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GSM8K-V","title":"GSM8K-V"},{"benchmarks":[{"benchmark_id":"opencompass-1328-gta","domain":"智能体","name":"GTA","released":"2024-07-11","url":"https://hub.opencompass.org.cn/dataset-detail/GTA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GTA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GTA","title":"GTA"},{"benchmarks":[{"benchmark_id":"opencompass-1376-genai-bench","domain":"多模态","name":"GenAI-Bench","released":"2024-06-19","url":"https://hub.opencompass.org.cn/dataset-detail/GenAI-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GenAI-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GenAI-Bench","title":"GenAI-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2081-general-bench","domain":"多模态","name":"General-Bench","released":"2025-05-07","url":"https://hub.opencompass.org.cn/dataset-detail/General-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/General-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/General-Bench","title":"General-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1865-gitgoodbench","domain":"代码","name":"GitGoodBench","released":"2025-05-28","url":"https://hub.opencompass.org.cn/dataset-detail/GitGoodBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GitGoodBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GitGoodBench","title":"GitGoodBench"},{"benchmarks":[{"benchmark_id":"opencompass-1128-gorilla","domain":"智能体","name":"Gorilla","released":"2023-05-24","url":"https://hub.opencompass.org.cn/dataset-detail/Gorilla"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Gorilla","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Gorilla","title":"Gorilla"},{"benchmarks":[{"benchmark_id":"opencompass-1098-grailqa","domain":"知识","name":"GrailQA","released":"2021-02-22","url":"https://hub.opencompass.org.cn/dataset-detail/GrailQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GrailQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GrailQA","title":"GrailQA"},{"benchmarks":[{"benchmark_id":"opencompass-2027-groundingsuite","domain":"多模态","name":"GroundingSuite","released":"2025-07-05","url":"https://hub.opencompass.org.cn/dataset-detail/GroundingSuite"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GroundingSuite","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/GroundingSuite","title":"GroundingSuite"},{"benchmarks":[{"benchmark_id":"opencompass-2035-gym4real","domain":"其他","name":"Gym4ReaL","released":"2025-06-30","url":"https://hub.opencompass.org.cn/dataset-detail/Gym4ReaL"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Gym4ReaL","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Gym4ReaL","title":"Gym4ReaL"},{"benchmarks":[{"benchmark_id":"opencompass-1842-hardmath2","domain":"推理","name":"HARDMath2","released":"2025-05-17","url":"https://hub.opencompass.org.cn/dataset-detail/HARDMath2"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HARDMath2","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HARDMath2","title":"HARDMath2"},{"benchmarks":[{"benchmark_id":"opencompass-2031-herb","domain":"其他","name":"HERB","released":"2025-06-29","url":"https://hub.opencompass.org.cn/dataset-detail/HERB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HERB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HERB","title":"HERB"},{"benchmarks":[{"benchmark_id":"opencompass-1355-hallusionbench","domain":"多模态","name":"HallusionBench","released":"2023-10-23","url":"https://hub.opencompass.org.cn/dataset-detail/HallusionBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HallusionBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HallusionBench","title":"HallusionBench"},{"benchmarks":[{"benchmark_id":"opencompass-1121-halueval","domain":"安全","name":"HaluEval","released":"2023-10-23","url":"https://hub.opencompass.org.cn/dataset-detail/HaluEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HaluEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HaluEval","title":"HaluEval"},{"benchmarks":[{"benchmark_id":"opencompass-531-hellaswag","domain":"推理","name":"HellaSwag","released":"2019-05-19","url":"https://hub.opencompass.org.cn/dataset-detail/HellaSwag"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HellaSwag","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HellaSwag","title":"HellaSwag"},{"benchmarks":[{"benchmark_id":"opencompass-1097-hellobench","domain":"长文本","name":"HelloBench","released":"2024-09-24","url":"https://hub.opencompass.org.cn/dataset-detail/HelloBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HelloBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HelloBench","title":"HelloBench"},{"benchmarks":[{"benchmark_id":"opencompass-1581-holobench","domain":"推理","name":"HoloBench","released":"2024-10-15","url":"https://hub.opencompass.org.cn/dataset-detail/HoloBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HoloBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HoloBench","title":"HoloBench"},{"benchmarks":[{"benchmark_id":"opencompass-1108-hotpotqa","domain":"强推理","name":"HotpotQA","released":"2018-09-25","url":"https://hub.opencompass.org.cn/dataset-detail/HotpotQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HotpotQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HotpotQA","title":"HotpotQA"},{"benchmarks":[{"benchmark_id":"opencompass-1992-htfllib","domain":"安全","name":"HtFLlib","released":"2025-06-04","url":"https://hub.opencompass.org.cn/dataset-detail/HtFLlib"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HtFLlib","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HtFLlib","title":"HtFLlib"},{"benchmarks":[{"benchmark_id":"opencompass-537-humaneval","domain":"代码","name":"HumanEval","released":"2021-07-08","url":"https://hub.opencompass.org.cn/dataset-detail/HumanEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HumanEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HumanEval","title":"HumanEval"},{"benchmarks":[{"benchmark_id":"opencompass-543-humaneval-x","domain":"代码","name":"HumanEval-X","released":"2023-03-30","url":"https://hub.opencompass.org.cn/dataset-detail/HumanEval-X"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HumanEval-X","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HumanEval-X","title":"HumanEval-X"},{"benchmarks":[{"benchmark_id":"opencompass-1782-hypobench","domain":"其他","name":"HypoBench","released":"2025-04-15","url":"https://hub.opencompass.org.cn/dataset-detail/HypoBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HypoBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HypoBench","title":"HypoBench"},{"benchmarks":[{"benchmark_id":"opencompass-1786-hypoeval","domain":"其他","name":"HypoEval","released":"2025-04-09","url":"https://hub.opencompass.org.cn/dataset-detail/HypoEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HypoEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/HypoEval","title":"HypoEval"},{"benchmarks":[{"benchmark_id":"opencompass-1136-ifeval","domain":"指令跟随","name":"IFEval","released":"2023-11-14","url":"https://hub.opencompass.org.cn/dataset-detail/IFEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IFEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IFEval","title":"IFEval"},{"benchmarks":[{"benchmark_id":"opencompass-1617-ifir","domain":"指令跟随","name":"IFIR","released":"2025-03-06","url":"https://hub.opencompass.org.cn/dataset-detail/IFIR"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IFIR","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IFIR","title":"IFIR"},{"benchmarks":[{"benchmark_id":"opencompass-1274-imdl-benco","domain":"多模态","name":"IMDL-BenCo","released":"2024-06-15","url":"https://hub.opencompass.org.cn/dataset-detail/IMDL-BenCo"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IMDL-BenCo","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IMDL-BenCo","title":"IMDL-BenCo"},{"benchmarks":[{"benchmark_id":"opencompass-1840-iqbench","domain":"多模态","name":"IQBench","released":"2025-05-17","url":"https://hub.opencompass.org.cn/dataset-detail/IQBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IQBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IQBench","title":"IQBench"},{"benchmarks":[{"benchmark_id":"opencompass-2091-is-bench","domain":"多模态","name":"IS-Bench","released":"2025-07-07","url":"https://hub.opencompass.org.cn/dataset-detail/IS-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IS-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IS-Bench","title":"IS-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2077-itbench","domain":"代码","name":"ITBench","released":"2025-02-07","url":"https://hub.opencompass.org.cn/dataset-detail/ITBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ITBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ITBench","title":"ITBench"},{"benchmarks":[{"benchmark_id":"opencompass-1320-iac-eval","domain":"代码","name":"IaC-Eval","released":"2024-09-26","url":"https://hub.opencompass.org.cn/dataset-detail/IaC-Eval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IaC-Eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IaC-Eval","title":"IaC-Eval"},{"benchmarks":[{"benchmark_id":"opencompass-1670-indicmmlu-pro","domain":"语言","name":"IndicMMLU-Pro","released":"2025-01-28","url":"https://hub.opencompass.org.cn/dataset-detail/IndicMMLU-Pro"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IndicMMLU-Pro","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IndicMMLU-Pro","title":"IndicMMLU-Pro"},{"benchmarks":[{"benchmark_id":"opencompass-1322-infibench","domain":"代码","name":"InfiBench","released":"2024-03-11","url":"https://hub.opencompass.org.cn/dataset-detail/InfiBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InfiBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InfiBench","title":"InfiBench"},{"benchmarks":[{"benchmark_id":"opencompass-1085-infobench","domain":"指令跟随","name":"InfoBench","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/InfoBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InfoBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InfoBench","title":"InfoBench"},{"benchmarks":[{"benchmark_id":"opencompass-1149-instrusum","domain":"理解","name":"InstruSum","released":"2024-06-16","url":"https://hub.opencompass.org.cn/dataset-detail/InstruSum"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InstruSum","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InstruSum","title":"InstruSum"},{"benchmarks":[{"benchmark_id":"opencompass-1976-intphys2","domain":"理解","name":"IntPhys2","released":"2025-06-11","url":"https://hub.opencompass.org.cn/dataset-detail/IntPhys2"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IntPhys2","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/IntPhys2","title":"IntPhys2"},{"benchmarks":[{"benchmark_id":"opencompass-2151-interndata-a1","domain":"多模态","name":"InternData-A1","released":"2025-07-26","url":"https://hub.opencompass.org.cn/dataset-detail/InternData-A1"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InternData-A1","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InternData-A1","title":"InternData-A1"},{"benchmarks":[{"benchmark_id":"opencompass-2153-interndata-m1","domain":"多模态","name":"InternData-M1","released":"2025-07-26","url":"https://hub.opencompass.org.cn/dataset-detail/InternData-M1"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InternData-M1","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InternData-M1","title":"InternData-M1"},{"benchmarks":[{"benchmark_id":"opencompass-2152-interndata-n1","domain":"多模态","name":"InternData-N1","released":"2025-07-26","url":"https://hub.opencompass.org.cn/dataset-detail/InternData-N1"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InternData-N1","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/InternData-N1","title":"InternData-N1"},{"benchmarks":[{"benchmark_id":"opencompass-2073-j1-bench","domain":"智能体","name":"J1-Bench","released":"2025-07-17","url":"https://hub.opencompass.org.cn/dataset-detail/J1-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/J1-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/J1-Bench","title":"J1-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1554-jl1-cd","domain":"其他","name":"JL1-CD","released":"2025-02-19","url":"https://hub.opencompass.org.cn/dataset-detail/JL1-CD"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/JL1-CD","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/JL1-CD","title":"JL1-CD"},{"benchmarks":[{"benchmark_id":"opencompass-2130-job-complex","domain":"其他","name":"JOB-Complex","released":"2025-07-10","url":"https://hub.opencompass.org.cn/dataset-detail/JOB-Complex"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/JOB-Complex","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/JOB-Complex","title":"JOB-Complex"},{"benchmarks":[{"benchmark_id":"opencompass-1337-jailtrickbench","domain":"安全","name":"JailTrickBench","released":"2024-06-13","url":"https://hub.opencompass.org.cn/dataset-detail/JailTrickBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/JailTrickBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/JailTrickBench","title":"JailTrickBench"},{"benchmarks":[{"benchmark_id":"opencompass-1583-judgebench","domain":"其他","name":"JudgeBench","released":"2024-10-16","url":"https://hub.opencompass.org.cn/dataset-detail/JudgeBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/JudgeBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/JudgeBench","title":"JudgeBench"},{"benchmarks":[{"benchmark_id":"opencompass-2076-k-sort-arena","domain":"多模态","name":"K-Sort-Arena","released":"2024-09-02","url":"https://hub.opencompass.org.cn/dataset-detail/K-Sort-Arena"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/K-Sort-Arena","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/K-Sort-Arena","title":"K-Sort-Arena"},{"benchmarks":[{"benchmark_id":"opencompass-1543-kitab-bench","domain":"多模态","name":"KITAB-Bench","released":"2025-02-20","url":"https://hub.opencompass.org.cn/dataset-detail/KITAB-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KITAB-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KITAB-Bench","title":"KITAB-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2126-kmmlu-redux","domain":"知识","name":"KMMLU-Redux","released":"2025-07-11","url":"https://hub.opencompass.org.cn/dataset-detail/KMMLU-Redux"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KMMLU-Redux","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KMMLU-Redux","title":"KMMLU-Redux"},{"benchmarks":[{"benchmark_id":"opencompass-1706-koffvqa","domain":"多模态","name":"KOFFVQA","released":"2025-03-31","url":"https://hub.opencompass.org.cn/dataset-detail/KOFFVQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KOFFVQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KOFFVQA","title":"KOFFVQA"},{"benchmarks":[{"benchmark_id":"opencompass-1245-kor-bench","domain":"强推理","name":"KOR-Bench","released":"2024-10-09","url":"https://hub.opencompass.org.cn/dataset-detail/KOR-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KOR-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KOR-Bench","title":"KOR-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1635-knowlogic","domain":"推理","name":"KnowLogic","released":"2025-03-08","url":"https://hub.opencompass.org.cn/dataset-detail/KnowLogic"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KnowLogic","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/KnowLogic","title":"KnowLogic"},{"benchmarks":[{"benchmark_id":"opencompass-541-l-eval","domain":"长文本","name":"L-Eval","released":"2023-10-04","url":"https://hub.opencompass.org.cn/dataset-detail/L-Eval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/L-Eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/L-Eval","title":"L-Eval"},{"benchmarks":[{"benchmark_id":"opencompass-523-lambada","domain":"理解","name":"LAMBADA","released":"2016-06-20","url":"https://hub.opencompass.org.cn/dataset-detail/LAMBADA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LAMBADA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LAMBADA","title":"LAMBADA"},{"benchmarks":[{"benchmark_id":"opencompass-520-lcsts","domain":"理解","name":"LCSTS","released":"2015-06-19","url":"https://hub.opencompass.org.cn/dataset-detail/LCSTS"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LCSTS","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LCSTS","title":"LCSTS"},{"benchmarks":[{"benchmark_id":"opencompass-2422-lens","domain":"多模态","name":"LENS","released":"2025-06-01","url":"https://hub.opencompass.org.cn/dataset-detail/LENS"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LENS","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LENS","title":"LENS"},{"benchmarks":[{"benchmark_id":"opencompass-1238-lingoly","domain":"推理","name":"LINGOLY","released":"2024-06-10","url":"https://hub.opencompass.org.cn/dataset-detail/LINGOLY"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LINGOLY","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LINGOLY","title":"LINGOLY"},{"benchmarks":[{"benchmark_id":"opencompass-1838-llm-babybench","domain":"强推理","name":"LLM-BabyBench","released":"2025-05-17","url":"https://hub.opencompass.org.cn/dataset-detail/LLM-BabyBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLM-BabyBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLM-BabyBench","title":"LLM-BabyBench"},{"benchmarks":[{"benchmark_id":"opencompass-1752-llm-srbench","domain":"推理","name":"LLM-SRBench","released":"2025-04-14","url":"https://hub.opencompass.org.cn/dataset-detail/LLM-SRBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLM-SRBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLM-SRBench","title":"LLM-SRBench"},{"benchmarks":[{"benchmark_id":"opencompass-1325-llm-uncertainty-bench","domain":"其他","name":"LLM-Uncertainty-Bench","released":"2024-01-23","url":"https://hub.opencompass.org.cn/dataset-detail/LLM-Uncertainty-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLM-Uncertainty-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLM-Uncertainty-Bench","title":"LLM-Uncertainty-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2047-llmthinkbench","domain":"推理","name":"LLMThinkBench","released":"2025-07-05","url":"https://hub.opencompass.org.cn/dataset-detail/LLMThinkBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLMThinkBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLMThinkBench","title":"LLMThinkBench"},{"benchmarks":[{"benchmark_id":"opencompass-1360-llava-bench","domain":"多模态","name":"LLaVA-Bench","released":"2023-04-17","url":"https://hub.opencompass.org.cn/dataset-detail/LLaVA-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLaVA-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LLaVA-Bench","title":"LLaVA-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2088-lmact","domain":"长文本","name":"LMAct","released":"2024-12-02","url":"https://hub.opencompass.org.cn/dataset-detail/LMAct"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LMAct","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LMAct","title":"LMAct"},{"benchmarks":[{"benchmark_id":"opencompass-1572-loki","domain":"多模态","name":"LOKI","released":"2024-10-13","url":"https://hub.opencompass.org.cn/dataset-detail/LOKI"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LOKI","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LOKI","title":"LOKI"},{"benchmarks":[{"benchmark_id":"opencompass-1335-ltmbenchmark","domain":"智能体","name":"LTMbenchmark","released":"2024-09-30","url":"https://hub.opencompass.org.cn/dataset-detail/LTMbenchmark"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LTMbenchmark","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LTMbenchmark","title":"LTMbenchmark"},{"benchmarks":[{"benchmark_id":"opencompass-564-lv-eval","domain":"长文本","name":"LV-Eval","released":"2024-02-18","url":"https://hub.opencompass.org.cn/dataset-detail/LV-Eval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LV-Eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LV-Eval","title":"LV-Eval"},{"benchmarks":[{"benchmark_id":"opencompass-1924-lamp-qa","domain":"其他","name":"LaMP-QA","released":"2025-05-30","url":"https://hub.opencompass.org.cn/dataset-detail/LaMP-QA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LaMP-QA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LaMP-QA","title":"LaMP-QA"},{"benchmarks":[{"benchmark_id":"opencompass-2089-lara","domain":"长文本","name":"LaRA","released":"2025-02-14","url":"https://hub.opencompass.org.cn/dataset-detail/LaRA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LaRA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LaRA","title":"LaRA"},{"benchmarks":[{"benchmark_id":"opencompass-2129-langnavbench","domain":"理解","name":"LangNavBench","released":"2025-07-09","url":"https://hub.opencompass.org.cn/dataset-detail/LangNavBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LangNavBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LangNavBench","title":"LangNavBench"},{"benchmarks":[{"benchmark_id":"opencompass-1246-livebench","domain":"知识","name":"LiveBench","released":"2024-06-12","url":"https://hub.opencompass.org.cn/dataset-detail/LiveBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LiveBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LiveBench","title":"LiveBench"},{"benchmarks":[{"benchmark_id":"opencompass-1413-livecodebench","domain":"代码","name":"LiveCodeBench","released":"2024-03-12","url":"https://hub.opencompass.org.cn/dataset-detail/LiveCodeBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LiveCodeBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LiveCodeBench","title":"LiveCodeBench"},{"benchmarks":[{"benchmark_id":"opencompass-1787-livelongbench","domain":"推理","name":"LiveLongBench","released":"2025-04-24","url":"https://hub.opencompass.org.cn/dataset-detail/LiveLongBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LiveLongBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LiveLongBench","title":"LiveLongBench"},{"benchmarks":[{"benchmark_id":"opencompass-1397-livemathbench","domain":"强推理","name":"LiveMathBench","released":"2024-12-17","url":"https://hub.opencompass.org.cn/dataset-detail/LiveMathBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LiveMathBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LiveMathBench","title":"LiveMathBench"},{"benchmarks":[{"benchmark_id":"opencompass-542-longbench","domain":"长文本","name":"LongBench","released":"2023-08-28","url":"https://hub.opencompass.org.cn/dataset-detail/LongBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LongBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LongBench","title":"LongBench"},{"benchmarks":[{"benchmark_id":"opencompass-2067-longvale","domain":"多模态","name":"LongVALE","released":"2025-04-17","url":"https://hub.opencompass.org.cn/dataset-detail/LongVALE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LongVALE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LongVALE","title":"LongVALE"},{"benchmarks":[{"benchmark_id":"opencompass-1510-longvideobench","domain":"多模态","name":"LongVideoBench","released":"2024-07-22","url":"https://hub.opencompass.org.cn/dataset-detail/LongVideoBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LongVideoBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LongVideoBench","title":"LongVideoBench"},{"benchmarks":[{"benchmark_id":"opencompass-1944-loopnav","domain":"多模态","name":"LoopNav","released":"2025-05-15","url":"https://hub.opencompass.org.cn/dataset-detail/LoopNav"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LoopNav","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/LoopNav","title":"LoopNav"},{"benchmarks":[{"benchmark_id":"opencompass-1147-m3t","domain":"理解","name":"M3T","released":"2024-06-16","url":"https://hub.opencompass.org.cn/dataset-detail/M3T"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/M3T","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/M3T","title":"M3T"},{"benchmarks":[{"benchmark_id":"opencompass-1609-mask","domain":"其他","name":"MASK","released":"2025-03-05","url":"https://hub.opencompass.org.cn/dataset-detail/MASK"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MASK","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MASK","title":"MASK"},{"benchmarks":[{"benchmark_id":"opencompass-534-math","domain":"数学","name":"MATH","released":"2021-11-08","url":"https://hub.opencompass.org.cn/dataset-detail/MATH"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MATH","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MATH","title":"MATH"},{"benchmarks":[{"benchmark_id":"opencompass-1848-mavos-dd","domain":"多模态","name":"MAVOS-DD","released":"2025-05-16","url":"https://hub.opencompass.org.cn/dataset-detail/MAVOS-DD"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MAVOS-DD","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MAVOS-DD","title":"MAVOS-DD"},{"benchmarks":[{"benchmark_id":"opencompass-538-mbpp","domain":"代码","name":"MBPP","released":"2021-08-16","url":"https://hub.opencompass.org.cn/dataset-detail/MBPP"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MBPP","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MBPP","title":"MBPP"},{"benchmarks":[{"benchmark_id":"opencompass-1606-mcitebench","domain":"多模态","name":"MCiteBench","released":"2025-03-05","url":"https://hub.opencompass.org.cn/dataset-detail/MCiteBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MCiteBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MCiteBench","title":"MCiteBench"},{"benchmarks":[{"benchmark_id":"opencompass-1893-medal","domain":"其他","name":"MEDAL","released":"2025-05-28","url":"https://hub.opencompass.org.cn/dataset-detail/MEDAL"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MEDAL","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MEDAL","title":"MEDAL"},{"benchmarks":[{"benchmark_id":"opencompass-2079-mer-unibench","domain":"多模态","name":"MER-UniBench","released":"2025-01-27","url":"https://hub.opencompass.org.cn/dataset-detail/MER-UniBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MER-UniBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MER-UniBench","title":"MER-UniBench"},{"benchmarks":[{"benchmark_id":"opencompass-2085-mib","domain":"理解","name":"MIB","released":"2025-04-17","url":"https://hub.opencompass.org.cn/dataset-detail/MIB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MIB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MIB","title":"MIB"},{"benchmarks":[{"benchmark_id":"opencompass-1781-mieb","domain":"多模态","name":"MIEB","released":"2025-04-14","url":"https://hub.opencompass.org.cn/dataset-detail/MIEB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MIEB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MIEB","title":"MIEB"},{"benchmarks":[{"benchmark_id":"opencompass-1844-miracl-vision","domain":"多模态","name":"MIRACL-VISION","released":"2025-05-22","url":"https://hub.opencompass.org.cn/dataset-detail/MIRACL-VISION"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MIRACL-VISION","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MIRACL-VISION","title":"MIRACL-VISION"},{"benchmarks":[{"benchmark_id":"opencompass-1142-mirage","domain":"其他","name":"MIRAGE","released":"2024-08-16","url":"https://hub.opencompass.org.cn/dataset-detail/MIRAGE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MIRAGE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MIRAGE","title":"MIRAGE"},{"benchmarks":[{"benchmark_id":"opencompass-1623-mj-bench","domain":"多模态","name":"MJ-Bench","released":"2024-07-05","url":"https://hub.opencompass.org.cn/dataset-detail/MJ-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MJ-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MJ-Bench","title":"MJ-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1099-mkqa","domain":"知识","name":"MKQA","released":"2021-08-17","url":"https://hub.opencompass.org.cn/dataset-detail/MKQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MKQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MKQA","title":"MKQA"},{"benchmarks":[{"benchmark_id":"opencompass-1779-mlrc-bench","domain":"智能体","name":"MLRC-Bench","released":"2025-04-13","url":"https://hub.opencompass.org.cn/dataset-detail/MLRC-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MLRC-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MLRC-Bench","title":"MLRC-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1512-mlvu","domain":"多模态","name":"MLVU","released":"2024-06-06","url":"https://hub.opencompass.org.cn/dataset-detail/MLVU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MLVU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MLVU","title":"MLVU"},{"benchmarks":[{"benchmark_id":"opencompass-1558-mm-alignbench","domain":"多模态","name":"MM-AlignBench","released":"2025-02-25","url":"https://hub.opencompass.org.cn/dataset-detail/MM-AlignBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MM-AlignBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MM-AlignBench","title":"MM-AlignBench"},{"benchmarks":[{"benchmark_id":"opencompass-1566-mm-iq","domain":"多模态","name":"MM-IQ","released":"2025-02-02","url":"https://hub.opencompass.org.cn/dataset-detail/MM-IQ"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MM-IQ","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MM-IQ","title":"MM-IQ"},{"benchmarks":[{"benchmark_id":"opencompass-1539-mm-rlhf","domain":"多模态","name":"MM-RLHF","released":"2025-02-04","url":"https://hub.opencompass.org.cn/dataset-detail/MM-RLHF"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MM-RLHF","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MM-RLHF","title":"MM-RLHF"},{"benchmarks":[{"benchmark_id":"opencompass-1356-mm-vet","domain":"多模态","name":"MM-Vet","released":"2023-08-04","url":"https://hub.opencompass.org.cn/dataset-detail/MM-Vet"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MM-Vet","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MM-Vet","title":"MM-Vet"},{"benchmarks":[{"benchmark_id":"opencompass-1585-mmad","domain":"多模态","name":"MMAD","released":"2024-10-12","url":"https://hub.opencompass.org.cn/dataset-detail/MMAD"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMAD","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMAD","title":"MMAD"},{"benchmarks":[{"benchmark_id":"opencompass-1910-mmar","domain":"强推理","name":"MMAR","released":"2025-05-11","url":"https://hub.opencompass.org.cn/dataset-detail/MMAR"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMAR","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMAR","title":"MMAR"},{"benchmarks":[{"benchmark_id":"opencompass-1206-mmbench","domain":"多模态","name":"MMBench","released":"2023-07-12","url":"https://hub.opencompass.org.cn/dataset-detail/MMBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMBench","title":"MMBench"},{"benchmarks":[{"benchmark_id":"opencompass-1207-mmbench-video","domain":"多模态","name":"MMBench-Video","released":"2024-06-20","url":"https://hub.opencompass.org.cn/dataset-detail/MMBench-Video"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMBench-Video","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMBench-Video","title":"MMBench-Video"},{"benchmarks":[{"benchmark_id":"opencompass-1334-mmdu","domain":"多模态","name":"MMDU","released":"2024-06-17","url":"https://hub.opencompass.org.cn/dataset-detail/MMDU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMDU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMDU","title":"MMDU"},{"benchmarks":[{"benchmark_id":"opencompass-1357-mme","domain":"多模态","name":"MME","released":"2023-06-23","url":"https://hub.opencompass.org.cn/dataset-detail/MME"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MME","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MME","title":"MME"},{"benchmarks":[{"benchmark_id":"opencompass-1565-mme-cot","domain":"强推理","name":"MME-CoT","released":"2025-02-13","url":"https://hub.opencompass.org.cn/dataset-detail/MME-CoT"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MME-CoT","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MME-CoT","title":"MME-CoT"},{"benchmarks":[{"benchmark_id":"opencompass-1451-mme-realworld","domain":"多模态","name":"MME-RealWorld","released":"2024-08-23","url":"https://hub.opencompass.org.cn/dataset-detail/MME-RealWorld"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MME-RealWorld","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MME-RealWorld","title":"MME-RealWorld"},{"benchmarks":[{"benchmark_id":"opencompass-1523-mmie","domain":"多模态","name":"MMIE","released":"2024-10-15","url":"https://hub.opencompass.org.cn/dataset-detail/MMIE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMIE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMIE","title":"MMIE"},{"benchmarks":[{"benchmark_id":"opencompass-1547-mmir","domain":"多模态","name":"MMIR","released":"2025-02-22","url":"https://hub.opencompass.org.cn/dataset-detail/MMIR"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMIR","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMIR","title":"MMIR"},{"benchmarks":[{"benchmark_id":"opencompass-1453-mmiu","domain":"多模态","name":"MMIU","released":"2024-08-05","url":"https://hub.opencompass.org.cn/dataset-detail/MMIU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMIU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMIU","title":"MMIU"},{"benchmarks":[{"benchmark_id":"opencompass-1578-mmke-bench","domain":"其他","name":"MMKE-Bench","released":"2025-02-27","url":"https://hub.opencompass.org.cn/dataset-detail/MMKE-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMKE-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMKE-Bench","title":"MMKE-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-498-mmlu","domain":"学科","name":"MMLU","released":"2020-09-07","url":"https://hub.opencompass.org.cn/dataset-detail/MMLU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMLU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMLU","title":"MMLU"},{"benchmarks":[{"benchmark_id":"opencompass-1276-mmlu-pro","domain":"学科","name":"MMLU-Pro","released":"2024-06-03","url":"https://hub.opencompass.org.cn/dataset-detail/MMLU-Pro"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMLU-Pro","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMLU-Pro","title":"MMLU-Pro"},{"benchmarks":[{"benchmark_id":"opencompass-1851-mmlongbench","domain":"多模态","name":"MMLongBench","released":"2025-05-16","url":"https://hub.opencompass.org.cn/dataset-detail/MMLongBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMLongBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMLongBench","title":"MMLongBench"},{"benchmarks":[{"benchmark_id":"opencompass-1275-mmlongbench-doc","domain":"长文本","name":"MMLongBench-Doc","released":"2024-07-01","url":"https://hub.opencompass.org.cn/dataset-detail/MMLongBench-Doc"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMLongBench-Doc","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMLongBench-Doc","title":"MMLongBench-Doc"},{"benchmarks":[{"benchmark_id":"opencompass-1248-mmmu","domain":"知识","name":"MMMU","released":"2023-11-27","url":"https://hub.opencompass.org.cn/dataset-detail/MMMU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMMU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMMU","title":"MMMU"},{"benchmarks":[{"benchmark_id":"opencompass-1829-mms-vpr","domain":"多模态","name":"MMS-VPR","released":"2025-05-18","url":"https://hub.opencompass.org.cn/dataset-detail/MMS-VPR"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMS-VPR","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMS-VPR","title":"MMS-VPR"},{"benchmarks":[{"benchmark_id":"opencompass-1922-mmsi-bench","domain":"多模态","name":"MMSI-Bench","released":"2025-05-30","url":"https://hub.opencompass.org.cn/dataset-detail/MMSI-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMSI-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMSI-Bench","title":"MMSI-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1584-mmsearch","domain":"多模态","name":"MMSearch","released":"2024-09-19","url":"https://hub.opencompass.org.cn/dataset-detail/MMSearch"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMSearch","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMSearch","title":"MMSearch"},{"benchmarks":[{"benchmark_id":"opencompass-1175-mmstar","domain":"其他","name":"MMStar","released":"2024-04-09","url":"https://hub.opencompass.org.cn/dataset-detail/MMStar"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMStar","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMStar","title":"MMStar"},{"benchmarks":[{"benchmark_id":"opencompass-1364-mmt-bench","domain":"多模态","name":"MMT-Bench","released":"2024-04-24","url":"https://hub.opencompass.org.cn/dataset-detail/MMT-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMT-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMT-Bench","title":"MMT-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1736-mmtb","domain":"智能体","name":"MMTB","released":"2025-03-30","url":"https://hub.opencompass.org.cn/dataset-detail/MMTB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMTB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MMTB","title":"MMTB"},{"benchmarks":[{"benchmark_id":"opencompass-1999-morse-500","domain":"强推理","name":"MORSE-500","released":"2025-06-05","url":"https://hub.opencompass.org.cn/dataset-detail/MORSE-500"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MORSE-500","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MORSE-500","title":"MORSE-500"},{"benchmarks":[{"benchmark_id":"opencompass-930-mr-ben-meta-reasoning-benchmark","domain":"推理","name":"MR-Ben-Meta-Reasoning-Benchmark","released":"2024-06-20","url":"https://hub.opencompass.org.cn/dataset-detail/MR-Ben-Meta-Reasoning-Benchmark"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MR-Ben-Meta-Reasoning-Benchmark","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MR-Ben-Meta-Reasoning-Benchmark","title":"MR-Ben-Meta-Reasoning-Benchmark"},{"benchmarks":[{"benchmark_id":"opencompass-1587-mr-gsm8k","domain":"推理","name":"MR-GSM8K","released":"2023-12-28","url":"https://hub.opencompass.org.cn/dataset-detail/MR-GSM8K"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MR-GSM8K","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MR-GSM8K","title":"MR-GSM8K"},{"benchmarks":[{"benchmark_id":"opencompass-1586-mrag-bench","domain":"多模态","name":"MRAG-Bench","released":"2024-10-10","url":"https://hub.opencompass.org.cn/dataset-detail/MRAG-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MRAG-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MRAG-Bench","title":"MRAG-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1102-ms-marco","domain":"知识","name":"MS_MARCO","released":"2018-10-31","url":"https://hub.opencompass.org.cn/dataset-detail/MS_MARCO"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MS_MARCO","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MS_MARCO","title":"MS_MARCO"},{"benchmarks":[{"benchmark_id":"opencompass-1172-mt-bench-101","domain":"理解","name":"MT-Bench-101","released":"2024-06-25","url":"https://hub.opencompass.org.cn/dataset-detail/MT-Bench-101"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MT-Bench-101","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MT-Bench-101","title":"MT-Bench-101"},{"benchmarks":[{"benchmark_id":"opencompass-1929-mtcmb","domain":"推理","name":"MTCMB","released":"2025-05-15","url":"https://hub.opencompass.org.cn/dataset-detail/MTCMB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MTCMB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MTCMB","title":"MTCMB"},{"benchmarks":[{"benchmark_id":"opencompass-2034-mteb","domain":"其他","name":"MTEB","released":"2025-06-26","url":"https://hub.opencompass.org.cn/dataset-detail/MTEB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MTEB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MTEB","title":"MTEB"},{"benchmarks":[{"benchmark_id":"opencompass-1502-mtvqa","domain":"多模态","name":"MTVQA","released":"2024-06-04","url":"https://hub.opencompass.org.cn/dataset-detail/MTVQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MTVQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MTVQA","title":"MTVQA"},{"benchmarks":[{"benchmark_id":"opencompass-1509-mvbench","domain":"多模态","name":"MVBench","released":"2023-11-28","url":"https://hub.opencompass.org.cn/dataset-detail/MVBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MVBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MVBench","title":"MVBench"},{"benchmarks":[{"benchmark_id":"opencompass-1537-mvl-sib","domain":"多模态","name":"MVL-SIB","released":"2025-02-18","url":"https://hub.opencompass.org.cn/dataset-detail/MVL-SIB"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MVL-SIB","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MVL-SIB","title":"MVL-SIB"},{"benchmarks":[{"benchmark_id":"opencompass-1905-mvpbench","domain":"多模态","name":"MVPBench","released":"2025-06-02","url":"https://hub.opencompass.org.cn/dataset-detail/MVPBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MVPBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MVPBench","title":"MVPBench"},{"benchmarks":[{"benchmark_id":"opencompass-1694-maritimebench","domain":"学科","name":"MaritimeBench","released":"2025-04-01","url":"https://hub.opencompass.org.cn/dataset-detail/MaritimeBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MaritimeBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MaritimeBench","title":"MaritimeBench"},{"benchmarks":[{"benchmark_id":"opencompass-1847-massive-steps","domain":"多模态","name":"Massive-STEPS","released":"2025-05-16","url":"https://hub.opencompass.org.cn/dataset-detail/Massive-STEPS"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Massive-STEPS","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Massive-STEPS","title":"Massive-STEPS"},{"benchmarks":[{"benchmark_id":"opencompass-1645-mastermindeval","domain":"推理","name":"MastermindEval","released":"2025-03-07","url":"https://hub.opencompass.org.cn/dataset-detail/MastermindEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MastermindEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MastermindEval","title":"MastermindEval"},{"benchmarks":[{"benchmark_id":"opencompass-1089-mathbench","domain":"数学","name":"MathBench","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/MathBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathBench","title":"MathBench"},{"benchmarks":[{"benchmark_id":"opencompass-1115-mathqa","domain":"数学","name":"MathQA","released":"2019-05-31","url":"https://hub.opencompass.org.cn/dataset-detail/MathQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathQA","title":"MathQA"},{"benchmarks":[{"benchmark_id":"opencompass-1371-mathverse","domain":"多模态","name":"MathVerse","released":"2024-03-21","url":"https://hub.opencompass.org.cn/dataset-detail/MathVerse"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathVerse","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathVerse","title":"MathVerse"},{"benchmarks":[{"benchmark_id":"opencompass-1370-mathvision","domain":"多模态","name":"MathVision","released":"2024-02-22","url":"https://hub.opencompass.org.cn/dataset-detail/MathVision"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathVision","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathVision","title":"MathVision"},{"benchmarks":[{"benchmark_id":"opencompass-1178-mathvista","domain":"强推理","name":"MathVista","released":"2023-10-03","url":"https://hub.opencompass.org.cn/dataset-detail/MathVista"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathVista","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MathVista","title":"MathVista"},{"benchmarks":[{"benchmark_id":"opencompass-1641-medagents-bench","domain":"推理","name":"MedAgents-Bench","released":"2025-03-16","url":"https://hub.opencompass.org.cn/dataset-detail/MedAgents-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedAgents-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedAgents-Bench","title":"MedAgents-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1874-medarabiq","domain":"其他","name":"MedArabiQ","released":"2025-05-06","url":"https://hub.opencompass.org.cn/dataset-detail/MedArabiQ"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedArabiQ","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedArabiQ","title":"MedArabiQ"},{"benchmarks":[{"benchmark_id":"opencompass-1287-medbench","domain":"知识","name":"MedBench","released":"2023-12-09","url":"https://hub.opencompass.org.cn/dataset-detail/MedBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedBench","title":"MedBench"},{"benchmarks":[{"benchmark_id":"opencompass-1920-medbookvqa","domain":"多模态","name":"MedBookVQA","released":"2025-05-17","url":"https://hub.opencompass.org.cn/dataset-detail/MedBookVQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedBookVQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedBookVQA","title":"MedBookVQA"},{"benchmarks":[{"benchmark_id":"opencompass-1832-medbrowsecomp","domain":"其他","name":"MedBrowseComp","released":"2025-05-20","url":"https://hub.opencompass.org.cn/dataset-detail/MedBrowseComp"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedBrowseComp","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedBrowseComp","title":"MedBrowseComp"},{"benchmarks":[{"benchmark_id":"opencompass-1241-medcalc-bench","domain":"推理","name":"MedCalc-Bench","released":"2024-06-17","url":"https://hub.opencompass.org.cn/dataset-detail/MedCalc-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedCalc-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedCalc-Bench","title":"MedCalc-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2417-medhalltune","domain":"医学","name":"MedHallTune","released":"2025-02-28","url":"https://hub.opencompass.org.cn/dataset-detail/MedHallTune"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedHallTune","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedHallTune","title":"MedHallTune"},{"benchmarks":[{"benchmark_id":"opencompass-1548-medhallu","domain":"其他","name":"MedHallu","released":"2025-02-20","url":"https://hub.opencompass.org.cn/dataset-detail/MedHallu"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedHallu","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedHallu","title":"MedHallu"},{"benchmarks":[{"benchmark_id":"opencompass-1326-medjourney","domain":"推理","name":"MedJourney","released":"2024-09-26","url":"https://hub.opencompass.org.cn/dataset-detail/MedJourney"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedJourney","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedJourney","title":"MedJourney"},{"benchmarks":[{"benchmark_id":"opencompass-1331-medsafetybench","domain":"安全","name":"MedSafetyBench","released":"2024-03-06","url":"https://hub.opencompass.org.cn/dataset-detail/MedSafetyBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedSafetyBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedSafetyBench","title":"MedSafetyBench"},{"benchmarks":[{"benchmark_id":"opencompass-1891-medxpertqa","domain":"学科","name":"MedXpertQA","released":"2025-02-09","url":"https://hub.opencompass.org.cn/dataset-detail/MedXpertQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedXpertQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MedXpertQA","title":"MedXpertQA"},{"benchmarks":[{"benchmark_id":"opencompass-1658-milic-eval","domain":"其他","name":"MiLiC-Eval","released":"2025-03-03","url":"https://hub.opencompass.org.cn/dataset-detail/MiLiC-Eval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MiLiC-Eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MiLiC-Eval","title":"MiLiC-Eval"},{"benchmarks":[{"benchmark_id":"opencompass-1667-microvqa","domain":"多模态","name":"MicroVQA","released":"2025-03-17","url":"https://hub.opencompass.org.cn/dataset-detail/MicroVQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MicroVQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MicroVQA","title":"MicroVQA"},{"benchmarks":[{"benchmark_id":"opencompass-1125-mind2web","domain":"其他","name":"Mind2Web","released":"2023-12-09","url":"https://hub.opencompass.org.cn/dataset-detail/Mind2Web"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Mind2Web","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Mind2Web","title":"Mind2Web"},{"benchmarks":[{"benchmark_id":"opencompass-1853-minilongbench","domain":"理解","name":"MiniLongBench","released":"2025-05-15","url":"https://hub.opencompass.org.cn/dataset-detail/MiniLongBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MiniLongBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MiniLongBench","title":"MiniLongBench"},{"benchmarks":[{"benchmark_id":"opencompass-1273-molpuzzle","domain":"推理","name":"MolPuzzle","released":"2024-09-26","url":"https://hub.opencompass.org.cn/dataset-detail/MolPuzzle"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MolPuzzle","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MolPuzzle","title":"MolPuzzle"},{"benchmarks":[{"benchmark_id":"opencompass-1704-mono2stereo","domain":"其他","name":"Mono2Stereo","released":"2025-03-30","url":"https://hub.opencompass.org.cn/dataset-detail/Mono2Stereo"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Mono2Stereo","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Mono2Stereo","title":"Mono2Stereo"},{"benchmarks":[{"benchmark_id":"opencompass-1682-motionbench","domain":"多模态","name":"MotionBench","released":"2025-01-06","url":"https://hub.opencompass.org.cn/dataset-detail/MotionBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MotionBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MotionBench","title":"MotionBench"},{"benchmarks":[{"benchmark_id":"opencompass-2154-motionmillion","domain":"多模态","name":"MotionMillion","released":"2025-07-26","url":"https://hub.opencompass.org.cn/dataset-detail/MotionMillion"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MotionMillion","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MotionMillion","title":"MotionMillion"},{"benchmarks":[{"benchmark_id":"opencompass-1751-multiloko","domain":"语言","name":"MultiLoKo","released":"2025-04-15","url":"https://hub.opencompass.org.cn/dataset-detail/MultiLoKo"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MultiLoKo","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/MultiLoKo","title":"MultiLoKo"},{"benchmarks":[{"benchmark_id":"opencompass-1783-nppc","domain":"推理","name":"NPPC","released":"2025-04-15","url":"https://hub.opencompass.org.cn/dataset-detail/NPPC"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NPPC","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NPPC","title":"NPPC"},{"benchmarks":[{"benchmark_id":"opencompass-513-nq","domain":"知识","name":"NQ","released":"2019-06-03","url":"https://hub.opencompass.org.cn/dataset-detail/NQ"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NQ","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NQ","title":"NQ"},{"benchmarks":[{"benchmark_id":"opencompass-1081-naturalcodebench","domain":"代码","name":"NaturalCodeBench","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/NaturalCodeBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NaturalCodeBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NaturalCodeBench","title":"NaturalCodeBench"},{"benchmarks":[{"benchmark_id":"opencompass-1117-naturalproofs","domain":"数学","name":"NaturalProofs","released":"2021-06-07","url":"https://hub.opencompass.org.cn/dataset-detail/NaturalProofs"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NaturalProofs","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NaturalProofs","title":"NaturalProofs"},{"benchmarks":[{"benchmark_id":"opencompass-1074-newsbench","domain":"创作","name":"NewsBench","released":"2024-06-04","url":"https://hub.opencompass.org.cn/dataset-detail/NewsBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NewsBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NewsBench","title":"NewsBench"},{"benchmarks":[{"benchmark_id":"opencompass-1533-nutritionqa","domain":"多模态","name":"NutritionQA","released":"2025-02-19","url":"https://hub.opencompass.org.cn/dataset-detail/NutritionQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NutritionQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/NutritionQA","title":"NutritionQA"},{"benchmarks":[{"benchmark_id":"opencompass-525-ocnli","domain":"推理","name":"OCNLI","released":"2020-10-12","url":"https://hub.opencompass.org.cn/dataset-detail/OCNLI"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OCNLI","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OCNLI","title":"OCNLI"},{"benchmarks":[{"benchmark_id":"opencompass-557-ocrbench","domain":"语言","name":"OCRBench","released":"2024-01-17","url":"https://hub.opencompass.org.cn/dataset-detail/OCRBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OCRBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OCRBench","title":"OCRBench"},{"benchmarks":[{"benchmark_id":"opencompass-1990-opt-bench","domain":"智能体","name":"OPT-BENCH","released":"2025-06-12","url":"https://hub.opencompass.org.cn/dataset-detail/OPT-BENCH"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OPT-BENCH","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OPT-BENCH","title":"OPT-BENCH"},{"benchmarks":[{"benchmark_id":"opencompass-2086-or-bench","domain":"理解","name":"OR-Bench","released":"2024-05-31","url":"https://hub.opencompass.org.cn/dataset-detail/OR-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OR-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OR-Bench","title":"OR-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2005-oss-bench","domain":"代码","name":"OSS-Bench","released":"2025-06-30","url":"https://hub.opencompass.org.cn/dataset-detail/OSS-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OSS-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OSS-Bench","title":"OSS-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1705-olymmath","domain":"强推理","name":"OlymMATH","released":"2025-03-28","url":"https://hub.opencompass.org.cn/dataset-detail/OlymMATH"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OlymMATH","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OlymMATH","title":"OlymMATH"},{"benchmarks":[{"benchmark_id":"opencompass-1070-olympiadbench","domain":"学科","name":"OlympiadBench","released":"2024-06-06","url":"https://hub.opencompass.org.cn/dataset-detail/OlympiadBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OlympiadBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OlympiadBench","title":"OlympiadBench"},{"benchmarks":[{"benchmark_id":"opencompass-1329-olympicarena","domain":"推理","name":"OlympicArena","released":"2024-06-18","url":"https://hub.opencompass.org.cn/dataset-detail/OlympicArena"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OlympicArena","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OlympicArena","title":"OlympicArena"},{"benchmarks":[{"benchmark_id":"opencompass-1244-omni-math","domain":"强推理","name":"Omni-MATH","released":"2024-10-10","url":"https://hub.opencompass.org.cn/dataset-detail/Omni-MATH"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Omni-MATH","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Omni-MATH","title":"Omni-MATH"},{"benchmarks":[{"benchmark_id":"opencompass-1553-omnialign-v","domain":"多模态","name":"OmniAlign-V","released":"2025-02-18","url":"https://hub.opencompass.org.cn/dataset-detail/OmniAlign-V"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniAlign-V","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniAlign-V","title":"OmniAlign-V"},{"benchmarks":[{"benchmark_id":"opencompass-1981-omnibench","domain":"智能体","name":"OmniBench","released":"2025-06-10","url":"https://hub.opencompass.org.cn/dataset-detail/OmniBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniBench","title":"OmniBench"},{"benchmarks":[{"benchmark_id":"opencompass-1942-omnidocbench","domain":"多模态","name":"OmniDocBench","released":"2024-12-02","url":"https://hub.opencompass.org.cn/dataset-detail/OmniDocBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniDocBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniDocBench","title":"OmniDocBench"},{"benchmarks":[{"benchmark_id":"opencompass-1801-omnigirl","domain":"多模态","name":"OmniGIRL","released":"2025-05-08","url":"https://hub.opencompass.org.cn/dataset-detail/OmniGIRL"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniGIRL","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniGIRL","title":"OmniGIRL"},{"benchmarks":[{"benchmark_id":"opencompass-2142-omnimmi","domain":"多模态","name":"OmniMMI","released":"2025-04-01","url":"https://hub.opencompass.org.cn/dataset-detail/OmniMMI"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniMMI","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OmniMMI","title":"OmniMMI"},{"benchmarks":[{"benchmark_id":"opencompass-631-openfindata","domain":"知识","name":"OpenFinData","released":"2023-12-29","url":"https://hub.opencompass.org.cn/dataset-detail/OpenFinData"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OpenFinData","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OpenFinData","title":"OpenFinData"},{"benchmarks":[{"benchmark_id":"opencompass-1754-openturingbench","domain":"其他","name":"OpenTuringBench","released":"2025-04-15","url":"https://hub.opencompass.org.cn/dataset-detail/OpenTuringBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OpenTuringBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OpenTuringBench","title":"OpenTuringBench"},{"benchmarks":[{"benchmark_id":"opencompass-1979-openunlearning","domain":"其他","name":"OpenUnlearning","released":"2025-06-14","url":"https://hub.opencompass.org.cn/dataset-detail/OpenUnlearning"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OpenUnlearning","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OpenUnlearning","title":"OpenUnlearning"},{"benchmarks":[{"benchmark_id":"opencompass-518-openbookqa","domain":"理解","name":"OpenbookQA","released":"2018-09-08","url":"https://hub.opencompass.org.cn/dataset-detail/OpenbookQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OpenbookQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/OpenbookQA","title":"OpenbookQA"},{"benchmarks":[{"benchmark_id":"opencompass-1917-orak","domain":"智能体","name":"Orak","released":"2025-06-04","url":"https://hub.opencompass.org.cn/dataset-detail/Orak"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Orak","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Orak","title":"Orak"},{"benchmarks":[{"benchmark_id":"opencompass-1283-p-mmeval","domain":"语言","name":"P-MMEval","released":"2024-11-14","url":"https://hub.opencompass.org.cn/dataset-detail/P-MMEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/P-MMEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/P-MMEval","title":"P-MMEval"},{"benchmarks":[{"benchmark_id":"opencompass-1076-pca-bench","domain":"其他","name":"PCA-Bench","released":"2024-02-21","url":"https://hub.opencompass.org.cn/dataset-detail/PCA-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PCA-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PCA-Bench","title":"PCA-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2350-picabench","domain":"多模态","name":"PICABench","released":"2025-10-27","url":"https://hub.opencompass.org.cn/dataset-detail/PICABench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PICABench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PICABench","title":"PICABench"},{"benchmarks":[{"benchmark_id":"opencompass-532-piqa","domain":"推理","name":"PIQA","released":"2019-11-26","url":"https://hub.opencompass.org.cn/dataset-detail/PIQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PIQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PIQA","title":"PIQA"},{"benchmarks":[{"benchmark_id":"opencompass-1362-pope","domain":"多模态","name":"POPE","released":"2023-05-17","url":"https://hub.opencompass.org.cn/dataset-detail/POPE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/POPE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/POPE","title":"POPE"},{"benchmarks":[{"benchmark_id":"opencompass-1681-prmbench-preview","domain":"学科","name":"PRMBench_Preview","released":"2025-01-06","url":"https://hub.opencompass.org.cn/dataset-detail/PRMBench_Preview"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PRMBench_Preview","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PRMBench_Preview","title":"PRMBench_Preview"},{"benchmarks":[{"benchmark_id":"opencompass-1730-paperbench","domain":"理解","name":"PaperBench","released":"2025-04-07","url":"https://hub.opencompass.org.cn/dataset-detail/PaperBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PaperBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PaperBench","title":"PaperBench"},{"benchmarks":[{"benchmark_id":"opencompass-1836-pashtoocr","domain":"多模态","name":"PashtoOCR","released":"2025-05-01","url":"https://hub.opencompass.org.cn/dataset-detail/PashtoOCR"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PashtoOCR","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PashtoOCR","title":"PashtoOCR"},{"benchmarks":[{"benchmark_id":"opencompass-1959-personalens","domain":"语言","name":"PersonaLens","released":"2025-06-11","url":"https://hub.opencompass.org.cn/dataset-detail/PersonaLens"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PersonaLens","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PersonaLens","title":"PersonaLens"},{"benchmarks":[{"benchmark_id":"opencompass-2087-perteval-scfm","domain":"其他","name":"PertEval-scFM","released":"2024-10-03","url":"https://hub.opencompass.org.cn/dataset-detail/PertEval-scFM"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PertEval-scFM","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PertEval-scFM","title":"PertEval-scFM"},{"benchmarks":[{"benchmark_id":"opencompass-2080-phygenbench","domain":"多模态","name":"PhyGenBench","released":"2024-10-07","url":"https://hub.opencompass.org.cn/dataset-detail/PhyGenBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PhyGenBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PhyGenBench","title":"PhyGenBench"},{"benchmarks":[{"benchmark_id":"opencompass-1576-physreason","domain":"学科","name":"PhysReason","released":"2025-02-17","url":"https://hub.opencompass.org.cn/dataset-detail/PhysReason"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PhysReason","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PhysReason","title":"PhysReason"},{"benchmarks":[{"benchmark_id":"opencompass-1251-planbench","domain":"强推理","name":"PlanBench","released":"2022-06-21","url":"https://hub.opencompass.org.cn/dataset-detail/PlanBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PlanBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PlanBench","title":"PlanBench"},{"benchmarks":[{"benchmark_id":"opencompass-1675-pokerbench","domain":"其他","name":"PokerBench","released":"2025-01-24","url":"https://hub.opencompass.org.cn/dataset-detail/PokerBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PokerBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PokerBench","title":"PokerBench"},{"benchmarks":[{"benchmark_id":"opencompass-1579-postersum","domain":"多模态","name":"PosterSum","released":"2025-02-24","url":"https://hub.opencompass.org.cn/dataset-detail/PosterSum"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PosterSum","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/PosterSum","title":"PosterSum"},{"benchmarks":[{"benchmark_id":"opencompass-1632-probench","domain":"多模态","name":"ProBench","released":"2025-03-10","url":"https://hub.opencompass.org.cn/dataset-detail/ProBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ProBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ProBench","title":"ProBench"},{"benchmarks":[{"benchmark_id":"opencompass-1633-projudge","domain":"多模态","name":"ProJudge","released":"2025-03-09","url":"https://hub.opencompass.org.cn/dataset-detail/ProJudge"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ProJudge","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ProJudge","title":"ProJudge"},{"benchmarks":[{"benchmark_id":"opencompass-1621-processbench","domain":"学科","name":"ProcessBench","released":"2024-12-10","url":"https://hub.opencompass.org.cn/dataset-detail/ProcessBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ProcessBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ProcessBench","title":"ProcessBench"},{"benchmarks":[{"benchmark_id":"opencompass-1120-proofnet","domain":"数学","name":"ProofNet","released":"2023-02-24","url":"https://hub.opencompass.org.cn/dataset-detail/ProofNet"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ProofNet","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ProofNet","title":"ProofNet"},{"benchmarks":[{"benchmark_id":"opencompass-1000-q-bench","domain":"理解","name":"Q-Bench","released":"2024-08-16","url":"https://hub.opencompass.org.cn/dataset-detail/Q-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Q-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Q-Bench","title":"Q-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1103-qasc","domain":"知识","name":"QASC","released":"2020-02-04","url":"https://hub.opencompass.org.cn/dataset-detail/QASC"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/QASC","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/QASC","title":"QASC"},{"benchmarks":[{"benchmark_id":"opencompass-516-race-high","domain":"理解","name":"RACE(High)","released":"2017-04-15","url":"https://hub.opencompass.org.cn/dataset-detail/RACE%28High%29"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RACE%28High%29","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RACE%28High%29","title":"RACE(High)"},{"benchmarks":[{"benchmark_id":"opencompass-517-race-middle","domain":"理解","name":"RACE(Middle)","released":"2017-04-15","url":"https://hub.opencompass.org.cn/dataset-detail/RACE%28Middle%29"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RACE%28Middle%29","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RACE%28Middle%29","title":"RACE(Middle)"},{"benchmarks":[{"benchmark_id":"opencompass-1918-rdb2g-bench","domain":"代码","name":"RDB2G-Bench","released":"2025-06-02","url":"https://hub.opencompass.org.cn/dataset-detail/RDB2G-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RDB2G-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RDB2G-Bench","title":"RDB2G-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1252-re-bench","domain":"强推理","name":"RE-Bench","released":"2024-11-22","url":"https://hub.opencompass.org.cn/dataset-detail/RE-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RE-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RE-Bench","title":"RE-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1755-real","domain":"智能体","name":"REAL","released":"2025-04-17","url":"https://hub.opencompass.org.cn/dataset-detail/REAL"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/REAL","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/REAL","title":"REAL"},{"benchmarks":[{"benchmark_id":"opencompass-2033-rexbench","domain":"代码","name":"RExBench","released":"2025-06-27","url":"https://hub.opencompass.org.cn/dataset-detail/RExBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RExBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RExBench","title":"RExBench"},{"benchmarks":[{"benchmark_id":"opencompass-1656-rfuav","domain":"其他","name":"RFUAV","released":"2025-03-18","url":"https://hub.opencompass.org.cn/dataset-detail/RFUAV"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RFUAV","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RFUAV","title":"RFUAV"},{"benchmarks":[{"benchmark_id":"opencompass-2048-risebench","domain":"推理","name":"RISEBench","released":"2025-04-03","url":"https://hub.opencompass.org.cn/dataset-detail/RISEBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RISEBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RISEBench","title":"RISEBench"},{"benchmarks":[{"benchmark_id":"opencompass-1524-rm-bench","domain":"语言","name":"RM-Bench","released":"2024-10-21","url":"https://hub.opencompass.org.cn/dataset-detail/RM-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RM-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RM-Bench","title":"RM-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1678-rsmmvp","domain":"多模态","name":"RSMMVP","released":"2025-03-24","url":"https://hub.opencompass.org.cn/dataset-detail/RSMMVP"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RSMMVP","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RSMMVP","title":"RSMMVP"},{"benchmarks":[{"benchmark_id":"opencompass-528-rte","domain":"推理","name":"RTE","released":null,"url":"https://hub.opencompass.org.cn/dataset-detail/RTE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RTE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RTE","title":"RTE"},{"benchmarks":[{"benchmark_id":"opencompass-1738-rulistening","domain":"理解","name":"RUListening","released":"2025-04-01","url":"https://hub.opencompass.org.cn/dataset-detail/RUListening"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RUListening","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RUListening","title":"RUListening"},{"benchmarks":[{"benchmark_id":"opencompass-1707-rxrx3-core","domain":"其他","name":"RXRX3-CORE","released":"2025-03-26","url":"https://hub.opencompass.org.cn/dataset-detail/RXRX3-CORE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RXRX3-CORE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RXRX3-CORE","title":"RXRX3-CORE"},{"benchmarks":[{"benchmark_id":"opencompass-530-record","domain":"推理","name":"ReCoRD","released":"2018-10-30","url":"https://hub.opencompass.org.cn/dataset-detail/ReCoRD"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ReCoRD","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ReCoRD","title":"ReCoRD"},{"benchmarks":[{"benchmark_id":"opencompass-1124-realtoxicityprompts","domain":"安全","name":"RealToxicityPrompts","released":"2020-11-16","url":"https://hub.opencompass.org.cn/dataset-detail/RealToxicityPrompts"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RealToxicityPrompts","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RealToxicityPrompts","title":"RealToxicityPrompts"},{"benchmarks":[{"benchmark_id":"opencompass-1361-realworldqa","domain":"多模态","name":"RealworldQA","released":"2024-04-12","url":"https://hub.opencompass.org.cn/dataset-detail/RealworldQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RealworldQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RealworldQA","title":"RealworldQA"},{"benchmarks":[{"benchmark_id":"opencompass-1333-redcode","domain":"安全","name":"RedCode","released":"2024-11-12","url":"https://hub.opencompass.org.cn/dataset-detail/RedCode"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RedCode","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RedCode","title":"RedCode"},{"benchmarks":[{"benchmark_id":"opencompass-1317-repliqa","domain":"推理","name":"RepLiQA","released":"2024-06-17","url":"https://hub.opencompass.org.cn/dataset-detail/RepLiQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RepLiQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RepLiQA","title":"RepLiQA"},{"benchmarks":[{"benchmark_id":"opencompass-1087-reveal","domain":"推理","name":"Reveal","released":"2024-05-21","url":"https://hub.opencompass.org.cn/dataset-detail/Reveal"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Reveal","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Reveal","title":"Reveal"},{"benchmarks":[{"benchmark_id":"opencompass-1915-rewardbench","domain":"指令跟随","name":"RewardBench","released":"2025-06-02","url":"https://hub.opencompass.org.cn/dataset-detail/RewardBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RewardBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RewardBench","title":"RewardBench"},{"benchmarks":[{"benchmark_id":"opencompass-2371-rigorousbench","domain":"智能体","name":"RigorousBench","released":"2025-10-02","url":"https://hub.opencompass.org.cn/dataset-detail/RigorousBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RigorousBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RigorousBench","title":"RigorousBench"},{"benchmarks":[{"benchmark_id":"opencompass-1091-rolellm","domain":"其他","name":"RoleLLM","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/RoleLLM"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RoleLLM","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/RoleLLM","title":"RoleLLM"},{"benchmarks":[{"benchmark_id":"opencompass-1003-s-eval","domain":"安全","name":"S-Eval","released":"2024-05-23","url":"https://hub.opencompass.org.cn/dataset-detail/S-Eval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/S-Eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/S-Eval","title":"S-Eval"},{"benchmarks":[{"benchmark_id":"opencompass-1772-s1-bench","domain":"强推理","name":"S1-Bench","released":"2025-04-15","url":"https://hub.opencompass.org.cn/dataset-detail/S1-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/S1-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/S1-Bench","title":"S1-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2083-saebench","domain":"理解","name":"SAEBench","released":"2025-03-12","url":"https://hub.opencompass.org.cn/dataset-detail/SAEBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SAEBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SAEBench","title":"SAEBench"},{"benchmarks":[{"benchmark_id":"opencompass-1077-salad-bench","domain":"安全","name":"SALAD-Bench","released":"2024-02-07","url":"https://hub.opencompass.org.cn/dataset-detail/SALAD-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SALAD-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SALAD-Bench","title":"SALAD-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1732-scam","domain":"多模态","name":"SCAM","released":"2025-04-07","url":"https://hub.opencompass.org.cn/dataset-detail/SCAM"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SCAM","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SCAM","title":"SCAM"},{"benchmarks":[{"benchmark_id":"opencompass-1986-sec-bench","domain":"安全","name":"SEC-bench","released":"2025-06-13","url":"https://hub.opencompass.org.cn/dataset-detail/SEC-bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SEC-bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SEC-bench","title":"SEC-bench"},{"benchmarks":[{"benchmark_id":"opencompass-1359-seed-bench","domain":"多模态","name":"SEED-Bench","released":"2023-07-30","url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SEED-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench","title":"SEED-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1363-seed-bench-2","domain":"多模态","name":"SEED-Bench-2","released":"2023-11-28","url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2","title":"SEED-Bench-2"},{"benchmarks":[{"benchmark_id":"opencompass-1395-seed-bench-2-plus","domain":"多模态","name":"SEED-Bench-2-Plus","released":"2024-04-25","url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2-Plus"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2-Plus","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2-Plus","title":"SEED-Bench-2-Plus"},{"benchmarks":[{"benchmark_id":"opencompass-1961-sfe","domain":"多模态","name":"SFE","released":"2025-06-12","url":"https://hub.opencompass.org.cn/dataset-detail/SFE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SFE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SFE","title":"SFE"},{"benchmarks":[{"benchmark_id":"opencompass-1330-sg-bench","domain":"安全","name":"SG-Bench","released":"2024-10-29","url":"https://hub.opencompass.org.cn/dataset-detail/SG-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SG-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SG-Bench","title":"SG-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2325-sgi-bench","domain":"多模态","name":"SGI-Bench","released":"2025-12-22","url":"https://hub.opencompass.org.cn/dataset-detail/SGI-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SGI-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SGI-Bench","title":"SGI-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-533-siqa","domain":"推理","name":"SIQA","released":"2019-09-09","url":"https://hub.opencompass.org.cn/dataset-detail/SIQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SIQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SIQA","title":"SIQA"},{"benchmarks":[{"benchmark_id":"opencompass-2029-smmile","domain":"多模态","name":"SMMILE","released":"2025-06-26","url":"https://hub.opencompass.org.cn/dataset-detail/SMMILE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SMMILE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SMMILE","title":"SMMILE"},{"benchmarks":[{"benchmark_id":"opencompass-1845-stark-10k","domain":"多模态","name":"STARK_10k","released":"2025-05-16","url":"https://hub.opencompass.org.cn/dataset-detail/STARK_10k"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/STARK_10k","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/STARK_10k","title":"STARK_10k"},{"benchmarks":[{"benchmark_id":"opencompass-1113-svamp","domain":"数学","name":"SVAMP","released":"2021-04-15","url":"https://hub.opencompass.org.cn/dataset-detail/SVAMP"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SVAMP","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SVAMP","title":"SVAMP"},{"benchmarks":[{"benchmark_id":"opencompass-1989-swe-factory","domain":"代码","name":"SWE-Factory","released":"2025-06-12","url":"https://hub.opencompass.org.cn/dataset-detail/SWE-Factory"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SWE-Factory","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SWE-Factory","title":"SWE-Factory"},{"benchmarks":[{"benchmark_id":"opencompass-1984-swe-bench-live","domain":"代码","name":"SWE-bench-Live","released":"2025-06-01","url":"https://hub.opencompass.org.cn/dataset-detail/SWE-bench-Live"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SWE-bench-Live","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SWE-bench-Live","title":"SWE-bench-Live"},{"benchmarks":[{"benchmark_id":"opencompass-1073-safetybench","domain":"安全","name":"SafetyBench","released":"2024-06-24","url":"https://hub.opencompass.org.cn/dataset-detail/SafetyBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SafetyBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SafetyBench","title":"SafetyBench"},{"benchmarks":[{"benchmark_id":"opencompass-1323-scifibench","domain":"多模态","name":"SciFIBench","released":"2024-05-14","url":"https://hub.opencompass.org.cn/dataset-detail/SciFIBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SciFIBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SciFIBench","title":"SciFIBench"},{"benchmarks":[{"benchmark_id":"opencompass-1100-scienceqa","domain":"知识","name":"ScienceQA","released":"2022-10-17","url":"https://hub.opencompass.org.cn/dataset-detail/ScienceQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ScienceQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ScienceQA","title":"ScienceQA"},{"benchmarks":[{"benchmark_id":"opencompass-948-secbench","domain":"安全","name":"SecBench","released":"2024-01-19","url":"https://hub.opencompass.org.cn/dataset-detail/SecBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SecBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SecBench","title":"SecBench"},{"benchmarks":[{"benchmark_id":"opencompass-2348-shell","domain":"安全","name":"Shell","released":"2025-12-25","url":"https://hub.opencompass.org.cn/dataset-detail/Shell"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Shell","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Shell","title":"Shell"},{"benchmarks":[{"benchmark_id":"opencompass-1321-shoppingmmlu","domain":"其他","name":"ShoppingMMLU","released":"2024-10-28","url":"https://hub.opencompass.org.cn/dataset-detail/ShoppingMMLU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ShoppingMMLU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ShoppingMMLU","title":"ShoppingMMLU"},{"benchmarks":[{"benchmark_id":"opencompass-2217-smartbench","domain":"多模态","name":"SmartBench","released":"2025-09-26","url":"https://hub.opencompass.org.cn/dataset-detail/SmartBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SmartBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SmartBench","title":"SmartBench"},{"benchmarks":[{"benchmark_id":"opencompass-2092-spatial457","domain":"多模态","name":"Spatial457","released":"2025-04-09","url":"https://hub.opencompass.org.cn/dataset-detail/Spatial457"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Spatial457","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Spatial457","title":"Spatial457"},{"benchmarks":[{"benchmark_id":"opencompass-1133-spider","domain":"智能体","name":"Spider","released":"2019-02-02","url":"https://hub.opencompass.org.cn/dataset-detail/Spider"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Spider","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Spider","title":"Spider"},{"benchmarks":[{"benchmark_id":"opencompass-1270-spider2-v","domain":"智能体","name":"Spider2-V","released":"2024-07-15","url":"https://hub.opencompass.org.cn/dataset-detail/Spider2-V"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Spider2-V","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Spider2-V","title":"Spider2-V"},{"benchmarks":[{"benchmark_id":"opencompass-1152-sportqa","domain":"理解","name":"SportQA","released":"2024-06-16","url":"https://hub.opencompass.org.cn/dataset-detail/SportQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SportQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SportQA","title":"SportQA"},{"benchmarks":[{"benchmark_id":"opencompass-1267-spreadsheetbench","domain":"其他","name":"SpreadsheetBench","released":"2024-06-21","url":"https://hub.opencompass.org.cn/dataset-detail/SpreadsheetBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SpreadsheetBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SpreadsheetBench","title":"SpreadsheetBench"},{"benchmarks":[{"benchmark_id":"opencompass-1084-stabletoolbench","domain":"其他","name":"StableToolBench","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/StableToolBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StableToolBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StableToolBench","title":"StableToolBench"},{"benchmarks":[{"benchmark_id":"opencompass-1107-strategyqa","domain":"知识","name":"StrategyQA","released":"2021-01-06","url":"https://hub.opencompass.org.cn/dataset-detail/StrategyQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StrategyQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StrategyQA","title":"StrategyQA"},{"benchmarks":[{"benchmark_id":"opencompass-1542-structflowbench","domain":"指令跟随","name":"StructFlowBench","released":"2025-02-20","url":"https://hub.opencompass.org.cn/dataset-detail/StructFlowBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StructFlowBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StructFlowBench","title":"StructFlowBench"},{"benchmarks":[{"benchmark_id":"opencompass-2082-structtokenbench","domain":"其他","name":"StructTokenBench","released":"2025-02-28","url":"https://hub.opencompass.org.cn/dataset-detail/StructTokenBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StructTokenBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StructTokenBench","title":"StructTokenBench"},{"benchmarks":[{"benchmark_id":"opencompass-1082-studenteval","domain":"代码","name":"StudentEval","released":"2024-08-11","url":"https://hub.opencompass.org.cn/dataset-detail/StudentEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StudentEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StudentEval","title":"StudentEval"},{"benchmarks":[{"benchmark_id":"opencompass-1744-stylerec","domain":"理解","name":"StyleRec","released":"2025-04-15","url":"https://hub.opencompass.org.cn/dataset-detail/StyleRec"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StyleRec","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/StyleRec","title":"StyleRec"},{"benchmarks":[{"benchmark_id":"opencompass-1538-supergpqa","domain":"学科","name":"SuperGPQA","released":"2025-02-20","url":"https://hub.opencompass.org.cn/dataset-detail/SuperGPQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SuperGPQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SuperGPQA","title":"SuperGPQA"},{"benchmarks":[{"benchmark_id":"opencompass-1608-swiltra-bench","domain":"语言","name":"SwiLTra-Bench","released":"2025-03-03","url":"https://hub.opencompass.org.cn/dataset-detail/SwiLTra-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SwiLTra-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/SwiLTra-Bench","title":"SwiLTra-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-540-t-eval","domain":"智能体","name":"T-Eval","released":"2024-01-15","url":"https://hub.opencompass.org.cn/dataset-detail/T-Eval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/T-Eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/T-Eval","title":"T-Eval"},{"benchmarks":[{"benchmark_id":"opencompass-1846-tglg","domain":"多模态","name":"TGLG","released":"2025-05-09","url":"https://hub.opencompass.org.cn/dataset-detail/TGLG"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TGLG","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TGLG","title":"TGLG"},{"benchmarks":[{"benchmark_id":"opencompass-2050-thunder","domain":"理解","name":"THUNDER","released":"2025-07-10","url":"https://hub.opencompass.org.cn/dataset-detail/THUNDER"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/THUNDER","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/THUNDER","title":"THUNDER"},{"benchmarks":[{"benchmark_id":"opencompass-1132-tabfact","domain":"智能体","name":"TabFact","released":"2020-06-14","url":"https://hub.opencompass.org.cn/dataset-detail/TabFact"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TabFact","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TabFact","title":"TabFact"},{"benchmarks":[{"benchmark_id":"opencompass-2025-tableeval","domain":"推理","name":"TableEval","released":"2025-06-05","url":"https://hub.opencompass.org.cn/dataset-detail/TableEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TableEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TableEval","title":"TableEval"},{"benchmarks":[{"benchmark_id":"opencompass-1164-taskbench","domain":"智能体","name":"TaskBench","released":"2023-11-30","url":"https://hub.opencompass.org.cn/dataset-detail/TaskBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TaskBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TaskBench","title":"TaskBench"},{"benchmarks":[{"benchmark_id":"opencompass-1574-text2world","domain":"其他","name":"Text2World","released":"2025-02-18","url":"https://hub.opencompass.org.cn/dataset-detail/Text2World"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Text2World","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Text2World","title":"Text2World"},{"benchmarks":[{"benchmark_id":"opencompass-1734-thai-local-benchmark","domain":"语言","name":"Thai_local_benchmark","released":"2025-04-08","url":"https://hub.opencompass.org.cn/dataset-detail/Thai_local_benchmark"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Thai_local_benchmark","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Thai_local_benchmark","title":"Thai_local_benchmark"},{"benchmarks":[{"benchmark_id":"opencompass-1134-theoremqa","domain":"推理","name":"TheoremQA","released":"2023-12-06","url":"https://hub.opencompass.org.cn/dataset-detail/TheoremQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TheoremQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TheoremQA","title":"TheoremQA"},{"benchmarks":[{"benchmark_id":"opencompass-2391-threat-signature-eval","domain":"视觉定位","name":"Threat-Signature_Eval","released":"2025-11-19","url":"https://hub.opencompass.org.cn/dataset-detail/Threat-Signature_Eval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Threat-Signature_Eval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Threat-Signature_Eval","title":"Threat-Signature_Eval"},{"benchmarks":[{"benchmark_id":"opencompass-1668-timetravel","domain":"多模态","name":"TimeTravel","released":"2025-02-20","url":"https://hub.opencompass.org.cn/dataset-detail/TimeTravel"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TimeTravel","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TimeTravel","title":"TimeTravel"},{"benchmarks":[{"benchmark_id":"opencompass-1712-toolhop","domain":"智能体","name":"ToolHop","released":"2025-01-05","url":"https://hub.opencompass.org.cn/dataset-detail/ToolHop"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ToolHop","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ToolHop","title":"ToolHop"},{"benchmarks":[{"benchmark_id":"opencompass-1607-toolret","domain":"其他","name":"ToolRet","released":"2025-03-03","url":"https://hub.opencompass.org.cn/dataset-detail/ToolRet"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ToolRet","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ToolRet","title":"ToolRet"},{"benchmarks":[{"benchmark_id":"opencompass-1855-transbench","domain":"语言","name":"TransBench","released":"2025-05-20","url":"https://hub.opencompass.org.cn/dataset-detail/TransBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TransBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TransBench","title":"TransBench"},{"benchmarks":[{"benchmark_id":"opencompass-2039-translaw","domain":"语言","name":"TransLaw","released":"2025-07-01","url":"https://hub.opencompass.org.cn/dataset-detail/TransLaw"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TransLaw","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TransLaw","title":"TransLaw"},{"benchmarks":[{"benchmark_id":"opencompass-512-triviaqa","domain":"知识","name":"TriviaQA","released":"2017-05-09","url":"https://hub.opencompass.org.cn/dataset-detail/TriviaQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TriviaQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TriviaQA","title":"TriviaQA"},{"benchmarks":[{"benchmark_id":"opencompass-1096-truthfulqa","domain":"知识","name":"TruthfulQA","released":"2022-05-08","url":"https://hub.opencompass.org.cn/dataset-detail/TruthfulQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TruthfulQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TruthfulQA","title":"TruthfulQA"},{"benchmarks":[{"benchmark_id":"opencompass-508-tydiqa","domain":"知识","name":"TyDiQA","released":"2020-03-10","url":"https://hub.opencompass.org.cn/dataset-detail/TyDiQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TyDiQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/TyDiQA","title":"TyDiQA"},{"benchmarks":[{"benchmark_id":"opencompass-1742-u-niah","domain":"长文本","name":"U-NIAH","released":"2025-03-01","url":"https://hub.opencompass.org.cn/dataset-detail/U-NIAH"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/U-NIAH","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/U-NIAH","title":"U-NIAH"},{"benchmarks":[{"benchmark_id":"opencompass-1088-uhgeval","domain":"知识","name":"UHGEval","released":"2024-05-24","url":"https://hub.opencompass.org.cn/dataset-detail/UHGEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/UHGEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/UHGEval","title":"UHGEval"},{"benchmarks":[{"benchmark_id":"opencompass-2014-utboost","domain":"代码","name":"UTBoost","released":"2025-06-10","url":"https://hub.opencompass.org.cn/dataset-detail/UTBoost"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/UTBoost","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/UTBoost","title":"UTBoost"},{"benchmarks":[{"benchmark_id":"opencompass-1332-unibench","domain":"多模态","name":"UniBench","released":"2024-08-09","url":"https://hub.opencompass.org.cn/dataset-detail/UniBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/UniBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/UniBench","title":"UniBench"},{"benchmarks":[{"benchmark_id":"opencompass-1636-urbanvideo-bench","domain":"多模态","name":"UrbanVideo-Bench","released":"2025-03-08","url":"https://hub.opencompass.org.cn/dataset-detail/UrbanVideo-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/UrbanVideo-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/UrbanVideo-Bench","title":"UrbanVideo-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1654-v-star","domain":"多模态","name":"V-STaR","released":"2025-03-14","url":"https://hub.opencompass.org.cn/dataset-detail/V-STaR"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/V-STaR","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/V-STaR","title":"V-STaR"},{"benchmarks":[{"benchmark_id":"opencompass-1375-vbench","domain":"多模态","name":"VBench","released":"2023-11-29","url":"https://hub.opencompass.org.cn/dataset-detail/VBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VBench","title":"VBench"},{"benchmarks":[{"benchmark_id":"opencompass-1875-vibe","domain":"其他","name":"VIBE","released":"2025-05-29","url":"https://hub.opencompass.org.cn/dataset-detail/VIBE"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VIBE","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VIBE","title":"VIBE"},{"benchmarks":[{"benchmark_id":"opencompass-2063-visco","domain":"多模态","name":"VISCO","released":"2024-12-03","url":"https://hub.opencompass.org.cn/dataset-detail/VISCO"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VISCO","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VISCO","title":"VISCO"},{"benchmarks":[{"benchmark_id":"opencompass-2388-vknowu","domain":"多模态","name":"VKnowU","released":"2025-11-25","url":"https://hub.opencompass.org.cn/dataset-detail/VKnowU"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VKnowU","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VKnowU","title":"VKnowU"},{"benchmarks":[{"benchmark_id":"opencompass-1541-vlm2-bench","domain":"多模态","name":"VLM2-Bench","released":"2025-02-17","url":"https://hub.opencompass.org.cn/dataset-detail/VLM2-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VLM2-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VLM2-Bench","title":"VLM2-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-2383-vrbench","domain":"多模态","name":"VRBench","released":"2025-06-12","url":"https://hub.opencompass.org.cn/dataset-detail/VRBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VRBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VRBench","title":"VRBench"},{"benchmarks":[{"benchmark_id":"opencompass-1717-vilbench","domain":"多模态","name":"ViLBench","released":"2025-03-26","url":"https://hub.opencompass.org.cn/dataset-detail/ViLBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ViLBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ViLBench","title":"ViLBench"},{"benchmarks":[{"benchmark_id":"opencompass-1993-vistorybench","domain":"多模态","name":"ViStoryBench","released":"2025-06-25","url":"https://hub.opencompass.org.cn/dataset-detail/ViStoryBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ViStoryBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ViStoryBench","title":"ViStoryBench"},{"benchmarks":[{"benchmark_id":"opencompass-1358-video-mme","domain":"多模态","name":"Video-MME","released":"2024-03-31","url":"https://hub.opencompass.org.cn/dataset-detail/Video-MME"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Video-MME","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Video-MME","title":"Video-MME"},{"benchmarks":[{"benchmark_id":"opencompass-1277-videogui","domain":"创作","name":"VideoGUI","released":"2024-06-14","url":"https://hub.opencompass.org.cn/dataset-detail/VideoGUI"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VideoGUI","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VideoGUI","title":"VideoGUI"},{"benchmarks":[{"benchmark_id":"opencompass-1926-videomathqa","domain":"多模态","name":"VideoMathQA","released":"2025-06-05","url":"https://hub.opencompass.org.cn/dataset-detail/VideoMathQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VideoMathQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VideoMathQA","title":"VideoMathQA"},{"benchmarks":[{"benchmark_id":"opencompass-1900-videoreasonbench","domain":"强推理","name":"VideoReasonBench","released":"2025-06-05","url":"https://hub.opencompass.org.cn/dataset-detail/VideoReasonBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VideoReasonBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VideoReasonBench","title":"VideoReasonBench"},{"benchmarks":[{"benchmark_id":"opencompass-1748-visualpuzzles","domain":"强推理","name":"VisualPuzzles","released":"2025-04-16","url":"https://hub.opencompass.org.cn/dataset-detail/VisualPuzzles"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VisualPuzzles","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VisualPuzzles","title":"VisualPuzzles"},{"benchmarks":[{"benchmark_id":"opencompass-1634-visualsimpleqa","domain":"多模态","name":"VisualSimpleQA","released":"2025-03-09","url":"https://hub.opencompass.org.cn/dataset-detail/VisualSimpleQA"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VisualSimpleQA","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/VisualSimpleQA","title":"VisualSimpleQA"},{"benchmarks":[{"benchmark_id":"opencompass-507-wsc","domain":"语言","name":"WSC","released":null,"url":"https://hub.opencompass.org.cn/dataset-detail/WSC"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WSC","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WSC","title":"WSC"},{"benchmarks":[{"benchmark_id":"opencompass-1982-webui-bench","domain":"代码","name":"WebUI-Bench","released":"2025-06-09","url":"https://hub.opencompass.org.cn/dataset-detail/WebUI-Bench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WebUI-Bench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WebUI-Bench","title":"WebUI-Bench"},{"benchmarks":[{"benchmark_id":"opencompass-1279-whodunitbench","domain":"智能体","name":"WhodunitBench","released":"2024-09-26","url":"https://hub.opencompass.org.cn/dataset-detail/WhodunitBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WhodunitBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WhodunitBench","title":"WhodunitBench"},{"benchmarks":[{"benchmark_id":"opencompass-504-wic","domain":"语言","name":"WiC","released":"2018-08-28","url":"https://hub.opencompass.org.cn/dataset-detail/WiC"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WiC","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WiC","title":"WiC"},{"benchmarks":[{"benchmark_id":"opencompass-1318-wikicontradict","domain":"推理","name":"WikiContradict","released":"2024-06-19","url":"https://hub.opencompass.org.cn/dataset-detail/WikiContradict"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WikiContradict","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WikiContradict","title":"WikiContradict"},{"benchmarks":[{"benchmark_id":"opencompass-1129-wikisql","domain":"其他","name":"WikiSQL","released":"2017-11-09","url":"https://hub.opencompass.org.cn/dataset-detail/WikiSQL"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WikiSQL","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WikiSQL","title":"WikiSQL"},{"benchmarks":[{"benchmark_id":"opencompass-1555-wildbench","domain":"理解","name":"WildBench","released":"2024-06-07","url":"https://hub.opencompass.org.cn/dataset-detail/WildBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WildBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WildBench","title":"WildBench"},{"benchmarks":[{"benchmark_id":"opencompass-2445-wildclawbench","domain":"智能体","name":"WildClawBench","released":"2026-04-07","url":"https://hub.opencompass.org.cn/dataset-detail/WildClawBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WildClawBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WildClawBench","title":"WildClawBench"},{"benchmarks":[{"benchmark_id":"opencompass-1109-winogrande","domain":"推理","name":"WinoGrande","released":"2019-11-21","url":"https://hub.opencompass.org.cn/dataset-detail/WinoGrande"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WinoGrande","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WinoGrande","title":"WinoGrande"},{"benchmarks":[{"benchmark_id":"opencompass-1938-worldgenbench","domain":"多模态","name":"WorldGenBench","released":"2025-06-16","url":"https://hub.opencompass.org.cn/dataset-detail/WorldGenBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WorldGenBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WorldGenBench","title":"WorldGenBench"},{"benchmarks":[{"benchmark_id":"opencompass-1735-worldscore","domain":"创作","name":"WorldScore","released":"2025-04-01","url":"https://hub.opencompass.org.cn/dataset-detail/WorldScore"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WorldScore","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WorldScore","title":"WorldScore"},{"benchmarks":[{"benchmark_id":"opencompass-1701-writingbench","domain":"长文本","name":"WritingBench","released":"2025-03-07","url":"https://hub.opencompass.org.cn/dataset-detail/WritingBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WritingBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/WritingBench","title":"WritingBench"},{"benchmarks":[{"benchmark_id":"opencompass-521-xsum","domain":"理解","name":"XSum","released":"2018-08-27","url":"https://hub.opencompass.org.cn/dataset-detail/XSum"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/XSum","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/XSum","title":"XSum"},{"benchmarks":[{"benchmark_id":"opencompass-1017-yue-benchmark","domain":"语言","name":"Yue_Benchmark","released":"2024-08-31","url":"https://hub.opencompass.org.cn/dataset-detail/Yue_Benchmark"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Yue_Benchmark","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/Yue_Benchmark","title":"Yue_Benchmark"},{"benchmarks":[{"benchmark_id":"opencompass-1532-zerobench","domain":"多模态","name":"ZeroBench","released":"2025-02-13","url":"https://hub.opencompass.org.cn/dataset-detail/ZeroBench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ZeroBench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ZeroBench","title":"ZeroBench"},{"benchmarks":[{"benchmark_id":"opencompass-2127-clembench","domain":"创作","name":"clembench","released":"2025-07-11","url":"https://hub.opencompass.org.cn/dataset-detail/clembench"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/clembench","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/clembench","title":"clembench"},{"benchmarks":[{"benchmark_id":"opencompass-2128-lm-evaluation-harness","domain":"知识","name":"lm-evaluation-harness","released":"2025-07-11","url":"https://hub.opencompass.org.cn/dataset-detail/lm-evaluation-harness"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/lm-evaluation-harness","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/lm-evaluation-harness","title":"lm-evaluation-harness"},{"benchmarks":[{"benchmark_id":"opencompass-1518-minictx","domain":"强推理","name":"miniCTX","released":"2024-08-03","url":"https://hub.opencompass.org.cn/dataset-detail/miniCTX"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/miniCTX","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/miniCTX","title":"miniCTX"},{"benchmarks":[{"benchmark_id":"opencompass-1839-tiny-qa-benchmark-pp","domain":"推理","name":"tiny_qa_benchmark_pp","released":"2025-05-17","url":"https://hub.opencompass.org.cn/dataset-detail/tiny_qa_benchmark_pp"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/tiny_qa_benchmark_pp","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/tiny_qa_benchmark_pp","title":"tiny_qa_benchmark_pp"},{"benchmarks":[{"benchmark_id":"opencompass-1625-ubuntu-osworld","domain":"多模态","name":"ubuntu_osworld","released":"2024-04-11","url":"https://hub.opencompass.org.cn/dataset-detail/ubuntu_osworld"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ubuntu_osworld","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/ubuntu_osworld","title":"ubuntu_osworld"},{"benchmarks":[{"benchmark_id":"opencompass-1072-xcodeeval","domain":"代码","name":"xCodeEval","released":"2023-11-06","url":"https://hub.opencompass.org.cn/dataset-detail/xCodeEval"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/xCodeEval","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/xCodeEval","title":"xCodeEval"},{"benchmarks":[{"benchmark_id":"opencompass-1784-xverify","domain":"推理","name":"xVerify","released":"2025-04-14","url":"https://hub.opencompass.org.cn/dataset-detail/xVerify"}],"document_type":"registry_page","id":"opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/xVerify","source":"opencompass_hub","source_url":"https://hub.opencompass.org.cn/dataset-detail/xVerify","title":"xVerify"}],"domains":{"3d":1,"AI Scientist":9,"Academic document parsing":1,"Advertising & Pricing":2,"Agent safety":1,"Agents":264,"Agents & Tool Use":65,"Architecture-specific programming":1,"Autonomous Driving":15,"Bilingual reasoning":1,"Biology & Drug Discovery":147,"CAD":12,"Chip Design & EDA":20,"Code":28,"Code & Software":89,"Code generation":2,"Contact-rich control":1,"Cybersecurity":50,"Distribution shift":1,"Embodied manipulation":1,"Energy & Grid":1,"Factuality":78,"Finance":74,"General AI":1281,"Geometric reasoning":12,"Information retrieval":2,"Language & Knowledge":99,"Logistics & Operations":4,"Manufacturing & Process Control":1,"Materials & Chemistry":4,"Mathematics & Formal Science":4,"Mobile & Personal Computing":1,"Multimodal":32,"Planning":2,"Reasoning":15,"Robot manipulation":49,"Robotics":24,"Robotics & Embodied AI":44,"Robustness":1,"Safety & Trustworthiness":1,"Software & AI Compute":8,"agent":11,"agentic":8,"agents":57,"ai_research":2,"audio":2,"biology":1,"code":15,"coding":10,"coding_agent":15,"computer_use":1,"creativity":1,"factuality":2,"finance":3,"frontend_development":1,"general":16,"health":3,"healthcare":6,"human_preference":2,"image-generation":2,"image_to_text":4,"instruction-following":1,"instruction_following":6,"intelligence-index":11,"knowledge":8,"language":11,"legal":22,"long_context":60,"math":84,"medical":1,"memory":3,"multilingual":1,"multimodal":173,"other":4,"physics":7,"productivity":6,"professional":4,"psychology":1,"question_answering":1,"reasoning":225,"research":1,"robotics":1,"safety":18,"science":11,"scientific_agent":1,"security":7,"spatial_reasoning":8,"speech_to_text":4,"structured_output":5,"summarization":2,"tool_use":7,"video":1,"vision":6,"代码":29,"其他":45,"创作":7,"医学":1,"多模态":132,"学科":19,"安全":21,"强推理":20,"指令跟随":5,"推理":49,"数学":11,"智能体":30,"理解":32,"知识":28,"科学":1,"视觉定位":1,"语言":19,"长文本":11},"entries":[{"aliases":["GPQA","GPQA-Diamond","GPQA Diamond"],"benchmark_id":"gpqa_diamond","caveat":"198 questions in the Diamond split, so run-to-run variance is wide and a single reported number hides it. Approaching saturation at the 2026 frontier, where reported scores cluster in the high 80s and 90s.","document_count":30,"document_ids":["model_reports:anthropic_claude_3_7_sonnet","model_reports:anthropic_claude_4_system_card","model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:deepseek_v4_1_flash_technical_report","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_2_5_report","model_reports:google_gemini_3_1_pro_model_card","model_reports:google_gemma_4_model_card","model_reports:meta_llama_4","model_reports:microsoft_phi_4_mini_reasoning_model_card","model_reports:mistral_large_3","model_reports:mistral_medium_3","model_reports:moonshot_kimi_k3_model_card","model_reports:openai_gpt_4_1","model_reports:openai_gpt_5_6_release","model_reports:openai_gpt_5_system_card","model_reports:openai_o3_o4_mini_system_card","model_reports:qwen3_5_model_card","model_reports:qwen3_8_model_card","model_reports:qwen3_technical_report","model_reports:tencent_hy4_preview","model_reports:xai_grok_4_5","model_reports:xai_grok_4_model_card","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_2_model_card","model_reports:zai_glm_5_model_card","model_reports:zai_glm_5_paper"],"document_share":0.008169934640522876,"domain":"science","name":"GPQA Diamond","organization_count":13,"organizations":["Anthropic","ByteDance","DeepSeek","Google","Meta","Microsoft","Mistral","Moonshot AI","OpenAI","Qwen","Tencent","Z.ai","xAI"],"rank":1,"released":"2023-11-20","source":"model_reports","url":"https://arxiv.org/abs/2311.12022"},{"aliases":["HLE","Humanity's Last Exam","HLE-Full","HLE-Verified"],"benchmark_id":"hle","caveat":"HLE evaluates answers to academic questions across many subjects. Its finalized original public release contains 2,500 questions; HLE-Rolling and HLE-Diamond are separate releases. Check the question set, tool access, answer-grading procedure and run settings attached to each result.","document_count":22,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5","model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_v4_1_flash_technical_report","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_pro_0813_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_2_5_report","model_reports:google_gemini_3_1_pro_model_card","model_reports:google_gemma_4_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:openai_gpt_5_6_release","model_reports:openai_gpt_5_system_card","model_reports:qwen3_5_model_card","model_reports:qwen3_8_model_card","model_reports:tencent_hy4_preview","model_reports:xai_grok_4_5","model_reports:xai_grok_4_model_card","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_2_model_card","model_reports:zai_glm_5_3_flash_model_card","model_reports:zai_glm_5_model_card","model_reports:zai_glm_5_paper"],"document_share":0.005991285403050109,"domain":"reasoning","name":"Humanity's Last Exam","organization_count":10,"organizations":["Anthropic","ByteDance","DeepSeek","Google","Moonshot AI","OpenAI","Qwen","Tencent","Z.ai","xAI"],"rank":2,"released":"2025-01-23","source":"model_reports","url":"https://lastexam.ai/"},{"aliases":["Terminal-Bench","Terminal Bench","TerminalBench","Terminal-Bench 2.0","Terminal-Bench 2.1","Terminal Bench 2","TerminalBench Hard","Terminal-Bench 3.0","Terminal Bench 3","Frontier-Bench","FrontierBench","Frontier-Bench v0.1","FrontierSWE"],"benchmark_id":"terminal_bench","caveat":"Environment image, timeout and permitted commands change results independently of the model. Version and harness both matter: 2.0 and 2.1 are different instruments, and cards report Terminus, Claude Code and Codex harness numbers for the same version. Frontier-Bench is this series' next version rather than a separate benchmark: frontierbench.ai now bills it as \"Terminal-Bench 3.0 (formerly Frontier-Bench)\", built by the makers of Harbor and Terminal-Bench, so its mentions are counted here. Merging it did not raise this count: all three cards that named Frontier-Bench (Claude Opus 5, Claude Fable 5 and Mythos 5, Kimi K3) also report Terminal-Bench in the same document, and the counting unit is the document, so each still adds one. Being a new version, its task set is not comparable to a 2.x number. Not to be confused with Cognition's FrontierCode, a separate instrument recorded separately despite the shared prefix.","document_count":21,"document_ids":["model_reports:anthropic_claude_4_system_card","model_reports:anthropic_claude_fable_5_mythos_5","model_reports:anthropic_claude_opus_5_system_card","model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_v4_1_flash_technical_report","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_pro_0813_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_3_1_pro_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:openai_gpt_5_6_release","model_reports:openai_gpt_5_6_system_card","model_reports:qwen3_5_model_card","model_reports:qwen3_8_model_card","model_reports:tencent_hy4_preview","model_reports:xai_grok_4_5","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_2_model_card","model_reports:zai_glm_5_3_flash_model_card","model_reports:zai_glm_5_model_card","model_reports:zai_glm_5_paper"],"document_share":0.005718954248366013,"domain":"agent","name":"Terminal-Bench","organization_count":10,"organizations":["Anthropic","ByteDance","DeepSeek","Google","Moonshot AI","OpenAI","Qwen","Tencent","Z.ai","xAI"],"rank":3,"released":"2025-05-19","source":"model_reports","url":"https://www.tbench.ai/"},{"aliases":["AIME","AIME 2024","AIME 2025","AIME 2026"],"benchmark_id":"aime","caveat":"pass@k, majority vote and Python tool access each shift this by double digits.","document_count":19,"document_ids":["model_reports:ai2_olmo_3","model_reports:anthropic_claude_3_7_sonnet","model_reports:anthropic_claude_4_system_card","model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:google_gemini_2_5_report","model_reports:google_gemma_4_model_card","model_reports:microsoft_phi_4_mini_reasoning_model_card","model_reports:openai_gpt_4_1","model_reports:openai_gpt_5_system_card","model_reports:openai_o3_o4_mini_system_card","model_reports:qwen3_5_model_card","model_reports:qwen3_technical_report","model_reports:xai_grok_4_model_card","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_2_model_card","model_reports:zai_glm_5_model_card","model_reports:zai_glm_5_paper"],"document_share":0.0051742919389978215,"domain":"math","name":"AIME","organization_count":10,"organizations":["Ai2","Anthropic","ByteDance","DeepSeek","Google","Microsoft","OpenAI","Qwen","Z.ai","xAI"],"rank":4,"released":"2024-02-01","source":"model_reports","url":"https://maa.org/maa-invitational-competitions/"},{"aliases":["SWE-bench Verified","SWE-bench-Verified","SWE Bench Verified","SWE Verified"],"benchmark_id":"swe_bench_verified","caveat":"Scaffold, tool permissions and time limit are part of the result; vendors run their own harnesses.","document_count":19,"document_ids":["model_reports:anthropic_claude_3_7_sonnet","model_reports:anthropic_claude_4_system_card","model_reports:anthropic_claude_opus_5_system_card","model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_2_5_report","model_reports:google_gemini_3_1_pro_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:openai_gpt_4_1","model_reports:openai_gpt_5_6_release","model_reports:openai_gpt_5_system_card","model_reports:openai_o3_o4_mini_system_card","model_reports:qwen3_5_model_card","model_reports:xai_grok_4_model_card","model_reports:zai_glm_5_model_card","model_reports:zai_glm_5_paper"],"document_share":0.0051742919389978215,"domain":"coding_agent","name":"SWE-bench Verified","organization_count":9,"organizations":["Anthropic","ByteDance","DeepSeek","Google","Moonshot AI","OpenAI","Qwen","Z.ai","xAI"],"rank":5,"released":"2024-08-13","source":"model_reports","url":"https://openai.com/index/introducing-swe-bench-verified/"},{"aliases":["LiveCodeBench","LCB"],"benchmark_id":"livecodebench","caveat":"Time-sliced problem set. A score without its problem window is not comparable to any other score.","document_count":18,"document_ids":["model_reports:ai2_olmo_3","model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_2_5_report","model_reports:google_gemma_4_model_card","model_reports:meta_llama_4","model_reports:mistral_large_3","model_reports:mistral_medium_3","model_reports:openai_gpt_5_system_card","model_reports:openai_o3_o4_mini_system_card","model_reports:qwen2_5_coder_report","model_reports:qwen3_5_model_card","model_reports:qwen3_8_model_card","model_reports:qwen3_technical_report","model_reports:xai_grok_4_model_card"],"document_share":0.004901960784313725,"domain":"coding","name":"LiveCodeBench","organization_count":9,"organizations":["Ai2","ByteDance","DeepSeek","Google","Meta","Mistral","OpenAI","Qwen","xAI"],"rank":6,"released":"2024-03-12","source":"model_reports","url":"https://livecodebench.github.io/"},{"aliases":["MMLU-Pro","MMLU Pro"],"benchmark_id":"mmlu_pro","caveat":"Static closed-set multiple choice; contamination risk rises over time.","document_count":13,"document_ids":["model_reports:ai2_olmo_3","model_reports:anthropic_claude_3_7_sonnet","model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemma_4_model_card","model_reports:meta_llama_3_1","model_reports:meta_llama_4","model_reports:mistral_medium_3","model_reports:qwen3_5_model_card","model_reports:qwen3_technical_report"],"document_share":0.003540305010893246,"domain":"knowledge","name":"MMLU-Pro","organization_count":8,"organizations":["Ai2","Anthropic","ByteDance","DeepSeek","Google","Meta","Mistral","Qwen"],"rank":7,"released":"2024-06-03","source":"model_reports","url":"https://github.com/TIGER-AI-Lab/MMLU-Pro"},{"aliases":["BrowseComp"],"benchmark_id":"browsecomp","caveat":"Live web. The result depends on what the internet contained on the day of the run.","document_count":12,"document_ids":["model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_3_1_pro_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:openai_gpt_5_6_release","model_reports:openai_gpt_5_6_system_card","model_reports:openai_gpt_5_system_card","model_reports:openai_o3_o4_mini_system_card","model_reports:qwen3_5_model_card","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_model_card","model_reports:zai_glm_5_paper"],"document_share":0.0032679738562091504,"domain":"agent","name":"BrowseComp","organization_count":6,"organizations":["DeepSeek","Google","Moonshot AI","OpenAI","Qwen","Z.ai"],"rank":8,"released":"2025-04-10","source":"model_reports","url":"https://openai.com/index/browsecomp/"},{"aliases":["MATH-500","MATH 500","MATH"],"benchmark_id":"math_500","caveat":"Largely saturated at the frontier.","document_count":11,"document_ids":["model_reports:anthropic_claude_3_7_sonnet","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:google_gemini_1_5_report","model_reports:meta_llama_3_1","model_reports:microsoft_phi_4_mini_reasoning_model_card","model_reports:mistral_large_2","model_reports:mistral_medium_3","model_reports:qwen2_5_coder_report","model_reports:qwen3_technical_report"],"document_share":0.0029956427015250544,"domain":"math","name":"MATH-500","organization_count":7,"organizations":["Anthropic","DeepSeek","Google","Meta","Microsoft","Mistral","Qwen"],"rank":9,"released":"2021-03-05","source":"model_reports","url":"https://github.com/openai/prm800k"},{"aliases":["SWE-bench Pro","SWE Pro","SWE-Bench Pro (Public)"],"benchmark_id":"swe_bench_pro","caveat":"Harness and split version must be recorded with any score.","document_count":10,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5","model_reports:anthropic_claude_opus_5_system_card","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_3_1_pro_model_card","model_reports:qwen3_8_model_card","model_reports:tencent_hy4_preview","model_reports:xai_grok_4_5","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_2_model_card"],"document_share":0.0027233115468409588,"domain":"coding_agent","name":"SWE-bench Pro","organization_count":7,"organizations":["Anthropic","DeepSeek","Google","Qwen","Tencent","Z.ai","xAI"],"rank":10,"released":"2025-09-23","source":"model_reports","url":"https://github.com/scaleapi/SWE-bench_Pro-os"},{"aliases":["MMMU","MMMU-Pro"],"benchmark_id":"mmmu","caveat":"Image preprocessing and chain-of-thought settings affect results.","document_count":10,"document_ids":["model_reports:anthropic_claude_3_7_sonnet","model_reports:anthropic_claude_4_system_card","model_reports:google_gemini_1_5_report","model_reports:google_gemini_2_5_report","model_reports:meta_llama_4","model_reports:openai_gpt_4_1","model_reports:openai_gpt_5_system_card","model_reports:openai_o3_o4_mini_system_card","model_reports:qwen3_5_model_card","model_reports:xai_grok_4_model_card"],"document_share":0.0027233115468409588,"domain":"multimodal","name":"MMMU","organization_count":6,"organizations":["Anthropic","Google","Meta","OpenAI","Qwen","xAI"],"rank":11,"released":"2023-11-27","source":"model_reports","url":"https://mmmu-benchmark.github.io/"},{"aliases":["SimpleQA","SimpleQA Verified"],"benchmark_id":"simpleqa","caveat":"Knowledge cutoff must be recorded alongside the score.","document_count":10,"document_ids":["model_reports:ai2_olmo_3","model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_2_5_report","model_reports:mistral_large_3","model_reports:openai_gpt_5_system_card","model_reports:openai_o3_o4_mini_system_card"],"document_share":0.0027233115468409588,"domain":"factuality","name":"SimpleQA","organization_count":6,"organizations":["Ai2","ByteDance","DeepSeek","Google","Mistral","OpenAI"],"rank":12,"released":"2024-10-30","source":"model_reports","url":"https://openai.com/index/introducing-simpleqa/"},{"aliases":["IFEval"],"benchmark_id":"ifeval","caveat":"Verifiable-constraint subset only; does not measure answer quality.","document_count":9,"document_ids":["model_reports:ai2_olmo_3","model_reports:anthropic_claude_3_7_sonnet","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:meta_llama_3_1","model_reports:mistral_medium_3","model_reports:openai_gpt_4_1","model_reports:qwen3_5_model_card","model_reports:qwen3_technical_report"],"document_share":0.0024509803921568627,"domain":"instruction_following","name":"IFEval","organization_count":7,"organizations":["Ai2","Anthropic","DeepSeek","Meta","Mistral","OpenAI","Qwen"],"rank":13,"released":"2023-11-14","source":"model_reports","url":"https://github.com/google-research/google-research/tree/master/instruction_following_eval"},{"aliases":["MMLU"],"benchmark_id":"mmlu","caveat":"Saturated at the frontier and heavily contaminated. Useful as a historical baseline, not as a ranking signal.","document_count":9,"document_ids":["model_reports:ai2_olmo_3","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:google_gemini_1_5_report","model_reports:meta_llama_3_1","model_reports:mistral_large_2","model_reports:openai_gpt_4_1","model_reports:qwen2_5_coder_report"],"document_share":0.0024509803921568627,"domain":"knowledge","name":"MMLU","organization_count":7,"organizations":["Ai2","DeepSeek","Google","Meta","Mistral","OpenAI","Qwen"],"rank":14,"released":"2020-09-07","source":"model_reports","url":"https://github.com/hendrycks/test"},{"aliases":["HumanEval","HumanEval+"],"benchmark_id":"humaneval","caveat":"Saturated. Retained because open-weight cards still report it.","document_count":9,"document_ids":["model_reports:ai2_olmo_3","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:google_gemini_1_5_report","model_reports:meta_llama_3_1","model_reports:mistral_large_2","model_reports:mistral_medium_3","model_reports:qwen2_5_coder_report","model_reports:qwen3_technical_report"],"document_share":0.0024509803921568627,"domain":"coding","name":"HumanEval","organization_count":6,"organizations":["Ai2","DeepSeek","Google","Meta","Mistral","Qwen"],"rank":15,"released":"2021-07-07","source":"model_reports","url":"https://github.com/openai/human-eval"},{"aliases":["MCP Atlas","MCP-Atlas","MCPAtlas","MCP-Atlas (Public Set)"],"benchmark_id":"mcp_atlas","caveat":"Public and full splits are both reported under one name; the tool server set is part of the measurement.","document_count":8,"document_ids":["model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_3_1_pro_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:tencent_hy4_preview","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_2_model_card","model_reports:zai_glm_5_model_card"],"document_share":0.002178649237472767,"domain":"tool_use","name":"MCP Atlas","organization_count":5,"organizations":["DeepSeek","Google","Moonshot AI","Tencent","Z.ai"],"rank":16,"released":"2025-11-18","source":"model_reports","url":"https://huggingface.co/datasets/Sierra-Research/MCP-Atlas"},{"aliases":["MRCR","OpenAI-MRCR","MRCR v2"],"benchmark_id":"mrcr","caveat":"Needle position and generation seed change the result.","document_count":8,"document_ids":["model_reports:anthropic_claude_4_system_card","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_2_5_report","model_reports:google_gemini_3_1_pro_model_card","model_reports:google_gemma_4_model_card","model_reports:openai_gpt_4_1","model_reports:openai_gpt_5_system_card"],"document_share":0.002178649237472767,"domain":"long_context","name":"MRCR","organization_count":4,"organizations":["Anthropic","DeepSeek","Google","OpenAI"],"rank":17,"released":"2025-04-14","source":"model_reports","url":"https://huggingface.co/datasets/openai/mrcr"},{"aliases":["Tool Decathlon","Toolathlon","Toolathlon-Verified"],"benchmark_id":"tool_decathlon","caveat":"Aggregates ten heterogeneous tool suites; the per-suite spread is wide.","document_count":8,"document_ids":["model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_pro_0813_model_card","model_reports:deepseek_v4_technical_report","model_reports:moonshot_kimi_k3_model_card","model_reports:qwen3_5_model_card","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_2_model_card","model_reports:zai_glm_5_model_card"],"document_share":0.002178649237472767,"domain":"tool_use","name":"Tool Decathlon","organization_count":4,"organizations":["DeepSeek","Moonshot AI","Qwen","Z.ai"],"rank":18,"released":"2025-10-28","source":"model_reports","url":"https://toolathlon.xyz/"},{"aliases":["GSM8K"],"benchmark_id":"gsm8k","caveat":"Saturated. A legacy baseline for small open models.","document_count":7,"document_ids":["model_reports:ai2_olmo_3","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:google_gemini_1_5_report","model_reports:meta_llama_3_1","model_reports:mistral_large_2","model_reports:qwen2_5_coder_report"],"document_share":0.001906318082788671,"domain":"math","name":"GSM8K","organization_count":6,"organizations":["Ai2","DeepSeek","Google","Meta","Mistral","Qwen"],"rank":19,"released":"2021-10-27","source":"model_reports","url":"https://github.com/openai/grade-school-math"},{"aliases":["MMMLU","Multilingual MMLU","MMLU-ProX"],"benchmark_id":"mmmlu","caveat":"A language-count average. Two cards reporting \"MMMLU\" may be averaging over different language sets, so the counts must match to compare.","document_count":7,"document_ids":["model_reports:anthropic_claude_4_system_card","model_reports:deepseek_v4_model_card","model_reports:google_gemini_3_1_pro_model_card","model_reports:google_gemma_4_model_card","model_reports:meta_llama_4","model_reports:mistral_large_3","model_reports:qwen3_5_model_card"],"document_share":0.001906318082788671,"domain":"multilingual","name":"MMMLU","organization_count":6,"organizations":["Anthropic","DeepSeek","Google","Meta","Mistral","Qwen"],"rank":20,"released":"2024-09-24","source":"model_reports","url":"https://huggingface.co/datasets/openai/MMMLU"},{"aliases":["AutomationBench","Zapier AutomationBench","HLEAutomationBench"],"benchmark_id":"automationbench","caveat":"Published by Zapier over its own automation surface, and cards report different task subsets of it.","document_count":7,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5","model_reports:anthropic_claude_opus_5_system_card","model_reports:deepseek_v4_1_flash_technical_report","model_reports:deepseek_v4_pro_0813_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:tencent_hy4_preview","model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.001906318082788671,"domain":"agent","name":"AutomationBench","organization_count":5,"organizations":["Anthropic","DeepSeek","Moonshot AI","Tencent","Z.ai"],"rank":21,"released":"2026-03-10","source":"model_reports","url":"https://github.com/zapier/automation-bench"},{"aliases":["GDPval","GDPval-AA","GDPval-AA v2"],"benchmark_id":"gdpval","caveat":"Graded by expert human comparison against real deliverables, and the \"-AA\" variants are run by Artificial Analysis rather than the vendor, so an Elo here is not comparable to a vendor-run pass rate.","document_count":7,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5","model_reports:anthropic_claude_opus_5_system_card","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:moonshot_kimi_k3_model_card","model_reports:tencent_hy4_preview","model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.001906318082788671,"domain":"professional","name":"GDPval","organization_count":5,"organizations":["Anthropic","DeepSeek","Moonshot AI","Tencent","Z.ai"],"rank":22,"released":"2025-09-25","source":"model_reports","url":"https://openai.com/index/gdpval/"},{"aliases":["HMMT","HMMT Feb 25","HMMT Nov 25","HMMT 2026 Feb"],"benchmark_id":"hmmt","caveat":"A dated competition, not a fixed set: each sitting is a different instrument and contamination rises once problems are published.","document_count":7,"document_ids":["model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:qwen3_5_model_card","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_2_model_card","model_reports:zai_glm_5_model_card"],"document_share":0.001906318082788671,"domain":"math","name":"HMMT","organization_count":4,"organizations":["ByteDance","DeepSeek","Qwen","Z.ai"],"rank":23,"released":"2025-02-15","source":"model_reports","url":"https://www.hmmt.org/"},{"aliases":["IMOAnswerBench","IMO AnswerBench"],"benchmark_id":"imo_answer_bench","caveat":"Final-answer grading on olympiad problems, so it does not check whether the proof reasoning was valid.","document_count":7,"document_ids":["model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:qwen3_5_model_card","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_2_model_card","model_reports:zai_glm_5_model_card"],"document_share":0.001906318082788671,"domain":"math","name":"IMOAnswerBench","organization_count":4,"organizations":["ByteDance","DeepSeek","Qwen","Z.ai"],"rank":24,"released":"2025-09-18","source":"model_reports","url":"https://huggingface.co/datasets/HuggingFaceH4/IMOAnswerBench"},{"aliases":["DROP"],"benchmark_id":"drop","caveat":"Legacy reading-comprehension baseline.","document_count":6,"document_ids":["model_reports:ai2_olmo_3","model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:google_gemini_1_5_report","model_reports:meta_llama_3_1"],"document_share":0.0016339869281045752,"domain":"reasoning","name":"DROP","organization_count":4,"organizations":["Ai2","DeepSeek","Google","Meta"],"rank":25,"released":"2019-03-01","source":"model_reports","url":"https://allenai.org/data/drop"},{"aliases":["tau2-bench","τ²-Bench","TAU2-Bench","Tau2","τ2-bench"],"benchmark_id":"tau2_bench","caveat":"Per-domain scores (Retail, Telecom, Airline) diverge sharply and several cards apply the Airline domain fixes from the Claude Opus 4.5 system card, so an average conceals both the domain mix and the patch level.","document_count":6,"document_ids":["model_reports:google_gemini_3_1_pro_model_card","model_reports:google_gemma_4_model_card","model_reports:qwen3_5_model_card","model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_model_card","model_reports:zai_glm_5_paper"],"document_share":0.0016339869281045752,"domain":"tool_use","name":"tau2-bench","organization_count":3,"organizations":["Google","Qwen","Z.ai"],"rank":26,"released":"2025-06-09","source":"model_reports","url":"https://arxiv.org/abs/2506.07982"},{"aliases":["MBPP","MBPP+"],"benchmark_id":"mbpp","caveat":"Saturated; short single-function tasks.","document_count":5,"document_ids":["model_reports:ai2_olmo_3","model_reports:deepseek_v3_report","model_reports:meta_llama_3_1","model_reports:mistral_large_2","model_reports:qwen2_5_coder_report"],"document_share":0.0013616557734204794,"domain":"coding","name":"MBPP","organization_count":5,"organizations":["Ai2","DeepSeek","Meta","Mistral","Qwen"],"rank":27,"released":"2021-08-16","source":"model_reports","url":"https://github.com/google-research/google-research/tree/master/mbpp"},{"aliases":["APEX-Agents","APEX","Apex","Apex Shortlist"],"benchmark_id":"apex_agents","caveat":"Expert-authored professional tasks with rubric grading; graders are LLMs.","document_count":5,"document_ids":["model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemini_3_1_pro_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:tencent_hy4_preview"],"document_share":0.0013616557734204794,"domain":"agent","name":"APEX-Agents","organization_count":4,"organizations":["DeepSeek","Google","Moonshot AI","Tencent"],"rank":28,"released":"2026-01-20","source":"model_reports","url":"https://arxiv.org/abs/2601.14242"},{"aliases":["SWE-bench Multilingual","SWE Multilingual"],"benchmark_id":"swe_bench_multilingual","caveat":"Per-language resolution rates differ widely, so an average hides which languages the model actually handles.","document_count":5,"document_ids":["model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:qwen3_5_model_card","model_reports:tencent_hy4_preview","model_reports:zai_glm_5_model_card"],"document_share":0.0013616557734204794,"domain":"coding_agent","name":"SWE-bench Multilingual","organization_count":4,"organizations":["DeepSeek","Qwen","Tencent","Z.ai"],"rank":29,"released":"2025-04-22","source":"model_reports","url":"https://www.swebench.com/multilingual.html"},{"aliases":["Codeforces","Codeforces ELO","Codeforces Rating"],"benchmark_id":"codeforces","caveat":"An Elo estimate against human contestants, not a fixed dataset. The problem set moves continuously and rating conversion differs by vendor.","document_count":5,"document_ids":["model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_v4_1_flash_technical_report","model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:google_gemma_4_model_card"],"document_share":0.0013616557734204794,"domain":"coding","name":"Codeforces","organization_count":3,"organizations":["ByteDance","DeepSeek","Google"],"rank":30,"released":"2010-02-19","source":"model_reports","url":"https://codeforces.com/"},{"aliases":["OSWorld","OSWorld-Verified","OSWorld 2.0"],"benchmark_id":"osworld","caveat":"Screen resolution, OS image and action space all change results. The Verified and 2.0 revisions are not comparable to the original.","document_count":5,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5","model_reports:anthropic_claude_opus_5_system_card","model_reports:moonshot_kimi_k3_model_card","model_reports:qwen3_5_model_card","model_reports:qwen3_8_model_card"],"document_share":0.0013616557734204794,"domain":"computer_use","name":"OSWorld","organization_count":3,"organizations":["Anthropic","Moonshot AI","Qwen"],"rank":31,"released":"2024-04-11","source":"model_reports","url":"https://os-world.github.io/"},{"aliases":["MMLU-Redux"],"benchmark_id":"mmlu_redux","caveat":"Error-corrected MMLU subset; reported mainly by open-weight cards.","document_count":5,"document_ids":["model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:qwen3_5_model_card","model_reports:qwen3_technical_report"],"document_share":0.0013616557734204794,"domain":"knowledge","name":"MMLU-Redux","organization_count":2,"organizations":["DeepSeek","Qwen"],"rank":32,"released":"2024-06-06","source":"model_reports","url":"https://github.com/aryopg/mmlu-redux"},{"aliases":["Aider Polyglot","Aider"],"benchmark_id":"aider_polyglot","caveat":"Edit-format sensitive; whole-file and diff modes differ substantially.","document_count":4,"document_ids":["model_reports:deepseek_r1_report","model_reports:google_gemini_2_5_report","model_reports:mistral_medium_3","model_reports:openai_gpt_4_1"],"document_share":0.0010893246187363835,"domain":"coding","name":"Aider Polyglot","organization_count":4,"organizations":["DeepSeek","Google","Mistral","OpenAI"],"rank":33,"released":"2024-12-21","source":"model_reports","url":"https://aider.chat/docs/leaderboards/"},{"aliases":["ARC-AGI","ARC-AGI-1","ARC-AGI-2"],"benchmark_id":"arc_agi","caveat":"Compute per task is reported alongside score by the maintainers and should not be dropped.","document_count":4,"document_ids":["model_reports:bytedance_seed2_0_model_card","model_reports:google_gemini_2_5_report","model_reports:openai_o3_o4_mini_system_card","model_reports:xai_grok_4_model_card"],"document_share":0.0010893246187363835,"domain":"reasoning","name":"ARC-AGI","organization_count":4,"organizations":["ByteDance","Google","OpenAI","xAI"],"rank":34,"released":"2019-11-05","source":"model_reports","url":"https://arcprize.org/"},{"aliases":["PostTrainBench","PostTrainBench Lite"],"benchmark_id":"posttrainbench","caveat":"Measures a model's ability to run post-training itself. Hardware differs between reported runs (H100 vs H20), which moves the score.","document_count":4,"document_ids":["model_reports:moonshot_kimi_k3_model_card","model_reports:openai_gpt_5_6_system_card","model_reports:tencent_hy4_preview","model_reports:zai_glm_5_2_model_card"],"document_share":0.0010893246187363835,"domain":"ai_research","name":"PostTrainBench","organization_count":4,"organizations":["Moonshot AI","OpenAI","Tencent","Z.ai"],"rank":35,"released":"2026-01-20","source":"model_reports","url":"https://posttrainbench.com/"},{"aliases":["Video-MME","VideoMME"],"benchmark_id":"video_mme","caveat":"Frame sampling rate dominates long-video results.","document_count":4,"document_ids":["model_reports:google_gemini_2_5_report","model_reports:moonshot_kimi_k3_model_card","model_reports:openai_gpt_4_1","model_reports:qwen3_5_model_card"],"document_share":0.0010893246187363835,"domain":"multimodal","name":"Video-MME","organization_count":4,"organizations":["Google","Moonshot AI","OpenAI","Qwen"],"rank":36,"released":"2024-05-31","source":"model_reports","url":"https://video-mme.github.io/"},{"aliases":["AGIEval"],"benchmark_id":"agieval","caveat":"Human-exam derived; overlaps heavily with MMLU-style coverage.","document_count":4,"document_ids":["model_reports:ai2_olmo_3","model_reports:deepseek_v3_report","model_reports:deepseek_v4_model_card","model_reports:meta_llama_3_1"],"document_share":0.0010893246187363835,"domain":"knowledge","name":"AGIEval","organization_count":3,"organizations":["Ai2","DeepSeek","Meta"],"rank":37,"released":"2023-04-13","source":"model_reports","url":"https://github.com/ruixiangcui/AGIEval"},{"aliases":["Arena-Hard","ArenaHard"],"benchmark_id":"arena_hard","caveat":"LLM-judge dependent, with known style and length bias.","document_count":4,"document_ids":["model_reports:deepseek_r1_report","model_reports:deepseek_v3_report","model_reports:meta_llama_3_1","model_reports:qwen3_technical_report"],"document_share":0.0010893246187363835,"domain":"human_preference","name":"Arena-Hard","organization_count":3,"organizations":["DeepSeek","Meta","Qwen"],"rank":38,"released":"2024-04-19","source":"model_reports","url":"https://github.com/lmarena/arena-hard-auto"},{"aliases":["BFCL","Berkeley Function Calling Leaderboard"],"benchmark_id":"bfcl","caveat":"Schema complexity and execution checking vary by version.","document_count":4,"document_ids":["model_reports:ai2_olmo_3","model_reports:meta_llama_3_1","model_reports:qwen3_5_model_card","model_reports:qwen3_technical_report"],"document_share":0.0010893246187363835,"domain":"tool_use","name":"BFCL","organization_count":3,"organizations":["Ai2","Meta","Qwen"],"rank":39,"released":"2024-02-26","source":"model_reports","url":"https://gorilla.cs.berkeley.edu/leaderboard.html"},{"aliases":["MathVision","MATH-Vision","MathVision (mini)"],"benchmark_id":"mathvision","caveat":"Prompt-format sensitive: vendors report boxed and unboxed variants and sometimes take the higher of the two.","document_count":4,"document_ids":["model_reports:google_gemma_4_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:qwen3_5_model_card","model_reports:qwen3_8_model_card"],"document_share":0.0010893246187363835,"domain":"multimodal","name":"MathVision","organization_count":3,"organizations":["Google","Moonshot AI","Qwen"],"rank":40,"released":"2024-02-22","source":"model_reports","url":"https://mathllm.github.io/mathvision/"},{"aliases":["MathVista"],"benchmark_id":"mathvista","caveat":"Mixes OCR quality with visual reasoning.","document_count":4,"document_ids":["model_reports:google_gemini_1_5_report","model_reports:google_gemini_2_5_report","model_reports:meta_llama_4","model_reports:qwen3_5_model_card"],"document_share":0.0010893246187363835,"domain":"multimodal","name":"MathVista","organization_count":3,"organizations":["Google","Meta","Qwen"],"rank":41,"released":"2023-10-03","source":"model_reports","url":"https://mathvista.github.io/"},{"aliases":["MMMU-Pro","MMMU Pro"],"benchmark_id":"mmmu_pro","caveat":"Harder vision-required subset of MMMU. Input image ordering changes the result, so the protocol has to travel with the number.","document_count":4,"document_ids":["model_reports:google_gemini_3_1_pro_model_card","model_reports:google_gemma_4_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:qwen3_5_model_card"],"document_share":0.0010893246187363835,"domain":"multimodal","name":"MMMU-Pro","organization_count":3,"organizations":["Google","Moonshot AI","Qwen"],"rank":42,"released":"2024-09-04","source":"model_reports","url":"https://arxiv.org/abs/2409.02813"},{"aliases":["OmniDocBench","OmniDocBench1.5","OmniDocBench 1.5"],"benchmark_id":"omnidocbench","caveat":"Reported as an edit distance where lower is better, so it inverts the direction of every other row a card puts beside it.","document_count":4,"document_ids":["model_reports:google_gemma_4_model_card","model_reports:moonshot_kimi_k3_model_card","model_reports:qwen3_5_model_card","model_reports:qwen3_8_model_card"],"document_share":0.0010893246187363835,"domain":"multimodal","name":"OmniDocBench","organization_count":3,"organizations":["Google","Moonshot AI","Qwen"],"rank":43,"released":"2024-12-10","source":"model_reports","url":"https://github.com/opendatalab/OmniDocBench"},{"aliases":["Agents' Last Exam","ALE","Agent's Last Exam","ALE-CLI"],"benchmark_id":"agents_last_exam","caveat":"Multi-step agentic benchmark using Claude Code harness. Tool Search disabled. Scores depend on harness, reasoning effort, context length, and timeout settings.","document_count":3,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report","model_reports:tencent_hy4_preview","model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0008169934640522876,"domain":"coding_agent","name":"Agents' Last Exam","organization_count":3,"organizations":["DeepSeek","Tencent","Z.ai"],"rank":44,"released":"2025-01-23","source":"model_reports","url":"https://lastexam.ai/"},{"aliases":["CritPt"],"benchmark_id":"critpt","caveat":"Research-level physics reasoning; small expert-authored set.","document_count":3,"document_ids":["model_reports:moonshot_kimi_k3_model_card","model_reports:tencent_hy4_preview","model_reports:zai_glm_5_2_model_card"],"document_share":0.0008169934640522876,"domain":"science","name":"CritPt","organization_count":3,"organizations":["Moonshot AI","Tencent","Z.ai"],"rank":45,"released":"2025-09-30","source":"model_reports","url":"https://arxiv.org/abs/2509.26574"},{"aliases":["DeepSWE"],"benchmark_id":"deepswe","caveat":"Original repository-level engineering tasks from Datacurve. Version 1.1 grades committed patches in a separate verifier container. Version, agent framework, sampling policy and budget must accompany pass@1 or pass@4 results; a passing test suite is not a complete code-quality review.","document_count":3,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report","model_reports:tencent_hy4_preview","model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0008169934640522876,"domain":"coding_agent","name":"DeepSWE","organization_count":3,"organizations":["DeepSeek","Tencent","Z.ai"],"rank":46,"released":"2026-05-26","source":"model_reports","url":"https://deepswe.datacurve.ai/blog/deepswe"},{"aliases":["GPQA main","GPQA full"],"benchmark_id":"gpqa","caveat":"The full split, distinct from the Diamond subset most frontier cards report.","document_count":3,"document_ids":["model_reports:ai2_olmo_3","model_reports:meta_llama_3_1","model_reports:qwen3_5_model_card"],"document_share":0.0008169934640522876,"domain":"science","name":"GPQA (full)","organization_count":3,"organizations":["Ai2","Meta","Qwen"],"rank":47,"released":"2023-11-20","source":"model_reports","url":"https://arxiv.org/abs/2311.12022"},{"aliases":["NL2Repo","NL-to-Repo"],"benchmark_id":"nl2repo","caveat":"Evaluates generating a repository from a natural language description. Temperature=1.0, top_p=1.0, max_new_tokens=64k under 1M context.","document_count":3,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report","model_reports:tencent_hy4_preview","model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0008169934640522876,"domain":"coding_agent","name":"NL2Repo","organization_count":3,"organizations":["DeepSeek","Tencent","Z.ai"],"rank":48,"released":"2025-06-15","source":"model_reports","url":"https://github.com/nl2repo/nl2repo"},{"aliases":["SuperGPQA","Super-GPQA"],"benchmark_id":"super_gpqa","caveat":"285 graduate disciplines; breadth comes at the cost of per-field sample size.","document_count":3,"document_ids":["model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_v4_model_card","model_reports:qwen3_5_model_card"],"document_share":0.0008169934640522876,"domain":"knowledge","name":"SuperGPQA","organization_count":3,"organizations":["ByteDance","DeepSeek","Qwen"],"rank":49,"released":"2025-02-20","source":"model_reports","url":"https://arxiv.org/abs/2502.14739"},{"aliases":["HealthBench","HealthBench Hard","HealthBench Consensus"],"benchmark_id":"healthbench","caveat":"Physician-written rubrics scored by an LLM grader, and the length-adjusted variant is what recent cards report. Published by OpenAI.","document_count":3,"document_ids":["model_reports:bytedance_seed2_0_model_card","model_reports:openai_gpt_5_6_system_card","model_reports:openai_gpt_5_system_card"],"document_share":0.0008169934640522876,"domain":"health","name":"HealthBench","organization_count":2,"organizations":["ByteDance","OpenAI"],"rank":50,"released":"2025-05-12","source":"model_reports","url":"https://openai.com/index/healthbench/"},{"aliases":["IFBench"],"benchmark_id":"ifbench","caveat":"Verifiable constraints on unseen instruction types; does not measure answer quality.","document_count":3,"document_ids":["model_reports:ai2_olmo_3","model_reports:qwen3_5_model_card","model_reports:qwen3_8_model_card"],"document_share":0.0008169934640522876,"domain":"instruction_following","name":"IFBench","organization_count":2,"organizations":["Ai2","Qwen"],"rank":51,"released":"2025-07-03","source":"model_reports","url":"https://arxiv.org/abs/2507.02833"},{"aliases":["LongBench","LongBench v2"],"benchmark_id":"longbench","caveat":"Nominal context length is not effective context length.","document_count":3,"document_ids":["model_reports:deepseek_v4_model_card","model_reports:deepseek_v4_technical_report","model_reports:qwen3_5_model_card"],"document_share":0.0008169934640522876,"domain":"long_context","name":"LongBench","organization_count":2,"organizations":["DeepSeek","Qwen"],"rank":52,"released":"2023-08-28","source":"model_reports","url":"https://github.com/THUDM/LongBench"},{"aliases":["tau-bench","τ-bench","tau2-bench","TAU-bench"],"benchmark_id":"tau_bench","caveat":"Depends on a simulated user and policy; the simulator model is part of the measurement.","document_count":3,"document_ids":["model_reports:anthropic_claude_3_7_sonnet","model_reports:anthropic_claude_4_system_card","model_reports:openai_gpt_5_system_card"],"document_share":0.0008169934640522876,"domain":"tool_use","name":"tau-bench","organization_count":2,"organizations":["Anthropic","OpenAI"],"rank":53,"released":"2024-06-17","source":"model_reports","url":"https://github.com/sierra-research/tau-bench"},{"aliases":["AA-LCR"],"benchmark_id":"aa_lcr","caveat":"Run by Artificial Analysis rather than the vendor. Third-party execution is the point, but it also means the vendor did not control the setup.","document_count":2,"document_ids":["model_reports:moonshot_kimi_k3_model_card","model_reports:qwen3_5_model_card"],"document_share":0.0005446623093681918,"domain":"long_context","name":"AA-LCR","organization_count":2,"organizations":["Moonshot AI","Qwen"],"rank":54,"released":"2025-09-16","source":"model_reports","url":"https://artificialanalysis.ai/evaluations/aa-lcr"},{"aliases":["ARC-AGI-2","ARC-AGI 2"],"benchmark_id":"arc_agi_2","caveat":"Cost per task is part of the official result and is routinely dropped when the score is quoted on its own.","document_count":2,"document_ids":["model_reports:google_gemini_3_1_pro_model_card","model_reports:xai_grok_4_5"],"document_share":0.0005446623093681918,"domain":"reasoning","name":"ARC-AGI-2","organization_count":2,"organizations":["Google","xAI"],"rank":55,"released":"2025-03-24","source":"model_reports","url":"https://arcprize.org/arc-agi/2/"},{"aliases":["BabyVision"],"benchmark_id":"babyvision","caveat":"Multimodal vision benchmark. Temperature=1.0, top_p=0.95, max context 164K tokens. Images resized to shorter side at least 1.5K pixels.","document_count":2,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report","model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0005446623093681918,"domain":"multimodal","name":"BabyVision","organization_count":2,"organizations":["DeepSeek","Z.ai"],"rank":56,"released":"2025-12-01","source":"model_reports","url":"https://github.com/babyvision/babyvision"},{"aliases":["BioMysteryBench"],"benchmark_id":"biomysterybench","caveat":"Reported in two splits (\"hard\" and \"human solved\") that differ by more than 35 points, so a bare score is unreadable without its split. Anthropic notes its own safety refusals depress this number, which means the score mixes capability with policy. First-party to Anthropic, which built it and publishes the dataset. The task set was revised after an answer-key audit, so the item count moved and scores are not comparable across dataset versions.","document_count":2,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5","model_reports:tencent_hy4_preview"],"document_share":0.0005446623093681918,"domain":"biology","name":"BioMysteryBench","organization_count":2,"organizations":["Anthropic","Tencent"],"rank":57,"released":"2026-04-29","source":"model_reports","url":"https://huggingface.co/datasets/Anthropic/BioMysteryBench-full"},{"aliases":["BrowseComp-ZH","BrowseComp-zh","BrowseComp-Zh"],"benchmark_id":"browsecomp_zh","caveat":"Chinese-language live web. Same day-to-day web drift as BrowseComp, over a different index.","document_count":2,"document_ids":["model_reports:qwen3_5_model_card","model_reports:zai_glm_5_model_card"],"document_share":0.0005446623093681918,"domain":"agent","name":"BrowseComp-ZH","organization_count":2,"organizations":["Qwen","Z.ai"],"rank":58,"released":"2025-04-27","source":"model_reports","url":"https://arxiv.org/abs/2504.19314"},{"aliases":["Chartography"],"benchmark_id":"chartography","caveat":"Chart comprehension benchmark with tool access. Scores depend on context length and tool configuration.","document_count":2,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report","model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0005446623093681918,"domain":"vision","name":"Chartography","organization_count":2,"organizations":["DeepSeek","Z.ai"],"rank":59,"released":"2025-06-01","source":"model_reports","url":"https://github.com/Chartography/Chartography"},{"aliases":["ChartQA"],"benchmark_id":"chartqa","caveat":"Largely saturated; relaxed-accuracy tolerance affects the reported figure.","document_count":2,"document_ids":["model_reports:google_gemini_1_5_report","model_reports:meta_llama_4"],"document_share":0.0005446623093681918,"domain":"multimodal","name":"ChartQA","organization_count":2,"organizations":["Google","Meta"],"rank":60,"released":"2022-03-19","source":"model_reports","url":"https://github.com/vis-nlp/ChartQA"},{"aliases":[],"benchmark_id":"cybergym","caveat":"Real-world cybersecurity tasks; tool permissions and time limits are part of the result, not incidental to it.","document_count":2,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report","model_reports:tencent_hy4_preview"],"document_share":0.0005446623093681918,"domain":"security","name":"CyberGym","organization_count":2,"organizations":["DeepSeek","Tencent"],"rank":61,"released":null,"source":"model_reports","url":"https://openreview.net/forum?id=2YvbLQEdYt"},{"aliases":["DeepSearchQA"],"benchmark_id":"deepsearchqa","caveat":"Live-web deep research; F1 grading depends on the reference answer set.","document_count":2,"document_ids":["model_reports:anthropic_claude_opus_5_system_card","model_reports:moonshot_kimi_k3_model_card"],"document_share":0.0005446623093681918,"domain":"agent","name":"DeepSearchQA","organization_count":2,"organizations":["Anthropic","Moonshot AI"],"rank":62,"released":"2026-02-10","source":"model_reports","url":"https://huggingface.co/datasets/PokeeAI/DeepSearchQA"},{"aliases":["DocVQA"],"benchmark_id":"docvqa","caveat":"Saturated at the frontier; ANLS scoring is lenient to OCR near-misses.","document_count":2,"document_ids":["model_reports:google_gemini_1_5_report","model_reports:meta_llama_4"],"document_share":0.0005446623093681918,"domain":"multimodal","name":"DocVQA","organization_count":2,"organizations":["Google","Meta"],"rank":63,"released":"2020-07-01","source":"model_reports","url":"https://www.docvqa.org/"},{"aliases":["ExploitBench","ExploitBench (Cap%)"],"benchmark_id":"exploitbench","caveat":"Offensive-security capability measured as a risk indicator against a preparedness threshold, not a leaderboard to top. Reported as a capability percentage. Published by Seunghyun Lee and David Brumley (Carnegie Mellon; Brumley also lists Bugcrowd) as arXiv:2605.14153 with code at github.com/exploitbench/exploitbench. 41 V8 N-day vulnerabilities scored by 16 grader-verified capability flags; vendor runs report the capability percentage without publishing per-flag results, so a reported number is not reproducible from the paper alone.","document_count":2,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5","model_reports:openai_gpt_5_6_system_card"],"document_share":0.0005446623093681918,"domain":"security","name":"ExploitBench","organization_count":2,"organizations":["Anthropic","OpenAI"],"rank":64,"released":"2026-05-13","source":"model_reports","url":"https://exploitbench.ai/"},{"aliases":["MathArena Apex","MathArena Apex 2025","Apex 2025"],"benchmark_id":"matharena_apex_2025","caveat":"The hardest MathArena split; even near-ceiling models sit well below typical math-benchmark numbers.","document_count":2,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report","model_reports:tencent_hy4_preview"],"document_share":0.0005446623093681918,"domain":"math","name":"MathArena Apex 2025","organization_count":2,"organizations":["DeepSeek","Tencent"],"rank":65,"released":null,"source":"model_reports","url":"https://matharena.ai/apex"},{"aliases":["MCPMark","MCP-Mark","MCPMark-Verified"],"benchmark_id":"mcp_mark","caveat":"Depends on live third-party MCP servers, so runs are not reproducible over time.","document_count":2,"document_ids":["model_reports:moonshot_kimi_k3_model_card","model_reports:qwen3_5_model_card"],"document_share":0.0005446623093681918,"domain":"tool_use","name":"MCPMark","organization_count":2,"organizations":["Moonshot AI","Qwen"],"rank":66,"released":"2025-09-30","source":"model_reports","url":"https://mcpmark.ai/"},{"aliases":["OfficeQA Pro","OfficeQA"],"benchmark_id":"office_qa_pro","caveat":"Visual QA benchmark on professional document corpus (PDF without embedded text). Scores depend on context length and image resolution.","document_count":2,"document_ids":["model_reports:tencent_hy4_preview","model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0005446623093681918,"domain":"vision","name":"OfficeQA Pro","organization_count":2,"organizations":["Tencent","Z.ai"],"rank":67,"released":"2025-06-01","source":"model_reports","url":"https://github.com/OfficeQA/OfficeQA-Pro"},{"aliases":[],"benchmark_id":"programbench","caveat":"Cleanroom program-rebuild tasks; most models score low, so small differences sit within run-to-run noise.","document_count":2,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report","model_reports:tencent_hy4_preview"],"document_share":0.0005446623093681918,"domain":"coding","name":"ProgramBench","organization_count":2,"organizations":["DeepSeek","Tencent"],"rank":68,"released":"2026-05-05","source":"model_reports","url":"https://arxiv.org/abs/2605.03546"},{"aliases":["SciCode"],"benchmark_id":"scicode","caveat":"Scientific code generation graded by test execution; subproblem context matters.","document_count":2,"document_ids":["model_reports:google_gemini_3_1_pro_model_card","model_reports:moonshot_kimi_k3_model_card"],"document_share":0.0005446623093681918,"domain":"science","name":"SciCode","organization_count":2,"organizations":["Google","Moonshot AI"],"rank":69,"released":"2024-07-18","source":"model_reports","url":"https://scicode-bench.github.io/"},{"aliases":["Toolathlon Verified","Toolathlon"],"benchmark_id":"toolathlon_verified","caveat":"Pass@1 averaged over 3 independent runs. Official evaluation service.","document_count":2,"document_ids":["model_reports:tencent_hy4_preview","model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0005446623093681918,"domain":"tool_use","name":"Toolathlon Verified","organization_count":2,"organizations":["Tencent","Z.ai"],"rank":70,"released":"2025-11-01","source":"model_reports","url":"https://github.com/toolathlon/toolathlon"},{"aliases":["WideSearch"],"benchmark_id":"widesearch","caveat":"Wide-coverage web collection; scored on completeness of an enumerated answer set.","document_count":2,"document_ids":["model_reports:qwen3_5_model_card","model_reports:tencent_hy4_preview"],"document_share":0.0005446623093681918,"domain":"agent","name":"WideSearch","organization_count":2,"organizations":["Qwen","Tencent"],"rank":71,"released":"2025-08-11","source":"model_reports","url":"https://widesearch-seed.github.io/"},{"aliases":["ZeroBench","ZeroBench-main"],"benchmark_id":"zerobench","caveat":"Challenging visual reasoning benchmark designed to be impossible for frontier models at release; evaluates complex reasoning over images.","document_count":2,"document_ids":["model_reports:bytedance_seed2_0_model_card","model_reports:deepseek_v4_1_flash_technical_report"],"document_share":0.0005446623093681918,"domain":"multimodal","name":"ZeroBench","organization_count":2,"organizations":["ByteDance","DeepSeek"],"rank":72,"released":"2025-02-13","source":"model_reports","url":"https://github.com/jonathan-roberts1/zerobench"},{"aliases":["CursorBench","Cursor Bench","CursorBench 3.2"],"benchmark_id":"cursor_bench","caveat":"Maintained by Cursor and measured inside the Cursor product, so it reflects that scaffold rather than the bare model.","document_count":2,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5","model_reports:anthropic_claude_opus_5_system_card"],"document_share":0.0005446623093681918,"domain":"coding_agent","name":"CursorBench","organization_count":1,"organizations":["Anthropic"],"rank":73,"released":"2025-10-29","source":"model_reports","url":"https://cursor.com/blog/cursor-bench"},{"aliases":["MLE-Bench","MLE-bench Revised","MLE-bench Lite"],"benchmark_id":"mle_bench","caveat":"Kaggle-derived ML engineering tasks; wall-clock budget and hardware are part of the result. Published by OpenAI.","document_count":2,"document_ids":["model_reports:openai_gpt_5_6_system_card","model_reports:openai_gpt_5_system_card"],"document_share":0.0005446623093681918,"domain":"ai_research","name":"MLE-bench","organization_count":1,"organizations":["OpenAI"],"rank":74,"released":"2024-10-09","source":"model_reports","url":"https://openai.com/index/mle-bench/"},{"aliases":["Vending Bench","Vending Bench 2"],"benchmark_id":"vending_bench","caveat":"Reported in simulated dollars, not a percentage, and run over long horizons where a single early failure dominates the total.","document_count":2,"document_ids":["model_reports:zai_glm_5_1_model_card","model_reports:zai_glm_5_model_card"],"document_share":0.0005446623093681918,"domain":"agent","name":"Vending Bench","organization_count":1,"organizations":["Z.ai"],"rank":75,"released":"2025-02-18","source":"model_reports","url":"https://andonlabs.com/evals/vending-bench-2"},{"aliases":[],"benchmark_id":"claire-radar-2609-04611","caveat":"A benchmark where a developer agent builds a complete customer-service agent from business records, client requirements, a production API, inherited codebase, and cost/model limits, scored by deployment against held-out simulated users across 53 tasks in four domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04611"],"document_share":0.0002723311546840959,"domain":"Agents","name":"$\\tau^\\tau$-Bench: An Environment for End-To-End, Realistic Agent Construction","organization_count":1,"organizations":["claire_radar"],"rank":76,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.04611"},{"aliases":[],"benchmark_id":"claire-radar-2610-07423","caveat":"Field-effect transistor (FET) layouts on exfoliated two-dimensional flakes are typically drawn by hand for each flake, placing contacts and gates to match its position and outline in optical micrographs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.07423"],"document_share":0.0002723311546840959,"domain":"General AI","name":"2d-fet-bench","organization_count":1,"organizations":["claire_radar"],"rank":77,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.07423"},{"aliases":[],"benchmark_id":"claire-radar-2608-08814","caveat":"360CityArena evaluates embodied agents in a photorealistic virtual urban environment built from 360-degree video of Tokyo's Akihabara district. It includes 175 tasks across environment understanding, path reasoning, and spatial reasoning, testing localization, landmark search, path planning, and relational spatial reasoning. Scoring uses mean evaluation score with varied success criteria.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08814"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"360CityArena","organization_count":1,"organizations":["claire_radar"],"rank":78,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08814"},{"aliases":[],"benchmark_id":"claire-radar-2606-01057","caveat":"3DCodeBench evaluates vision-language model agents on procedural 3D modeling by converting text and image references into Blender Python code. It includes 212 object categories with ground-truth scripts, and scores outputs on executability, image similarity (SigLIP-2/DINOv3), 3D shape distance (Chamfer/Uni3D), and LLM-as-judge, plus a human-preference ranking platform.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01057"],"document_share":0.0002723311546840959,"domain":"General AI","name":"3DCodeBench","organization_count":1,"organizations":["claire_radar"],"rank":79,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01057"},{"aliases":["4DSynth: Controllable Procedural World Synthesis for Dynamic Embodied Simulation"],"benchmark_id":"claire-radar-2608-26947","caveat":"Evaluates embodied agents on interactive navigation tasks in procedurally generated 4D environments with independently tunable difficulty axes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26947"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"4DSynth-Nav","organization_count":1,"organizations":["claire_radar"],"rank":80,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26947"},{"aliases":[],"benchmark_id":"claire-radar-2609-09964","caveat":"Evaluation object not clearly defined; abstract mentions alignment of Romanized transliteration with dialectal text, Standard Bangla, English, and subjectivity labels across five Bangla regional varieties, but no explicit task or scoring setup is provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09964"],"document_share":0.0002723311546840959,"domain":"General AI","name":"5-Dialects-BN","organization_count":1,"organizations":["claire_radar"],"rank":81,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09964"},{"aliases":[],"benchmark_id":"claire-radar-2605-24045","caveat":"InteractBind provides a large-scale dataset of ~100k protein-ligand pairs with fine-grained binding-site localization tasks and interaction maps for six non-covalent interaction types, plus affinity and similarity-controlled splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24045"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"A Large-Scale Dataset and Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":82,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.24045"},{"aliases":[],"benchmark_id":"claire-radar-2610-13466","caveat":"CAD segmentation datasets are predominantly synthetic or automatically labelled, leaving it unclear how well existing methods transfer to real-world engineering tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13466"],"document_share":0.0002723311546840959,"domain":"CAD","name":"A Real-World CAD Segmentation Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":83,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13466"},{"aliases":[],"benchmark_id":"opencompass-1580-a-bench","caveat":"A-Bench is a benchmark designed to diagnose whether LMMs are masters at evaluating AIGIs. 2,864 AIGIs from 16 text-to-image models are sampled, each paired with question-answers annotated by human experts, and tested across 18 leading LMMs. A-Bench是一个旨在诊断 LMMs 是否擅长评估 AIGIs 的基准，从 16 个文本到图像模型中采样了 2,864 个 AIGIs，每个都与由人类专家标注的问题-答案配对。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/A-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"A-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":84,"released":"2024-06-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/A-Bench"},{"aliases":[],"benchmark_id":"opencompass-1367-a-okvqa","caveat":"A-OKVQA assesses commonsense reasoning abilities. It is a crowdsourced dataset composed of a diverse set of about 25K questions requiring a broad base of commonsense and world knowledge to answer. A-OKVQA用于评估多模态大模型的常识及推理能力，由25K个不同的问题组成，需要对图像中描述的场景进行某种形式的常识性推理来回答。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/A-OKVQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"A-OKVQA","organization_count":1,"organizations":["opencompass_hub"],"rank":85,"released":"2022-06-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/A-OKVQA"},{"aliases":[],"benchmark_id":"claire-radar-2605-22321","caveat":"A3S-Bench is an executable security test suite for evaluating autonomous agents across multi-turn scenarios and agent-level risks using an action-grounded oracle.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22321"],"document_share":0.0002723311546840959,"domain":"Agents","name":"A3S-Bench","organization_count":1,"organizations":["claire_radar"],"rank":86,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22321"},{"aliases":[],"benchmark_id":"artificial-analysis-aa-analystagent","caveat":"Quantitative analysis on spreadsheets & documents","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/aa-analyst-agent"],"document_share":0.0002723311546840959,"domain":"agentic","name":"AA-AnalystAgent","organization_count":1,"organizations":["artificial_analysis"],"rank":87,"released":null,"source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/aa-analyst-agent"},{"aliases":[],"benchmark_id":"artificial-analysis-aa-briefcase","caveat":"Agentic knowledge work, Elo","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/aa-briefcase"],"document_share":0.0002723311546840959,"domain":"agentic","name":"AA-Briefcase","organization_count":1,"organizations":["artificial_analysis"],"rank":88,"released":"2026-06-18","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/aa-briefcase"},{"aliases":[],"benchmark_id":"llm-stats-aa-briefcase","caveat":"AA-Briefcase is an Artificial Analysis evaluation of AI systems on professional knowledge-work tasks, reported as an Elo score.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aa-briefcase?top_n=500"],"document_share":0.0002723311546840959,"domain":"productivity","name":"AA-Briefcase","organization_count":1,"organizations":["llm_stats"],"rank":89,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-briefcase?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-aa-index","caveat":"No official academic documentation found for this benchmark. Extensive research through ArXiv, IEEE/ACL/NeurIPS papers, and university research sites yielded no peer-reviewed sources for an 'aa-index' benchmark. This entry requires verification from official academic sources.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aa-index?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"AA-Index","organization_count":1,"organizations":["llm_stats"],"rank":90,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-index?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-aa-lcr","caveat":"Long context reasoning","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"AA-LCR","organization_count":1,"organizations":["artificial_analysis"],"rank":91,"released":"2025-08-05","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning"},{"aliases":[],"benchmark_id":"llm-stats-aa-lcr","caveat":"Agent Arena Long Context Reasoning benchmark","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aa-lcr?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"AA-LCR","organization_count":1,"organizations":["llm_stats"],"rank":92,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-lcr?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-aa-omniscience-accuracy","caveat":"Knowledge","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/omniscience"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"AA-Omniscience Accuracy","organization_count":1,"organizations":["artificial_analysis"],"rank":93,"released":"2025-11-16","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/omniscience"},{"aliases":[],"benchmark_id":"llm-stats-aa-omniscience-index","caveat":"AA-Omniscience Index is Artificial Analysis's knowledge-reliability metric. It rewards correct answers, penalizes hallucinations, and does not penalize abstention. Scores range from -100 to 100, where 0 means as many correct as incorrect answers.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aa-omniscience-index?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"AA-Omniscience Index","organization_count":1,"organizations":["llm_stats"],"rank":94,"released":"2025-11-16","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aa-omniscience-index?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-aa-omniscience-non-hallucination","caveat":"1 - hallucination rate","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/omniscience"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"AA-Omniscience Non-Hallucination Rate","organization_count":1,"organizations":["artificial_analysis"],"rank":95,"released":"2025-11-16","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/omniscience"},{"aliases":[],"benchmark_id":"claire-radar-2606-07462","caveat":"AARRI-Bench evaluates LLM agents on entry-level research intern tasks, measuring success rate in containerized environments with fixed tasks and scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07462"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AARRI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":96,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07462"},{"aliases":["ABACUS_Bench"],"benchmark_id":"claire-radar-huggingface-aymen-lassoued-abacus-bench","caveat":"Evaluates instruction-based image editing models on add, remove, and change tasks, measuring exact object count correctness and scene preservation without VLM judges.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/Aymen-Lassoued/ABACUS_Bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ABACUS","organization_count":1,"organizations":["claire_radar"],"rank":97,"released":"2026-09-09","source":"claire_radar","url":"https://huggingface.co/datasets/Aymen-Lassoued/ABACUS_Bench"},{"aliases":[],"benchmark_id":"claire-radar-2606-11150","caveat":"ABC-Bench evaluates LLM agents on biosecurity-relevant tasks including liquid handling robot code generation, DNA fragment design, and DNA synthesis screening evasion, with wet-lab validation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11150"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ABC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":98,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11150"},{"aliases":[],"benchmark_id":"opencompass-1148-abspyramid","caveat":"ABSPYRAMID is a unified entailment graph of 221K textual descriptions of abstraction knowledge. ABSPYRAMID collects abstract knowledge for three components\nof diverse events to comprehensively evaluate the abstraction ability of language\nmodels in the open domain. ABSPYRAMID 是包含 221,000 条文本描述的抽象知识,收集了多种事件的三个组成部分的抽象知识，以全面评估语言模型在开放域中的抽象能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AbsPyramid"],"document_share":0.0002723311546840959,"domain":"知识","name":"AbsPyramid","organization_count":1,"organizations":["opencompass_hub"],"rank":99,"released":"2024-06-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AbsPyramid"},{"aliases":[],"benchmark_id":"llm-stats-acebench","caveat":"ACEBench is a comprehensive benchmark for evaluating Large Language Models' tool usage capabilities across three primary evaluation types: Normal (basic tool usage scenarios), Special (tool usage with ambiguous or incomplete instructions), and Agent (multi-agent interactions simulating real-world dialogues). The benchmark covers 4,538 APIs across 8 major domains and 68 sub-domains including technology, finance, entertainment, society, health, culture, and environment, supporting both English and Chinese languages.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/acebench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ACEBench","organization_count":1,"organizations":["llm_stats"],"rank":100,"released":"2025-01-22","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/acebench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-06184","caveat":"Generalization in multi-arm collaboration can be studied as composing familiar atomic skills in new ways across arms.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.06184"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ACG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":101,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.06184"},{"aliases":[],"benchmark_id":"claire-radar-2608-09476","caveat":"Self-evolving benchmark of behavioral safety in cowork agents, evaluating risk from execution trajectories across 600 cases, 213 scenarios, 15 risk behaviors, and 48 web-service APIs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09476"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ActBench","organization_count":1,"organizations":["claire_radar"],"rank":102,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09476"},{"aliases":[],"benchmark_id":"opencompass-1319-actionatlas","caveat":"ActionAtlas is a multiple-choice video question answering benchmark, including 934 videos showcasing 580 unique actions across 56 sports, with a total of 1896 actions within choices. ActionAtlas是一个多项选择视频问答基准测试，包括934个视频，展示了56项运动中的580个独特动作，选项共包含1896个动作。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ActionAtlas"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ActionAtlas","organization_count":1,"organizations":["opencompass_hub"],"rank":103,"released":"2024-10-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ActionAtlas"},{"aliases":[],"benchmark_id":"claire-radar-2609-34547","caveat":"Video-capable vision-language models score above 80\\% on popular benchmarks yet struggle with spatial-temporal binding: associating the right action with the right person at the right moment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34547"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ActionLens","organization_count":1,"organizations":["claire_radar"],"rank":104,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34547"},{"aliases":[],"benchmark_id":"claire-radar-2608-04682","caveat":"Active-SWE evaluates coding agents on proactive bug fixing: detecting and fixing multiple bugs without issue reports. It includes 1,663 tasks across six bug categories and eight languages, with stages for recorded bugs, potential bugs, and judge validation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04682"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Active-SWE","organization_count":1,"organizations":["claire_radar"],"rank":105,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04682"},{"aliases":[],"benchmark_id":"claire-radar-2609-24124","caveat":"ActiveArena-Bench evaluates active perception in robotic manipulation through 35 tasks across five categories covering visual exploration and interactive information acquisition.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24124"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"ActiveArena-Bench","organization_count":1,"organizations":["claire_radar"],"rank":106,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24124"},{"aliases":[],"benchmark_id":"claire-radar-2607-10180","caveat":"ActiveFly-Bench is a benchmark for UAV embodied perception, decomposing active perception into three tasks: Aerial Embodied Question Answering (Air-EQA), Observation Behavior Planning (OBP), and Fine-grained Language-guided UAV Control (FLUC). It includes datasets from real-world and simulated outdoor environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10180"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"ActiveFly-Bench","organization_count":1,"organizations":["claire_radar"],"rank":107,"released":"2026-07-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.10180"},{"aliases":[],"benchmark_id":"claire-radar-2605-24043","caveat":"ActiveSciBench evaluates budget-constrained, closed-loop scientific discovery across enzyme-kinetics and gene-regulatory-network tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24043"],"document_share":0.0002723311546840959,"domain":"AI Scientist","name":"ActiveSciBench","organization_count":1,"organizations":["claire_radar"],"rank":108,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.24043"},{"aliases":["An Exam for Active Observers"],"benchmark_id":"claire-radar-2607-16165","caveat":"ActiveVision evaluates whether multimodal large language models can perform repeated, active visual perception across 17 tasks in three categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.16165"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ActiveVision","organization_count":1,"organizations":["claire_radar"],"rank":109,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.16165"},{"aliases":[],"benchmark_id":"llm-stats-activitynet","caveat":"A large-scale video benchmark for human activity understanding. Provides samples from 203 activity classes with an average of 137 untrimmed videos per class and 1.41 activity instances per video, for a total of 849 video hours. The benchmark covers a wide range of complex human activities that are of interest to people in their daily living and can be used to compare algorithms for three scenarios: untrimmed video classification, trimmed activity classification, and activity detection.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/activitynet?top_n=500"],"document_share":0.0002723311546840959,"domain":"video","name":"ActivityNet","organization_count":1,"organizations":["llm_stats"],"rank":110,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/activitynet?top_n=500"},{"aliases":["Understanding Autonomous Driving Datasets by Describing Differences between Image Subsets in Natural Language"],"benchmark_id":"claire-radar-2609-03677","caveat":"Evaluates set-difference captioning on autonomous driving image subsets, with three splits based on annotation filtering, CLIP filtering, and web scraping, yielding natural-language descriptions of differences between target and reference sets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03677"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"AD-Diff Bench","organization_count":1,"organizations":["claire_radar"],"rank":111,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03677"},{"aliases":[],"benchmark_id":"opencompass-1155-ada-leval","caveat":"Ada-LEval is a length-adaptable benchmark for evaluating the long-context understanding\nof LLMs. Ada-LEval includes two challenging subsets, TSort and BestAnswer, which enable\na more reliable evaluation of LLMs’ long context capabilities. Ada-LEval 用于评估大型语言模型（LLMs）对长上下文的理解能力。Ada-LEval 包含两个具有挑战性的子集，TSort 和 BestAnswer，能够更可靠地评估 LLMs 的长上下文能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Ada-LEval"],"document_share":0.0002723311546840959,"domain":"长文本","name":"Ada-LEval","organization_count":1,"organizations":["opencompass_hub"],"rank":112,"released":"2024-06-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Ada-LEval"},{"aliases":[],"benchmark_id":"claire-radar-2606-21144","caveat":"AdaMem-Bench simulates weeks of interaction with week-by-week question answering to evaluate memory policies for personalized long-horizon LLM agents. It measures QA accuracy and memory volume across different models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21144"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AdaMem-Bench","organization_count":1,"organizations":["claire_radar"],"rank":113,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21144"},{"aliases":[],"benchmark_id":"claire-radar-2606-05622","caveat":"Evaluates adaptive planning of LLM agents under progressively disclosed world and user constraints across 307 household tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05622"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"AdaPlanBench","organization_count":1,"organizations":["claire_radar"],"rank":114,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05622"},{"aliases":[],"benchmark_id":"claire-radar-2609-36488","caveat":"Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36488"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AdaptArena","organization_count":1,"organizations":["claire_radar"],"rank":115,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36488"},{"aliases":["Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security"],"benchmark_id":"claire-radar-2607-18063","caveat":"A 21-scenario benchmark evaluating memoryless LLM defenders against adaptive attackers that observe earlier responses and change tactics across multiple rounds.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18063"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Adaptive Adversaries","organization_count":1,"organizations":["claire_radar"],"rank":116,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.18063"},{"aliases":[],"benchmark_id":"claire-radar-2608-26204","caveat":"Computer Use Agents (CUAs) are increasingly deployed to navigate mobile and desktop applications on behalf of users, yet no benchmark comprehensively evaluates whether they can safely interact with visual interfaces while handling ambiguou…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26204"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"ADeptS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":117,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.26204"},{"aliases":[],"benchmark_id":"claire-radar-2610-10047","caveat":"Product-centric advertisement video generation aims to create promotional videos that preserve fine-grained product identity while presenting selling points through coherent multi-shot narratives.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10047"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AdSpark","organization_count":1,"organizations":["claire_radar"],"rank":118,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10047"},{"aliases":[],"benchmark_id":"llm-stats-advancedif","caveat":"AdvancedIF is a rubric-based benchmark measuring complex, multi-turn, and system-prompted instruction following ability, scored with a calibrated LLM judge against per-instruction rubrics.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/advancedif?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"AdvancedIF","organization_count":1,"organizations":["llm_stats"],"rank":119,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/advancedif?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-11849","caveat":"AdvancedMathBench is a benchmark suite for advanced mathematical reasoning, containing ProverBench (296 proof problems) and VerifierBench (888 proof trajectories with expert labels), with an automatic verification pipeline.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11849"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"AdvancedMathBench","organization_count":1,"organizations":["claire_radar"],"rank":120,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11849"},{"aliases":[],"benchmark_id":"claire-radar-2606-24589","caveat":"AdversaBench is an automated LLM red-teaming evaluation that mutates seed prompts and confirms resulting failures with multiple judges and a tiebreaker.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24589"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"AdversaBench","organization_count":1,"organizations":["claire_radar"],"rank":121,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24589"},{"aliases":[],"benchmark_id":"claire-radar-2608-00832","caveat":"AdvPlan-Bench is an offline benchmark for adversarial evaluation of structured plan-generation agents. It uses typed action chains, adversarial response sets, selector diagnostics, and metrics like BLUE-vs-RED advantage and Nash-gap. Includes 150 synthetic scenarios across five planning templates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00832"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AdvPlan-Bench","organization_count":1,"organizations":["claire_radar"],"rank":122,"released":"2026-08-01","source":"claire_radar","url":"https://arxiv.org/abs/2608.00832"},{"aliases":[],"benchmark_id":"claire-radar-2607-14726","caveat":"AE-UAV is an air-to-air event-based UAV tracking benchmark with 178 flight sequences and continuous-time cubic B-spline annotations, supporting evaluation at arbitrary temporal resolutions. It includes multimodal auxiliary data and predefined train/validation/test splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14726"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AE-UAV","organization_count":1,"organizations":["claire_radar"],"rank":123,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14726"},{"aliases":[],"benchmark_id":"opencompass-2452-aecbench","caveat":"AECBench is an open-source benchmark for evaluating LLMs in architecture, engineering, and construction (AEC), covering 23 tasks and about 4,800 samples across five cognitive levels. AECBench 是面向建筑、工程与施工（AEC）领域的大语言模型评测基准，覆盖 5 个认知层级、23 类任务和约 4,800 个样本，用于评估模型在知识记忆、理解、推理、计算与应用方面的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AECBench"],"document_share":0.0002723311546840959,"domain":"科学","name":"AECBench","organization_count":1,"organizations":["opencompass_hub"],"rank":124,"released":"2026-04-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AECBench"},{"aliases":["Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding"],"benchmark_id":"claire-radar-2607-04383","caveat":"AEGBench is a difficulty-stratified evaluation for open-vocabulary audio event grounding, where systems localize queried sound events in time.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04383"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AEGBench","organization_count":1,"organizations":["claire_radar"],"rank":125,"released":"2026-07-05","source":"claire_radar","url":"https://arxiv.org/abs/2607.04383"},{"aliases":[],"benchmark_id":"llm-stats-community-5f95f778-c521-43fa-b80e-6a55465601e3","caveat":null,"document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3A5f95f778-c521-43fa-b80e-6a55465601e3?top_n=500"],"document_share":0.0002723311546840959,"domain":"other","name":"ael_gate_benchmark_cases_template","organization_count":1,"organizations":["llm_stats"],"rank":126,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A5f95f778-c521-43fa-b80e-6a55465601e3?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-16349","caveat":"AeroCopilotBench is a two-tier benchmark for evaluating LLM agents as aviation copilots. Tier-1 uses 1,200 multiple-choice questions for knowledge assessment, while Tier-2 includes 73 procedural tasks in an interactive virtual cockpit environment, with safety-gated evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16349"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AeroCopilotBench","organization_count":1,"organizations":["claire_radar"],"rank":127,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16349"},{"aliases":[],"benchmark_id":"claire-radar-2608-14721","caveat":"AeroGround evaluates vision-language models on aerial-ground collaborative reasoning using a simulated dataset of ~29,000 multimodal observation groups and 2,250 QA instances covering cross-view correspondence, spatial understanding, and reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14721"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"AeroGround","organization_count":1,"organizations":["claire_radar"],"rank":128,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.14721"},{"aliases":[],"benchmark_id":"claire-radar-2609-36915","caveat":"Aerial manipulators extend robotic manipulation into 3D workspaces that are difficult for ground-based robots to access, creating new opportunities for general-purpose manipulation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36915"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"AeroManip-VLA","organization_count":1,"organizations":["claire_radar"],"rank":129,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36915"},{"aliases":[],"benchmark_id":"llm-stats-aethercode","caveat":"AetherCode is a competitive-programming benchmark of olympiad-level algorithmic coding problems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aethercode?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"AetherCode","organization_count":1,"organizations":["llm_stats"],"rank":130,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aethercode?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-24954","caveat":"Evaluates generative meteorological reasoning through 7,732 expert-written forecast discussions paired with AI weather inputs, using metrics for numerical accuracy, style, and grounding.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24954"],"document_share":0.0002723311546840959,"domain":"AI Scientist","name":"AFDBench","organization_count":1,"organizations":["claire_radar"],"rank":131,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24954"},{"aliases":[],"benchmark_id":"claire-radar-2606-14240","caveat":"Affordance20Q is a benchmark for evaluating affordance reasoning in LLMs using a 20-questions game. It comprises 1,009 games over 454 objects and 59 affordances, where models identify a hidden object's affordance by asking yes/no questions about physical properties.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14240"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AFFORDANCE20Q","organization_count":1,"organizations":["claire_radar"],"rank":132,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14240"},{"aliases":[],"benchmark_id":"opencompass-506-afqmc","caveat":"AFQMC is an Ant Financial chinese semantic similarity task, which requires to judge whether two sentences have the same meaning or not. AFQMC一个蚂蚁金服中文语义相似度任务，要求判断两个句子是否具有相同的语义。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AFQMC"],"document_share":0.0002723311546840959,"domain":"语言","name":"AFQMC","organization_count":1,"organizations":["opencompass_hub"],"rank":133,"released":"2020-04-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AFQMC"},{"aliases":["Callability Is Not Operability: Controlled Interface Interventions for LLM Agents"],"benchmark_id":"claire-radar-2608-23628","caveat":"Holds task, backend, initial state, injected failure, agent, and language model fixed while varying the tool interface to measure callability versus operability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.23628"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AFT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":134,"released":"2026-08-23","source":"claire_radar","url":"https://arxiv.org/abs/2608.23628"},{"aliases":[],"benchmark_id":"claire-radar-2607-01152","caveat":"AGC-Bench evaluates artificial general creativity across 78 datasets covering brainstorming, problem solving, STEM, narrative, figurative language, and humor. It uses an agentic harness and a public leaderboard with an open-weight judge model.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01152"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AGC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":135,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.01152"},{"aliases":[],"benchmark_id":"claire-radar-github-kyashrathore-agent-app-benchmark","caveat":"Evaluates GUI application performance for coding agents using deterministic historical session workloads, measuring app start, session switching, memory, and CPU metrics.","document_count":1,"document_ids":["claire_radar:https://github.com/kyashrathore/agent-app-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Agent App Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":136,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/kyashrathore/agent-app-benchmark"},{"aliases":["agent-failure-lab"],"benchmark_id":"claire-radar-github-mostre-agent-failure-lab","caveat":"Evaluates coding agent failures across 12 categories by converting failed sessions into isolated benchmark tasks with verification commands.","document_count":1,"document_ids":["claire_radar:https://github.com/MOSTRE/agent-failure-lab"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Agent Failure Lab","organization_count":1,"organizations":["claire_radar"],"rank":137,"released":"2026-09-18","source":"claire_radar","url":"https://github.com/MOSTRE/agent-failure-lab"},{"aliases":["agent-memory-benchmark"],"benchmark_id":"claire-radar-github-datapace-ai-agent-memory-benchmark","caveat":"Evaluates agent memory systems on a 10-question pilot using LongMemEval, measuring answer accuracy under three judge rules plus token usage, latency, ingestion cost, forgetting curve, and superseded-value handling.","document_count":1,"document_ids":["claire_radar:https://github.com/datapace-ai/agent-memory-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Agent Memory Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":138,"released":"2026-09-08","source":"claire_radar","url":"https://github.com/datapace-ai/agent-memory-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2606-04874","caveat":"Agent Planning Benchmark (APB) is a diagnostic benchmark with 4,209 multimodal cases across 22 domains, evaluating planning capabilities in five settings including tool noise and unsolvable tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04874"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Agent Planning Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":139,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04874"},{"aliases":[],"benchmark_id":"claire-radar-2607-24882","caveat":"File-level retrieval benchmark for coding agents, covering four positive tasks (code2test, comment2context, trace2code, edit2ripple) and a selective-retrieval subset with natural no-gold and counterfactual controls across 25 repositories, 427 samples, with frozen base-commit corpora.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24882"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Agent Retrieval Bench","organization_count":1,"organizations":["claire_radar"],"rank":140,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24882"},{"aliases":[],"benchmark_id":"llm-stats-agent-startup-bench","caveat":"Agent Startup Bench measures AI agents on high-economic-value, startup-style tasks that require autonomous planning and execution to deliver practical, verifiable results.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/agent-startup-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Agent Startup Bench","organization_count":1,"organizations":["llm_stats"],"rank":141,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/agent-startup-bench?top_n=500"},{"aliases":["agent-studio-latency-benchmark"],"benchmark_id":"claire-radar-github-jadenguitarman-agent-studio-latency-benchmark","caveat":"Evaluates latency of a direct Agent Studio integration by running a fixed query set under controlled conditions and reporting timings for Search, Agent Studio, and request-selected index comparisons.","document_count":1,"document_ids":["claire_radar:https://github.com/jadenguitarman/agent-studio-latency-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Agent Studio latency benchmark","organization_count":1,"organizations":["claire_radar"],"rank":142,"released":"2026-09-23","source":"claire_radar","url":"https://github.com/jadenguitarman/agent-studio-latency-benchmark"},{"aliases":["agent-systems-benchmark"],"benchmark_id":"claire-radar-github-martin-beck-agent-systems-benchmark","caveat":"A Linux terminal framework for measuring how AI coding agents scale under concurrent sessions while maintaining task quality, latency, and resource consumption within declared bounds.","document_count":1,"document_ids":["claire_radar:https://github.com/martin-beck/agent-systems-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Agent Systems Benchmark (ASB)","organization_count":1,"organizations":["claire_radar"],"rank":143,"released":"2026-09-06","source":"claire_radar","url":"https://github.com/martin-beck/agent-systems-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-github-titusblair-agent-arena","caveat":"A harness for evaluating multi-agent strategies on financial data tasks, comparing single, fanout, and plan-execute-review approaches with measured cost and confidence intervals.","document_count":1,"document_ids":["claire_radar:https://github.com/titusblair/agent-arena"],"document_share":0.0002723311546840959,"domain":"Agents","name":"agent-arena","organization_count":1,"organizations":["claire_radar"],"rank":144,"released":"2026-09-15","source":"claire_radar","url":"https://github.com/titusblair/agent-arena"},{"aliases":[],"benchmark_id":"claire-radar-github-dkautomation23-agent-hijack-bench","caveat":"Measures how often a local LLM agent obeys an instruction hidden in a website's llms.txt instead of the user's request, using a canary token and substring scoring.","document_count":1,"document_ids":["claire_radar:https://github.com/dkautomation23/agent-hijack-bench"],"document_share":0.0002723311546840959,"domain":"Agents","name":"agent-hijack-bench","organization_count":1,"organizations":["claire_radar"],"rank":145,"released":"2026-09-20","source":"claire_radar","url":"https://github.com/dkautomation23/agent-hijack-bench"},{"aliases":[],"benchmark_id":"claire-radar-github-sentinelden-agent-injection-bench","caveat":"Evaluates prompt-injection robustness of tool-calling agent stacks using a 24-scenario corpus (21 adversarial, 3 control) delivered through channels such as tool results, documents, web pages, filenames and multi-turn dialogue, with deterministic canary, tool-call, exfiltration and refusal success conditions; results report compromise and over-refusal rates per defense.","document_count":1,"document_ids":["claire_radar:https://github.com/sentinelden/agent-injection-bench"],"document_share":0.0002723311546840959,"domain":"Agents","name":"agent-injection-bench","organization_count":1,"organizations":["claire_radar"],"rank":146,"released":"2026-09-11","source":"claire_radar","url":"https://github.com/sentinelden/agent-injection-bench"},{"aliases":[],"benchmark_id":"claire-radar-github-ajbermudezh22-agent-model-bench","caveat":"Evaluates language models on tool calling and strict JSON adherence using 32 cases, scoring exact tool and argument matches and JSON parsing with strict and content-based criteria.","document_count":1,"document_ids":["claire_radar:https://github.com/ajbermudezh22/agent-model-bench"],"document_share":0.0002723311546840959,"domain":"Agents","name":"agent-model-bench","organization_count":1,"organizations":["claire_radar"],"rank":147,"released":"2026-08-21","source":"claire_radar","url":"https://github.com/ajbermudezh22/agent-model-bench"},{"aliases":[],"benchmark_id":"claire-radar-2607-10059","caveat":"AgentAbstain is a paired-task benchmark for evaluating LLM agents' ability to abstain from acting in scenarios such as ambiguity, conflicting constraints, or tool failures. It includes 263 paired tasks across 42 sandbox environments, with a proposed pipeline for generating fresh task instances.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10059"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentAbstain","organization_count":1,"organizations":["claire_radar"],"rank":148,"released":"2026-07-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.10059"},{"aliases":[],"benchmark_id":"opencompass-1242-agentboard","caveat":"AgentBoard is tailored to analytical evaluation of LLM agents. It offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit that features easy assessment of agents for multi-faceted analysis through interactive visualization. AgentBoard专用于LLM Agent的分析评估，它提供了一个精细指标用于捕获增量进步，以及一个全面的评估工具包，能基于交互式可视化评估进行多方面分析。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgentBoard"],"document_share":0.0002723311546840959,"domain":"智能体","name":"AgentBoard","organization_count":1,"organizations":["opencompass_hub"],"rank":149,"released":"2024-06-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AgentBoard"},{"aliases":[],"benchmark_id":"claire-radar-2608-14680","caveat":"AGENTCHAOSBENCH is a dataset of sanitized execution traces from five agentic applications with injected runtime faults, used to evaluate fault detection and localization from telemetry.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14680"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AGENTCHAOSBENCH","organization_count":1,"organizations":["claire_radar"],"rank":150,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.14680"},{"aliases":[],"benchmark_id":"claire-radar-2606-23189","caveat":"AgentCIBench is an executable evaluation harness for testing whether computer-use agents improperly disclose information across application contexts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23189"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentCIBench","organization_count":1,"organizations":["claire_radar"],"rank":151,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23189"},{"aliases":[],"benchmark_id":"claire-radar-2610-05586","caveat":"As Large Language Models (LLMs) gain tool use capabilities such as web search, they can retrieve and cross-reference public information, creating privacy risks beyond memorization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05586"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentDoxx","organization_count":1,"organizations":["claire_radar"],"rank":152,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05586"},{"aliases":[],"benchmark_id":"claire-radar-2609-06972","caveat":"Step-labeled corpus of 12,536 synthetic LLM agent tool-call trajectories in five domains, labeled per step for benign, injection point, hijacked, or failed injection, supporting detection, localization, and attempt-vs-success evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06972"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentDrift","organization_count":1,"organizations":["claire_radar"],"rank":153,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.06972"},{"aliases":[],"benchmark_id":"claire-radar-2606-16723","caveat":"AgentFairBench evaluates demographic disparity in the actions of LLM agents across hiring, lending, and medical triage. It uses synthetic, demographic-neutral profiles in counterfactual matched sets varying name-coded race/gender. Metrics include counterfactual flip rate, mean absolute score difference, action-rate disparity, and tool-invocation disparity, with bootstrap confidence intervals and FDR control.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16723"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentFairBench","organization_count":1,"organizations":["claire_radar"],"rank":154,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16723"},{"aliases":[],"benchmark_id":"claire-radar-2609-32795","caveat":"Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32795"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentHabit","organization_count":1,"organizations":["claire_radar"],"rank":155,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32795"},{"aliases":[],"benchmark_id":"opencompass-1351-agentharm","caveat":"AgentHarm tests the robustness of LLMs to jailbreak attacks. It includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment. AgentHarm用于评估LLM智能体对越狱攻击的鲁棒性，包括110套恶意智能体任务（其中有440个强化任务），涵盖欺诈、网络犯罪和骚扰等11个危害类别。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgentHarm"],"document_share":0.0002723311546840959,"domain":"安全","name":"AgentHarm","organization_count":1,"organizations":["opencompass_hub"],"rank":156,"released":"2024-10-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AgentHarm"},{"aliases":[],"benchmark_id":"opencompass-2061-agenthazard","caveat":"移动端 GUI Agent 通过与设备环境的交互来完成任务，在完成任务的过程中会遇到一些未知或不可信的信息来源，这些信息可能含有攻击性的内容，致使 Agent 无法正常完成任务，甚至对用户的隐私和财产带来危害。本评测集兼具动态执行环境和静态评测数据集，旨在为移动端 GUI Agent 提供一个仿真度高的模拟环境，以评估其在真实场景下执行的行为和安全性。 移动端 GUI Agent 通过与设备环境的交互来完成任务，在完成任务的过程中会遇到一些未知或不可信的信息来源，这些信息可能含有攻击性的内容，致使 Agent 无法正常完成任务，甚至对用户的隐私和财产带来危害。本评测集兼具动态执行环境和静态评测数据集，旨在为移动端 GUI Agent 提供一个仿真度高的模拟环境，以评估其在真实场景下执行的行为和安全性。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgentHazard"],"document_share":0.0002723311546840959,"domain":"多模态","name":"AgentHazard","organization_count":1,"organizations":["opencompass_hub"],"rank":157,"released":"2025-07-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AgentHazard"},{"aliases":[],"benchmark_id":"claire-radar-2605-25707","caveat":"Evaluates the robustness of computer use agents under common environment corruptions such as pop-ups, resolution changes, and competing applications. The benchmark introduces 9 configurable corruptions and evaluates agent performance on desktop tasks using multimodal LLM-based agents, measuring task completion rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25707"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentHijack","organization_count":1,"organizations":["claire_radar"],"rank":158,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25707"},{"aliases":[],"benchmark_id":"claire-radar-2609-34428","caveat":"Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34428"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentHop","organization_count":1,"organizations":["claire_radar"],"rank":159,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34428"},{"aliases":[],"benchmark_id":"claire-radar-2610-11050","caveat":"Computer-use agents are capable of completing complex tasks, increasing the use of automatic judges to determine success, either for training or for evaluation without human involvement.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11050"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentHorizon","organization_count":1,"organizations":["claire_radar"],"rank":160,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11050"},{"aliases":[],"benchmark_id":"claire-radar-2607-29626","caveat":"AgentHPOBench evaluates LLM agents as sequential hyperparameter optimizers across 30 executable ML tasks. Agents observe accumulated configurations, metrics, and logs, then propose the next configuration. Scoring compares agents and conventional HPO baselines under a unified protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29626"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentHPOBench","organization_count":1,"organizations":["claire_radar"],"rank":161,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29626"},{"aliases":[],"benchmark_id":"claire-radar-github-bluebunnyanon-agent-benchmark-2d-maze","caveat":"Evaluates interactive multimodal agents in structured 2D maze environments requiring long-horizon planning, mechanism interaction, and error recovery, scored by mechanism-aware progress.","document_count":1,"document_ids":["claire_radar:https://github.com/bluebunnyanon/agent-benchmark-2d-maze"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Agentic Evaluation in 2D Mazes","organization_count":1,"organizations":["claire_radar"],"rank":162,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/bluebunnyanon/agent-benchmark-2d-maze"},{"aliases":["agentic-infrastructure-change-benchmark"],"benchmark_id":"claire-radar-github-aah20-agentic-infrastructure-change-benchmark","caveat":"Evaluates AI agent proposals for cloud, Kubernetes, and network infrastructure changes across correctness, safety, reliability, economics, efficiency, and evidence using deterministic scenario-based checks.","document_count":1,"document_ids":["claire_radar:https://github.com/AAH20/agentic-infrastructure-change-benchmark"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Agentic Infrastructure ChangeBench","organization_count":1,"organizations":["claire_radar"],"rank":163,"released":"2026-09-05","source":"claire_radar","url":"https://github.com/AAH20/agentic-infrastructure-change-benchmark"},{"aliases":["agentic-netdevops-benchmark"],"benchmark_id":"claire-radar-github-andersballegaard-agentic-netdevops-benchmark","caveat":"A multi-stage environment with a Linux management host and five VyOS routers, requiring agents to complete netdevops tasks from information gathering to evaluation.","document_count":1,"document_ids":["claire_radar:https://github.com/AndersBallegaard/agentic-netdevops-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Agentic NetDevOps Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":164,"released":"2026-09-08","source":"claire_radar","url":"https://github.com/AndersBallegaard/agentic-netdevops-benchmark"},{"aliases":["agentic-security-benchmark"],"benchmark_id":"claire-radar-github-gitayg-agentic-security-benchmark","caveat":"Measures detection and prevention of agentic-AI security products on 286 attack and 875 benign samples across five AMTSO attack vectors via a scoring adapter harness.","document_count":1,"document_ids":["claire_radar:https://github.com/gitayg/agentic-security-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Agentic Security Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":165,"released":"2026-09-07","source":"claire_radar","url":"https://github.com/gitayg/agentic-security-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2610-12183","caveat":"Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensive and limited.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12183"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"AgenticBBO-Bench","organization_count":1,"organizations":["claire_radar"],"rank":166,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.12183"},{"aliases":[],"benchmark_id":"claire-radar-2607-01647","caveat":"Evaluates LLM-based data agents on realistic data science workflows across 15 domains, with fine-grained ground-truth labels and skill-level scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01647"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgenticDataBench","organization_count":1,"organizations":["claire_radar"],"rank":167,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.01647"},{"aliases":[],"benchmark_id":"claire-radar-2606-24026","caveat":"Evaluates language model agents on explaining components of transformer circuits, with 84 semi-synthetic circuits and 163 component-level annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24026"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgenticInterpBench","organization_count":1,"organizations":["claire_radar"],"rank":168,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24026"},{"aliases":["AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents"],"benchmark_id":"claire-radar-2607-02255","caveat":"AgenticSTS is a reproducible long-horizon agent testbed that evaluates bounded, typed-retrieval memory configurations through game outcomes and controlled ablations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02255"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgenticSTS","organization_count":1,"organizations":["claire_radar"],"rank":169,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02255"},{"aliases":["ai-agent-infrastructure-benchmark"],"benchmark_id":"claire-radar-github-aah20-ai-agent-infrastructure-benchmark","caveat":"Evaluates AI agent infrastructure across isolation, identity, network, authority, durability, observability, and unit economics using weighted scores with hard safety gates.","document_count":1,"document_ids":["claire_radar:https://github.com/AAH20/ai-agent-infrastructure-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentInfraBench","organization_count":1,"organizations":["claire_radar"],"rank":170,"released":"2026-09-05","source":"claire_radar","url":"https://github.com/AAH20/ai-agent-infrastructure-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2608-26623","caveat":"LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26623"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentJudgeBench","organization_count":1,"organizations":["claire_radar"],"rank":171,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26623"},{"aliases":[],"benchmark_id":"claire-radar-2607-06624","caveat":"AgentLens evaluates interactive coding agents across entire trajectories, combining formal verification with LLM-written trajectory reviews and side-by-side comparisons to score dimensions such as instruction compliance, tool use, and interaction style.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06624"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentLens","organization_count":1,"organizations":["claire_radar"],"rank":172,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06624"},{"aliases":[],"benchmark_id":"claire-radar-2608-00009","caveat":"AgentMemBench evaluates five long-term memory management strategies for conversational AI agents across three public datasets (LoCoMo, MultiDoc2Dial, MSC), covering multi-session dialogue, document grounding, and persona-grounded chat. Scoring uses retrieval metrics (Recall@k, MRR, nDCG@k), Answer F1, LLM-judge faithfulness, memory footprint, and latency over 491 annotated question turns.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00009"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentMemBench","organization_count":1,"organizations":["claire_radar"],"rank":173,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2608.00009"},{"aliases":[],"benchmark_id":"claire-radar-2610-06406","caveat":"As agents take on long-horizon tasks, users shift from making individual decisions to overseeing autonomous execution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.06406"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentMonBench","organization_count":1,"organizations":["claire_radar"],"rank":174,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.06406"},{"aliases":["aa-agentperf-local"],"benchmark_id":"claire-radar-github-artificialanalysis-aa-agentperf-local","caveat":"Replays recorded agent conversations against an OpenAI-compatible model server and reports throughput and latency, with a default comparable replay of eight recorded agent tasks and 168 model turns.","document_count":1,"document_ids":["claire_radar:https://github.com/ArtificialAnalysis/aa-agentperf-local"],"document_share":0.0002723311546840959,"domain":"Agents","name":"agentperf-local","organization_count":1,"organizations":["claire_radar"],"rank":175,"released":"2026-09-26","source":"claire_radar","url":"https://github.com/ArtificialAnalysis/aa-agentperf-local"},{"aliases":[],"benchmark_id":"claire-radar-2610-04379","caveat":"We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.04379"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentPersonaBench","organization_count":1,"organizations":["claire_radar"],"rank":176,"released":"2026-10-03","source":"claire_radar","url":"https://arxiv.org/abs/2610.04379"},{"aliases":[],"benchmark_id":"claire-radar-2606-02240","caveat":"AgentRedBench evaluates LLM agents against indirect prompt injection and underspecified-authorization attacks across 24 enterprise SaaS integrations. It defines 215 attack scenarios with immutable versioning, and tracks attack success rate (ASR) for models and defenses. Open-source codebase and schemas enable replay.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02240"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentRedBench","organization_count":1,"organizations":["claire_radar"],"rank":177,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02240"},{"aliases":[],"benchmark_id":"claire-radar-2608-15286","caveat":"AgentRelBench measures repeated agent runs in a fixed task suite, computing severity-weighted damage from database state diffs with no LLM judging.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15286"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentRelBench","organization_count":1,"organizations":["claire_radar"],"rank":178,"released":"2026-08-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.15286"},{"aliases":[],"benchmark_id":"opencompass-1778-agentrewardbench","caveat":"AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents. AgentRewardBench contains 1302 trajectories across 5 benchmarks and 4 LLMs. AgentRewardBench 是首个用于评估大型语言模型（LLM）评判者评估网络代理有效性的基准测试。AgentRewardBench 包含来自 5 个基准测试和 4 个大型语言模型的 1302 条轨迹。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgentRewardBench"],"document_share":0.0002723311546840959,"domain":"推理","name":"AgentRewardBench","organization_count":1,"organizations":["opencompass_hub"],"rank":179,"released":"2025-04-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AgentRewardBench"},{"aliases":[],"benchmark_id":"llm-stats-agents-last-exam","caveat":"Agents' Last Exam is a challenging benchmark for AI agents on hard, long-horizon tasks that test sustained reasoning, planning, and tool use, reported with and without tool access.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/agents-last-exam?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Agents' Last Exam","organization_count":1,"organizations":["llm_stats"],"rank":180,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/agents-last-exam?top_n=500"},{"aliases":["Agents' Last Exam"],"benchmark_id":"claire-radar-2606-05405","caveat":"A living benchmark of 1K+ long-horizon professional tasks with verifiable outcomes, covering 55 sub-fields in 13 industry clusters based on O*NET / SOC 2018.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05405"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Agents' Last Exam (ALE)","organization_count":1,"organizations":["claire_radar"],"rank":181,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05405"},{"aliases":[],"benchmark_id":"claire-radar-2607-27294","caveat":"AgentS4D evaluates runtime safety of LLM-based workspace agents across a four-dimensional framework, with 328 risk-injected cases and seven lifecycle checkpoints, measuring unsafe behavior and evidence across six risk-entry sources and nine harms.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27294"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentS4D","organization_count":1,"organizations":["claire_radar"],"rank":182,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27294"},{"aliases":["AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints"],"benchmark_id":"claire-radar-2608-00805","caveat":"A resource-aware agent benchmark with 16 Dockerized task environments that profiles correctness alongside latency, cost, compute, memory, and network use under declared budgets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00805"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"AgentSLABench","organization_count":1,"organizations":["claire_radar"],"rank":183,"released":"2026-08-01","source":"claire_radar","url":"https://arxiv.org/abs/2608.00805"},{"aliases":[],"benchmark_id":"claire-radar-2610-13361","caveat":"As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13361"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentSWE","organization_count":1,"organizations":["claire_radar"],"rank":184,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13361"},{"aliases":[],"benchmark_id":"claire-radar-2608-15127","caveat":"AgentSysBench is a benchmark suite and measurement toolkit for characterizing agentic workloads on LLM serving systems. It includes ten representative agentic applications and unified instrumentation, identifying six properties that distinguish agentic workloads from conventional LLM inference.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15127"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentSysBench","organization_count":1,"organizations":["claire_radar"],"rank":185,"released":"2026-08-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.15127"},{"aliases":[],"benchmark_id":"claire-radar-2609-32915","caveat":"Browser-use agents often carry information in their context as they move between websites.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32915"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgentTell","organization_count":1,"organizations":["claire_radar"],"rank":186,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32915"},{"aliases":[],"benchmark_id":"claire-radar-2610-09944","caveat":"An essential control of AI agents is their ability to manage runtime.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.09944"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentTime","organization_count":1,"organizations":["claire_radar"],"rank":187,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.09944"},{"aliases":[],"benchmark_id":"claire-radar-2609-21386","caveat":"Evaluates MLLM agents on 100 multi-hop video question answering tasks with 26 answer choices per question, requiring spatial, temporal, and causal reasoning over video content.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21386"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentVidBench","organization_count":1,"organizations":["claire_radar"],"rank":188,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21386"},{"aliases":[],"benchmark_id":"claire-radar-2606-24597","caveat":"AgentWorldBench evaluates language world models on simulation fidelity across 7 domains (MCP, Search, Terminal, SWE, Android, Web, OS) using real-world trajectories and rubric-based scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24597"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgentWorldBench","organization_count":1,"organizations":["claire_radar"],"rank":189,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24597"},{"aliases":["AGI Maze Prediction Datasets: A Compact Benchmark for Learning World Dynamics with Transformers"],"benchmark_id":"claire-radar-2609-02339","caveat":"Evaluates predictive models on maze prediction tasks to study world dynamics learning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02339"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AGI Maze Prediction Datasets and Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":190,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02339"},{"aliases":[],"benchmark_id":"llm-stats-agieval","caveat":"A human-centric benchmark for evaluating foundation models on standardized exams including college entrance exams (Gaokao, SAT), law school admission tests (LSAT), math competitions, lawyer qualification tests, and civil service exams. Contains 20 tasks (18 multiple-choice, 2 cloze) designed to assess understanding, knowledge, reasoning, and calculation abilities in real-world academic and professional contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/agieval?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"AGIEval","organization_count":1,"organizations":["llm_stats"],"rank":191,"released":"2023-04-13","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/agieval?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-497-agieval","caveat":"AGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving. This benchmark is derived from 20 official, public, and high-standard admission and qualification exams intended for general human test-takers, such as general college admission tests (e.g., Chinese College Entrance Exam (Gaokao) and American SAT), law school admission tests, math competitions, lawyer qualification tests, and national civil service exams. AGIEval是一个以人为中心的基准测试，专门设计用于评估基础模型在涉及人类认知和问题解决的任务中的一般能力。该基准测试源自20个官方、公开和高标准的入学和资格考试，例如普通大学入学考试（例如中国高考和美国SAT）、法学院入学考试、数学竞赛、律师资格考试以及国家公务员考试","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AGIEval"],"document_share":0.0002723311546840959,"domain":"学科","name":"AGIEval","organization_count":1,"organizations":["opencompass_hub"],"rank":192,"released":"2023-09-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AGIEval"},{"aliases":[],"benchmark_id":"claire-radar-2605-26302","caveat":"Evaluates the reliability of deployed AI agents over extended operational lifetimes. The benchmark organizes agent aging into four mechanisms—compression, interference, revision, and maintenance—and uses temporal dependency graphs and paired counterfactual probes to produce diagnostic profiles of the memory pipeline's write, retrieval, and utilization stages. It includes multiple scenarios and memory policies, with scoring based on task performance over sessions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26302"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AgingBench","organization_count":1,"organizations":["claire_radar"],"rank":193,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26302"},{"aliases":[],"benchmark_id":"opencompass-1753-agmmu","caveat":"A Comprehensive Agricultural Multimodal Understanding and Reasoning Benchmark 农业综合多模态理解和推理基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AgMMU"],"document_share":0.0002723311546840959,"domain":"多模态","name":"AgMMU","organization_count":1,"organizations":["opencompass_hub"],"rank":194,"released":"2025-04-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AgMMU"},{"aliases":[],"benchmark_id":"claire-radar-2606-24526","caveat":"Agora evaluates agentic document reasoning across eight domain collections of 9,664 authentic workplace documents. It includes 362 questions requiring location of sparse evidence and reconciliation of terminology, units, and time conventions. The benchmark is designed to exceed model context windows, necessitating deliberate exploration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24526"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AGORA","organization_count":1,"organizations":["claire_radar"],"rank":195,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24526"},{"aliases":[],"benchmark_id":"claire-radar-2609-08402","caveat":"AGOS-Bench evaluates VLMs on the urban Air-Ground Object Search task where a UAV and UGV coordinate to locate a target vehicle from multi-view visual references, reporting success rate and success weighted by path length.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08402"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"AGOS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":196,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08402"},{"aliases":[],"benchmark_id":"claire-radar-2605-22366","caveat":"AgroTools evaluates tool-augmented multimodal agents in agriculture, with 539 QA instances, 1,097 images, 14 executable tools, and structured tool-use traces for process and outcome evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22366"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgroTools","organization_count":1,"organizations":["claire_radar"],"rank":197,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22366"},{"aliases":["AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding"],"benchmark_id":"claire-radar-2605-22034","caveat":"AgroVG is an agricultural visual grounding benchmark of 10,071 image-query pairs from ten source datasets. It covers box and instance-mask grounding, including queries with no target present.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22034"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AgroVG","organization_count":1,"organizations":["claire_radar"],"rank":198,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22034"},{"aliases":[],"benchmark_id":"claire-radar-2607-02271","caveat":"AGVBench compares vein-recognition augmentation strategies across datasets and model families using accuracy, calibration, and adversarial-robustness considerations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02271"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AGVBench","organization_count":1,"organizations":["claire_radar"],"rank":199,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02271"},{"aliases":[],"benchmark_id":"claire-radar-github-aiondadotcom-ai-sim-benchmark","caveat":"Evaluates autonomous coding agents on a reproducible 3D water-simulation challenge, measuring maximum capability through correctness of the generated simulation, with interventions counted against each run.","document_count":1,"document_ids":["claire_radar:https://github.com/AiondaDotCom/ai-sim-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AI Coding Agent Water Simulation Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":200,"released":"2026-08-21","source":"claire_radar","url":"https://github.com/AiondaDotCom/ai-sim-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-github-enterpilot-ai-gateway-reproducible-benchmark","caveat":"Compares four open-source AI gateway proxies (GoModel, LiteLLM, Portkey, Bifrost) for routing overhead on a fixed AWS c7i.large instance with a mock backend. Measures latency percentiles, throughput, memory, cold start, and image size across streaming and non-streaming workloads.","document_count":1,"document_ids":["claire_radar:https://github.com/ENTERPILOT/ai-gateway-reproducible-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"AI gateway reproducible benchmark","organization_count":1,"organizations":["claire_radar"],"rank":201,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/ENTERPILOT/ai-gateway-reproducible-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2608-03070","caveat":"The AI Security Leaderboard evaluates frontier AI model safeguards against the FAR.AI Minimal Standard for Safeguards across severe misuse requests (CBRNE threats and offensive cybersecurity), testing for universal jailbreaks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03070"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"AI Security Leaderboard","organization_count":1,"organizations":["claire_radar"],"rank":202,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03070"},{"aliases":[],"benchmark_id":"llm-stats-ai2-reasoning-challenge-arc","caveat":"A dataset of 7,787 genuine grade-school level, multiple-choice science questions assembled to encourage research in advanced question-answering. The dataset is partitioned into a Challenge Set and Easy Set, where the Challenge Set contains only questions answered incorrectly by both retrieval-based and word co-occurrence algorithms. Covers multiple scientific domains including biology, physics, earth science, and chemistry, requiring scientific reasoning, causal understanding, and conceptual knowledge beyond simple fact retrieval. Includes a supporting corpus of over 14 million science sentences.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ai2-reasoning-challenge-%28arc%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"AI2 Reasoning Challenge (ARC)","organization_count":1,"organizations":["llm_stats"],"rank":203,"released":"2018-03-14","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2-reasoning-challenge-%28arc%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ai2d","caveat":"AI2D is a dataset of 4,903 illustrative diagrams from grade school natural sciences (such as food webs, human physiology, and life cycles) with over 15,000 multiple choice questions and answers. The benchmark evaluates diagram understanding and visual reasoning capabilities, requiring models to interpret diagrammatic elements, relationships, and structure to answer questions about scientific concepts represented in visual form.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ai2d?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"AI2D","organization_count":1,"organizations":["llm_stats"],"rank":204,"released":"2016-03-24","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ai2d?top_n=500"},{"aliases":["AI4AI Bench","AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement"],"benchmark_id":"ai4ai_bench","caveat":"Ten fixed research projects, with model, harness and reasoning effort jointly evaluated under 4 h exploration on one B300 and up to 12 h fresh execution. The mean normalized score uses 0.1 for the original recipe, not percent solved; some proxy/final evaluations share a corpus.","document_count":1,"document_ids":["model_reports:ai4ai_bench_paper_v1_table3"],"document_share":0.0002723311546840959,"domain":"agents","name":"AI4AI-Bench","organization_count":1,"organizations":["Navers Lab / Einsia.AI"],"rank":205,"released":"2026-08-20","source":"model_reports","url":"https://lab.einsia.ai/ai4ai/"},{"aliases":[],"benchmark_id":"claire-radar-2608-20318","caveat":"AI4AI-Bench evaluates LLM agents on algorithmic design across 10 frozen research repositories. Agents rewrite training algorithms within 4 hours, then scored by fixed evaluators against baseline algorithms, with submissions released for repeatable measurement.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.20318"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"AI4AI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":206,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.20318"},{"aliases":["Auditory Illusion Benchmark for Large Audio Language Models"],"benchmark_id":"claire-radar-2609-02277","caveat":"Evaluates large audio language models on 14,829 forced-choice trials across ten auditory illusions, scoring human-likeness and reality alignment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02277"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"AIB","organization_count":1,"organizations":["claire_radar"],"rank":207,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02277"},{"aliases":[],"benchmark_id":"claire-radar-2606-04867","caveat":"AICompanionBench provides a dataset of 2,123 human-AI companion conversations with safety risk annotations to evaluate LLMs-as-judges for detecting unsafe interactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04867"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"AICompanionBench","organization_count":1,"organizations":["claire_radar"],"rank":208,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04867"},{"aliases":[],"benchmark_id":"opencompass-2396-aidabench","caveat":"AIDABench provides a professional benchmarking suite for AI office data analytics tools, focusing on the core data-processing needs in everyday office workflows, covering high-frequency and reproducible data-processing scenarios commonly seen in real business contexts. AIDABench致力于为AI办公数据分析工具提供专业评测基准，聚焦日常办公中的核心数据处理需求。评测输入以 Excel 文件为主，辅以少量DOC、 PDF、图片及无文件文本等形态，覆盖真实业务中高频、可复现的数据处理场景。核心考察能力包括：数据结构识别、数据清洗、条件筛选、分组聚合、描述统计、排序与排名。任务交付类型包括：问答（Q/A）、文件生成、数据可视化。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AIDABench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"AIDABench","organization_count":1,"organizations":["opencompass_hub"],"rank":209,"released":"2026-02-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AIDABench"},{"aliases":[],"benchmark_id":"llm-stats-aider","caveat":"Aider is a comprehensive code editing benchmark based on 133 practice exercises from Exercism's Python repository, designed to evaluate AI models' ability to translate natural language coding requests into executable code that passes unit tests. The benchmark measures end-to-end code editing capabilities, including GPT's ability to edit existing code and format code changes for automated saving to local files. The Aider Polyglot variant extends this evaluation across 225 challenging exercises spanning C++, Go, Java, JavaScript, Python, and Rust, making it a standard benchmark for assessing multilingual code editing performance in AI research.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aider?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Aider","organization_count":1,"organizations":["llm_stats"],"rank":210,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aider?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-aider-polyglot","caveat":"A coding benchmark that evaluates LLMs on 225 challenging Exercism programming exercises across C++, Go, Java, JavaScript, Python, and Rust. Models receive two attempts to solve each problem, with test error feedback provided after the first attempt if it fails. The benchmark measures both initial problem-solving ability and capacity to edit code based on error feedback, providing an end-to-end evaluation of code generation and editing capabilities across multiple programming languages.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"Aider-Polyglot","organization_count":1,"organizations":["llm_stats"],"rank":211,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-aider-polyglot-edit","caveat":"A challenging multi-language coding benchmark that evaluates models' code editing abilities across C++, Go, Java, JavaScript, Python, and Rust. Contains 225 of Exercism's most difficult programming problems, selected as problems that were solved by 3 or fewer out of 7 top coding models. The benchmark focuses on code editing tasks and measures both correctness of solutions and proper edit format usage. Designed to re-calibrate evaluation scales so top models score between 5-50%.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot-edit?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"Aider-Polyglot Edit","organization_count":1,"organizations":["llm_stats"],"rank":212,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aider-polyglot-edit?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-aime","caveat":"American Invitational Mathematics Examination (AIME) benchmark for evaluating mathematical reasoning capabilities of large language models. Contains 30 challenging mathematical problems from AIME 2024 competition that require multi-step reasoning and advanced mathematical insight. Each problem has an integer answer between 000-999.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aime?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"AIME","organization_count":1,"organizations":["llm_stats"],"rank":213,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aime?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-aime-2024","caveat":"American Invitational Mathematics Examination 2024, consisting of 30 challenging mathematical reasoning problems from AIME I and AIME II competitions. Each problem requires an integer answer between 0-999 and tests advanced mathematical reasoning across algebra, geometry, combinatorics, and number theory. Used as a benchmark for evaluating mathematical reasoning capabilities in large language models at Olympiad-level difficulty.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aime-2024?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"AIME 2024","organization_count":1,"organizations":["llm_stats"],"rank":214,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2024?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-aime-2025","caveat":"Mathematical reasoning","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/aime-2025"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"AIME 2025","organization_count":1,"organizations":["artificial_analysis"],"rank":215,"released":"2025-02-12","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/aime-2025"},{"aliases":[],"benchmark_id":"llm-stats-aime-2025","caveat":"All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aime-2025?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"AIME 2025","organization_count":1,"organizations":["llm_stats"],"rank":216,"released":"2025-02-12","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2025?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-aime-2026","caveat":"All 30 problems from the 2026 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aime-2026?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"AIME 2026","organization_count":1,"organizations":["llm_stats"],"rank":217,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aime-2026?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-22671","caveat":"AIR-BENCH Live is a self-evolving safety benchmark for foundation models, with an automated pipeline that updates risk taxonomy and prompts based on new regulations. It evaluates model safety across multilingual prompts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22671"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AIR-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":218,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.22671"},{"aliases":[],"benchmark_id":"llm-stats-air-bench","caveat":"AIR-Bench 2024 is a safety benchmark grounded in risk categories derived from government regulations and company policies. It evaluates policy-grounded refusal across a broad regulatory and policy-derived harm taxonomy, using category-specific LLM-judge prompts that reward safe engagement rather than only penalizing unsafe responses.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/air-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"AIR-Bench","organization_count":1,"organizations":["llm_stats"],"rank":219,"released":"2024-07-11","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/air-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1069-air-bench","caveat":"AIR-Bench is the first benchmark designed to evaluate the\nability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench 是第一个旨在评估 LALMs 理解各种音频信号（包括人类语言、自然声音和音乐）能力的基准，并进一步评估其以文本形式与人类互动的能力。AIR-Bench 包括两个维度：基础基准和聊天基准。前者由19个任务组成，包含约19,000个单选题，旨在检查LALMs的基本单任务能力。后者包含2,000个开放式问答数据实例，直接评估模型对复杂音频的理解及其遵循指令的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AIR-Bench"],"document_share":0.0002723311546840959,"domain":"知识","name":"AIR-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":220,"released":"2024-02-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AIR-Bench"},{"aliases":[],"benchmark_id":"opencompass-1557-airbench-2024","caveat":"AIR-Bench 2024, the first policy-aligned AI safety benchmark, structures 8 government regulations and 16 corporate policies into four security tiers, with 5,694 diverse prompts spanning these categories. AIR-Bench 2024是首个与新兴政府法规和企业政策相一致的 AI 安全基准， 将 8 项政府法规和 16 项企业政策分解为四级安全分类，涵盖了这些类别的 5,694 个多样化的提示。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AIRBench-2024"],"document_share":0.0002723311546840959,"domain":"安全","name":"AIRBench-2024","organization_count":1,"organizations":["opencompass_hub"],"rank":221,"released":"2024-08-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AIRBench-2024"},{"aliases":[],"benchmark_id":"claire-radar-2606-28049","caveat":"AirGroundBench evaluates multi-view spatial intelligence in multimodal large language models through UAV-UGV collaborative tasks, including 62,000 dual-view multiple-choice questions and 115 navigation episodes across 11 simulated environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.28049"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"AirGroundBench","organization_count":1,"organizations":["claire_radar"],"rank":222,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.28049"},{"aliases":[],"benchmark_id":"claire-radar-2610-10421","caveat":"Goal-oriented Vision-and-Language Navigation (VLN) requires agents to locate and reach targets described in natural language without prescribed routes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10421"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AirGroundVLN","organization_count":1,"organizations":["claire_radar"],"rank":223,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10421"},{"aliases":[],"benchmark_id":"opencompass-1995-airtbench","caveat":"AIRTBench is a benchmark designed to evaluate large language models (LLMs) on their autonomous AI red teaming capabilities. AIRTBench 是一个专为评估大型语言模型（LLM）在“红队”安全任务中的自主攻击能力而设计的评测基准。本基准包含 70 个黑盒 CTF（夺旗赛）挑战，模拟真实 AI/ML 系统漏洞环境，要求模型独立编写 Python 代码进行漏洞发现、利用与夺旗操作，体现其计划、推理与系统操控等综合能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AIRTBench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"AIRTBench","organization_count":1,"organizations":["opencompass_hub"],"rank":224,"released":"2025-06-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AIRTBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-20498","caveat":"AISE-Bench is a benchmark for evaluating multi-step API-using LLM agents in information seeking on academic knowledge graphs. It comprises 1,133 QA pairs with API trajectories, validated parameters, and grounded answers, and evaluates answer quality, reference grounding, API-planning correctness, and execution success.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20498"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"AISE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":225,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.20498"},{"aliases":[],"benchmark_id":"llm-stats-aitz-em","caveat":"Android-In-The-Zoo (AitZ) benchmark for evaluating autonomous GUI agents on smartphones. Contains 18,643 screen-action pairs with chain-of-action-thought annotations spanning over 70 Android apps. Designed to connect perception (screen layouts and UI elements) with cognition (action decision-making) for natural language-triggered smartphone task completion.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/aitz-em?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"AITZ_EM","organization_count":1,"organizations":["llm_stats"],"rank":226,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/aitz-em?top_n=500"},{"aliases":["A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images"],"benchmark_id":"claire-radar-2608-14075","caveat":"ALD/E-ImageMiner is a benchmark of 1,951 scientific figures from 205 publications, expert-annotated for classification, table extraction, summarization, and visual question answering.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14075"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ALD/E-ImageMiner","organization_count":1,"organizations":["claire_radar"],"rank":227,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14075"},{"aliases":[],"benchmark_id":"opencompass-1969-ale-bench","caveat":"ALE-Bench is a benchmark for evaluating AI systems on score-based algorithmic programming contests. ALE-Bench 是一个用于评估 AI 系统在基于分数的算法编程竞赛中的基准测试。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ALE-Bench"],"document_share":0.0002723311546840959,"domain":"代码","name":"ALE-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":228,"released":"2025-06-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ALE-Bench"},{"aliases":["Benchmarking Open-Ended Multi-Agent Coordination in Language Agents"],"benchmark_id":"claire-radar-2606-08340","caveat":"alem is a JAX-based, long-horizon environment for evaluating coordination among language-agent teams through survival, exploration, crafting, trading, and combat.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08340"],"document_share":0.0002723311546840959,"domain":"Agents","name":"alem","organization_count":1,"organizations":["claire_radar"],"rank":229,"released":"2026-06-06","source":"claire_radar","url":"https://arxiv.org/abs/2606.08340"},{"aliases":[],"benchmark_id":"claire-radar-2608-29397","caveat":"Evaluates language-model agents on 240 algorithmic worlds across ten combinatorial optimization families, assessing structured final decisions for feasibility and exact global optimality using certified optima.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29397"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AlgoWorlds","organization_count":1,"organizations":["claire_radar"],"rank":230,"released":"2026-08-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.29397"},{"aliases":[],"benchmark_id":"llm-stats-alignbench","caveat":"AlignBench is a comprehensive multi-dimensional benchmark for evaluating Chinese alignment of Large Language Models. It contains 8 main categories: Fundamental Language Ability, Advanced Chinese Understanding, Open-ended Questions, Writing Ability, Logical Reasoning, Mathematics, Task-oriented Role Play, and Professional Knowledge. The benchmark includes 683 real-scenario rooted queries with human-verified references and uses a rule-calibrated multi-dimensional LLM-as-Judge approach with Chain-of-Thought for evaluation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/alignbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"AlignBench","organization_count":1,"organizations":["llm_stats"],"rank":231,"released":"2023-11-30","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/alignbench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1075-alignbench","caveat":"ALIGNBENCH is a comprehensive multidimensional benchmark for evaluating LLMs’ alignment in Chinese. We tailor a humanin-the-loop data curation pipeline, containing 8 main categories, 683 real-scenario rooted queries and corresponding human verified references. AlignBench 是一个用于评估中文大语言模型对齐性能的全面、多维度的评测基准。AlignBench 构建了人类参与的数据构建流程，来保证评测数据的动态更新。AlignBench 采用多维度、规则校准的模型评价方法（LLM-as-Judge），并且结合思维链（Chain-of-Thought）生成对模型回复的多维度分析和最终的综合评分，增强了评测的高可靠性和可解释性。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AlignBench"],"document_share":0.0002723311546840959,"domain":"理解","name":"AlignBench","organization_count":1,"organizations":["opencompass_hub"],"rank":232,"released":"2024-08-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AlignBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-14573","caveat":"The evaluation object is unclear from the provided information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14573"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Alipay-PIBench","organization_count":1,"organizations":["claire_radar"],"rank":233,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14573"},{"aliases":[],"benchmark_id":"claire-radar-2607-23332","caveat":"A paired benchmark tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in an abstract text-based formulation and a code-construction task.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23332"],"document_share":0.0002723311546840959,"domain":"Finance","name":"AllocBench","organization_count":1,"organizations":["claire_radar"],"rank":234,"released":"2026-07-25","source":"claire_radar","url":"https://arxiv.org/abs/2607.23332"},{"aliases":[],"benchmark_id":"claire-radar-github-jbpayton-almanac","caveat":"Evaluates agent memory systems on generated multi-month conversation lives, testing time, plans, provenance, absence, tasks, and hygiene via 215 questions.","document_count":1,"document_ids":["claire_radar:https://github.com/jbpayton/almanac"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Almanac","organization_count":1,"organizations":["claire_radar"],"rank":235,"released":"2026-09-25","source":"claire_radar","url":"https://github.com/jbpayton/almanac"},{"aliases":[],"benchmark_id":"claire-radar-2606-05531","caveat":"Evaluates vision-language models on six cognitive levels (Remember to Create) with bilingual English-Arabic image-question-answer tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05531"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Almieyar-Oryx-BloomBench","organization_count":1,"organizations":["claire_radar"],"rank":236,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05531"},{"aliases":[],"benchmark_id":"llm-stats-alpacaeval-2-0","caveat":"AlpacaEval 2.0 is a length-controlled automatic evaluator for instruction-following language models that uses GPT-4 Turbo to assess model responses against a baseline. It evaluates models on 805 diverse instruction-following tasks including creative writing, classification, programming, and general knowledge questions. The benchmark achieves 0.98 Spearman correlation with ChatBot Arena while being fast (< 3 minutes) and affordable (< $10 in OpenAI credits). It addresses length bias in automatic evaluation through length-controlled win-rates and uses weighted scoring based on response quality.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/alpacaeval-2.0?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"AlpacaEval 2.0","organization_count":1,"organizations":["llm_stats"],"rank":237,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/alpacaeval-2.0?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-github-hervemottaran-alzheimer-classification-benchmark","caveat":"Comparative evaluation of 11 classifiers for Alzheimer's disease detection on a synthetic health dataset, using KNIME workflows and a fixed experimental framework with sensitivity as primary metric.","document_count":1,"document_ids":["claire_radar:https://github.com/hervemottaran/alzheimer-classification-benchmark"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"alzheimer-classification-benchmark","organization_count":1,"organizations":["claire_radar"],"rank":238,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/hervemottaran/alzheimer-classification-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-00641","caveat":"Evaluates aerial manipulation policy learning with a modular simulation suite covering embodiments, 12 tasks, disturbances, controllers, and baseline algorithms.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00641"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":239,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.00641"},{"aliases":[],"benchmark_id":"opencompass-1280-ambrosia","caveat":"AMBROSIA is a new benchmark for recognizing and interpreting ambiguous requests in text-to-SQL. It contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries. AMBROSIA是识别和解释text-to-SQL中歧义请求的新基准，其中包含三种不同类型的歧义（范围歧义、附件歧义和模糊性）问题、它们的解释和相应的SQL查询。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AMBROSIA"],"document_share":0.0002723311546840959,"domain":"推理","name":"AMBROSIA","organization_count":1,"organizations":["opencompass_hub"],"rank":240,"released":"2024-06-27","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AMBROSIA"},{"aliases":[],"benchmark_id":"llm-stats-amc-2022-23","caveat":"American Mathematics Competition problems from the 2022-23 academic year, consisting of multiple-choice mathematics competition problems designed for high school students. These problems require advanced mathematical reasoning, problem-solving strategies, and mathematical knowledge covering topics like algebra, geometry, number theory, and combinatorics. The benchmark is derived from the official AMC competitions sponsored by the Mathematical Association of America.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/amc-2022-23?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"AMC_2022_23","organization_count":1,"organizations":["llm_stats"],"rank":241,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/amc-2022-23?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-15191","caveat":"AmchiBias is a benchmark for measuring socio-cultural stereotypical bias for Goan identity groups, with 313 minimal pairs across eight sociodemographic dimensions in English and Devanagari Konkani.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15191"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AmchiBias","organization_count":1,"organizations":["claire_radar"],"rank":242,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2606.15191"},{"aliases":["Amharic-Tokenization-Benchmark"],"benchmark_id":"claire-radar-github-bedr-ux-amharic-tokenization-benchmark","caveat":"Evaluates tokenizer fertility and fragmentation for Amharic versus English using 10 parallel sentence pairs, reporting fertility ratio and characters per token for standard, multilingual, and Amharic-specialized LLM tokenizers.","document_count":1,"document_ids":["claire_radar:https://github.com/bedr-ux/Amharic-Tokenization-Benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Amharic-Tokenization-Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":243,"released":"2026-09-15","source":"claire_radar","url":"https://github.com/bedr-ux/Amharic-Tokenization-Benchmark"},{"aliases":["AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis"],"benchmark_id":"claire-radar-2605-30599","caveat":"An open-source medical-unlearning benchmark containing 70,560 question-answer pairs from 8,820 patient notes across 11 disease categories, with factual-recall and clinical-reasoning evaluations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30599"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"AMNESIA","organization_count":1,"organizations":["claire_radar"],"rank":244,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30599"},{"aliases":[],"benchmark_id":"llm-stats-amo-bench","caveat":"AMO Bench is an olympiad-level mathematics benchmark that evaluates advanced mathematical problem-solving and multi-step reasoning on competition-style problems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/amo-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"AMO Bench","organization_count":1,"organizations":["llm_stats"],"rank":245,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/amo-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-huggingface-djalexj-amo-bench-p","caveat":"AMO-Bench-P is a 39-problem subset of AMO-Bench with parser-graded answers (numbers, sets, variables), excluding description-type answers that require an LLM judge.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/djalexj/amo-bench-p"],"document_share":0.0002723311546840959,"domain":"General AI","name":"amo-bench-p","organization_count":1,"organizations":["claire_radar"],"rank":246,"released":"2026-08-20","source":"claire_radar","url":"https://huggingface.co/datasets/djalexj/amo-bench-p"},{"aliases":[],"benchmark_id":"claire-radar-2607-25518","caveat":"AMPBench-MT evaluates antimicrobial peptide prediction across binary recognition, species-conditioned potency regression, and endpoint-specific safety readouts (hemolysis, toxicity, selectivity) under a sequence-homology-controlled protocol. It includes 13 source databases and multiple task configurations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25518"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"AMPBench-MT","organization_count":1,"organizations":["claire_radar"],"rank":247,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25518"},{"aliases":[],"benchmark_id":"opencompass-1966-amsbench","caveat":"AMSbench is a benchmark of ~8000 questions to evaluate multi-modal LLMs on analog/mixed-signal circuit tasks like schematic recognition, analysis, and design. AMSbench 是一个包含约8000道题目的基准测试集，用于评估多模态大语言模型在模拟/混合信号电路任务中的表现，包括识图、分析与设计。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AMSbench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"AMSbench","organization_count":1,"organizations":["opencompass_hub"],"rank":248,"released":"2025-06-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AMSbench"},{"aliases":["analog-design-bench"],"benchmark_id":"claire-radar-huggingface-arcadia-2026-analog-design-bench","caveat":"Fifty analog and mixed-signal circuit design tasks in the SkyWater SKY130 process where an agent delivers a SPICE netlist that a hidden ngspice verifier checks against every specification.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/Arcadia-2026/analog-design-bench"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"Analog Design Bench","organization_count":1,"organizations":["claire_radar"],"rank":249,"released":"2026-09-26","source":"claire_radar","url":"https://huggingface.co/datasets/Arcadia-2026/analog-design-bench"},{"aliases":[],"benchmark_id":"claire-radar-2608-14320","caveat":"AnchorBench evaluates anchoring effects in LLMs across multiple pathways and anchor relevance, measuring answer shifts through controlled prompts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14320"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AnchorBench","organization_count":1,"organizations":["claire_radar"],"rank":250,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14320"},{"aliases":[],"benchmark_id":"claire-radar-2608-27169","caveat":"Ancient Chinese artifact text recognition is fundamental to heritage digitization, and benchmarks for ancient texts are essential for evaluating current model capabilities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27169"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Ancient-Bench","organization_count":1,"organizations":["claire_radar"],"rank":251,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27169"},{"aliases":[],"benchmark_id":"llm-stats-android-control-high-em","caveat":"Android device control benchmark using high exact match evaluation metric for assessing agent performance on mobile interface tasks","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/android-control-high-em?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Android Control High_EM","organization_count":1,"organizations":["llm_stats"],"rank":252,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-high-em?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-android-control-low-em","caveat":"Android control benchmark evaluating autonomous agents on mobile device interaction tasks with low exact match scoring criteria","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/android-control-low-em?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Android Control Low_EM","organization_count":1,"organizations":["llm_stats"],"rank":253,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/android-control-low-em?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-androidbench","caveat":"AndroidBench evaluates coding agents on Android application development tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/androidbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"AndroidBench","organization_count":1,"organizations":["llm_stats"],"rank":254,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/androidbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-27761","caveat":"AndroidDaily evaluates mobile GUI agents on 350 daily-use tasks across 94 closed-source Android applications. Automatic scoring is based on a three-tiered guideline system (operational obligations, output quality, negative constraints), with step-level diagnostic judgments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27761"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"AndroidDaily","organization_count":1,"organizations":["claire_radar"],"rank":255,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27761"},{"aliases":[],"benchmark_id":"llm-stats-androidworld","caveat":"AndroidWorld evaluates an agent's ability to operate in real Android GUI environments, completing multi-step tasks by perceiving screen content and executing touch/type actions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/androidworld?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"AndroidWorld","organization_count":1,"organizations":["llm_stats"],"rank":256,"released":"2024-05-23","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-androidworld-sr","caveat":"AndroidWorld Success Rate (SR) benchmark - A dynamic benchmarking environment for autonomous agents operating on Android devices. Evaluates agents on 116 programmatic tasks across 20 real-world Android apps using multimodal inputs (screen screenshots, accessibility trees, and natural language instructions). Measures success rate of agents completing tasks like sending messages, creating calendar events, and navigating mobile interfaces. Published at ICLR 2025. Best current performance: 30.6% success rate (M3A agent) vs 80.0% human performance.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/androidworld-sr?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"AndroidWorld_SR","organization_count":1,"organizations":["llm_stats"],"rank":257,"released":"2024-05-23","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/androidworld-sr?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-32740","caveat":"Intraoperative anesthesia requires systems to interpret evolving multimodal evidence, recommend timely management, and revise decisions as patient states change, yet existing benchmarks usually isolate perception or single-point reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32740"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"AnesTRACE","organization_count":1,"organizations":["claire_radar"],"rank":258,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32740"},{"aliases":[],"benchmark_id":"opencompass-1876-aneumo","caveat":"Based on 427 real aneurysm geometries, we synthesized 10,660 3D shapes via controlled deformation to simulate aneurysm evolution. CFD computations were performed on each shape under eight steady-state mass flow conditions, generating a total of 85,280 blood flow dynamics data covering key parameters Based on 427 real aneurysm geometries, we synthesized 10,660 3D shapes via controlled deformation to simulate aneurysm evolution. CFD computations were performed on each shape under eight steady-state mass flow conditions, generating a total of 85,280 blood flow dynamics data covering key parameters","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Aneumo"],"document_share":0.0002723311546840959,"domain":"多模态","name":"Aneumo","organization_count":1,"organizations":["opencompass_hub"],"rank":259,"released":"2025-05-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Aneumo"},{"aliases":[],"benchmark_id":"claire-radar-2606-28593","caveat":"Animation2Code evaluates temporal visual reasoning in video-to-code generation. It includes 1,069 web animation videos with corresponding HTML/CSS/JavaScript implementations, and uses appearance and temporal similarity metrics to assess model performance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.28593"],"document_share":0.0002723311546840959,"domain":"Code","name":"Animation2Code","organization_count":1,"organizations":["claire_radar"],"rank":260,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.28593"},{"aliases":[],"benchmark_id":"claire-radar-2609-23345","caveat":"Evaluates provenance detection for anime images across Real, Inpainting, and Text2Image categories using statistical features derived from five semantic dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23345"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AniPrO-SFD-Bench","organization_count":1,"organizations":["claire_radar"],"rank":261,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23345"},{"aliases":[],"benchmark_id":"claire-radar-2607-25911","caveat":"AnnoBench evaluates visualization annotation generation across four representation formats, five chart description conditions, and two prompt specification levels. It uses a VLM-as-a-judge protocol aligned with human assessment to score annotation quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25911"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AnnoBench","organization_count":1,"organizations":["claire_radar"],"rank":262,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25911"},{"aliases":[],"benchmark_id":"claire-radar-github-abdullahuseyinli-dot-annomi-counselling-dialogue-analysis","caveat":"Evaluates therapist-behaviour classification on the AnnoMI counselling-dialogue corpus using transcript-grouped splits, comparing elastic-net and RoBERTa models with metrics including macro-F1 and accuracy.","document_count":1,"document_ids":["claire_radar:https://github.com/abdullahuseyinli-dot/annomi-counselling-dialogue-analysis"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"AnnoMI Counselling Dialogue Analysis","organization_count":1,"organizations":["claire_radar"],"rank":263,"released":"2026-08-21","source":"claire_radar","url":"https://github.com/abdullahuseyinli-dot/annomi-counselling-dialogue-analysis"},{"aliases":[],"benchmark_id":"claire-radar-2608-03700","caveat":"AntiSkillBench is a benchmark for evaluating privacy leakage and impersonation risks in persona-skills. It includes 7,500 persona-grounded dialogue traces and evaluates across three skill-distillation strategies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03700"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AntiSkillBench","organization_count":1,"organizations":["claire_radar"],"rank":264,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03700"},{"aliases":["Any2Poster: Any-Source Poster Generation Across Modalities and Domains"],"benchmark_id":"claire-radar-2606-02915","caveat":"A benchmark for poster generation from eight source modalities and five domains, scored with quiz-based fidelity probes and VLM judgments of visual quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02915"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Any2Poster Bench","organization_count":1,"organizations":["claire_radar"],"rank":265,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02915"},{"aliases":[],"benchmark_id":"claire-radar-2609-34139","caveat":"Mobile GUI agents deployed in real settings must work across different applications that support the same functionality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34139"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"AnyAppBench","organization_count":1,"organizations":["claire_radar"],"rank":266,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34139"},{"aliases":[],"benchmark_id":"claire-radar-2607-02269","caveat":"AnyGroundBench evaluates video grounding in vision-language models across five specialized domains (animal, industry, sports, surgery, public security) with spatio-temporal annotations. It provides training and test splits per domain for zero-shot and in-context learning evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02269"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AnyGroundBench","organization_count":1,"organizations":["claire_radar"],"rank":267,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02269"},{"aliases":[],"benchmark_id":"claire-radar-2606-21147","caveat":"AOR-Bench is a benchmark of 3,000 pseudo-harmful audio samples across six categories, designed to evaluate over-refusal in large audio language models. It assesses whether models incorrectly reject benign audio queries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21147"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AOR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":268,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21147"},{"aliases":[],"benchmark_id":"llm-stats-apex","caveat":"Apex is a challenging frontier reasoning benchmark testing advanced multi-step problem solving across difficult STEM and logical tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/apex?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Apex","organization_count":1,"organizations":["llm_stats"],"rank":269,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/apex?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-27189","caveat":"APEX-Accounting evaluates frontier models on expert-authored accounting and bookkeeping work conducted across simulated accounting systems and supporting files.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27189"],"document_share":0.0002723311546840959,"domain":"General AI","name":"APEX-Accounting","organization_count":1,"organizations":["claire_radar"],"rank":270,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27189"},{"aliases":[],"benchmark_id":"llm-stats-apex-agents","caveat":"APEX-Agents is a benchmark evaluating AI agents on long horizon professional tasks that require sustained reasoning, planning, and execution across complex multi-step workflows.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/apex-agents?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"APEX-Agents","organization_count":1,"organizations":["llm_stats"],"rank":271,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-agents?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-apex-agents-aa","caveat":"Long-horizon agentic tasks","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/apex-agents-aa"],"document_share":0.0002723311546840959,"domain":"agentic","name":"APEX-Agents-AA","organization_count":1,"organizations":["artificial_analysis"],"rank":272,"released":null,"source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/apex-agents-aa"},{"aliases":[],"benchmark_id":"claire-radar-2610-13623","caveat":"Professional agents increasingly work through command-line interfaces and multiple tools over shared workspaces containing files, messages, and business applications.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13623"],"document_share":0.0002723311546840959,"domain":"Agents","name":"APEX-Harm","organization_count":1,"organizations":["claire_radar"],"rank":273,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13623"},{"aliases":[],"benchmark_id":"llm-stats-apex-swe","caveat":"APEX-SWE evaluates AI agents on software engineering tasks requiring multi-step coding, debugging, and verification.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/apex-swe?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"APEX-SWE","organization_count":1,"organizations":["llm_stats"],"rank":274,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/apex-swe?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-34973","caveat":"Full-duplex voice agents can now listen, speak, use tools, and act during spoken interactions, but fluent dialogue does not guarantee correct completion of delegated professional workflows.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34973"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"APEX-Voice","organization_count":1,"organizations":["claire_radar"],"rank":275,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34973"},{"aliases":[],"benchmark_id":"claire-radar-2609-24165","caveat":"APEXA-Bench evaluates multi-agent LLM automation on 58 synchrotron data reduction tasks organized by a four-class physical-consequence taxonomy, with grading against NIST-traceable lattice constants in a cross-detector slice.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24165"],"document_share":0.0002723311546840959,"domain":"Agents","name":"APEXA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":276,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24165"},{"aliases":[],"benchmark_id":"llm-stats-api-bank","caveat":"A comprehensive benchmark for tool-augmented LLMs that evaluates API planning, retrieval, and calling capabilities. Contains 314 tool-use dialogues with 753 API calls across 73 API tools, designed to assess how effectively LLMs can utilize external tools and overcome obstacles in tool leveraging.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/api-bank?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"API-Bank","organization_count":1,"organizations":["llm_stats"],"rank":277,"released":"2023-04-14","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/api-bank?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-29128","caveat":"Evaluates tool-using agents on 467 long-horizon REST API tasks with deterministic provenance-sensitive grading, measuring reliability, efficiency, robustness, and other engineering capabilities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29128"],"document_share":0.0002723311546840959,"domain":"General AI","name":"APIFlow-Bench","organization_count":1,"organizations":["claire_radar"],"rank":278,"released":"2026-08-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.29128"},{"aliases":[],"benchmark_id":"claire-radar-github-liningbest-apitest","caveat":"Not applicable; excluded.","document_count":1,"document_ids":["claire_radar:https://github.com/liningbest/apitest"],"document_share":0.0002723311546840959,"domain":"General AI","name":"apitest","organization_count":1,"organizations":["claire_radar"],"rank":279,"released":"2026-09-13","source":"claire_radar","url":"https://github.com/liningbest/apitest"},{"aliases":[],"benchmark_id":"claire-radar-2609-37559","caveat":"To serve as real-world personal assistants, streaming video models need persistent memory that retains past experiences for later use.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37559"],"document_share":0.0002723311546840959,"domain":"General AI","name":"APM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":280,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37559"},{"aliases":[],"benchmark_id":"opencompass-1093-apps","caveat":"APPS is a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. APPS 是一个代码生成评测基准，该评测基准测量模型根据任意自然语言规范生成令人满意的 Python 代码的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/APPS"],"document_share":0.0002723311546840959,"domain":"代码","name":"APPS","organization_count":1,"organizations":["opencompass_hub"],"rank":281,"released":"2021-11-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/APPS"},{"aliases":[],"benchmark_id":"claire-radar-2609-07712","caveat":"Controllable simulated apps for mobile GUI agent evaluation, containing 557 tasks across 17 Chinese and English apps with deterministic backend data and outcome-based verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07712"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"AppSim-Bench","organization_count":1,"organizations":["claire_radar"],"rank":282,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07712"},{"aliases":[],"benchmark_id":"claire-radar-2607-24663","caveat":"APS-Bench is a 50-question QA dataset with auditable gold answers for evaluating retrieval-augmented generation over scientific facility operations knowledge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24663"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"APS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":283,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24663"},{"aliases":[],"benchmark_id":"claire-radar-2610-02770","caveat":"Document databases such as MongoDB are core infrastructure for modern applications, and natural-language interfaces to them---text-to-MQL---would let non-experts query complex, semi-structured data without mastering the query language.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.02770"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AptMQL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":284,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.02770"},{"aliases":[],"benchmark_id":"opencompass-1116-aqua-rat","caveat":"AQUA-RAT contains the algebraic word problems. The dataset consists of about 100,000 algebraic word problems with natural language rationales. AQUA-RAT 包含代数文字问题。该数据集由约 100,000 道带有自然语言推理的代数文字问题组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AQUA-RAT"],"document_share":0.0002723311546840959,"domain":"数学","name":"AQUA-RAT","organization_count":1,"organizations":["opencompass_hub"],"rank":285,"released":"2017-10-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AQUA-RAT"},{"aliases":["ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs"],"benchmark_id":"claire-radar-2608-12788","caveat":"ARAC-Bench evaluates auto-research systems on alignment and completeness of research trajectories across proposal, experiment, and synthesis stages using rubrics derived from academic cognition skills.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12788"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ARAC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":286,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.12788"},{"aliases":[],"benchmark_id":"claire-radar-2607-20764","caveat":"ArbiGraph is a benchmark generator for evaluating tool-assisted language agents' context management via scalable task graphs with exact automatic verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20764"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ArbiGraph","organization_count":1,"organizations":["claire_radar"],"rank":287,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.20764"},{"aliases":[],"benchmark_id":"llm-stats-arc","caveat":"The Abstraction and Reasoning Corpus (ARC) is a benchmark designed to measure human-like general fluid intelligence through grid-based reasoning tasks. It consists of 800 tasks (400 training, 400 evaluation) where each task presents input-output grids that require understanding abstract patterns and transformations. Test-takers must produce exactly correct output grids for all test inputs in a task to solve it, with 3 trials allowed per test input. ARC aims to enable fair comparisons of general intelligence between AI systems and humans using priors designed to be as close as possible to innate human priors.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Arc","organization_count":1,"organizations":["llm_stats"],"rank":288,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arc?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-arc-agi","caveat":"The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) is a benchmark designed to test general intelligence and abstract reasoning capabilities through visual grid-based transformation tasks. Each task consists of 2-5 demonstration pairs showing input grids transformed into output grids according to underlying rules, with test-takers required to infer these rules and apply them to novel test inputs. The benchmark uses colored grids (up to 30x30) with 10 discrete colors/symbols, designed to measure human-like general fluid intelligence and skill-acquisition efficiency with minimal prior knowledge.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-agi?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ARC-AGI","organization_count":1,"organizations":["llm_stats"],"rank":289,"released":"2019-11-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-arc-agi-v2","caveat":"ARC-AGI-2 is an upgraded benchmark for measuring abstract reasoning and problem-solving abilities in AI systems through visual grid transformation tasks. It evaluates fluid intelligence via input-output grid pairs (1x1 to 30x30) using colored cells (0-9), requiring models to identify underlying transformation rules from demonstration examples and apply them to test cases. Designed to be easy for humans but challenging for AI, focusing on core cognitive abilities like spatial reasoning, pattern recognition, and compositional generalization.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ARC-AGI v2","organization_count":1,"organizations":["llm_stats"],"rank":290,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-v2?top_n=500"},{"aliases":["ARC-AGI-3","ARC-AGI 3"],"benchmark_id":"arc_agi_3","caveat":"Interactive multi-step format, so the agent harness is part of the measurement. Scores are low and spread wide, which makes small absolute gaps look larger than they are.","document_count":1,"document_ids":["model_reports:anthropic_claude_opus_5_system_card"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ARC-AGI-3","organization_count":1,"organizations":["Anthropic"],"rank":291,"released":"2026-01-15","source":"model_reports","url":"https://arcprize.org/arc-agi/3/"},{"aliases":[],"benchmark_id":"llm-stats-arc-agi-3","caveat":"ARC-AGI-3 is the third-generation Abstraction and Reasoning Corpus benchmark, an interactive-reasoning evaluation designed to measure fluid, novel problem-solving ability that remains far from saturated for frontier models.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-3?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ARC-AGI-3","organization_count":1,"organizations":["llm_stats"],"rank":292,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-agi-3?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-arc-c","caveat":"The AI2 Reasoning Challenge (ARC) Challenge Set is a multiple-choice question-answering benchmark containing grade-school level science questions that require advanced reasoning capabilities. ARC-C specifically contains questions that were answered incorrectly by both retrieval-based and word co-occurrence algorithms, making it a particularly challenging subset designed to test commonsense reasoning abilities in AI systems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-c?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ARC-C","organization_count":1,"organizations":["llm_stats"],"rank":293,"released":"2018-03-14","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-c?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-502-arc-c","caveat":"The AI2’s Reasoning Challenge (ARC) dataset is a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9. The dataset is split in two partitions: Easy and Challenge, where the latter partition contains the more difficult questions that require reasoning. Most of the questions have 4 answer choices, with <1% of all the questions having either 3 or 5 answer choices. AI2的推理挑战（ARC）数据集是一个多项选择问题回答数据集，包含了从三年级到九年级的科学考试中提取的问题。该数据集分为两个部分：简单和挑战，其中后者包含了需要推理能力的更难的问题。大多数问题有4个答案选项，仅有不到1％的问题有3个或5个答案选项。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ARC-c"],"document_share":0.0002723311546840959,"domain":"学科","name":"ARC-c","organization_count":1,"organizations":["opencompass_hub"],"rank":294,"released":"2018-03-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ARC-c"},{"aliases":[],"benchmark_id":"llm-stats-arc-e","caveat":"ARC-E (AI2 Reasoning Challenge - Easy Set) is a subset of grade-school level, multiple-choice science questions that requires knowledge and reasoning capabilities. Part of the AI2 Reasoning Challenge dataset containing 5,197 questions that test scientific reasoning and factual knowledge. The Easy Set contains questions that are answerable by retrieval-based and word co-occurrence algorithms, making them more accessible than the Challenge Set.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arc-e?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ARC-E","organization_count":1,"organizations":["llm_stats"],"rank":295,"released":"2018-03-14","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arc-e?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-503-arc-e","caveat":"The AI2’s Reasoning Challenge (ARC) dataset is a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9. The dataset is split in two partitions: Easy and Challenge, where the latter partition contains the more difficult questions that require reasoning. Most of the questions have 4 answer choices, with <1% of all the questions having either 3 or 5 answer choices. AI2的推理挑战（ARC）数据集是一个多项选择问题回答数据集，包含了从三年级到九年级的科学考试中提取的问题。该数据集分为两个部分：简单和挑战，其中后者包含了需要推理能力的更难的问题。大多数问题有4个答案选项，仅有不到1％的问题有3个或5个答案选项。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ARC-e"],"document_share":0.0002723311546840959,"domain":"学科","name":"ARC-e","organization_count":1,"organizations":["opencompass_hub"],"rank":296,"released":"2018-03-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ARC-e"},{"aliases":[],"benchmark_id":"llm-stats-arcagi2","caveat":"ARC-AGI-2 is the second-generation Abstraction and Reasoning Corpus benchmark measuring fluid, general reasoning and abstraction.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arcagi2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ArcAGI2","organization_count":1,"organizations":["llm_stats"],"rank":297,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arcagi2?top_n=500"},{"aliases":["ARCGym: Benchmarking Deep Reinforcement Learning in Autonomous Robotic Colonoscopy"],"benchmark_id":"claire-radar-2609-22803","caveat":"Benchmark for deep reinforcement learning in autonomous robotic colonoscopy, focusing on image-based navigation in clinically derived deformable colon anatomies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.22803"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"ARCGym","organization_count":1,"organizations":["claire_radar"],"rank":298,"released":"2026-09-19","source":"claire_radar","url":"https://arxiv.org/abs/2609.22803"},{"aliases":[],"benchmark_id":"claire-radar-2609-34047","caveat":"Multimodal models increasingly interpret visual environments, but their ability to recognize the same building across photographs, floor plans, elevations, sections, and renderings remains poorly characterized.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34047"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ARCH-B","organization_count":1,"organizations":["claire_radar"],"rank":299,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34047"},{"aliases":[],"benchmark_id":"claire-radar-2608-06772","caveat":"ArchEGraph evaluates geometry-topology-physics aligned building energy modeling through graph reconstruction and topology-informed load prediction tasks, with standardized protocols and generalization experiments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06772"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ArchEGraph","organization_count":1,"organizations":["claire_radar"],"rank":300,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.06772"},{"aliases":[],"benchmark_id":"llm-stats-arena-hard","caveat":"Arena-Hard-Auto is an automatic evaluation benchmark for instruction-tuned LLMs consisting of 500 challenging real-world prompts curated by BenchBuilder. It includes open-ended software engineering problems, mathematical questions, and creative writing tasks. The benchmark uses LLM-as-a-Judge methodology with GPT-4.1 and Gemini-2.5 as automatic judges to approximate human preference. Arena-Hard achieves 98.6% correlation with human preference rankings and provides 3x higher separation of model performances compared to MT-Bench, making it highly effective for distinguishing between models of similar quality.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arena-hard?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Arena Hard","organization_count":1,"organizations":["llm_stats"],"rank":301,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-arena-hard-v2","caveat":"Arena-Hard-Auto v2 is a challenging benchmark consisting of 500 carefully curated prompts sourced from Chatbot Arena and WildChat-1M, designed to evaluate large language models on real-world user queries. The benchmark covers diverse domains including open-ended software engineering problems, mathematics, creative writing, and technical problem-solving. It uses LLM-as-a-Judge for automatic evaluation, achieving 98.6% correlation with human preference rankings while providing 3x higher separation of model performances compared to MT-Bench. The benchmark emphasizes prompt specificity, complexity, and domain knowledge to better distinguish between model capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arena-hard-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Arena-Hard v2","organization_count":1,"organizations":["llm_stats"],"rank":302,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arena-hard-v2?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2075-arena-hard-auto","caveat":"Arena-Hard-Auto is an automated benchmark for instruction-tuned LLMs, designed to efficiently approximate human preferences. Arena-Hard-Auto 是一个用于评估 LLM 的基准，自动甄选 500 条高难度开放式提示，从模型区分度、人类偏好一致性与提示质量三维度进行严苛评测。依托 BenchBuilder 管道、主题建模与 LLM 裁判，实现众包数据→筛选→评分的全自动闭环。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Arena-Hard-Auto"],"document_share":0.0002723311546840959,"domain":"安全","name":"Arena-Hard-Auto","organization_count":1,"organizations":["opencompass_hub"],"rank":303,"released":"2024-04-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Arena-Hard-Auto"},{"aliases":[],"benchmark_id":"claire-radar-2609-38409","caveat":"Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable rewards, particularly in mathematics, code, and formal logic.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38409"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ArgGYM","organization_count":1,"organizations":["claire_radar"],"rank":304,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.38409"},{"aliases":[],"benchmark_id":"opencompass-2370-argusinspection","caveat":"As Multimodal Large Language Models (MLLMs) continue to evolve, their cognitive and reasoning capabilities have seen remarkable progress. However, challenges in visual fine-grained perception and commonsense causal inference persist. This paper introduces Argus Inspection, a multimodal benchmark wit As Multimodal Large Language Models (MLLMs) continue to evolve, their cognitive and reasoning capabilities have seen remarkable progress. However, challenges in visual fine-grained perception and commonsense causal inference persist. This paper introduces Argus Inspection, a multimodal benchmark wit","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ArgusInspection"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ArgusInspection","organization_count":1,"organizations":["opencompass_hub"],"rank":305,"released":"2025-10-27","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ArgusInspection"},{"aliases":[],"benchmark_id":"llm-stats-arkitscenes","caveat":"ARKitScenes evaluates 3D scene understanding and spatial reasoning in AR/VR contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arkitscenes?top_n=500"],"document_share":0.0002723311546840959,"domain":"spatial_reasoning","name":"ARKitScenes","organization_count":1,"organizations":["llm_stats"],"rank":306,"released":"2021-11-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arkitscenes?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-24481","caveat":"A benchmark for robot manipulation policies evaluated on a fleet of low-cost SO-101 cells. It compares 7 policies across 12 tasks in single-arm and bimanual configurations, with 2,518 policy rollouts and 600 reference demonstrations, all labeled successful, suboptimal, or failure. Data is released in LeRobot and RoboMeter formats.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24481"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"ArmnetBench","organization_count":1,"organizations":["claire_radar"],"rank":307,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24481"},{"aliases":[],"benchmark_id":"claire-radar-2606-26793","caveat":"ART-SafeBench evaluates attacks against multimodal agentic RAG systems across four attack surfaces, using attack success rate and query cost as comparable outcomes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26793"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"ART-SafeBench","organization_count":1,"organizations":["claire_radar"],"rank":308,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.26793"},{"aliases":[],"benchmark_id":"claire-radar-2608-03358","caveat":"ArtECulture is a benchmark for culture-conditioned visual emotion understanding with 6,792 artworks and culture-specific labels across three cultures. It evaluates MLLMs on predicting cultural emotion perception and explaining rationale.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03358"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ArtECulture","organization_count":1,"organizations":["claire_radar"],"rank":309,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03358"},{"aliases":[],"benchmark_id":"llm-stats-artifacts-bench","caveat":"Artifacts Bench evaluates a model's ability to generate visual code artifacts, measuring the quality of generated interactive and visual front-end outputs from natural-language requests.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/artifacts-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"frontend_development","name":"Artifacts Bench","organization_count":1,"organizations":["llm_stats"],"rank":310,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/artifacts-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2052-artifactsbench","caveat":"ArtifactsBench is a benchmark with 1,825 tasks for evaluating LLM-generated visual and interactive code. It addresses the gap of traditional benchmarks that focus only on algorithmic correctness by assessing visual fidelity and user interaction. rtifactsBench 是一个专注于弥合传统代码评测中“视觉-交互”鸿沟的新型基准。它旨在全面评估大语言模型（LLM）生成动态可视化与交互式代码的能力，而非仅仅考核算法正确性。\n该基准包含1825个真实世界的任务，并开创了一套自动化多模态评估流程：系统会自动渲染代码、捕捉其视觉与交互行为，再由一个多模态大模型（MLLM）依据详细清单进行评分。该流程与人类专家判断的一致性高达94.4%，证明了其高度可靠性。\nArtifactsBench 已将数据集、评估框架及工具链完全开源，为社区提供了一个可扩展且精准的工具，以推动能创造丰富用户体验的新一代生成模型的发展。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ArtifactsBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ArtifactsBench","organization_count":1,"organizations":["opencompass_hub"],"rank":311,"released":"2025-07-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ArtifactsBench"},{"aliases":[],"benchmark_id":"llm-stats-artificial-analysis","caveat":"Artificial Analysis benchmark evaluates AI models across quality, speed, and pricing dimensions, providing a composite assessment of model capabilities for real-world usage.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/artificial-analysis?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"Artificial Analysis","organization_count":1,"organizations":["llm_stats"],"rank":312,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/artificial-analysis?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-30030","caveat":"A validation framework for assessing whether synthetic populations support intended analyses.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30030"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Artificial Societies Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":313,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.30030"},{"aliases":[],"benchmark_id":"arxivmath","caveat":"Competition-style mathematics drawn from arXiv; test-time compute budget is part of the result.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"math","name":"ArXivMath","organization_count":1,"organizations":["Tencent"],"rank":314,"released":null,"source":"model_reports","url":"https://matharena.ai/arxivmath"},{"aliases":[],"benchmark_id":"llm-stats-arxivmath","caveat":"ArXivMath is a final-answer benchmark of research-level mathematics maintained by MathArena. Problems are extracted monthly from recent arXiv paper abstracts, then filtered through automated and manual checks to ensure they are self-contained, non-trivial, and verifiable. Because problems are drawn from active research, the benchmark is more realistic and more closely connected to mathematical research than contest or olympiad benchmarks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/arxivmath?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"ArXivMath","organization_count":1,"organizations":["llm_stats"],"rank":315,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/arxivmath?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1114-asdiv","caveat":"ASDiv is a new MWP corpus that contains diverse lexicon patterns with wide problem type coverage. Each problem provides consistent equations and answers. It is further annotated with the corresponding problem type and grade level. ASDiv 是一个新的数学文字问题（MWP）语料库，包含多样的词汇模式，覆盖广泛的问题类型。每个问题提供对应的方程和答案。它进一步标注了相应的问题类型和年级水平，可用于测试系统的能力，并指明问题的难度等级。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ASDiv"],"document_share":0.0002723311546840959,"domain":"数学","name":"ASDiv","organization_count":1,"organizations":["opencompass_hub"],"rank":316,"released":"2020-07-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ASDiv"},{"aliases":["ASI-Bench","ASI Bench"],"benchmark_id":"asi_bench","caveat":"First benchmark that jointly evaluates general intelligence, innovation, and autonomous execution, via 60 project-level scientific research tasks spanning 11 domains, progressively withdrawing human guidance. Very new (arXiv 2608.17271, code at github.com/apexin-ai/ASI-Bench) and intended to probe superintelligence-level behaviour, so high variance across seeds and scaffolds should be expected before any vendor adopts it. Only the seed-31415 instances are public; the seed-42 references stay private for official leaderboard scoring, so a locally computed score and a leaderboard score are not the same measurement. This registry's maintainer is one of the benchmark's authors.","document_count":1,"document_ids":["model_reports:asi_bench_results_table"],"document_share":0.0002723311546840959,"domain":"science","name":"ASI-Bench","organization_count":1,"organizations":["Apexin AI"],"rank":317,"released":"2026-08-18","source":"model_reports","url":"https://arxiv.org/abs/2608.17271"},{"aliases":[],"benchmark_id":"claire-radar-2608-17271","caveat":"ASI-Bench evaluates AI systems on 60 project-level scientific research tasks across 11 domains, with four guidance levels B1-B4 measuring autonomous execution and innovation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17271"],"document_share":0.0002723311546840959,"domain":"AI Scientist","name":"ASI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":318,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17271"},{"aliases":[],"benchmark_id":"claire-radar-2607-03831","caveat":"ASOB-Bench evaluates diffusion classifiers along three bias dimensions: attribute binding, size-order bias, and background dependency, using newly constructed datasets and extending existing frameworks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03831"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ASOB-Bench","organization_count":1,"organizations":["claire_radar"],"rank":319,"released":"2026-07-04","source":"claire_radar","url":"https://arxiv.org/abs/2607.03831"},{"aliases":[],"benchmark_id":"claire-radar-2608-31111","caveat":"ASPIRE evaluates LLM agents and models on self-evolution from vague natural-language goals, using a hidden expert-authored set of 520 items across six goals in an interactive environment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.31111"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ASPIRE","organization_count":1,"organizations":["claire_radar"],"rank":320,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.31111"},{"aliases":[],"benchmark_id":"claire-radar-github-anuragkokate09-asr-model-benchmark","caveat":"Compares Whisper Small, Faster-Whisper Small, and Wav2Vec2 Base on 20 clean and 20 noisy LibriSpeech samples using WER, inference latency, RTF, and memory usage on CPU.","document_count":1,"document_ids":["claire_radar:https://github.com/Anuragkokate09/asr-model-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ASR Model Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":321,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/Anuragkokate09/asr-model-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-11877","caveat":"AssayBench-Loop is a large-scale benchmark for adaptive hit discovery using 1,389 CRISPR screens across five phenotype categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11877"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AssayBench-Loop","organization_count":1,"organizations":["claire_radar"],"rank":322,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11877"},{"aliases":[],"benchmark_id":"claire-radar-2609-16075","caveat":"AssemblyGrid v1 is a benchmark for repeated multi-robot production that combines explicit process progression, decentralized observations, material transfer, temporary multi-robot coalitions, productive concurrency, and geometry-dependent feasibility within one task-level formulation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16075"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"AssemblyGrid v1","organization_count":1,"organizations":["claire_radar"],"rank":323,"released":"2026-09-13","source":"claire_radar","url":"https://arxiv.org/abs/2609.16075"},{"aliases":[],"benchmark_id":"claire-radar-2605-27472","caveat":"Evaluates LLM generation of SystemVerilog assertions from structured design specifications and RTL, across two tasks: bug-prevention and bug-hunting. The benchmark includes 83 real-world designs with golden and mutated RTL, and assesses syntactic validity, formal provability, coverage, and mutation-based bug detection.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27472"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"AssertLLM2","organization_count":1,"organizations":["claire_radar"],"rank":324,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27472"},{"aliases":[],"benchmark_id":"opencompass-1991-assetopsbench","caveat":"AssetOpsBench is a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) and AI agents in complex asset operation and maintenance tasks. AssetOpsBench 是一个专注于评估大语言模型（LLM）和智能体在资产运维领域复杂任务中实际表现的多维度评测基准。该基准旨在检验模型在工业场景下的任务规划、多步推理、工具调用、安全合规性以及领域知识理解等核心能力，覆盖设备维护、异常诊断、风险评估等典型运维场景。测试集包含 1,000 个高质量样本，涉及 5 大类任务和 20 余种细分领域，数据来源于真实运维手册、工单记录及专家验证案例。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AssetOpsBench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"AssetOpsBench","organization_count":1,"organizations":["opencompass_hub"],"rank":325,"released":"2025-06-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AssetOpsBench"},{"aliases":[],"benchmark_id":"claire-radar-2605-24573","caveat":"AstroMind evaluates LLM reasoning about spacecraft behavior across intent inference, maneuver parameter estimation, and threat assessment, using physics-grounded simulations with realistic sensing noise. Metrics capture semantic correctness and quantitative consistency under physical constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24573"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AstroMind","organization_count":1,"organizations":["claire_radar"],"rank":326,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24573"},{"aliases":[],"benchmark_id":"claire-radar-2606-05920","caveat":"Evaluates code agents on 50 web tasks with underspecified intent and multi-round refinement via browser-rendered behavior.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05920"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Asuka-Bench","organization_count":1,"organizations":["claire_radar"],"rank":327,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05920"},{"aliases":["Towards Detecting AI-Assisted Responses in Online Surveys"],"benchmark_id":"claire-radar-2609-17317","caveat":"Dataset for detecting AI-assisted responses in online surveys.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17317"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ASURRE","organization_count":1,"organizations":["claire_radar"],"rank":328,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17317"},{"aliases":[],"benchmark_id":"claire-radar-2609-32662","caveat":"Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32662"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AsynCodeBench","organization_count":1,"organizations":["claire_radar"],"rank":329,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32662"},{"aliases":[],"benchmark_id":"claire-radar-2605-27995","caveat":"AsyncTool is a benchmark for evaluating asynchronous function calling in multi-task tool-use environments. It presents multiple tasks with simulated tool response latency, assessing step-level tool-call correctness, sub-task completion, and task-level end-to-end success, along with efficiency-oriented metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27995"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"AsyncTool","organization_count":1,"organizations":["claire_radar"],"rank":330,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27995"},{"aliases":[],"benchmark_id":"claire-radar-2605-22645","caveat":"AtelierEval evaluates prompting proficiency of humans and MLLMs for text-to-image systems via 360 tasks, using a skill-based agentic evaluator (AtelierJudge) that scores prompt-image pairs subjectively and objectively.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22645"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AtelierEval","organization_count":1,"organizations":["claire_radar"],"rank":331,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22645"},{"aliases":[],"benchmark_id":"llm-stats-community-ed90e889-4678-4fbd-98ab-0e654f4bf35e","caveat":null,"document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3Aed90e889-4678-4fbd-98ab-0e654f4bf35e?top_n=500"],"document_share":0.0002723311546840959,"domain":"other","name":"atlas","organization_count":1,"organizations":["llm_stats"],"rank":332,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3Aed90e889-4678-4fbd-98ab-0e654f4bf35e?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-18726","caveat":"AtmosCoder-Bench is an execution-grounded benchmark for LLMs on atmospheric science computation, with 436 problems and 3,910 variants, grading by executing code solutions against ground truth.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.18726"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AtmosCoder-Bench","organization_count":1,"organizations":["claire_radar"],"rank":333,"released":"2026-08-19","source":"claire_radar","url":"https://arxiv.org/abs/2608.18726"},{"aliases":[],"benchmark_id":"claire-radar-2606-16826","caveat":"ATOM-Bench evaluates atomic skills and compositional generalization in manipulation policies across 30 atomic tasks and 24 held-out compositional tasks, using paired single-arm and dual-arm robot tracks, with 3,000 human demonstrations and evaluation rollout data released.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16826"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"ATOM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":334,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16826"},{"aliases":[],"benchmark_id":"claire-radar-2607-14541","caveat":"Evaluates coding agents on generating GPU kernels from PyTorch references across 30 operators and 440 shapes derived from production inference traces. Scoring uses a three-stage evaluator measuring compile success, numerical correctness, and speed-of-light (SOL) efficiency against a cached roofline. The benchmark includes 4 DSL backends and supports multi-vendor GPUs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14541"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Atrex-Bench","organization_count":1,"organizations":["claire_radar"],"rank":335,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14541"},{"aliases":["kaggle-art-benchmark"],"benchmark_id":"claire-radar-github-mziqudhd92-kaggle-art-benchmark","caveat":"ART evaluates whether language models separate attacker-reachable vulnerabilities from patched twins, safe code, and vacuous noise, scoring vulnerability, patched, and filler accuracy via a weighted ART score and a Twin Gap metric.","document_count":1,"document_ids":["claire_radar:https://github.com/mziqudhd92/kaggle-art-benchmark"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"Attacker-Reachable Sink Triage (ART)","organization_count":1,"organizations":["claire_radar"],"rank":336,"released":"2026-09-24","source":"claire_radar","url":"https://github.com/mziqudhd92/kaggle-art-benchmark"},{"aliases":[],"benchmark_id":"llm-stats-attaq","caveat":"AttaQ is a unique dataset containing adversarial examples in the form of questions designed to provoke harmful or inappropriate responses from large language models. The benchmark evaluates safety vulnerabilities by using specialized clustering techniques that analyze both the semantic similarity of input attacks and the harmfulness of model responses, facilitating targeted improvements to model safety mechanisms.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/attaq?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"AttaQ","organization_count":1,"organizations":["llm_stats"],"rank":337,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/attaq?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-16127","caveat":"AuAu combines psychometric instruments, vignettes, and realistic prompts to assess authoritarian tendencies in LLM responses, measuring sub-concepts like aggression, submission, and conventionalism.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16127"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AuAu","organization_count":1,"organizations":["claire_radar"],"rank":338,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16127"},{"aliases":[],"benchmark_id":"opencompass-1835-audiojailbreak","caveat":"LAMs face jailbreak risks. AJailBench, our new benchmark, reveals leading LAMs lack robustness. Subtle audio perturbations significantly degrade their safety. We release AJailBench for research. LAM 面临越狱风险。我们新的基准测试 AJailBench 揭示，领先的 LAM 缺乏稳健性。细微的音频干扰会显著降低其安全性。我们发布 AJailBench 进行研究。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AudioJailbreak"],"document_share":0.0002723311546840959,"domain":"多模态","name":"AudioJailbreak","organization_count":1,"organizations":["opencompass_hub"],"rank":339,"released":"2025-05-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AudioJailbreak"},{"aliases":[],"benchmark_id":"opencompass-1908-audiotrust","caveat":"AudioTrust is a comprehensive trust evaluation framework for Audio Large Language Models (ALLMs) that effectively reveals potential risks in six dimensions: fairness, hallucination, security, privacy, robustness, and authentication. It aggregates over 4,420 real-world audio/text data samples, coveri AudioTrust针对Audio Large Language Models（ALLMs）的全方位可信评估框架，有效揭示音频大模型在公平性、幻觉、安全、隐私、鲁棒性和身份验证六大维度的潜在风险。汇集4,420+条真实场景音频/文本数据，覆盖日常对话、紧急呼叫、语音助手等18种实验设置，设计9项音频特定评测指标，构建自动化评估流水线。主要发现：闭源模型在鲁棒性和安全防护上表现更佳，开源模型对隐私和公平性仍存盲区；多数ALLMs对性别、口音、年龄等敏感属性存在系统性偏见。期待研究者基于AudioTrust继续优化音频大模型，共同推动更安全、可信的AI音频生态发展！","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AudioTrust"],"document_share":0.0002723311546840959,"domain":"多模态","name":"AudioTrust","organization_count":1,"organizations":["opencompass_hub"],"rank":340,"released":"2025-06-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AudioTrust"},{"aliases":[],"benchmark_id":"claire-radar-2609-06771","caveat":"A multilingual authorship representation benchmark with 428,150 documents across 10 languages, 9 primary genres, and four length buckets, supporting same-author retrieval and verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06771"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AuthBench","organization_count":1,"organizations":["claire_radar"],"rank":341,"released":"2026-09-06","source":"claire_radar","url":"https://arxiv.org/abs/2609.06771"},{"aliases":[],"benchmark_id":"claire-radar-2608-01679","caveat":"AuthMem-Bench evaluates authority collapse in persistent memory for LLM agents. It uses a paired benchmark holding claims and tasks fixed while varying source authority, measuring write-time collapse, authorization errors, and automatic authority preservation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01679"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AuthMem-Bench","organization_count":1,"organizations":["claire_radar"],"rank":342,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01679"},{"aliases":[],"benchmark_id":"claire-radar-2606-13104","caveat":"AuthorityBench evaluates how citation-based authority signals affect epistemic behavior in LLMs using 220,564 prompts across four domains, with a 2x2 factorial design crossing claim veracity and citation veracity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13104"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AuthorityBench","organization_count":1,"organizations":["claire_radar"],"rank":343,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13104"},{"aliases":[],"benchmark_id":"claire-radar-2605-25382","caveat":"AuthTrace is a diagnostic benchmark for evidence construction in thematically dense single-author corpora, with fan-in annotations and pack-level protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25382"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AuthTrace","organization_count":1,"organizations":["claire_radar"],"rank":344,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25382"},{"aliases":[],"benchmark_id":"opencompass-2078-autoadvexbench","caveat":"AutoAdvExBench is a benchmark designed to evaluate large language models' (LLMs) ability to autonomously exploit adversarial example defenses, directly measuring LLMs' success on tasks regularly performed by machine learning security experts. AutoAdvExBench 是一个评估大型语言模型（LLMs）自主利用对抗性样本防御能力的基准，直接衡量LLMs在机器学习安全专家任务上的成功率。它主要评估模型理解学术论文、代码实现及生成对抗性攻击的能力。测试集包含75个对抗性样本防御实现。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AutoAdvExBench"],"document_share":0.0002723311546840959,"domain":"安全","name":"AutoAdvExBench","organization_count":1,"organizations":["opencompass_hub"],"rank":345,"released":"2025-03-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AutoAdvExBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-35025","caveat":"Evaluates agents' ability to autonomously author executable training tasks that meet acceptance criteria, scored via gate, difficulty, and quality terms.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35025"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AutoDataBench","organization_count":1,"organizations":["claire_radar"],"rank":346,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35025"},{"aliases":[],"benchmark_id":"claire-radar-2609-40097","caveat":"Existing auto-research benchmarks often entangle multiple sources of improvement, including training frameworks, hyperparameters, compute budgets, and data, making it difficult to attribute why one frontier agent outperforms another to spe…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.40097"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AutoDataBench","organization_count":1,"organizations":["claire_radar"],"rank":347,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.40097"},{"aliases":[],"benchmark_id":"claire-radar-2606-05080","caveat":"AutoLab evaluates frontier models on long-horizon closed-loop optimization tasks across system optimization, CUDA kernel optimization, model development, and puzzle challenges, with 36 expert-curated tasks and a strict wall-clock budget.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05080"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AutoLab","organization_count":1,"organizations":["claire_radar"],"rank":348,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05080"},{"aliases":[],"benchmark_id":"llm-stats-autologi","caveat":"AutoLogi is an automated method for synthesizing open-ended logic puzzles to evaluate reasoning abilities of Large Language Models. The benchmark addresses limitations of existing multiple-choice reasoning evaluations by featuring program-based verification and controllable difficulty levels. It includes 1,575 English and 883 Chinese puzzles, enabling more reliable evaluation that better distinguishes models' reasoning capabilities across languages.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/autologi?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"AutoLogi","organization_count":1,"organizations":["llm_stats"],"rank":349,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/autologi?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-automationbench","caveat":"AutomationBench is a tool-use benchmark that evaluates AI agents on automating real-world workflows, testing their ability to orchestrate tools and complete multi-step automation tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/automationbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"AutomationBench","organization_count":1,"organizations":["llm_stats"],"rank":350,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-automationbench-aa","caveat":"Agentic SaaS workflows","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/automationbench-aa"],"document_share":0.0002723311546840959,"domain":"agentic","name":"AutomationBench-AA","organization_count":1,"organizations":["artificial_analysis"],"rank":351,"released":null,"source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/automationbench-aa"},{"aliases":[],"benchmark_id":"llm-stats-automationbench-aa","caveat":"AutomationBench-AA is Artificial Analysis's independently run version of AutomationBench, covering 657 real-world SaaS workflow tasks across 40 simulated applications (e.g. Gmail, Slack, Salesforce, HubSpot). It scores the share of objectives an agent completes without violating business guardrails, using a private held-out task set.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/automationbench-aa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"AutomationBench-AA","organization_count":1,"organizations":["llm_stats"],"rank":352,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/automationbench-aa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-01961","caveat":"AutoMedBench evaluates autonomous AI agents on end-to-end medical-AI research tasks spanning segmentation, image enhancement, VQA, report generation, and lesion detection. Tasks follow a five-stage workflow (Plan, Setup, Validate, Inference, Submit), with scoring based on both final task performance and stage-level rubric scores.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01961"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AutoMedBench","organization_count":1,"organizations":["claire_radar"],"rank":353,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01961"},{"aliases":[],"benchmark_id":"claire-radar-2609-32245","caveat":"Partial differential equations (PDEs) are essential for modeling complex physical systems, and neural solvers have recently emerged as powerful data-driven tools for numerically solving them.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32245"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AutoPDEBench","organization_count":1,"organizations":["claire_radar"],"rank":354,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32245"},{"aliases":[],"benchmark_id":"claire-radar-2609-18123","caveat":"A benchmark and measurement protocol for agent auto-tuning of LLM serving engines, featuring a proposal-evaluation loop with correctness checks, fair timing, profiling, and anti-cheating gates to ensure trustworthy results.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18123"],"document_share":0.0002723311546840959,"domain":"Agents","name":"AutoTuneBench","organization_count":1,"organizations":["claire_radar"],"rank":355,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18123"},{"aliases":[],"benchmark_id":"claire-radar-2608-11216","caveat":"AutoWorldModel-Bench evaluates coding agents on autonomously improving a base world model under fixed compute budget across eight game environments. Uses structured-state representation, with held-out test split and closed-loop iteration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11216"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AutoWorldModel-Bench","organization_count":1,"organizations":["claire_radar"],"rank":356,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.11216"},{"aliases":[],"benchmark_id":"claire-radar-2609-34854","caveat":"Multimodal foundation models are increasingly used for evaluating and generating user interfaces (UIs), often producing seemingly reasonable aesthetic judgments and visually plausible pages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34854"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AUV-Bench","organization_count":1,"organizations":["claire_radar"],"rank":357,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34854"},{"aliases":[],"benchmark_id":"opencompass-1396-av-odyssey-bench","caveat":"AV-Odyssey Bench. This benchmark encompasses 26 different tasks and 4,555 carefully crafted problems, each incorporating text, visual, and audio components. All data are newly collected and annotated by humans, not from any existing audio-visual dataset. AV-Odyssey Bench. This benchmark encompasses 26 different tasks and 4,555 carefully crafted problems, each incorporating text, visual, and audio components. All data are newly collected and annotated by humans, not from any existing audio-visual dataset.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AV-Odyssey-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"AV-Odyssey-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":358,"released":"2024-12-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AV-Odyssey-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2609-25864","caveat":"Audio-visual object removal benchmark with objective metrics and MLLM-based evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25864"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AV-Remove-Bench","organization_count":1,"organizations":["claire_radar"],"rank":359,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.25864"},{"aliases":[],"benchmark_id":"claire-radar-2609-06991","caveat":"Proposed safety benchmark for text-to-audio-video generation, with a four-axis, 13-category taxonomy and 5,200 manually reviewed prompts, evaluated under Full-AV, Video-Only, and Audio-Only views.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06991"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AV-SafetyBench","organization_count":1,"organizations":["claire_radar"],"rank":360,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.06991"},{"aliases":[],"benchmark_id":"claire-radar-2607-00726","caveat":"AV-SyncBench evaluates audio-visual synchronization by separating temporal and semantic consistency. It contains 3,269 videos and 38,390 samples across 10 scenarios and 5 tasks, with data filtered and verified for on-screen sound sources.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00726"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AV-SyncBench","organization_count":1,"organizations":["claire_radar"],"rank":361,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.00726"},{"aliases":[],"benchmark_id":"claire-radar-2605-24183","caveat":"AvalancheBench evaluates enterprise data agents on latent world recovery, scoring analytical understanding of segments, drivers, temporal events, and relationships from generated observations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24183"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AvalancheBench","organization_count":1,"organizations":["claire_radar"],"rank":362,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.24183"},{"aliases":[],"benchmark_id":"claire-radar-2608-09638","caveat":"Avalon-ToM-Bench evaluates fine-grained theory of mind in LLMs using a 2x2 taxonomy of epistemic/motivational reasoning crossed with inference/action, via human-crafted perspective-constrained queries from The Resistance: Avalon.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09638"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Avalon-ToM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":363,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09638"},{"aliases":[],"benchmark_id":"claire-radar-2605-24652","caveat":"AVBench evaluates audio-video generative models across ten fine-grained dimensions covering visual quality, audio quality, and cross-modal consistency for human-centric scenarios. It uses specialized evaluators trained via preference learning and provides continuous scores from prediction confidence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24652"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AVBench","organization_count":1,"organizations":["claire_radar"],"rank":364,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24652"},{"aliases":["AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward"],"benchmark_id":"claire-radar-2608-06930","caveat":"Evaluates atomic-level details in audiovisual captions using a specialized benchmark and metric, with associated dataset AVCap-100K and model AVCap.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06930"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AVCap-Bench","organization_count":1,"organizations":["claire_radar"],"rank":365,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.06930"},{"aliases":[],"benchmark_id":"claire-radar-2607-24821","caveat":"AVE-Compass evaluates audio-visual editing models on 145 source videos and 196 instructions with 2,688 checklist items, scoring Instruction Following, Fidelity Preserving, Realism, and Editing Intent via MLLM judging and automated metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24821"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AVE-Compass","organization_count":1,"organizations":["claire_radar"],"rank":366,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.24821"},{"aliases":[],"benchmark_id":"claire-radar-2606-07643","caveat":"AVI-Bench evaluates omni-multimodal LLMs on audio-visual tasks across perception, understanding, and reasoning stages, with an extension probing primitive audio-visual sensation using unfamiliar low-semantic stimuli.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07643"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AVI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":367,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.07643"},{"aliases":["When Writing Style Drifts: Benchmarking Authorship Verification under Distribution Shifts in Genre, Time and the AI-Era"],"benchmark_id":"claire-radar-2608-17979","caveat":"AVShift evaluates authorship verification across over 150,000 German text pairs spanning three genres and 21 years, with controlled cross-genre, temporal, and AI-era distribution shifts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17979"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AVShift","organization_count":1,"organizations":["claire_radar"],"rank":368,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17979"},{"aliases":[],"benchmark_id":"claire-radar-2609-35409","caveat":"As language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35409"],"document_share":0.0002723311546840959,"domain":"General AI","name":"AwarenessBench","organization_count":1,"organizations":["claire_radar"],"rank":369,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35409"},{"aliases":[],"benchmark_id":"opencompass-526-ax-b","caveat":"AX-b is a broad-coverage diagnostic task, which requires to determine the logical relation between the given sentence pair, with three relations: entailment, contradiction and neutral. This task is selected from a subset of the GLUE broad-coverage diagnostic dataset, mainly used to test the model's understanding ability in grammar, semantics, world knowledge and so on. AX-b是一个广覆盖诊断任务，要求根据给定的句子对，判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。这个任务是从GLUE的广覆盖诊断数据集中选取了一部分数据，主要用来测试模型在语法、语义、世界知识等方面的理解能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AX-b"],"document_share":0.0002723311546840959,"domain":"推理","name":"AX-b","organization_count":1,"organizations":["opencompass_hub"],"rank":370,"released":"2019-05-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AX-b"},{"aliases":[],"benchmark_id":"opencompass-527-ax-g","caveat":"AX-g is a Winogender diagnostic task, which requires to determine which noun the pronoun refers to according to the given sentence and pronoun. This task is selected from a subset of the Winogender dataset, mainly used to test the model's ability in dealing with gender bias and discrimination. AX-g是一个Winogender诊断任务，要求根据给定的句子和代词，判断代词指代的是哪个名词。这个任务是从Winogender数据集中选取了一部分数据，主要用来测试模型在处理性别偏见和性别歧视方面的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AX-g"],"document_share":0.0002723311546840959,"domain":"推理","name":"AX-g","organization_count":1,"organizations":["opencompass_hub"],"rank":371,"released":"2019-07-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AX-g"},{"aliases":[],"benchmark_id":"opencompass-2084-axbench","caveat":"AXBENCH is a benchmark for large-scale evaluation of Language Model (LLM) control methods using synthetic data, focusing on fine-grained steering for safety and reliability. AXBENCH 是一个旨在评估LLM控制能力的基准。它通过概念检测和模型操控（包含概念、指令、流畅度）评估，旨在实现安全可靠的细粒度操控。基准使用大规模合成数据集，并集成了多种基线方法。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/AXBENCH"],"document_share":0.0002723311546840959,"domain":"理解","name":"AXBENCH","organization_count":1,"organizations":["opencompass_hub"],"rank":372,"released":"2025-01-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/AXBENCH"},{"aliases":[],"benchmark_id":"claire-radar-2607-21588","caveat":"AXIS is a community-driven data engine and benchmark for robot manipulation, providing browser-based teleoperation for data collection, automated task generation and validation, and a dataset of 207 tasks and 50K+ trajectories, with a systematic held-out protocol for policy evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21588"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"AXIS","organization_count":1,"organizations":["claire_radar"],"rank":373,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21588"},{"aliases":["BABE"],"benchmark_id":"babe","caveat":"Biology Arena BEnchmark; measures experimental reasoning and causal inference over peer-reviewed biological research papers.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BABE","organization_count":1,"organizations":["ByteDance"],"rank":374,"released":"2026-02-05","source":"model_reports","url":"https://arxiv.org/abs/2602.05857"},{"aliases":[],"benchmark_id":"claire-radar-2609-23490","caveat":"A benchmark for evaluating multilingual LLM agents using a general localization workflow.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23490"],"document_share":0.0002723311546840959,"domain":"Agents","name":"BabelArena","organization_count":1,"organizations":["claire_radar"],"rank":375,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23490"},{"aliases":[],"benchmark_id":"claire-radar-2606-22329","caveat":"BabelJudge audits LLM-as-a-judge reliability across languages and agent trajectories, measuring position bias, verbosity bias, order inconsistency, and cross-lingual degradation without human labels, providing a composite bias-penalised reliability score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22329"],"document_share":0.0002723311546840959,"domain":"Agents","name":"BabelJudge","organization_count":1,"organizations":["claire_radar"],"rank":376,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22329"},{"aliases":[],"benchmark_id":"opencompass-1266-babilong","caveat":"BABILong is designed to test language models' ability to reason across facts distributed in extremely long documents. It contains a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. BABILong旨在测试语言模型对分布在极长文档中的事实进行推理的能力，涵盖事实链接、简单归纳、推导、计数和处理列表/集合等20种各类推理任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BABILong"],"document_share":0.0002723311546840959,"domain":"长文本","name":"BABILong","organization_count":1,"organizations":["opencompass_hub"],"rank":377,"released":"2024-06-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/BABILong"},{"aliases":[],"benchmark_id":"llm-stats-babyvision","caveat":"A benchmark for early-stage visual reasoning and perception on child-like vision tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/babyvision?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"BabyVision","organization_count":1,"organizations":["llm_stats"],"rank":378,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/babyvision?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-11042","caveat":"BackendForge is a benchmark of 56 contract-defined backend generation tasks from real open-source applications. LLMs must generate Dockerized services evaluated through HTTP tests against an OpenAPI contract.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11042"],"document_share":0.0002723311546840959,"domain":"Code","name":"BackendForge","organization_count":1,"organizations":["claire_radar"],"rank":379,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11042"},{"aliases":["Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs"],"benchmark_id":"claire-radar-2608-11232","caveat":"An algorithmic-trading benchmark for LLM coding agents. It uses multiple-choice questions derived from backtests, with an independent checker that re-derives every answer, and scores models by accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11232"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Backtrader-Bench","organization_count":1,"organizations":["claire_radar"],"rank":380,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.11232"},{"aliases":[],"benchmark_id":"claire-radar-2609-24921","caveat":"A retrospective benchmark where systems recover problem-space and solution-space precursor signals for a mature target topic under a temporal evidence constraint.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24921"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BackTrend","organization_count":1,"organizations":["claire_radar"],"rank":381,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24921"},{"aliases":[],"benchmark_id":"claire-radar-2608-30724","caveat":"Evaluates whether LLM agents exploit optional shortcuts in three synthetic tabular ML tasks to inflate public test scores while failing hidden tests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30724"],"document_share":0.0002723311546840959,"domain":"Agents","name":"BAITBENCH","organization_count":1,"organizations":["claire_radar"],"rank":382,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30724"},{"aliases":[],"benchmark_id":"claire-radar-2608-27219","caveat":"Mental health assessment relies on episodic self-report scales, which convert subjective states such as stress into numerical scores but provide only sparse snapshots of wellbeing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27219"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BALMS","organization_count":1,"organizations":["claire_radar"],"rank":383,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27219"},{"aliases":["Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms"],"benchmark_id":"claire-radar-2608-22335","caveat":"Evaluates LLM safety on 879 Bengali prompts spanning 17 harm categories and five prompting conditions varying language, writing style, and authority framing.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22335v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BanglaSafe","organization_count":1,"organizations":["claire_radar"],"rank":384,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22335v1"},{"aliases":[],"benchmark_id":"claire-radar-2608-21880","caveat":"Evaluates Bangla LLM safety across six language forms using 2,366 prompts and a held-out 354-prompt split spanning unsafe, safe, and safe-sensitive requests, with deterministic response scoring and guardrail screening.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.21880v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BanglaVeilGuard","organization_count":1,"organizations":["claire_radar"],"rank":385,"released":"2026-08-22","source":"claire_radar","url":"http://arxiv.org/abs/2608.21880v1"},{"aliases":[],"benchmark_id":"claire-radar-2608-03884","caveat":"BanglaWild is a benchmark of 2,535 Bengali scene text images with verbatim gold transcriptions and diagnostic attributes. It evaluates OCR and vision-language models on in-the-wild scene text recognition.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03884"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BanglaWild","organization_count":1,"organizations":["claire_radar"],"rank":386,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03884"},{"aliases":["BankerToolBench","Banker Tool Bench","BTB"],"benchmark_id":"bankertoolbench","caveat":"End-to-end investment-banking tasks produce spreadsheets, presentations, and documents; the agent harness, financial-data tools, and rubric-grader configuration are part of the score.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"professional","name":"BankerToolBench","organization_count":1,"organizations":["Tencent"],"rank":387,"released":null,"source":"model_reports","url":"https://github.com/Handshake-AI-Research/bankertoolbench"},{"aliases":[],"benchmark_id":"llm-stats-bankertoolbench","caveat":"BankerToolBench is a public benchmark that evaluates models on banking and finance tool-use tasks. Models are scored against dataset rubrics, measuring their ability to correctly invoke tools and complete multi-step financial workflows.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bankertoolbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"finance","name":"BankerToolBench","organization_count":1,"organizations":["llm_stats"],"rank":388,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bankertoolbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-13550","caveat":"Recent speech LLMs can leverage rich paralinguistic cues, enabling more complex and personalized interactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13550"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BASIS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":389,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13550"},{"aliases":[],"benchmark_id":"claire-radar-2608-23435","caveat":"Evaluates basketball understanding through 7,980 questions over text, image, and video, covering ten tasks from event recognition to structured game knowledge.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23435v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BasketballBench","organization_count":1,"organizations":["claire_radar"],"rank":390,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23435v1"},{"aliases":[],"benchmark_id":"claire-radar-2608-12894","caveat":"BavGround evaluates regional cultural grounding and dialect competence in Bavarian across English, German, and Bavarian, with 618 multi-parallel multiple-choice questions across eight cultural domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12894"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BavGround","organization_count":1,"organizations":["claire_radar"],"rank":391,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.12894"},{"aliases":[],"benchmark_id":"claire-radar-2609-37292","caveat":"Physical active vision allows robots to change their viewpoint when task-relevant observations become unreliable, yet existing manipulation benchmarks provide limited support for studying how policies recover from occlusion during executio…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37292"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"BAVO-Bench","organization_count":1,"organizations":["claire_radar"],"rank":392,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37292"},{"aliases":[],"benchmark_id":"claire-radar-2610-06748","caveat":"In decentralized consumer-to-consumer (C2C) marketplaces, people list goods, negotiate with strangers, and rate one another, so trust rests on reputation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.06748"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"BazaarBench","organization_count":1,"organizations":["claire_radar"],"rank":393,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.06748"},{"aliases":[],"benchmark_id":"llm-stats-bbh","caveat":"Big-Bench Hard (BBH) is a suite of 23 challenging tasks selected from BIG-Bench for which prior language model evaluations did not outperform the average human-rater. These tasks require multi-step reasoning across diverse domains including arithmetic, logical reasoning, reading comprehension, and commonsense reasoning. The benchmark was designed to test capabilities believed to be beyond current language models and focuses on evaluating complex reasoning skills including temporal understanding, spatial reasoning, causal understanding, and deductive logical reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bbh?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"BBH","organization_count":1,"organizations":["llm_stats"],"rank":394,"released":"2022-10-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bbh?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-539-bbh","caveat":"BIG-Bench Hard (BBH) is a subset of the BIG-Bench, a diverse evaluation suite for language models. BBH focuses on a suite of 23 challenging tasks from BIG-Bench that were found to be beyond the capabilities of current language models. BIG Bench-Hard（BBH）是BIG Bench的一个子集，它是一个用于语言模型的多样化评估套件。BBH专注于BIG Bench的23项具有挑战性的任务，这些任务被发现超出了当前语言模型的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BBH"],"document_share":0.0002723311546840959,"domain":"推理","name":"BBH","organization_count":1,"organizations":["opencompass_hub"],"rank":395,"released":"2022-10-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/BBH"},{"aliases":[],"benchmark_id":"llm-stats-bc-vl","caveat":"BC-VL is a vision-language benchmark for knowledge-grounded multimodal question answering.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bc-vl?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"BC-VL","organization_count":1,"organizations":["llm_stats"],"rank":396,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bc-vl?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-beam-128k","caveat":"Beam 128K evaluates reasoning over long inputs at a 128K-token context length.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/beam-128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"Beam 128K","organization_count":1,"organizations":["llm_stats"],"rank":397,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/beam-128k?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-17895","caveat":"BEAR-Bench evaluates multimodal models on bilingual English-Russian text-dense business and scientific documents with 1,000 human-annotated questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17895"],"document_share":0.0002723311546840959,"domain":"Bilingual reasoning","name":"BEAR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":398,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17895"},{"aliases":[],"benchmark_id":"claire-radar-2608-30536","caveat":"Evaluates vision-language-action policies on 50 long-horizon mobile manipulation tasks via 235,492 skill instances, with trajectory-level and skill-level metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30536"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"Behavior-Skill","organization_count":1,"organizations":["claire_radar"],"rank":399,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30536"},{"aliases":[],"benchmark_id":"claire-radar-2608-26807","caveat":"Travel planning agents assist users in generating personalized travel plans by modeling their individual preferences.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26807"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Behavior2Trip","organization_count":1,"organizations":["claire_radar"],"rank":400,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26807"},{"aliases":[],"benchmark_id":"claire-radar-2606-02798","caveat":"BehaviorBench evaluates personalized decision modeling from real-world behavioral traces, with belief and trade prediction tasks from prediction-market and on-chain records across 2,000 wallets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02798"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BehaviorBench","organization_count":1,"organizations":["claire_radar"],"rank":401,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02798"},{"aliases":[],"benchmark_id":"claire-radar-2606-24162","caveat":"Evaluates foundation models on four behavioral science capabilities: behavior prediction and simulation, strategic decision-making, subject-trait inference, and behavioral knowledge application, assessing both individual-level and distributional alignment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24162"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"BehaviorBench","organization_count":1,"organizations":["claire_radar"],"rank":402,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24162"},{"aliases":["BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click"],"benchmark_id":"claire-radar-2608-26867","caveat":"BekchiAI-Benchmark evaluates LLM agent skills using 2,057 deterministic tool-using ReAct tasks across 7 categories, scoring accuracy plus tool-call adherence, URL hallucination, source-match, and token cost.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26867"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BekchiAI-Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":403,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26867"},{"aliases":[],"benchmark_id":"opencompass-1086-belebele","caveat":"BELEBELE, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset enables the evaluation of text models in\nhigh-, medium-, and low-resource languages. BELEBELE 是一个多项选择机器阅读理解（MRC）数据集，涵盖 122 种语言变体。该数据集显著扩展了自然语言理解（NLU）基准的语言覆盖范围，使得可以在高、中、低资源语言中评估文本模型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Belebele"],"document_share":0.0002723311546840959,"domain":"语言","name":"Belebele","organization_count":1,"organizations":["opencompass_hub"],"rank":404,"released":"2024-07-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Belebele"},{"aliases":["BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems"],"benchmark_id":"claire-radar-2606-20668","caveat":"An operational benchmark comparing moderation filters, jailbreak detectors, and general-purpose LLM supervisors across effectiveness, false positives, latency, and cost.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20668"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BELLS-O","organization_count":1,"organizations":["claire_radar"],"rank":405,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.20668"},{"aliases":[],"benchmark_id":"claire-radar-2609-15726","caveat":"This entry describes a simulation benchmark for visuo-tactile bimanual manipulation, but no task dataset, scorer, or public reuse path is evidenced.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15726"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"Bench2Dex","organization_count":1,"organizations":["claire_radar"],"rank":406,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15726"},{"aliases":[],"benchmark_id":"claire-radar-2608-23898","caveat":"Evaluates LLMs on 149 protocol-modification tasks reconstructed from real changes scientists made to published wet-lab protocols, using weighted rubric scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.23898"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BenchBench-Protocol","organization_count":1,"organizations":["claire_radar"],"rank":407,"released":"2026-08-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.23898"},{"aliases":[],"benchmark_id":"llm-stats-benchcad","caveat":"BenchCAD is a benchmark for programmatic CAD reasoning built from 17,900 execution-verified CadQuery programs spanning 106 industrial part families, roughly half anchored to real ISO, DIN, EN, ASME, and IEC specification tables. It decomposes CAD capability into matched tasks; the Vision2Code task requires models to generate CadQuery code from multi-view renders, scored by voxel IoU against the reference geometry.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/benchcad?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"BenchCAD","organization_count":1,"organizations":["llm_stats"],"rank":408,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-benchcad-with-python-tool","caveat":"BenchCAD variant evaluated with access to a Python tool for programmatic CAD reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/benchcad-with-python-tool?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"BenchCAD (with Python tool)","organization_count":1,"organizations":["llm_stats"],"rank":409,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/benchcad-with-python-tool?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-18270","caveat":"A payment-domain benchmark with 306 items (MCQ, open QA, and attack variants) that tests models' ability to use domain context correctly, distinguish parametric from context-grounded knowledge, and remain robust under controlled context/question attacks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18270"],"document_share":0.0002723311546840959,"domain":"Finance","name":"BENCHCOMPASS","organization_count":1,"organizations":["claire_radar"],"rank":410,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18270"},{"aliases":[],"benchmark_id":"opencompass-1562-benchmax","caveat":"BenchMAX is a comprehensive, high-quality, and multiway parallel multilingual benchmark comprising 10 tasks designed to assess crucial capabilities across 17 diverse language. BenchMAX 是一个全面、高质量的多向并行多语言基准，包含 10 个任务，旨在评估 17 种不同语言的关键能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BenchMAX"],"document_share":0.0002723311546840959,"domain":"语言","name":"BenchMAX","organization_count":1,"organizations":["opencompass_hub"],"rank":411,"released":"2025-02-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/BenchMAX"},{"aliases":[],"benchmark_id":"claire-radar-github-senssei-benchrig","caveat":"A cross-platform evaluation and hardware profiling rig for local LLMs, focusing on speed, coding, reasoning, and context scaling.","document_count":1,"document_ids":["claire_radar:https://github.com/senssei/benchrig"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"BenchRig","organization_count":1,"organizations":["claire_radar"],"rank":412,"released":"2026-09-18","source":"claire_radar","url":"https://github.com/senssei/benchrig"},{"aliases":[],"benchmark_id":"claire-radar-2606-10061","caveat":"BenSyc is a benchmark for studying conversational sycophancy in Bengali social contexts, with binary labels and a five-level taxonomy from invalidation to escalation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10061"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BenSyc","organization_count":1,"organizations":["claire_radar"],"rank":413,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.10061"},{"aliases":[],"benchmark_id":"llm-stats-beyond-aime","caveat":"Beyond AIME is a difficult mathematical reasoning benchmark designed to test deeper reasoning chains and harder decomposition than standard AIME-style problem sets.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/beyond-aime?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"Beyond AIME","organization_count":1,"organizations":["llm_stats"],"rank":414,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/beyond-aime?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-29990","caveat":"Evaluates large language models on 31 expert-authored prompts covering Saudi dialect phenomena, with expert-established ground truths and rubric-based scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29990"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models","organization_count":1,"organizations":["claire_radar"],"rank":415,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29990"},{"aliases":["BeyondAIME"],"benchmark_id":"beyond_aime","caveat":"Beyond AIME difficulty competition math; most models score low and gaps are dominated by run-to-run variance.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"math","name":"BeyondAIME","organization_count":1,"organizations":["ByteDance"],"rank":416,"released":null,"source":"model_reports","url":"https://huggingface.co/datasets/ByteDance-Seed/BeyondAIME"},{"aliases":[],"benchmark_id":"claire-radar-2608-20107","caveat":"Evaluates causally consistent video object removal using paired synthetic and real videos with aligned references, and includes the CORE metric for after-effect consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.20107"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BeyondMasks","organization_count":1,"organizations":["claire_radar"],"rank":417,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.20107"},{"aliases":[],"benchmark_id":"llm-stats-bfcl","caveat":"The Berkeley Function Calling Leaderboard (BFCL) is the first comprehensive and executable function call evaluation dedicated to assessing Large Language Models' ability to invoke functions. It evaluates serial and parallel function calls across multiple programming languages (Python, Java, JavaScript, REST API) using a novel Abstract Syntax Tree (AST) evaluation method. The benchmark consists of over 2,000 question-function-answer pairs covering diverse application domains and complex use cases including multiple function calls, parallel function calls, and multi-turn interactions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BFCL","organization_count":1,"organizations":["llm_stats"],"rank":418,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-bfcl-v2","caveat":"Berkeley Function Calling Leaderboard (BFCL) v2 is a comprehensive benchmark for evaluating large language models' function calling capabilities. It features 2,251 question-function-answer pairs with enterprise and OSS-contributed functions, addressing data contamination and bias through live, user-contributed scenarios. The benchmark evaluates AST accuracy, executable accuracy, irrelevance detection, and relevance detection across multiple programming languages (Python, Java, JavaScript) and includes complex real-world function calling scenarios with multi-lingual prompts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BFCL v2","organization_count":1,"organizations":["llm_stats"],"rank":419,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-bfcl-v3","caveat":"Berkeley Function Calling Leaderboard v3 (BFCL-v3) is an advanced benchmark that evaluates large language models' function calling capabilities through multi-turn and multi-step interactions. It introduces extended conversational exchanges where models must retain contextual information across turns and execute multiple internal function calls for complex user requests. The benchmark includes 1000 test cases across domains like vehicle control, trading bots, travel booking, and file system management, using state-based evaluation to verify both system state changes and execution path correctness.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BFCL-v3","organization_count":1,"organizations":["llm_stats"],"rank":420,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-bfcl-v4","caveat":"Berkeley Function Calling Leaderboard V4 (BFCL-V4) evaluates LLMs on their ability to accurately call functions and APIs, including simple, multiple, parallel, and nested function calls across diverse programming scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v4?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"BFCL-V4","organization_count":1,"organizations":["llm_stats"],"rank":421,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v4?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-bfcl-v3-multiturn","caveat":"Berkeley Function Calling Leaderboard (BFCL) V3 MultiTurn benchmark that evaluates large language models' ability to handle multi-turn and multi-step function calling scenarios. The benchmark introduces complex interactions requiring models to manage sequential function calls, handle conversational context across multiple turns, and make dynamic decisions about when and how to use available functions. BFCL V3 uses state-based evaluation by verifying the actual state of API systems after function execution, providing more realistic assessment of function calling capabilities in agentic applications.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3-multiturn?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BFCL_v3_MultiTurn","organization_count":1,"organizations":["llm_stats"],"rank":422,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bfcl-v3-multiturn?top_n=500"},{"aliases":["Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events"],"benchmark_id":"claire-radar-2608-06485","caveat":"Evaluates directional fidelity of event-induced personality change in LLM agents across 11 major life events, scored against Big Five trait directions from human longitudinal evidence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06485"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BFI-Adapt","organization_count":1,"organizations":["claire_radar"],"rank":423,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06485"},{"aliases":[],"benchmark_id":"claire-radar-2607-26232","caveat":"BG-REAL is a benchmark for background manipulation detection and localization in images. It contains 7,000 processed samples (6,000 public-data anchored, 1,000 synthetic) over six edit families with matched authentic controls, source-group splits, and quality control.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26232"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"BG-REAL","organization_count":1,"organizations":["claire_radar"],"rank":424,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.26232"},{"aliases":[],"benchmark_id":"claire-radar-2609-34274","caveat":"Artificial-intelligence (AI) agents hold promise for automating bioimage analysis, yet no benchmark evaluates whether they can carry out real-world analyses end to end.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34274"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"BIABench","organization_count":1,"organizations":["claire_radar"],"rank":425,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34274"},{"aliases":[],"benchmark_id":"llm-stats-big-bench-audio","caveat":"Big Bench Audio is an audio reasoning benchmark adapted from a subset of Big Bench Hard, with text questions converted to spoken audio. It evaluates the reasoning ability of speech-to-speech and audio language models on tasks delivered as audio input, with accuracy scored by an independent evaluation (Artificial Analysis).","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-bench-audio?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Big Bench Audio","organization_count":1,"organizations":["llm_stats"],"rank":426,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-audio?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-big-finance-bench","caveat":"Big Finance Bench evaluates models on complex financial-analysis tasks that require retrieving and reasoning over financial documents and performing multi-step quantitative work.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-finance-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Big Finance Bench","organization_count":1,"organizations":["llm_stats"],"rank":427,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/big-finance-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-big-bench","caveat":"Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark consisting of 204+ tasks designed to probe large language models and extrapolate their future capabilities. It covers diverse domains including linguistics, mathematics, common-sense reasoning, biology, physics, social bias, software development, and more. The benchmark focuses on tasks believed to be beyond current language model capabilities and includes both English and non-English tasks across multiple languages.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"BIG-Bench","organization_count":1,"organizations":["llm_stats"],"rank":428,"released":"2022-06-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench?top_n=500"},{"aliases":["BigBench Extra Hard","BBEH","BIG-Bench Extra Hard"],"benchmark_id":"bigbench_extra_hard","caveat":"Successor to BBH after saturation; per-task variance is high.","document_count":1,"document_ids":["model_reports:google_gemma_4_model_card"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BIG-Bench Extra Hard","organization_count":1,"organizations":["Google"],"rank":429,"released":"2025-02-26","source":"model_reports","url":"https://arxiv.org/abs/2502.19187"},{"aliases":[],"benchmark_id":"llm-stats-big-bench-extra-hard","caveat":"BIG-Bench Extra Hard (BBEH) is a challenging benchmark that replaces each task in BIG-Bench Hard with a novel task that probes similar reasoning capabilities but exhibits significantly increased difficulty. The benchmark contains 23 tasks testing diverse reasoning skills including many-hop reasoning, causal understanding, spatial reasoning, temporal arithmetic, geometric reasoning, linguistic reasoning, logic puzzles, and humor understanding. Designed to address saturation on existing benchmarks where state-of-the-art models achieve near-perfect scores, BBEH shows substantial room for improvement with best models achieving only 9.8-44.8% average accuracy.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-bench-extra-hard?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BIG-Bench Extra Hard","organization_count":1,"organizations":["llm_stats"],"rank":430,"released":"2025-02-26","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-extra-hard?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-big-bench-hard","caveat":"BIG-Bench Hard (BBH) is a subset of 23 challenging BIG-Bench tasks selected because prior language model evaluations did not outperform average human-rater performance. The benchmark contains 6,511 evaluation examples testing various forms of multi-step reasoning including arithmetic, logical reasoning (Boolean expressions, logical deduction), geometric reasoning, temporal reasoning, and language understanding. Tasks require capabilities such as causal judgment, object counting, navigation, pattern recognition, and complex problem solving.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/big-bench-hard?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"BIG-Bench Hard","organization_count":1,"organizations":["llm_stats"],"rank":431,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/big-bench-hard?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-bigcodebench","caveat":"A benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained programming tasks. Evaluates code generation with diverse function calls and complex instructions, featuring two variants: Complete (code completion based on comprehensive docstrings) and Instruct (generating code from natural language instructions).","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BigCodeBench","organization_count":1,"organizations":["llm_stats"],"rank":432,"released":"2024-06-22","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1253-bigcodebench","caveat":"BigCodeBench is a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. BigCodeBench用于评估LLM的代码生成能力，包含1140个可以调用139个库和7个域的多个函数来完成的细粒度任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BigCodeBench"],"document_share":0.0002723311546840959,"domain":"强推理","name":"BigCodeBench","organization_count":1,"organizations":["opencompass_hub"],"rank":433,"released":"2024-06-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/BigCodeBench"},{"aliases":[],"benchmark_id":"llm-stats-bigcodebench-full","caveat":"A comprehensive benchmark that evaluates large language models' ability to solve complex, practical programming tasks via code generation. Contains 1,140 fine-grained tasks across 7 domains using function calls from 139 libraries. Challenges LLMs to invoke multiple function calls as tools and handle complex instructions for realistic software engineering and general-purpose reasoning tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-full?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BigCodeBench-Full","organization_count":1,"organizations":["llm_stats"],"rank":434,"released":"2024-06-22","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-full?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-bigcodebench-hard","caveat":"BigCodeBench-Hard is a subset of 148 challenging programming tasks from BigCodeBench, designed to evaluate large language models' ability to solve complex, real-world programming problems. These tasks require diverse function calls from multiple libraries across 7 domains including computation, networking, data analysis, and visualization. The benchmark tests compositional reasoning and the ability to implement complex instructions that span 139 libraries with an average of 2.8 libraries per task.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-hard?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BigCodeBench-Hard","organization_count":1,"organizations":["llm_stats"],"rank":435,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bigcodebench-hard?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-03829","caveat":"BigFinanceBench evaluates financial-research agents on open-ended tasks with 928 items, each paired with a ground-truth answer and a point-weighted rubric decomposing the derivation into steps. Supports partial-credit scoring across 36,241 rubric points.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03829"],"document_share":0.0002723311546840959,"domain":"Finance","name":"BigFinanceBench","organization_count":1,"organizations":["claire_radar"],"rank":436,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03829"},{"aliases":[],"benchmark_id":"opencompass-1680-bigobench","caveat":"BigO(Bench)是一个包含约 300 个需要用 Python 解决的代码问题的基准测试，以及 3,105 个编码问题和 1,190,250 个解决方案用于训练，以评估LLMs能否找到代码解决方案的时间-空间复杂度，或者生成符合时间-空间复杂度要求的代码解决方案。 BigO(Bench)是一个包含约 300 个需要用 Python 解决的代码问题的基准测试，以及 3,105 个编码问题和 1,190,250 个解决方案用于训练，以评估LLMs能否找到代码解决方案的时间-空间复杂度，或者生成符合时间-空间复杂度要求的代码解决方案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BigOBench"],"document_share":0.0002723311546840959,"domain":"代码","name":"BigOBench","organization_count":1,"organizations":["opencompass_hub"],"rank":437,"released":"2025-03-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/BigOBench"},{"aliases":[],"benchmark_id":"claire-radar-2605-30900","caveat":"Evaluates physical reasoning in synthetic billiards environments, testing collision prediction, wall bounce reasoning, and final position estimation for multimodal LLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30900"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BilliardPhys-Bench","organization_count":1,"organizations":["claire_radar"],"rank":438,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.30900"},{"aliases":[],"benchmark_id":"claire-radar-2606-20146","caveat":"BIM-Edit evaluates large language models on natural-language editing of Industry Foundation Classes (IFC) building models. The benchmark includes 324 editing tasks across 11 realistic building models and 36 synthetic scenes. Tasks are categorized as direct, spatial, or topological instructions, and outputs are scored on geometric accuracy, semantic validity, and topological consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20146"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BIM-Edit","organization_count":1,"organizations":["claire_radar"],"rank":439,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.20146"},{"aliases":[],"benchmark_id":"claire-radar-2608-07038","caveat":"BinJudgeBench evaluates LLM-as-a-Judge for human-oriented binary reverse engineering, covering function name recovery, code summarization, and decompilation optimization, with correlation to human judgment as the metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07038"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"BinJudgeBench","organization_count":1,"organizations":["claire_radar"],"rank":440,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07038"},{"aliases":[],"benchmark_id":"llm-stats-biolp-bench","caveat":"BioLP-Bench is a model-graded evaluation measuring ability to find and correct mistakes in common biological laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/biolp-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"BioLP-Bench","organization_count":1,"organizations":["llm_stats"],"rank":441,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/biolp-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-biomysterybench","caveat":"BioMysteryBench evaluates a model's ability to reason through challenging molecular biology problems, reporting performance on a hard subset and on the subset of problems solved by human experts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/biomysterybench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BioMysteryBench","organization_count":1,"organizations":["llm_stats"],"rank":442,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/biomysterybench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-19180","caveat":"Dataset for evidence-grounded scientific reasoning over biophysical literature.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19180"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"BioPhys-Bridge","organization_count":1,"organizations":["claire_radar"],"rank":443,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.19180"},{"aliases":[],"benchmark_id":"claire-radar-2607-05462","caveat":"BioSecBench-Refusal evaluates AI agents on 61 Routine and 46 Red-Team biosecurity-related tasks, measuring refusal rates and risk identification across multiple model configurations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05462"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BioSecBench-Refusal","organization_count":1,"organizations":["claire_radar"],"rank":444,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.05462"},{"aliases":[],"benchmark_id":"claire-radar-2607-19262","caveat":"BioSecBench-Surveillance evaluates AI agents on pathogen genomic surveillance across 100 tasks spanning seven categories, including taxonomic classification and genetic-engineering detection. Agents receive raw sequencing data and surveillance context, and their structured answers are graded deterministically against ground truth.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19262"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"BioSecBench-Surveillance","organization_count":1,"organizations":["claire_radar"],"rank":445,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.19262"},{"aliases":[],"benchmark_id":"claire-radar-2610-07614","caveat":"We evaluate whether AI agents can match the reported findings of published biomedical studies using public data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.07614"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"BioStudyBench","organization_count":1,"organizations":["claire_radar"],"rank":446,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.07614"},{"aliases":[],"benchmark_id":"claire-radar-2608-29345","caveat":"Evaluates text-to-SQL systems on 1,393 tasks that require leveraging historical SQL logs, with fine-grained annotations for retrieval and knowledge utilization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29345"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"BIRD-History","organization_count":1,"organizations":["claire_radar"],"rank":447,"released":"2026-08-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.29345"},{"aliases":[],"benchmark_id":"llm-stats-bird-sql-dev","caveat":"BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQLs) is a comprehensive text-to-SQL benchmark containing 12,751 question-SQL pairs across 95 databases (33.4 GB total) spanning 37+ professional domains. It evaluates large language models' ability to convert natural language to executable SQL queries in real-world scenarios with complex database schemas and dirty data.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bird-sql-%28dev%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Bird-SQL (dev)","organization_count":1,"organizations":["llm_stats"],"rank":448,"released":"2023-05-04","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bird-sql-%28dev%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-bixbench","caveat":"BixBench is a benchmark for real-world bioinformatics and computational biology data analysis. It evaluates AI models on multi-step scientific workflows that require code execution, statistical reasoning, and biological domain knowledge to interpret experimental data.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/bixbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BixBench","organization_count":1,"organizations":["llm_stats"],"rank":449,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/bixbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-25286","caveat":"Evaluates AI agents on 20 research-study-scale computational biology tasks that require processing raw data into scientific results, scored by matching generated artifacts against study-derived targets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25286"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BixBench3","organization_count":1,"organizations":["claire_radar"],"rank":450,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25286"},{"aliases":["Blender World"],"benchmark_id":"blenderworld","caveat":"Agents create editable Blender scenes from text, image or video briefs across 223 tasks. Overall is a 0–100 score combining requirement satisfaction and visual quality equally, not the full-rubric pass rate. The publisher runs each task once with an automatic judge and different generation harnesses across models; compare the complete setup, not an isolated model.","document_count":1,"document_ids":["model_reports:blenderworld_v1_results_table"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"BlenderWorld","organization_count":1,"organizations":["BlenderWorld authors"],"rank":451,"released":"2026-10-14","source":"model_reports","url":"https://blenderworld.github.io/"},{"aliases":[],"benchmark_id":"claire-radar-2607-08317","caveat":"Evaluates reasoning blind spots in language, vision-language, and image-generation models across 235 samples with structured reference solutions and taxonomy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08317"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Blind-Spots-Bench","organization_count":1,"organizations":["claire_radar"],"rank":452,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08317"},{"aliases":[],"benchmark_id":"claire-radar-2609-16305","caveat":"This entry describes a benchmark for trajectory-level safety calibration of tool-using agents, but the provided abstract and links do not include released task data or scoring code.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16305"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BLINDSPOT","organization_count":1,"organizations":["claire_radar"],"rank":453,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.16305"},{"aliases":[],"benchmark_id":"llm-stats-blink","caveat":"BLINK: Multimodal Large Language Models Can See but Not Perceive. A benchmark for multimodal language models focusing on core visual perception abilities. Reformats 14 classic computer vision tasks into 3,807 multiple-choice questions paired with single or multiple images and visual prompting. Tasks include relative depth estimation, visual correspondence, forensics detection, multi-view reasoning, counting, object localization, and spatial reasoning that humans can solve 'within a blink'.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/blink?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"BLINK","organization_count":1,"organizations":["llm_stats"],"rank":454,"released":"2024-04-18","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/blink?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1365-blink","caveat":"BLINK focuses on MLLMs' core visual perception abilities. It contains 3,807 multiple-choice questions spanning 14 classic computer vision tasks. BLINK用于评估多模态大模型的视觉感知能力，包含来自14个经典计算机视觉任务的3807道多项选择题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BLINK"],"document_share":0.0002723311546840959,"domain":"多模态","name":"BLINK","organization_count":1,"organizations":["opencompass_hub"],"rank":455,"released":"2024-04-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/BLINK"},{"aliases":[],"benchmark_id":"claire-radar-2605-30907","caveat":"Evaluates LLM agents on synthesis, manipulation, and comprehension tasks over financial spreadsheet workbooks, with 131 tasks and granular rubric criteria validated by expert annotators.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30907"],"document_share":0.0002723311546840959,"domain":"Finance","name":"BlueFin","organization_count":1,"organizations":["claire_radar"],"rank":456,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.30907"},{"aliases":["Blueprint-Bench 2","Blueprint-Bench","BlueprintBench 2"],"benchmark_id":"blueprint_bench_2","caveat":"Spatial-reasoning set: 50 apartments, ~20 photos each, scored by a connectivity-graph grader, with a public leaderboard on the Andon Labs eval page (run in-house; dataset not openly downloadable). Builds on the original Blueprint-Bench paper (arXiv:2509.25229), a distinct instrument released 2025-09-24 whose scores must not be compared across versions.","document_count":1,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5"],"document_share":0.0002723311546840959,"domain":"spatial_reasoning","name":"Blueprint-Bench 2","organization_count":1,"organizations":["Anthropic"],"rank":457,"released":"2026-05-04","source":"model_reports","url":"https://andonlabs.com/evals/blueprint-bench-2"},{"aliases":[],"benchmark_id":"llm-stats-blueprint-bench-2","caveat":"Blueprint-Bench 2 is an agentic spatial reasoning benchmark that evaluates a model's ability to understand, plan, and reason over architectural blueprints and other structured spatial documents. Scores are reported as a normalized score.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/blueprint-bench-2?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Blueprint-Bench 2","organization_count":1,"organizations":["llm_stats"],"rank":458,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/blueprint-bench-2?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-22723","caveat":"BLUEX v2 evaluates large language models on open-ended, discursive questions from the second-phase entrance exams of UNICAMP and USP (2022–2025). The dataset includes 395 questions with 919 subquestions, covering nine subjects, with 55.7% of questions containing images represented as context-aware captions. Scoring uses an LLM-as-a-judge protocol with binary rubric criteria based on official reference answers, yielding a 0–10 score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22723"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"BLUEX v2","organization_count":1,"organizations":["claire_radar"],"rank":459,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22723"},{"aliases":[],"benchmark_id":"claire-radar-2605-22643","caveat":"Boiling the Frog evaluates whether tool-using AI models in corporate settings are susceptible to incremental attacks through multi-turn scenarios with persistent workspaces. It includes a three-level operational risk taxonomy and scores attack success rate (ASR) on resulting artifact state.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22643"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Boiling the Frog","organization_count":1,"organizations":["claire_radar"],"rank":460,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22643"},{"aliases":[],"benchmark_id":"llm-stats-boolq","caveat":"BoolQ is a reading comprehension dataset for yes/no questions containing 15,942 naturally occurring examples. Each example consists of a question, passage, and boolean answer, where questions are generated in unprompted and unconstrained settings. The dataset challenges models with complex, non-factoid information requiring entailment-like inference to solve.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/boolq?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BoolQ","organization_count":1,"organizations":["llm_stats"],"rank":461,"released":"2019-05-24","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/boolq?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-510-boolq","caveat":"BoolQ is a question answering dataset for yes/no questions containing 15942 examples. These questions are naturally occurring ---they are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context. BoolQ是一个包含15942个示例的是/否问题的问答数据集。这些问题是自然生成的——即在无prompt和无约束的环境中产生的。每个例子都是一个三元组(问题、段落、答案)，页面标题是可选的附加上下文。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BoolQ"],"document_share":0.0002723311546840959,"domain":"知识","name":"BoolQ","organization_count":1,"organizations":["opencompass_hub"],"rank":462,"released":"2019-05-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/BoolQ"},{"aliases":[],"benchmark_id":"claire-radar-2608-02670","caveat":"Boundary-Bench is an open-source plugin that adds configurable security policy levels to Terminal-Bench, enabling evaluation of coding agents under constraints like scoped credentials, restricted egress, read-only filesystems, and non-root execution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02670"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Boundary-Bench","organization_count":1,"organizations":["claire_radar"],"rank":463,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.02670"},{"aliases":[],"benchmark_id":"claire-radar-2608-27268","caveat":"Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27268"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BrailleBench","organization_count":1,"organizations":["claire_radar"],"rank":464,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27268"},{"aliases":["Brain-to-text '25","BtoT25"],"benchmark_id":"brain_to_text_25","caveat":"Evaluates one participant (T15), with about two thirds of 1,450 held-out sentences used for the private leaderboard. Speaking strategy, language model and corpus mix affect word error rate, so these scores are not directly comparable to Brain-to-Text '24.","document_count":1,"document_ids":["model_reports:brain_to_text_25_kaggle_leaderboard"],"document_share":0.0002723311546840959,"domain":"science","name":"Brain-to-Text '25","organization_count":1,"organizations":["UC Davis Neuroprosthetics Lab"],"rank":465,"released":"2025-07-07","source":"model_reports","url":"https://www.kaggle.com/competitions/brain-to-text-25"},{"aliases":["Brain-to-Text '24","BtoT24"],"benchmark_id":"brain_to_text_24","caveat":"Evaluates 1,200 held-out sentences from one participant (T12). Language model post-processing is part of each system, and results do not establish transfer to another participant or to the different Brain-to-Text '25 dataset.","document_count":1,"document_ids":["model_reports:brain_to_text_24_evalai_leaderboard"],"document_share":0.0002723311546840959,"domain":"science","name":"Brain-to-Text Benchmark '24","organization_count":1,"organizations":["Stanford Neural Prosthetics Translational Laboratory"],"rank":466,"released":"2023-06-01","source":"model_reports","url":"https://eval.ai/web/challenges/challenge-page/2099/overview"},{"aliases":[],"benchmark_id":"claire-radar-2608-04156","caveat":"BrainBench is a benchmark for instruction-conditioned EEG understanding covering four subsets across 17 datasets. It evaluates LLMs on producing scientific reports and artifacts from EEG recordings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04156"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BrainBench","organization_count":1,"organizations":["claire_radar"],"rank":467,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.04156"},{"aliases":[],"benchmark_id":"claire-radar-2610-06571","caveat":"Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences, localize findings within volumetric anatomy, and translate this evidence into report-gr…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.06571"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"BrainTRACE","organization_count":1,"organizations":["claire_radar"],"rank":468,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.06571"},{"aliases":[],"benchmark_id":"claire-radar-2609-22064","caveat":"Proposed benchmark for across-animal transfer on multi-region neural recordings from the International Brain Laboratory Brainwide Map dataset.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.22064"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BrainWideBench","organization_count":1,"organizations":["claire_radar"],"rank":469,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.22064"},{"aliases":["BranchDrive: A Branch-Structured Dataset for Action-Conditioned Driving Prediction"],"benchmark_id":"claire-radar-2609-27275","caveat":"BranchDrive is a branch-structured CARLA dataset that pairs one canonical pre-decision history with one nominal expert future and twelve physically executed intervention futures spanning acceleration, braking, and left- and right-steering policies at three magnitudes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27275"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BranchDrive","organization_count":1,"organizations":["claire_radar"],"rank":470,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27275"},{"aliases":[],"benchmark_id":"claire-radar-2609-17106","caveat":"A benchmark offering 3DGS scenes and tools for evaluating robotic active vision systems in 6DOF object pose estimation tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17106"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"BRAVE-6D","organization_count":1,"organizations":["claire_radar"],"rank":471,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17106"},{"aliases":[],"benchmark_id":"opencompass-1571-bright","caveat":"BRIGHT is the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. BRIGHT 是第一个需要大量推理来检索相关文档的文本检索基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BRIGHT"],"document_share":0.0002723311546840959,"domain":"强推理","name":"BRIGHT","organization_count":1,"organizations":["opencompass_hub"],"rank":472,"released":"2024-10-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/BRIGHT"},{"aliases":[],"benchmark_id":"brokenarxiv","caveat":"arXiv proofs with planted errors; tests whether a model catches a broken argument instead of reproducing it.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"math","name":"BrokenArXiv","organization_count":1,"organizations":["Tencent"],"rank":473,"released":null,"source":"model_reports","url":"https://matharena.ai/brokenarxiv"},{"aliases":[],"benchmark_id":"llm-stats-browsecomp","caveat":"BrowseComp is a benchmark comprising 1,266 questions that challenge AI agents to persistently navigate the internet in search of hard-to-find, entangled information. The benchmark measures agents' ability to exercise persistence in information gathering, demonstrate creativity in web navigation, and find concise, verifiable answers. Despite the difficulty of the questions, BrowseComp is simple and easy-to-use, as predicted answers are short and easily verifiable against reference answers.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BrowseComp","organization_count":1,"organizations":["llm_stats"],"rank":474,"released":"2025-04-10","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-browsecomp-long-128k","caveat":"A challenging benchmark for evaluating web browsing agents' ability to persistently navigate the internet and find hard-to-locate, entangled information. Comprises 1,266 questions requiring strategic reasoning, creative search, and interpretation of retrieved content, with short and easily verifiable answers.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BrowseComp Long Context 128k","organization_count":1,"organizations":["llm_stats"],"rank":475,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-128k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-browsecomp-long-256k","caveat":"BrowseComp is a benchmark for measuring the ability of agents to browse the web, comprising 1,266 questions that require persistently navigating the internet in search of hard-to-find, entangled information. Despite the difficulty of the questions, BrowseComp is simple and easy-to-use, as predicted answers are short and easily verifiable against reference answers. The benchmark focuses on questions where answers are obscure, time-invariant, and well-supported by evidence scattered across the open web.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-256k?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BrowseComp Long Context 256k","organization_count":1,"organizations":["llm_stats"],"rank":476,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-long-256k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-browsecomp-vl","caveat":"BrowseComp-VL is the vision-language variant of BrowseComp, evaluating multimodal models on web browsing comprehension tasks that require processing visual web page content alongside text.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-vl?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"BrowseComp-VL","organization_count":1,"organizations":["llm_stats"],"rank":477,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-vl?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-browsecomp-zh","caveat":"A high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents on the Chinese web, consisting of 289 multi-hop questions spanning 11 diverse domains including Film & TV, Technology, Medicine, and History. Questions are reverse-engineered from short, objective, and easily verifiable answers, requiring sophisticated reasoning and information reconciliation beyond basic retrieval. The benchmark addresses linguistic, infrastructural, and censorship-related complexities in Chinese web environments.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-zh?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"BrowseComp-zh","organization_count":1,"organizations":["llm_stats"],"rank":478,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/browsecomp-zh?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-27334","caveat":"Evaluates multi-turn agent performance on 532 telemetry-derived tasks across train/dev/test splits, with additional XAI4HEAT episodes, using deterministic replayable construction and verifiable gold answers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27334"],"document_share":0.0002723311546840959,"domain":"Agents","name":"BTS-AgentBench","organization_count":1,"organizations":["claire_radar"],"rank":479,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27334"},{"aliases":["Companion-style QA Assistance in Ego-Vision"],"benchmark_id":"claire-radar-2609-06721","caveat":"BuddyVQA evaluates model performance on companion-style question answering in streaming egocentric video, including ego-deictic and chained questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06721"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BuddyVQA","organization_count":1,"organizations":["claire_radar"],"rank":480,"released":"2026-09-06","source":"claire_radar","url":"https://arxiv.org/abs/2609.06721"},{"aliases":[],"benchmark_id":"claire-radar-2607-03523","caveat":"The work introduces BugSourceBench, a code repair benchmark with bugs from human-written, LM-generated, and human-edited LM-generated code. The benchmark evaluates the fix rate of language models on these bug sources. No scoring contract, dataset, or public access path is provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03523"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"BugSourceBench","organization_count":1,"organizations":["claire_radar"],"rank":481,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.03523"},{"aliases":[],"benchmark_id":"claire-radar-2608-02685","caveat":"BulkPR-Bench evaluates governance of interacting pull requests. It includes 581 candidate PRs on 18 repositories, with metrics RDS and Global-SGY measuring safe delivery. The benchmark uses executable repository execution with hidden safety checks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02685"],"document_share":0.0002723311546840959,"domain":"Agents","name":"BulkPR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":482,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02685"},{"aliases":["BullingerDB: A Dataset for Handwritten Text Recognition and Writer Retrieval"],"benchmark_id":"claire-radar-2605-30235","caveat":"A large-scale historical-document benchmark for multilingual handwritten text recognition and time-aware writer retrieval.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30235"],"document_share":0.0002723311546840959,"domain":"General AI","name":"BullingerDB","organization_count":1,"organizations":["claire_radar"],"rank":483,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30235"},{"aliases":[],"benchmark_id":"opencompass-1146-bust","caveat":"BUST is a comprehensive benchmark for evaluating synthetic text detectors, focusing on their effectiveness against outputs from various Large Language Models (LLMs). BUST 是一个综合基准，旨在评估合成文本检测器，BUST 使用多种指标来评估检测器，包括语言特征、可读性和作者态度。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/BUST"],"document_share":0.0002723311546840959,"domain":"理解","name":"BUST","organization_count":1,"organizations":["opencompass_hub"],"rank":484,"released":"2024-06-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/BUST"},{"aliases":[],"benchmark_id":"opencompass-1983-bytemorph","caveat":"ByteMorph is a benchmark for instruction-guided image editing, focusing on evaluating models’ capabilities in handling non-rigid motions such as camera viewpoint changes, object deformations, human articulations, and complex interactions. ByteMorph 是一个面向指令驱动图像编辑的基准，专注于评估模型在处理非刚性运动（如相机视角变化、物体变形、人类动作和复杂交互）方面的能力。 该基准包括超过 600 万对高分辨率图像编辑样本，涵盖多种动态编辑场景，支持对模型在多种非刚性运动类型下的表现进行细粒度评估。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ByteMorph"],"document_share":0.0002723311546840959,"domain":"创作","name":"ByteMorph","organization_count":1,"organizations":["opencompass_hub"],"rank":485,"released":"2025-06-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ByteMorph"},{"aliases":[],"benchmark_id":"llm-stats-c-eval","caveat":"C-Eval is a comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. It comprises 13,948 multiple-choice questions across 52 diverse disciplines spanning humanities, science, and engineering, with four difficulty levels: middle school, high school, college, and professional. The benchmark includes C-Eval Hard, a subset of very challenging subjects requiring advanced reasoning abilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/c-eval?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"C-Eval","organization_count":1,"organizations":["llm_stats"],"rank":486,"released":"2023-05-15","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/c-eval?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-496-c-eval","caveat":"C-Eval is a comprehensive Chinese evaluation suite for foundation models. It consists of 13948 multi-choice questions spanning 52 diverse disciplines and four difficulty levels. C-Eval 是一个全面的中文基础模型评估套件。它包含了13948个多项选择题，涵盖了52个不同的学科和四个难度级别。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/C-Eval"],"document_share":0.0002723311546840959,"domain":"学科","name":"C-Eval","organization_count":1,"organizations":["opencompass_hub"],"rank":487,"released":"2023-05-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/C-Eval"},{"aliases":[],"benchmark_id":"opencompass-1780-c-faith","caveat":"C-FAITH, a Chinese QA hallucination benchmark created from 1,399 knowledge documents obtained from web scraping, totaling 60,702 entries. C-FAITH，这是一个中国 QA 幻觉基准，由从网络抓取中获得的 1,399 份知识文档创建，总共 60,702 个条目。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/C-FAITH"],"document_share":0.0002723311546840959,"domain":"语言","name":"C-FAITH","organization_count":1,"organizations":["opencompass_hub"],"rank":488,"released":"2025-04-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/C-FAITH"},{"aliases":["Who Judges the Judges? A Chinese Safety QA Benchmark for Evaluating LLM Responses and Safety Judges"],"benchmark_id":"claire-radar-2609-01210","caveat":"A policy-grounded benchmark for response-level Chinese safety evaluation, assessing LLM responses and safety judges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01210"],"document_share":0.0002723311546840959,"domain":"General AI","name":"C-SafeQA","organization_count":1,"organizations":["claire_radar"],"rank":489,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01210"},{"aliases":[],"benchmark_id":"claire-radar-2608-05864","caveat":"C-SUITEBENCH evaluates multimodal LLMs as CEOs on five decision tasks under paired text-only and multimodal conditions across 50 scenarios, focusing on evidence-centric reasoning and constraint satisfaction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05864"],"document_share":0.0002723311546840959,"domain":"General AI","name":"C-SUITEBENCH","organization_count":1,"organizations":["claire_radar"],"rank":490,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05864"},{"aliases":[],"benchmark_id":"opencompass-514-c3","caveat":"A free-form multiple-Choice Chinese machine reading Comprehension dataset (C3), containing 13,369 documents (dialogues or more formally written mixed-genre texts) and their associated 19,577 multiple-choice free-form questions collected from Chinese-as-a-second-language examinations 一个自由形式的多项选择中文机器阅读理解数据集（C3），包含13369篇文献（对话或更正式的混合体裁文本）及其相关的19577道自由选择题，这些问题都是从汉语作为第二语言的考试中收集到的","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/C3"],"document_share":0.0002723311546840959,"domain":"理解","name":"C3","organization_count":1,"organizations":["opencompass_hub"],"rank":491,"released":"2019-04-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/C3"},{"aliases":[],"benchmark_id":"claire-radar-2606-25445","caveat":"C3-Bench evaluates context-aware change captioning with 4,996 human-labeled image pairs across 51 real-world contexts, using an LLM-as-Judge framework scoring correctness, specificity, fluency, relevance, and a reversibility metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25445"],"document_share":0.0002723311546840959,"domain":"General AI","name":"C3-Bench","organization_count":1,"organizations":["claire_radar"],"rank":492,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25445"},{"aliases":[],"benchmark_id":"claire-radar-2608-09296","caveat":"CADEngBench evaluates parametric CAD part generation and editing through B-Rep validity, engineering checks, parameter perturbation, and linear-static FEA, plus assembly reasoning via joint retrieval, grounding, and kinematic verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09296"],"document_share":0.0002723311546840959,"domain":"CAD","name":"CADEngBench","organization_count":1,"organizations":["claire_radar"],"rank":493,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09296"},{"aliases":[],"benchmark_id":"claire-radar-2608-17550","caveat":"CADP-Bench evaluates multimodal LLMs on compilable academic document parsing. It includes expert-verified full academic pages with tightly coupled text and structured elements, assessed through a re-injection compilation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17550"],"document_share":0.0002723311546840959,"domain":"Academic document parsing","name":"CADP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":494,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17550"},{"aliases":["CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design"],"benchmark_id":"claire-radar-2609-16251","caveat":"Evaluates long-horizon computer use in FreeCAD through 200 tasks spanning 11 mechanical-CAD workflow categories, with success determined by executable checks over saved artifacts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16251"],"document_share":0.0002723311546840959,"domain":"CAD","name":"CADWorld","organization_count":1,"organizations":["claire_radar"],"rank":495,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.16251"},{"aliases":[],"benchmark_id":"claire-radar-2606-00548","caveat":"CAFOSat is a dataset of over 45,000 image patches with infrastructure-level annotations for CAFO mapping across 20 states, benchmarking models for infrastructure-aware agricultural monitoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00548"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CAFOSat","organization_count":1,"organizations":["claire_radar"],"rank":496,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00548"},{"aliases":["CalArena: A Large-Scale Post-Hoc Calibration Benchmark"],"benchmark_id":"claire-radar-2605-30188","caveat":"CalArena is a standardized benchmark for comparing post-hoc calibration methods across tabular and computer-vision classification settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30188"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CalArena","organization_count":1,"organizations":["claire_radar"],"rank":497,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30188"},{"aliases":[],"benchmark_id":"claire-radar-2607-22635","caveat":"CallBench is a Chinese benchmark for evaluating dual-goal coordination in phone call assistants, with 50,000 multi-turn dialogues across six scenarios and a preset-aware turn-level scoring protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22635"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CallBench","organization_count":1,"organizations":["claire_radar"],"rank":498,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.22635"},{"aliases":[],"benchmark_id":"opencompass-1052-calm","caveat":"CaLM is the first comprehensive benchmark for evaluating the causal reasoning capabilities of language models. The CaLM framework establishes a foundational taxonomy consisting of four modules: causal target, adaptation, metric, and error. CaLM是上海人工智能实验室联合同济大学、上海交通大学、北京大学及商汤科技发布首个大模型因果推理开放评测体系。首次从因果推理角度提出评估框架，为AI研究者打造可靠评测工具，从而为推进大模型认知能力向人类水平看齐提供指标参考。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CaLM"],"document_share":0.0002723311546840959,"domain":"推理","name":"CaLM","organization_count":1,"organizations":["opencompass_hub"],"rank":499,"released":"2024-05-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CaLM"},{"aliases":[],"benchmark_id":"claire-radar-2606-04986","caveat":"CalorieBench-80K is a food image benchmark with calorie labels and dietary advice annotations, used to evaluate Food-R1 and other models for food analysis tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04986"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"CalorieBench-80K","organization_count":1,"organizations":["claire_radar"],"rank":500,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04986"},{"aliases":["Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation"],"benchmark_id":"claire-radar-2608-10932","caveat":"Evaluates temporally grounded compositional camera motion understanding using 4,229 single-shot clips with expert-annotated segments and 20 direction-aware labels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10932"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CamChoreo","organization_count":1,"organizations":["claire_radar"],"rank":501,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10932"},{"aliases":["Can AI Make Money in Crypto? Measuring the Gap from Backtests to Real Markets"],"benchmark_id":"claire-radar-2609-34510","caveat":"Evaluates machine learning, reinforcement learning, LLM-based, and agent-based trading methods in cryptocurrency markets across three stages: historical backtesting, prospective exchange-based paper trading, and real-money live trading.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34510"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Can AI Make Money in Crypto?","organization_count":1,"organizations":["claire_radar"],"rank":502,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34510"},{"aliases":[],"benchmark_id":"claire-radar-2609-10394","caveat":"Audio-visual speech recognition on natural dyadic videoconference conversations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10394"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Candor-LR","organization_count":1,"organizations":["claire_radar"],"rank":503,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10394"},{"aliases":[],"benchmark_id":"claire-radar-2605-30497","caveat":"CanLegalRAGBench evaluates retrieval-augmented generation for Canadian case law with realistic queries and expert-annotated answers. It tests retrieval and answer generation grounded in legal documents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30497"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CanLegalRAGBench","organization_count":1,"organizations":["claire_radar"],"rank":504,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30497"},{"aliases":[],"benchmark_id":"claire-radar-2607-20518","caveat":"Proposes a benchmark for AI-generated operator code on Huawei's Ascend NPU, covering 53 operators and 1060 test cases. Evaluation uses a three-dimensional weighted composite score for compilation, functional correctness, and performance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20518"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CANN Bench","organization_count":1,"organizations":["claire_radar"],"rank":505,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.20518"},{"aliases":[],"benchmark_id":"claire-radar-2608-08392","caveat":"CAP evaluates cross-site browser agents on 420 tasks across 108 real websites and 24 domains. Tasks require complex UI interactions and visual perception, with scoring via an agent-as-a-judge framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08392"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CAP","organization_count":1,"organizations":["claire_radar"],"rank":506,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08392"},{"aliases":["How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation"],"benchmark_id":"claire-radar-2609-01369","caveat":"Evaluates semantic correctness of open-domain QA answers using 8.8k examples derived from widely used QA datasets, with a framework for answer verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01369"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CAP-Correctness","organization_count":1,"organizations":["claire_radar"],"rank":507,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01369"},{"aliases":["capability-overload"],"benchmark_id":"claire-radar-github-bizhanchik-capability-overload","caveat":"Evaluates tool-selection reliability, task success, unnecessary tool calls, and token cost in LLM agents under varying catalog sizes (6-48) and semantic overlap levels (low/medium/high) across 40 tasks, with deterministic grading and multiple provider adapters.","document_count":1,"document_ids":["claire_radar:https://github.com/bizhanchik/capability-overload"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Capability Overload Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":508,"released":"2026-09-11","source":"claire_radar","url":"https://github.com/bizhanchik/capability-overload"},{"aliases":[],"benchmark_id":"claire-radar-2609-17688","caveat":"CapMem is a benchmark for caption-based episodic memory in egocentric video with 1,000 multiple-choice questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17688"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CapMem","organization_count":1,"organizations":["claire_radar"],"rank":509,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17688"},{"aliases":[],"benchmark_id":"claire-radar-2608-11074","caveat":"CapProbe is a full-scene dense QA benchmark for evaluating detailed image captions from Vision-Language Models. It decomposes images into semantic regions and generates multiple-choice questions across 10 semantic categories, with a language judge answering from captions. The benchmark comprises 346 images, 1,868 regions, and 25,650 questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11074"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CapProbe","organization_count":1,"organizations":["claire_radar"],"rank":510,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.11074"},{"aliases":["Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering"],"benchmark_id":"claire-radar-2609-09973","caveat":"Evaluates video caption quality through human-verified multiple-choice questions across 10 types and 24 domains, measuring information fidelity as coverage and factuality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09973"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CapQuiz","organization_count":1,"organizations":["claire_radar"],"rank":511,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09973"},{"aliases":[],"benchmark_id":"claire-radar-2606-21949","caveat":"CapRiCorn-1K evaluates video captioning quality and subject referential consistency across long videos (15s-10min) with audiovisual and visual-only settings. It uses LLM judge to compute accuracy, coverage, and referential consistency metrics based on manual annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21949"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CapRiCorn-1K","organization_count":1,"organizations":["claire_radar"],"rank":512,"released":"2026-06-20","source":"claire_radar","url":"https://arxiv.org/abs/2606.21949"},{"aliases":[],"benchmark_id":"llm-stats-capture-the-flag-challenges","caveat":"Capture-the-Flag Challenges is OpenAI's internal expansion of competitive, professional-level cybersecurity capture-the-flag tasks used to evaluate vulnerability identification and exploitation capability under the Preparedness Framework.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/capture-the-flag-challenges?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"Capture-the-Flag Challenges (Internal)","organization_count":1,"organizations":["llm_stats"],"rank":513,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/capture-the-flag-challenges?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-25763","caveat":"CaptureGuide-Bench evaluates capture-time photography guidance with two tasks: photographer-side composition decision/refinement and subject-side scene-conditioned pose recommendation, using metrics like IoU and plausibility.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25763"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CaptureGuide-Bench","organization_count":1,"organizations":["claire_radar"],"rank":514,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25763"},{"aliases":[],"benchmark_id":"claire-radar-2606-00123","caveat":"CardioLens evaluates MLLMs on multi-sequence cardiac MRI interpretation, covering image understanding, report generation, and disease diagnosis using QA pairs from private hospital archives.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00123"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"CardioLens","organization_count":1,"organizations":["claire_radar"],"rank":515,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2606.00123"},{"aliases":[],"benchmark_id":"claire-radar-2608-03731","caveat":"CARE-Bench is a source-grounded benchmark for patient-facing triage evaluation with 500 cases and 1,059 patient-disclosure prefixes. It evaluates LLMs on a four-label current-action task across 269 held-out rounds.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03731"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"CARE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":516,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03731"},{"aliases":[],"benchmark_id":"claire-radar-2606-29685","caveat":"CAREBench evaluates language models on upstream child-safety risks with 500 prompts across twelve categories, assessing recognition, refusal, de-escalation, and redirection.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29685"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CAREBench","organization_count":1,"organizations":["claire_radar"],"rank":517,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.29685"},{"aliases":[],"benchmark_id":"claire-radar-2605-23216","caveat":"CaST-Bench evaluates causal chain-grounded spatio-temporal reasoning in video question answering. It contains 2,066 questions over 1,015 videos with causal chains annotated as temporal segments and bounding-box tracks. The evaluation suite includes metrics for answer correctness and visual evidence grounding.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23216"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CaST-Bench","organization_count":1,"organizations":["claire_radar"],"rank":518,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23216"},{"aliases":[],"benchmark_id":"claire-radar-2606-01897","caveat":"CASTER-Bench evaluates whether user-generated content achieves positive community resonance based on multimodal attributes, using human annotations across diverse categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01897"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CASTER-Bench","organization_count":1,"organizations":["claire_radar"],"rank":519,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01897"},{"aliases":[],"benchmark_id":"claire-radar-2609-39533","caveat":"During reinforcement learning with verifiable rewards (RLVR), large language models (LLMs) can exploit loopholes in their environments to obtain high rewards without improving the intended capabilities, i.e., reward hacking.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39533"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"CATCH","organization_count":1,"organizations":["claire_radar"],"rank":520,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39533"},{"aliases":[],"benchmark_id":"claire-radar-2608-22808","caveat":"Assesses agent failure auditing across PRE, LIVE, and POST information states with seven task contracts covering evidential and Gold-derived diagnostics.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22808v1"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CatchBench","organization_count":1,"organizations":["claire_radar"],"rank":521,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.22808v1"},{"aliases":[],"benchmark_id":"claire-radar-2606-01810","caveat":"Causal-Plan-Bench evaluates embodied planning across four causal dimensions (executability, effects, composition, robustness) using 1,200 instances from 12 tasks, with MCQ and rubric-based scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01810"],"document_share":0.0002723311546840959,"domain":"Planning","name":"Causal-Plan-Bench","organization_count":1,"organizations":["claire_radar"],"rank":522,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01810"},{"aliases":["A Plausible Future Is Not a Causal Future: Can Video Models Generate How the World Unfolds?"],"benchmark_id":"claire-radar-2610-15562","caveat":"A simulator-grounded benchmark for assessing causal cascade generation in image-to-video models through event-rich scenes, motion metrics, and diagnostic question answering.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.15562"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CausalCascade","organization_count":1,"organizations":["claire_radar"],"rank":523,"released":"2026-10-11","source":"claire_radar","url":"https://arxiv.org/abs/2610.15562"},{"aliases":[],"benchmark_id":"claire-radar-2607-08093","caveat":"Evaluates causal reasoning in data-science workflows using synthetic scenes with hidden structural causal models, covering Pearl's three rungs and abstention scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08093"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CausalDS","organization_count":1,"organizations":["claire_radar"],"rank":524,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08093"},{"aliases":[],"benchmark_id":"claire-radar-2607-04293","caveat":"CausalGame is a benchmark for evaluating causal thinking in LLM agents through interactive games. It includes 14 scenarios with selection bias, measurement error, and hidden confounders. Agents design experimental protocols, collect data, and produce solutions with explanations, scored against analytical optima and causal-reasoning rubrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04293"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"CausalGame","organization_count":1,"organizations":["claire_radar"],"rank":525,"released":"2026-07-05","source":"claire_radar","url":"https://arxiv.org/abs/2607.04293"},{"aliases":[],"benchmark_id":"opencompass-1975-causalvqa","caveat":"CausalVQA tests causal reasoning in videos across five question types, and state-of-the-art multimodal models still trail human performance. CausalVQA 是面向视频问答的因果推理基准，涵盖反事实、假设、预判、规划、描述五类问题，强调真实物理场景。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CausalVQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"CausalVQA","organization_count":1,"organizations":["opencompass_hub"],"rank":526,"released":"2025-06-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CausalVQA"},{"aliases":[],"benchmark_id":"claire-radar-2610-06587","caveat":"AI voice assistants often use Automatic Speech Recognition (ASR) with LLM-based reasoning, yet existing systems struggle with regional British accents, including Scottish, Irish, and Welsh accents, since most ASR models are trained predomi…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.06587"],"document_share":0.0002723311546840959,"domain":"Finance","name":"CavaBench","organization_count":1,"organizations":["claire_radar"],"rank":527,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.06587"},{"aliases":[],"benchmark_id":"claire-radar-2609-32616","caveat":"LLM agents increasingly keep working after a task succeeds as they resume after compaction or take over handoffs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32616"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CAVE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":528,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32616"},{"aliases":[],"benchmark_id":"claire-radar-2609-27273","caveat":"Benchmark for computer-use agents in incentive-misaligned environments with nine marketplace environments and eight steering mechanisms.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27273"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CAVEAT","organization_count":1,"organizations":["claire_radar"],"rank":529,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27273"},{"aliases":[],"benchmark_id":"llm-stats-cbnsl","caveat":"Curriculum Learning of Bayesian Network Structures (CBNSL) benchmark for evaluating algorithms that learn Bayesian network structures from data using curriculum learning techniques. The benchmark uses networks from the bnlearn repository and evaluates structure learning performance using BDeu scoring metrics.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cbnsl?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"CBNSL","organization_count":1,"organizations":["llm_stats"],"rank":530,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cbnsl?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-15404","caveat":"CBX-Bench is a benchmark for quantitatively measuring the quality of Concept Bottleneck Model (CBM) explanations. It uses a council of five open-weight multimodal LLMs to score explanations given an image and class, validated against human preferences. The benchmark maintains a leaderboard for CBM explanation quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15404"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CBX-Bench","organization_count":1,"organizations":["claire_radar"],"rank":531,"released":"2026-08-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.15404"},{"aliases":[],"benchmark_id":"llm-stats-cc-bench-v2-backend","caveat":"CC-Bench-V2 Backend evaluates coding agents on backend development tasks, measuring practical engineering ability to implement server-side logic, APIs, and system components.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-backend?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"CC-Bench-V2 Backend","organization_count":1,"organizations":["llm_stats"],"rank":532,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-backend?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cc-bench-v2-frontend","caveat":"CC-Bench-V2 Frontend evaluates coding agents on frontend development tasks, measuring ability to build UI components, handle styling, and implement client-side logic.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-frontend?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"CC-Bench-V2 Frontend","organization_count":1,"organizations":["llm_stats"],"rank":533,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-frontend?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cc-bench-v2-repo","caveat":"CC-Bench-V2 Repo Exploration evaluates coding agents on repository-level understanding and navigation, measuring ability to explore, comprehend, and work across entire codebases.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-repo?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"CC-Bench-V2 Repo Exploration","organization_count":1,"organizations":["llm_stats"],"rank":534,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-bench-v2-repo?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cc-ocr","caveat":"A comprehensive OCR benchmark for evaluating Large Multimodal Models (LMMs) in literacy. Comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. Contains 39 subsets with 7,058 fully annotated images, 41% sourced from real applications. Tests capabilities including text grounding, multi-orientation text recognition, and detecting hallucination/repetition across diverse visual challenges.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cc-ocr?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"CC-OCR","organization_count":1,"organizations":["llm_stats"],"rank":535,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cc-ocr?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-30979","caveat":"Evaluates visual causal reasoning under constraints in single-image physical scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30979"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CCRV-Bench","organization_count":1,"organizations":["claire_radar"],"rank":536,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.30979"},{"aliases":["cdaf"],"benchmark_id":"claire-radar-github-uditakhourii-cdaf","caveat":"Benchmark compares sidecar-based video understanding against direct analysis on accuracy, prompt token usage, and latency using a reproducible setup with Gemini.","document_count":1,"document_ids":["claire_radar:https://github.com/UditAkhourii/cdaf"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CDAF — Cached Descriptive Asset Files","organization_count":1,"organizations":["claire_radar"],"rank":537,"released":"2026-08-26","source":"claire_radar","url":"https://github.com/UditAkhourii/cdaf"},{"aliases":[],"benchmark_id":"claire-radar-2606-31435","caveat":"CDR-Bench is a benchmark of 3,462 tasks for evaluating large language models on faithful execution of compositional, order-sensitive data refinement recipes across four domains and 29 operators, with deterministic reference outputs for exact evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31435"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CDR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":538,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31435"},{"aliases":[],"benchmark_id":"opencompass-1552-ceb","caveat":"CEB evaluates LLM bias compositionally, featuring 11k samples characterized across bias types, social groups, and tasks. CEB是一个用于大型语言模型偏差的组成评估基准，引入了包含 11,004 个样本的组成评估基准，从偏差类型、社会群体和任务三个维度描述每个数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CEB"],"document_share":0.0002723311546840959,"domain":"其他","name":"CEB","organization_count":1,"organizations":["opencompass_hub"],"rank":539,"released":"2024-07-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CEB"},{"aliases":[],"benchmark_id":"claire-radar-2606-17459","caveat":"CEO-Bench evaluates LLM agents on strategic resource allocation in multi-round organizational simulations with conflicting advisor inputs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17459"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CEO-Bench","organization_count":1,"organizations":["claire_radar"],"rank":540,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.17459"},{"aliases":[],"benchmark_id":"claire-radar-2606-18543","caveat":"CEO-Bench evaluates long-horizon agent capabilities by simulating a startup over 500 days. Agents manage pricing, marketing, budgeting, and other business aspects through a programmable Python interface, facing noisy data and changing market conditions. Performance is measured by final company balance against a rule-based baseline.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18543"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"CEO-Bench","organization_count":1,"organizations":["claire_radar"],"rank":541,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18543"},{"aliases":[],"benchmark_id":"claire-radar-2609-21344","caveat":"Lists 380 expert-written items across six sub-domains of cryptographic engineering security for IoT devices.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21344"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CESBench","organization_count":1,"organizations":["claire_radar"],"rank":542,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21344"},{"aliases":[],"benchmark_id":"claire-radar-2606-22000","caveat":"CFAgentBench is an executable environment for autonomous construction-finance agents, with 1,014 task specifications across 8 domains and 77 families. A subset of 40 tasks (54 with PM extension) has oracle-validated evaluators. Grading uses state diffs, forbidden-side-effect checks, and required-output regexes, with an LLM judge only for reply quality. A public split of 711 tasks is available, and a private split of 303 is reserved for remote scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22000"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CFAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":543,"released":"2026-06-20","source":"claire_radar","url":"https://arxiv.org/abs/2606.22000"},{"aliases":[],"benchmark_id":"llm-stats-cfeval","caveat":"CFEval benchmark for evaluating code generation and problem-solving capabilities","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cfeval?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"CFEval","organization_count":1,"organizations":["llm_stats"],"rank":544,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cfeval?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1907-cfinbench","caveat":"We present CFinBench: a meticulously crafted, the most comprehensive evaluation benchmark to date, for assessing the financial knowledge of LLMs under Chinese context. 为了更加全面地探究大语言模型在中文财经领域的能力，本工作提出了目前为止量级最大的中文财经评测基准（CFinBench）。该数据集共包含99,100个评测样本，并包含单选题、多选题和判断题在内的三种题型。该工作对当前主流的大模型从四个维度进行了详细评测：财经学科基础、财经资格认证、财经从业实践、财经法律法规。数据集和测评代码均已开源。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CFinBench"],"document_share":0.0002723311546840959,"domain":"知识","name":"CFinBench","organization_count":1,"organizations":["opencompass_hub"],"rank":545,"released":"2024-10-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CFinBench"},{"aliases":[],"benchmark_id":"opencompass-1079-cflue","caveat":"CFLUE is the Chinese Financial Language Understanding Evaluation benchmark, designed to assess the capability of LLMs across various dimensions. CFLUE 是中国金融语言理解评估基准，旨在评估大型语言模型（LLMs）在各个维度上的能力。具体而言，CFLUE 提供了针对知识评估和应用评估量身定制的数据集。在知识评估方面，它包含超过 38,000 道选择题及相关的解决方案解释。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CFLUE"],"document_share":0.0002723311546840959,"domain":"知识","name":"CFLUE","organization_count":1,"organizations":["opencompass_hub"],"rank":546,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CFLUE"},{"aliases":[],"benchmark_id":"claire-radar-2607-14975","caveat":"The evaluation object is unclear from the provided information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14975"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CFM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":547,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14975"},{"aliases":[],"benchmark_id":"opencompass-1513-cg-bench","caveat":"CG-Bench is meant for evaluating MLLMs' long video understanding, including 12,129 QA pairs from 1219 videos in 3 major question types: perception, reasoning, and hallucination. CG-Bench用于评估多模态大模型的长视频理解能力，基于1219个视频设计了12129个涵盖感知、推理和幻觉三种问题类型的QA对。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CG-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"CG-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":548,"released":"2024-12-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CG-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2607-02606","caveat":"ChainSWE evaluates coding agents on sequential, dependent bug fixes within a shared codebase. It includes 304 issues across 54 Python projects, forming chronological chains, and measures performance drop as chain length increases.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02606"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"ChainSWE","organization_count":1,"organizations":["claire_radar"],"rank":549,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.02606"},{"aliases":[],"benchmark_id":"claire-radar-2605-24305","caveat":"ChaosBench-Logic v2 evaluates LLM logical reasoning over dynamical systems with 40,886 questions across 165 systems, 27 first-order logic predicates, and 78 axiom edges. The CARE protocol measures calibration and adversarial robustness, reporting metrics such as MCC.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24305"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChaosBench-Logic","organization_count":1,"organizations":["claire_radar"],"rank":550,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24305"},{"aliases":[],"benchmark_id":"llm-stats-charadessta","caveat":"Charades-STA is a benchmark dataset for temporal activity localization via language queries, extending the Charades dataset with sentence temporal annotations. It contains 12,408 training and 3,720 testing segment-sentence pairs from videos with natural language descriptions and precise temporal boundaries for localizing activities based on language queries.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/charadessta?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"CharadesSTA","organization_count":1,"organizations":["llm_stats"],"rank":551,"released":"2017-05-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/charadessta?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1141-charm","caveat":"CHARM is the first benchmark for comprehensively and in-depth evaluating the commonsense reasoning ability of large language models (LLMs) in Chinese, which covers both globally known and Chinese-specific commonsense. CHARM 是首个全面深入评估大语言模型（LLMs）在中文中的常识推理能力的基准，涵盖了全球通用的常识和特定于中国的常识。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CHARM"],"document_share":0.0002723311546840959,"domain":"推理","name":"CHARM","organization_count":1,"organizations":["opencompass_hub"],"rank":552,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CHARM"},{"aliases":[],"benchmark_id":"claire-radar-2605-26994","caveat":"Interactive benchmark for dynamic chart understanding requiring GUI actions to obtain evidence, with 673 charts and 1,440 QA pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26994"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChartAct","organization_count":1,"organizations":["claire_radar"],"rank":553,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26994"},{"aliases":[],"benchmark_id":"claire-radar-2608-03464","caveat":"ChartAnno is a benchmark for evaluating MLLMs on chart annotation generation with 1,200 real-world charts and paired code/instructions. It evaluates annotation quality under different instruction specificity and input settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03464"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChartAnno","organization_count":1,"organizations":["claire_radar"],"rank":554,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03464"},{"aliases":[],"benchmark_id":"claire-radar-2606-01348","caveat":"ChartArena is a bilingual benchmark for chart parsing covering eight chart families (bar, line, pie, radar, box plot, combination, flowchart, mind map) across three visual scenarios (digital, printed, hand-drawn). It uses a format-agnostic evaluation protocol with structure-aware metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01348"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChartArena","organization_count":1,"organizations":["claire_radar"],"rank":555,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01348"},{"aliases":[],"benchmark_id":"claire-radar-2609-24210","caveat":"Evaluates LMM judges in chart-to-code workflows.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24210"],"document_share":0.0002723311546840959,"domain":"Code","name":"ChartJudgeBench","organization_count":1,"organizations":["claire_radar"],"rank":556,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24210"},{"aliases":[],"benchmark_id":"llm-stats-chartmuseum","caveat":"ChartMuseum is a chart question-answering benchmark of 1,162 expert-annotated questions over real-world chart images drawn from 184 sources, including academic figures, infographics, and unconventional chart designs. It specifically targets questions that require visual reasoning, such as comparing unlabeled visual elements, tracking trajectories, and judging spatial relationships.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/chartmuseum?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ChartMuseum","organization_count":1,"organizations":["llm_stats"],"rank":557,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/chartmuseum?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-10677","caveat":"Chartography evaluates vision-language models on 100 chart-understanding tasks sourced from professional practice, with questions authored by domain experts and triple-verified. The benchmark uses pass@1 accuracy as the scoring metric across 30 frontier model configurations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10677"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Chartography","organization_count":1,"organizations":["claire_radar"],"rank":558,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10677"},{"aliases":[],"benchmark_id":"llm-stats-chartqa","caveat":"ChartQA is a large-scale benchmark comprising 9.6K human-written questions and 23.1K questions generated from human-written chart summaries, designed to evaluate models' abilities in visual and logical reasoning over charts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/chartqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ChartQA","organization_count":1,"organizations":["llm_stats"],"rank":559,"released":"2022-03-19","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-chartqapro","caveat":"ChartQAPro is a challenging benchmark for question answering over diverse, real-world charts and infographics.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/chartqapro?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ChartQAPro","organization_count":1,"organizations":["llm_stats"],"rank":560,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/chartqapro?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-10301","caveat":"ChartSync evaluates image editing models on chart editing tasks, including text-only edits and visuo-logical cascading edits (VLCE) requiring synchronized text and geometry changes. The benchmark contains 870 expert-validated triplets across nine chart families, with objective visual metrics and a vision-language model judge for evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10301"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChartSync","organization_count":1,"organizations":["claire_radar"],"rank":561,"released":"2026-07-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.10301"},{"aliases":["CharXiv Reasoning","CharXiv"],"benchmark_id":"charxiv_reasoning","caveat":"Chart reasoning benchmark with tool access. Scores depend on context length and tool configuration.","document_count":1,"document_ids":["model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0002723311546840959,"domain":"vision","name":"CharXiv Reasoning","organization_count":1,"organizations":["Z.ai"],"rank":562,"released":"2025-06-01","source":"model_reports","url":"https://github.com/CharXiv/CharXiv"},{"aliases":[],"benchmark_id":"llm-stats-charxiv-d","caveat":"CharXiv-D is the descriptive questions subset of the CharXiv benchmark, designed to assess multimodal large language models' ability to extract basic information from scientific charts. It contains descriptive questions covering information extraction, enumeration, pattern recognition, and counting across 2,323 diverse charts from arXiv papers, all curated and verified by human experts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/charxiv-d?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"CharXiv-D","organization_count":1,"organizations":["llm_stats"],"rank":563,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-d?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-charxiv-r","caveat":"CharXiv-R is the reasoning component of the CharXiv benchmark, focusing on complex reasoning questions that require synthesizing information across visual chart elements. It evaluates multimodal large language models on their ability to understand and reason about scientific charts from arXiv papers through various reasoning tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/charxiv-r?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"CharXiv-R","organization_count":1,"organizations":["llm_stats"],"rank":564,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/charxiv-r?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1534-chase-code","caveat":"CHASE is a unified framework to synthetically generate challenging problems using LLMs without human involvement CHASE是一个无需人工参与的统一框架，用于合成生成具有挑战性的问题","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CHASE-Code"],"document_share":0.0002723311546840959,"domain":"代码","name":"CHASE-Code","organization_count":1,"organizations":["opencompass_hub"],"rank":565,"released":"2025-02-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CHASE-Code"},{"aliases":["Chatbot Arena","LMArena","LMSYS Arena"],"benchmark_id":"chatbot_arena","caveat":"Elo from real user traffic; sampling and prompt distribution are outside any vendor's control.","document_count":1,"document_ids":["model_reports:meta_llama_4"],"document_share":0.0002723311546840959,"domain":"human_preference","name":"Chatbot Arena","organization_count":1,"organizations":["Meta"],"rank":566,"released":"2023-05-03","source":"model_reports","url":"https://lmarena.ai/"},{"aliases":[],"benchmark_id":"claire-radar-2609-36308","caveat":"Reinforcement learning has helped AI agents solve increasingly difficult tasks, but high rewards do not always reflect the work users intended.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36308"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CheatBench","organization_count":1,"organizations":["claire_radar"],"rank":567,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.36308"},{"aliases":[],"benchmark_id":"claire-radar-2610-07557","caveat":"Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.07557"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"CheckerBench","organization_count":1,"organizations":["claire_radar"],"rank":568,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.07557"},{"aliases":[],"benchmark_id":"claire-radar-2606-21657","caveat":"Chehre is a video dataset of 2,111 facial expressions prompted by 40 emojis, with annotations transferred to synthetic faces. It defines two tasks: dominant expression recognition and distributional expression recognition, evaluating models' ability to predict human-rated labels and capture response diversity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21657"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Chehre","organization_count":1,"organizations":["claire_radar"],"rank":569,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21657"},{"aliases":[],"benchmark_id":"claire-radar-2607-28079","caveat":"Chem World integrates 17 chemical datasets with 800,000+ molecules for property prediction, offering a unified evaluation platform.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28079"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Chem World","organization_count":1,"organizations":["claire_radar"],"rank":570,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28079"},{"aliases":[],"benchmark_id":"claire-radar-2606-21109","caveat":"Chem2Gen-Bench evaluates chemical-to-genetic translation using 260,084 chemical and 1,099,045 genetic perturbation profiles in cell-target contexts. It measures pairwise alignment, retrieval success, and representation quality across matched perturbation settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21109"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Chem2Gen-Bench","organization_count":1,"organizations":["claire_radar"],"rank":571,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21109"},{"aliases":[],"benchmark_id":"opencompass-692-chembench","caveat":"ChemBench is a large-scale chemistry competency evaluation benchmark for language models, which includes nine chemistry core tasks and 4100 high-quality single-choice questions and answers. ChemBench是一个包含了九项化学核心任务，4100个高质量单选问答的大语言模型化学能力评测基准.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ChemBench"],"document_share":0.0002723311546840959,"domain":"知识","name":"ChemBench","organization_count":1,"organizations":["opencompass_hub"],"rank":572,"released":"2024-02-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ChemBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-23231","caveat":"Evaluates cross-lingual retrieval models on a multilingual chemical patent corpus with generated question-answer pairs; performance is measured by Recall@10 and MRR@10.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23231"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChemCLIR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":573,"released":"2026-09-19","source":"claire_radar","url":"https://arxiv.org/abs/2609.23231"},{"aliases":[],"benchmark_id":"claire-radar-2606-03660","caveat":"ChemCoTBench-V2 evaluates chemical reasoning at the process level, with 5,620 samples across molecular understanding, editing, optimization, and reaction prediction. It checks intermediate steps using deterministic chemistry rules and reference traces.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03660"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"ChemCoTBench-V2","organization_count":1,"organizations":["claire_radar"],"rank":574,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03660"},{"aliases":["cheme-ml-benchmarks"],"benchmark_id":"claire-radar-github-yukevindai-cheme-ml-benchmarks","caveat":"A locally runnable evaluation suite for machine learning methods on chemical engineering tasks, including concrete strength and airfoil self-noise datasets plus synthetic fixtures, with fixed splits, preprocessing, scoring metrics, and leaderboard generation.","document_count":1,"document_ids":["claire_radar:https://github.com/yukevindai/cheme-ml-benchmarks"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChemE ML Benchmarks","organization_count":1,"organizations":["claire_radar"],"rank":575,"released":"2026-09-07","source":"claire_radar","url":"https://github.com/yukevindai/cheme-ml-benchmarks"},{"aliases":[],"benchmark_id":"claire-radar-2605-30100","caveat":"Chess-World-Model evaluates state tracking by predicting the exact board state after sequences of legal moves from 10 million chess games, with accuracy as the metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30100"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Chess-World-Model","organization_count":1,"organizations":["claire_radar"],"rank":576,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30100"},{"aliases":[],"benchmark_id":"llm-stats-chexpert-cxr","caveat":"CheXpert is a large dataset of 224,316 chest radiographs from 65,240 patients for automated chest X-ray interpretation. The dataset includes uncertainty labels for 14 medical observations extracted from radiology reports. It serves as a benchmark for developing and evaluating automated chest radiograph interpretation models.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/chexpert-cxr?top_n=500"],"document_share":0.0002723311546840959,"domain":"healthcare","name":"CheXpert CXR","organization_count":1,"organizations":["llm_stats"],"rank":577,"released":"2019-01-21","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/chexpert-cxr?top_n=500"},{"aliases":["Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory"],"benchmark_id":"claire-radar-2609-03394","caveat":"Human-annotated benchmark of 1,000 sentences where each scene elicits opposing emotions in two agents, evaluated via macro-F1 for both LLMs and classifiers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03394"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CHIARO","organization_count":1,"organizations":["claire_radar"],"rank":578,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03394"},{"aliases":[],"benchmark_id":"opencompass-505-chid","caveat":"CHID is a chinese idiom reading comprehension task, which requires to select the correct idiom to fill in the blank according to the context, with 10 candidate idioms. CHID是一个中文成语阅读理解任务，要求根据上下文选择正确的成语填空，共有10个候选成语。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CHID"],"document_share":0.0002723311546840959,"domain":"语言","name":"CHID","organization_count":1,"organizations":["opencompass_hub"],"rank":579,"released":"2019-06-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CHID"},{"aliases":[],"benchmark_id":"claire-radar-2605-27805","caveat":"ChildEval is a benchmark for evaluating LLMs' ability to infer and follow child-centered preferences in long-context conversations. It contains 29K synthesized persona profiles of children aged 3-6, with explicit and implicit preference expressions across five top-level and fourteen sub-level categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27805"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChildEval","organization_count":1,"organizations":["claire_radar"],"rank":580,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27805"},{"aliases":[],"benchmark_id":"claire-radar-2609-06353","caveat":"Benchmark dataset proposed for collaborative behavior understanding in children during naturalistic interactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06353"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChildGaze","organization_count":1,"organizations":["claire_radar"],"rank":581,"released":"2026-09-06","source":"claire_radar","url":"https://arxiv.org/abs/2609.06353"},{"aliases":[],"benchmark_id":"claire-radar-2609-21637","caveat":"Intended to evaluate LLM understanding of competitive Chinese-language debate at match, stage, and speaker levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21637"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Chinese Competitive Debating Dataset and Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":582,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21637"},{"aliases":["ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning","ChitraMiti-12.8k"],"benchmark_id":"claire-radar-2609-12509","caveat":"ChitraMiti-12.8k is a synthetic Bengali planar geometry benchmark of 12,874 problems with structured 15-attribute diagram descriptions, complemented by NCTB-500, 500 textbook-extracted diagrams, targeting visual grounding and modality reliance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12509"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"ChitraMiti-12.8k","organization_count":1,"organizations":["claire_radar"],"rank":583,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12509"},{"aliases":[],"benchmark_id":"claire-radar-2606-17905","caveat":"ChLogic is an English–Chinese aligned benchmark for evaluating logical reasoning robustness across surface realizations. It includes 3,000 general, 2,000 difficult, and 1,500 Chinese-only items derived from formal logical templates, with each aligned item pairing one English reference with five Chinese variants.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17905"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChLogic","organization_count":1,"organizations":["claire_radar"],"rank":584,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.17905"},{"aliases":[],"benchmark_id":"claire-radar-2607-17999","caveat":"ChoroplethMap-Bench evaluates spatial understanding of foundation models with 2,400 synthetic choropleth maps, corresponding GeoJSON data, and 12,000 questions across five cognitive dimensions (Identify, Spatial Recognition, Compare, Rank, Delineate). Models are assessed under Data Only, Map Only, and Data + Map conditions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.17999"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChoroplethMap-Bench","organization_count":1,"organizations":["claire_radar"],"rank":585,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.17999"},{"aliases":[],"benchmark_id":"claire-radar-2607-15768","caveat":"ChronoBench evaluates long-term temporal understanding in remote sensing across four progressive cognitive levels: land cover perception, temporal recognition, long-term memory, and spatio-temporal reasoning. It comprises 12 sub-tasks and 17,689 QA pairs over 3,469 images spanning 500 regions across 39 U.S. cities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15768"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChronoBench","organization_count":1,"organizations":["claire_radar"],"rank":586,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.15768"},{"aliases":[],"benchmark_id":"opencompass-1278-chronomagic-bench","caveat":"ChronoMagic-Bench can evaluate the temporal and metamorphic capabilities of the T2V (text-to-video) models in time-lapse video generation, introducing 1,649 prompts and real-world videos as references. ChronoMagic-Bench用来评估 T2V （文本到视频 ）模型在延时视频生成中的时间和变形能力，引入了1649个提示和真实世界的视频作为参考。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ChronoMagic-Bench"],"document_share":0.0002723311546840959,"domain":"创作","name":"ChronoMagic-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":587,"released":"2024-06-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ChronoMagic-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2606-07962","caveat":"ChronoPhyBench evaluates multimodal LLMs on chronological physical dynamics reasoning via next-state prediction and VQA, using video frames and captions for single-image selection and multi-frame sorting.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07962"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"ChronoPhyBench","organization_count":1,"organizations":["claire_radar"],"rank":588,"released":"2026-06-06","source":"claire_radar","url":"https://arxiv.org/abs/2606.07962"},{"aliases":[],"benchmark_id":"claire-radar-2607-14770","caveat":"The evaluation object is unclear from the provided information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14770"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"ChronoQG","organization_count":1,"organizations":["claire_radar"],"rank":589,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14770"},{"aliases":[],"benchmark_id":"claire-radar-2608-09124","caveat":"ChronoState evaluates whether a frozen language model can compose hidden elapsed-time scalars with symbolic task state to select temporal actions, using forced-choice accuracy under direct supervision.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09124"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ChronoState","organization_count":1,"organizations":["claire_radar"],"rank":590,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09124"},{"aliases":[],"benchmark_id":"claire-radar-2609-11515","caveat":"Measures staleness errors in agentic AI answers under temporal drift across heterogeneous enterprise data sources.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11515"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"ChurnBench","organization_count":1,"organizations":["claire_radar"],"rank":591,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11515"},{"aliases":[],"benchmark_id":"llm-stats-ci-memories-coverage","caveat":"CI Memories measures privacy behavior in memory-enabled agents using contextual-integrity scenarios. This metric reports evaluation coverage.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-coverage?top_n=500"],"document_share":0.0002723311546840959,"domain":"memory","name":"CI Memories Coverage","organization_count":1,"organizations":["llm_stats"],"rank":592,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-coverage?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ci-memories-violation","caveat":"CI Memories measures privacy failures in memory-enabled agents using contextual-integrity scenarios. This metric is the violation rate; lower is better.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-violation?top_n=500"],"document_share":0.0002723311546840959,"domain":"memory","name":"CI Memories Violation Rate","organization_count":1,"organizations":["llm_stats"],"rank":593,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ci-memories-violation?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-11063","caveat":"CIAware-Bench measures control intervention awareness in language models through four task domains: essay writing, BigCodeBench, Bash Arena, and SHADE-Arena. Models are tested on distinguishing their own trajectories from those modified by a control protocol, with variations in watermarking, side-task presence, and protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11063"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CIAware-Bench","organization_count":1,"organizations":["claire_radar"],"rank":594,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11063"},{"aliases":[],"benchmark_id":"claire-radar-2609-21722","caveat":"Evaluates Chinese-to-English cross-lingual understanding of internet buzzwords through explanation, equivalence selection, and harmfulness classification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21722"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CIBuzzBench","organization_count":1,"organizations":["claire_radar"],"rank":595,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21722"},{"aliases":[],"benchmark_id":"claire-radar-2609-36218","caveat":"Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering, and cybersecurity, yet film remains comparatively underexplored despite requiring long-form narrative integration, multili…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36218"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CineSubBench","organization_count":1,"organizations":["claire_radar"],"rank":596,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.36218"},{"aliases":[],"benchmark_id":"claire-radar-2609-38863","caveat":"The two-dimensional irregular knapsack problem in a fixed circular container is an important combinatorial optimization problem for maximizing material utilization in manufacturing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38863"],"document_share":0.0002723311546840959,"domain":"CAD","name":"CircleNest-Bench","organization_count":1,"organizations":["claire_radar"],"rank":597,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.38863"},{"aliases":["CircuitReason-1k: Benchmarking Long-Horizon Visual-to-Symbolic Reasoning inElectrical Circuits"],"benchmark_id":"claire-radar-2608-09374","caveat":"Evaluates long-horizon visual-to-symbolic reasoning on 1,000 authentic circuit problems, combining typed scoring with multi-model semantic consensus, covering topology grounding, physical modeling, and symbolic computation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09374"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CircuitReason-1k","organization_count":1,"organizations":["claire_radar"],"rank":598,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09374"},{"aliases":[],"benchmark_id":"claire-radar-2608-00018","caveat":"CITBench evaluates LLMs on interactive tabular data processing, covering table matching, cleaning, augmentation, and transformation across 18 task types and 1,296 instances.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00018"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CITBench","organization_count":1,"organizations":["claire_radar"],"rank":599,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.00018"},{"aliases":["GridFlow: Structured Latent Flow for Seamless City-Scale 3D Point Cloud Generation"],"benchmark_id":"claire-radar-2608-29793","caveat":"City3D-MultiGen provides 163K densely annotated tiles with aligned point clouds, satellite images, semantic maps, and elevation data for city-scale 3D point cloud generation evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29793"],"document_share":0.0002723311546840959,"domain":"General AI","name":"City3D-MultiGen","organization_count":1,"organizations":["claire_radar"],"rank":600,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29793"},{"aliases":[],"benchmark_id":"claire-radar-2605-26036","caveat":"CityRep is a benchmark for urban representation learning across 8 cities and 8 tasks with spatially structured splits to mitigate leakage.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26036"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CITYREP","organization_count":1,"organizations":["claire_radar"],"rank":601,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26036"},{"aliases":[],"benchmark_id":"claire-radar-2606-02287","caveat":"CityTrajBench evaluates city-scale vehicle trajectory generation methods with standardized preprocessing, model adaptation, and multi-level evaluation across statistical, VAE, GAN, diffusion, and flow-matching models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02287"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CityTrajBench","organization_count":1,"organizations":["claire_radar"],"rank":602,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02287"},{"aliases":[],"benchmark_id":"claire-radar-2609-02459","caveat":"Evaluates language model agents in long-horizon, tool-mediated Civilization VI gameplay via 76 MCP tools.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02459"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CivBench","organization_count":1,"organizations":["claire_radar"],"rank":603,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02459"},{"aliases":[],"benchmark_id":"llm-stats-cl-bench","caveat":"CL-bench is an open-source benchmark with its own data and rubrics for evaluating models on coding and agentic tasks, scored using a setup fully aligned with the official procedure.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cl-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"CL-bench","organization_count":1,"organizations":["llm_stats"],"rank":604,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cl-bench-life","caveat":"CL-bench Life variant.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cl-bench-%28life%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"CL-bench (Life)","organization_count":1,"organizations":["llm_stats"],"rank":605,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cl-bench-%28life%29?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-00711","caveat":"ClarifyCodeBench evaluates LLMs on clarifying ambiguous requirements for code generation through interactive dialogues. It includes manual annotations of ambiguity types, clarification questions, and ground-truth answers, with metrics for interaction quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00711"],"document_share":0.0002723311546840959,"domain":"Code","name":"ClarifyCodeBench","organization_count":1,"organizations":["claire_radar"],"rank":606,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.00711"},{"aliases":[],"benchmark_id":"claire-radar-2609-19892","caveat":"ClashBench comprises 268 validated conflict cases across 55 resource types for evaluating agent behavior.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19892"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ClashBench","organization_count":1,"organizations":["claire_radar"],"rank":607,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.19892"},{"aliases":[],"benchmark_id":"claire-radar-2607-05185","caveat":"ClassicLogic is a benchmark suite of four classic logic puzzles (Sudoku, KenKen, Kakuro, Futoshiki) with a hierarchical knowledge base that defines complex strategies as compositions of simpler ones. It evaluates an agent's compositional generalization in problem-solving from basic rules to multi-step strategies across increasing puzzle difficulty.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05185"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ClassicLogic","organization_count":1,"organizations":["claire_radar"],"rank":608,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.05185"},{"aliases":["Cross-lingual Biography Enrichment via Claim Extraction and Alignment"],"benchmark_id":"claire-radar-2608-23390","caveat":"Evaluates cross-lingual biography enrichment on 300 Wikipedia biography pairs with claim annotations and a fine-grained claim-pair relation corpus.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23390v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CLAW-4L","organization_count":1,"organizations":["claire_radar"],"rank":609,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23390v1"},{"aliases":[],"benchmark_id":"claire-radar-2605-26086","caveat":"Claw-Anything evaluates always-on LLM personal assistants in simulated environments with long-horizon activity histories, interdependent backend services, and GUI/CLI across devices. It includes 200 human-verified tasks scored on completion, robustness, communication, and safety, with a live leaderboard.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26086"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Claw-Anything","organization_count":1,"organizations":["claire_radar"],"rank":610,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26086"},{"aliases":[],"benchmark_id":"llm-stats-claw-eval","caveat":"Claw-Eval tests real-world agentic task completion across complex multi-step scenarios, evaluating a model's ability to use tools, navigate environments, and complete end-to-end tasks autonomously.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/claw-eval?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Claw-Eval","organization_count":1,"organizations":["llm_stats"],"rank":611,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-12344","caveat":"Claw-SWE-Bench is a multilingual SWE-bench-style benchmark and adapter protocol for comparing agent harnesses on coding tasks, with 350 GitHub issue-resolution instances across 8 languages and 43 repositories. Score is Pass@1 on patch correctness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12344"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Claw-SWE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":612,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12344"},{"aliases":[],"benchmark_id":"claire-radar-2606-31174","caveat":"ClawArena-Team evaluates a single text-only LLM's ability to manage a fixed, locally served pool of subagents (LLM, VLM, omni) across 41 multi-turn, multimodal, multi-directory scenarios with 258 evaluation rounds and 72 staged updates. Scoring is execution-based via shell commands, producing a Subagent-Management Score (SMS) that multiplies task correctness by a least-privilege and modality-routing factor, without LLM judges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31174"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ClawArena-Team","organization_count":1,"organizations":["claire_radar"],"rank":613,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31174"},{"aliases":[],"benchmark_id":"llm-stats-claw-eval-mm","caveat":"ClawEval-MM is the multimodal variant of ClawEval, evaluating agentic problem solving with visual inputs.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/claw-eval-mm?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ClawEval-MM","organization_count":1,"organizations":["llm_stats"],"rank":614,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/claw-eval-mm?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-22510","caveat":"Evaluates agent configurations on 102 live-runtime and 68 frozen holdout scenarios, scoring execution traces with a safety-gated formula covering correctness, process quality, and efficiency.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22510v1"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ClawProBench","organization_count":1,"organizations":["claire_radar"],"rank":615,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22510v1"},{"aliases":[],"benchmark_id":"claire-radar-2607-28037","caveat":"ClawTrack is a dual-assessment benchmark for agents, measuring task outcomes and process quality across 320 tasks in 8 domains with 25+ mock services.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28037"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ClawTrack","organization_count":1,"organizations":["claire_radar"],"rank":616,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28037"},{"aliases":["From Prompt Injection to Persistent Control: Defending Agentic Harness Against Trojan Backdoors"],"benchmark_id":"claire-radar-2605-31042","caveat":"A reusable benchmark for measuring multi-step trojan attacks that establish persistent control through local agent workspace state.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31042"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ClawTrojan","organization_count":1,"organizations":["claire_radar"],"rank":617,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31042"},{"aliases":[],"benchmark_id":"claire-radar-2607-25294","caveat":"CLBench-V evaluates multimodal context learning across three dimensions: context grounding, new information application, and new knowledge learning. It includes 3,443 instances across 14 subdatasets spanning science, finance, long-document understanding, spatial reasoning, and web-based VQA.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25294"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CLBench-V","organization_count":1,"organizations":["claire_radar"],"rank":618,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25294"},{"aliases":[],"benchmark_id":"opencompass-1849-cleanpatrick","caveat":"CleanPatrick is a large-scale, real-world image data-cleaning benchmark with 496,377 binary annotations from 933 medical crowd workers for ranking off-topic, near-duplicate, and label-error issues. CleanPatrick是首个大规模图像数据清洗基准，基于公开的Fitzpatrick17k皮肤科数据集构建。该基准包含超过50万条来自933名医学众包工人的二元注释，涵盖三种数据质量问题：离题样本、近似重复样本和标签错误。通过医学专家验证，CleanPatrick提供了高质量的基准数据，用于评估图像数据清洗策略。基准测试结果表明，现有的数据清洗方法在近似重复检测中表现出色，但在标签错误检测方面仍面临挑战。CleanPatrick为数据清洗方法提供了标准化的评估框架，推动了更可靠的数据中心人工智能的发展。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CleanPatrick"],"document_share":0.0002723311546840959,"domain":"理解","name":"CleanPatrick","organization_count":1,"organizations":["opencompass_hub"],"rank":619,"released":"2025-05-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CleanPatrick"},{"aliases":[],"benchmark_id":"opencompass-2127-clembench","caveat":"clembench is a benchmark framework for evaluating LLMs through dialogue game-based interactions, assessing chat-optimized models as conversational agents via multi-turn interactive game scenario. clembench是一个基于对话游戏的大语言模型评测框架，通过多回合交互式游戏场景评测聊天优化模型的会话智能体能力，包含Wordle、Taboo等多种游戏任务，评测指令遵循、目标导向行为和精细化交互理解等核心维度，提供可重复、可控制的自玩评测环境和综合得分机制，支持英语及多语言扩展的开源评测平台。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/clembench"],"document_share":0.0002723311546840959,"domain":"创作","name":"clembench","organization_count":1,"organizations":["opencompass_hub"],"rank":620,"released":"2025-07-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/clembench"},{"aliases":[],"benchmark_id":"opencompass-1834-clever","caveat":"CLEVER is a benchmark suite for end-to-end code generation and formal verification in Lean 4, adapted from the HumanEval dataset. It requires models to generate implementations, formal specifications, and proofs—all verifiable by Lean's type checker, moving beyond test-case-driven evaluation. CLEVER is a benchmark suite for end-to-end code generation and formal verification in Lean 4, adapted from the HumanEval dataset. It requires models to generate implementations, formal specifications, and proofs—all verifiable by Lean's type checker, moving beyond test-case-driven evaluation.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CLEVER"],"document_share":0.0002723311546840959,"domain":"代码","name":"CLEVER","organization_count":1,"organizations":["opencompass_hub"],"rank":621,"released":"2025-05-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CLEVER"},{"aliases":[],"benchmark_id":"claire-radar-github-arjunkshah12345-hash-cli-bench","caveat":"CLI-Bench is a benchmark framework for coding-agent CLI harnesses, featuring deterministic verifiers, seeded tasks, and resource accounting metrics.","document_count":1,"document_ids":["claire_radar:https://github.com/arjunkshah12345-hash/cli-bench"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CLI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":622,"released":"2026-09-23","source":"claire_radar","url":"https://github.com/arjunkshah12345-hash/cli-bench"},{"aliases":[],"benchmark_id":"opencompass-1954-climateviz","caveat":"ClimateViz is a large-scale multimodal benchmark designed to evaluate the scientific fact-checking and statistical reasoning capabilities of large language and vision-language models. It focuses on real-world climate science data, with over 49,000 high quality natural language claims. ClimateViz 是一个多模态基准数据集，用于评估大模型在气候科学图表上的事实核查与统计推理能力。数据来源于 NOAA、英国气象局等权威机构，共包含约 2,800 张科学图表与近 5 万条主张，标注为支持（support）、反驳（refute）或信息不足（NEI）。\n\n该数据集支持三种输入格式：图表+主张、表格+主张、图表标题+表格+主张，涵盖趋势识别、时空推理与科学对比等核心任务。ClimateViz 适用于多模态大模型和语言模型的系统性评估。\n图表转表格 + 图表标题 + 主张（Caption + Table","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ClimateViz"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ClimateViz","organization_count":1,"organizations":["opencompass_hub"],"rank":623,"released":"2025-06-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ClimateViz"},{"aliases":[],"benchmark_id":"claire-radar-2606-02568","caveat":"ClinEnv evaluates LLMs as physicians in an interactive multi-stage EHR simulation, requiring queries to four specialized agents before committing to medical decisions, scored via ontology-grounded matching.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02568"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"ClinEnv","organization_count":1,"organizations":["claire_radar"],"rank":624,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02568"},{"aliases":[],"benchmark_id":"claire-radar-2606-14697","caveat":"ClinHallu is a benchmark for diagnosing stage-wise hallucinations in medical multimodal large language models. It contains 7,031 instances with structured reasoning traces decomposed into visual recognition, knowledge recall, and reasoning integration, along with stage-replacement interventions for measuring final answer changes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14697"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ClinHallu","organization_count":1,"organizations":["claire_radar"],"rank":625,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14697"},{"aliases":[],"benchmark_id":"claire-radar-2606-03157","caveat":"ClinicalMC evaluates LLM clinical decision-making across multi-course patient trajectories, with 1,275 Chinese and 5,804 English samples spanning four stages from admission to discharge. Includes triage, examination, diagnosis, treatment, and final diagnosis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03157"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ClinicalMC","organization_count":1,"organizations":["claire_radar"],"rank":626,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03157"},{"aliases":[],"benchmark_id":"claire-radar-2608-07796","caveat":"CliniCARE-Bench evaluates clinical agents on retrospective audit tasks over longitudinal EHR data, measuring verdict accuracy, evidence grounding, process adherence, calibrated abstention, reliability, and efficiency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07796"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"CliniCARE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":627,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07796"},{"aliases":[],"benchmark_id":"claire-radar-2607-26155","caveat":"ClinLens evaluates clinical data-science agents on 200 executable tasks over five linked MIMIC resources (EHR, notes, ECG, chest X-rays, echocardiograms), organized by a 4x5 taxonomy of patient-time scopes and analysis capabilities. Scoring uses a STRICTPASS metric requiring correct artifacts, semantics, and final answers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26155"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ClinLens","organization_count":1,"organizations":["claire_radar"],"rank":628,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.26155"},{"aliases":[],"benchmark_id":"claire-radar-2607-03650","caveat":"ClinOCR-Bench is a public dataset of 384 scanned clinical documents across six artifact subsets (normal, handwriting, poor quality, rotation, tables, mixed). It evaluates OCR systems on clinical text extraction, with ground truth transcriptions and template-aware train/test splits supporting 0-shot and 1-shot evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03650"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ClinOCR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":629,"released":"2026-07-04","source":"claire_radar","url":"https://arxiv.org/abs/2607.03650"},{"aliases":[],"benchmark_id":"claire-radar-2608-04302","caveat":"CLIP-CC-Bench evaluates paragraph-level video description quality using 5-hour movie content and expert-written references. It scores 17 VLMs via coarse- and fine-grained semantic matching with five LLM-based embedding judges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04302"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CLIP-CC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":630,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04302"},{"aliases":[],"benchmark_id":"claire-radar-2607-09880","caveat":"A benchmark for question answering over irregular clinical time series from ICU records, containing 6,600 QA instances across 11 clinical variables and 11 tasks. It evaluates answer accuracy and evidence use through explicit temporal evidence and answer derivation rules.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09880"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"CLIR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":631,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.09880"},{"aliases":[],"benchmark_id":"llm-stats-cloningscenarios","caveat":"CloningScenarios is an expert-level multi-step reasoning benchmark about difficult genetic cloning scenarios in multiple-choice format. It evaluates dual-use biological knowledge relevant to bioweapons development.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cloningscenarios?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CloningScenarios","organization_count":1,"organizations":["llm_stats"],"rank":632,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cloningscenarios?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-16632","caveat":"CLOSER-Bench evaluates budgeted cross-stage design closure for hardware agents with spec-to-RTL, RTL-to-GDS, and spec-to-GDS tasks, using open-source tools and recording quality, progress, tool cost, and recovery.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.16632"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"CLOSER-Bench","organization_count":1,"organizations":["claire_radar"],"rank":633,"released":"2026-07-18","source":"claire_radar","url":"https://arxiv.org/abs/2607.16632"},{"aliases":["clouda-ocr-benchmark"],"benchmark_id":"claire-radar-github-sahrasayed3-crypto-clouda-ocr-benchmark","caveat":"Evaluates distorted Arabic document OCR on 177 page images with reference transcriptions using normalized Arabic character error rate and a primary leaderboard for models with full coverage.","document_count":1,"document_ids":["claire_radar:https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Clouda OCR Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":634,"released":"2026-09-07","source":"claire_radar","url":"https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2606-13513","caveat":"CloudCons is an end-to-end benchmark for evaluating forecasting models in cloud resource consolidation. It encompasses datasets from Huawei Cloud, Microsoft Azure, and Google Borg with diverse workload patterns, and evaluates statistical, deep learning, and time series foundation models. The evaluation includes resource efficiency and service reliability metrics under a forecast-then-optimize paradigm.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13513"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CloudCons","organization_count":1,"organizations":["claire_radar"],"rank":635,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13513"},{"aliases":[],"benchmark_id":"claire-radar-2605-29933","caveat":"CLUBench evaluates clustering algorithms across 131 datasets (tabular, text, image) using 24 algorithms, measuring clustering performance via standard metrics like NMI, ARI, etc.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29933"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CLUBench","organization_count":1,"organizations":["claire_radar"],"rank":636,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29933"},{"aliases":[],"benchmark_id":"claire-radar-2605-25009","caveat":"A benchmark decomposing synthetic image detection into annotated forensic cognitive skills, used to evaluate a proposed detection framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25009"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ClueAegis-Bench","organization_count":1,"organizations":["claire_radar"],"rank":637,"released":"2026-05-24","source":"claire_radar","url":"https://arxiv.org/abs/2605.25009"},{"aliases":[],"benchmark_id":"llm-stats-cluewsc","caveat":"CLUEWSC2020 is the Chinese version of the Winograd Schema Challenge, part of the CLUE benchmark. It focuses on pronoun disambiguation and coreference resolution, requiring models to determine which noun a pronoun refers to in a sentence. The dataset contains 1,244 training samples and 304 development samples extracted from contemporary Chinese literature.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cluewsc?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CLUEWSC","organization_count":1,"organizations":["llm_stats"],"rank":638,"released":"2020-04-13","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cluewsc?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1137-cmb","caveat":"CMB is Comprehensive Medical Benchmark in Chinese, designed and rooted entirely within the native Chinese linguistic and cultural framework. CMB 是一个综合医学基准，专为中文而设计，并完全依赖于本土的中文语言和文化框架中。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CMB"],"document_share":0.0002723311546840959,"domain":"其他","name":"CMB","organization_count":1,"organizations":["opencompass_hub"],"rank":639,"released":"2024-04-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CMB"},{"aliases":[],"benchmark_id":"llm-stats-cmmlu","caveat":"CMMLU (Chinese Massive Multitask Language Understanding) is a comprehensive Chinese benchmark that evaluates the knowledge and reasoning capabilities of large language models across 67 different subject topics. The benchmark covers natural sciences, social sciences, engineering, and humanities with multiple-choice questions ranging from basic to advanced professional levels.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cmmlu?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CMMLU","organization_count":1,"organizations":["llm_stats"],"rank":640,"released":"2023-06-15","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cmmlu?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-499-cmmlu","caveat":"CMMLU is a comprehensive Chinese evaluation benchmark specifically designed to assess the knowledge and reasoning abilities of language models in the context of the Chinese language. CMMLU covers 67 topics ranging from basic subjects to advanced professional levels. It includes tasks that require calculations and reasoning in natural sciences, as well as tasks involving knowledge from humanities, social sciences, and practical aspects like Chinese driving rules. Moreover, many tasks within CMMLU have answers specific to China, which might not be universally applicable in other regions or languages. As a result, CMMLU serves as a fully localized Chinese evaluation benchmark. CMMLU是一个综合性的中文评估基准，专门用于评估语言模型在中文语境下的知识和推理能力。CMMLU涵盖了从基础学科到高级专业水平的67个主题。它包括：需要计算和推理的自然科学，需要知识的人文科学和社会科学,以及需要生活常识的中国驾驶规则等。此外，CMMLU中的许多任务具有中国特定的答案，可能在其他地区或语言中并不普遍适用。因此是一个完全中国化的中文测试基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CMMLU"],"document_share":0.0002723311546840959,"domain":"学科","name":"CMMLU","organization_count":1,"organizations":["opencompass_hub"],"rank":641,"released":"2023-06-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CMMLU"},{"aliases":[],"benchmark_id":"claire-radar-2609-10722","caveat":"CMNIE jointly annotates event triggers, event arguments, named entities, and entity relations in 13,000 Chinese military news instances for structured extraction evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10722"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CMNIE","organization_count":1,"organizations":["claire_radar"],"rank":642,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10722"},{"aliases":[],"benchmark_id":"opencompass-524-cmnli","caveat":"CMNLI  is a Chinese natural language inference task, which requires to determine the logical relation between two sentences, with three relations: entailment, contradiction and neutral. CMNLI是一个中文自然语言推理任务，要求根据两个句子判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CMNLI"],"document_share":0.0002723311546840959,"domain":"推理","name":"CMNLI","organization_count":1,"organizations":["opencompass_hub"],"rank":643,"released":"2020-04-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CMNLI"},{"aliases":[],"benchmark_id":"llm-stats-cmt-benchmark","caveat":"CMT-Benchmark evaluates models on condensed matter theory problems, testing advanced physics reasoning across areas such as many-body systems, quantum field theory, and statistical mechanics.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cmt-benchmark?top_n=500"],"document_share":0.0002723311546840959,"domain":"physics","name":"CMT-Benchmark","organization_count":1,"organizations":["llm_stats"],"rank":644,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cmt-benchmark?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-24714","caveat":"CN-NewsTTS Bench v0.1 evaluates Chinese news TTS pronunciation of dense written forms (scores, model names, ranges, units, percentages, abbreviations) from raw text, using 800 public test records and a 992-target auto-evaluable subset with ASR-ensemble transcripts and an automatic target scorer.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24714"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CN-NewsTTS Bench","organization_count":1,"organizations":["claire_radar"],"rank":645,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24714"},{"aliases":[],"benchmark_id":"claire-radar-2608-28053","caveat":"Benchmark of 4,759 Chinese neologisms with reference definitions and a two-tier evaluation separating definition generation from mechanism-based manipulation tasks.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.28053v1"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"CNeo-Bench","organization_count":1,"organizations":["claire_radar"],"rank":646,"released":"2026-08-28","source":"claire_radar","url":"http://arxiv.org/abs/2608.28053v1"},{"aliases":[],"benchmark_id":"llm-stats-cnmo-2024","caveat":"China Mathematical Olympiad 2024 - A challenging mathematics competition.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cnmo-2024?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"CNMO 2024","organization_count":1,"organizations":["llm_stats"],"rank":647,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cnmo-2024?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-28266","caveat":"Construct-level benchmark with 897 oracle-validated household task instances evaluating four coordination constructs (allocation, ordering, mutual exclusion, handoff) via task success and construct-level scores.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.28266v1"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CoCoBench","organization_count":1,"organizations":["claire_radar"],"rank":648,"released":"2026-08-28","source":"claire_radar","url":"http://arxiv.org/abs/2608.28266v1"},{"aliases":[],"benchmark_id":"claire-radar-2605-22068","caveat":"COCOTree evaluates open tree-structured visual decomposition, segmenting images into hierarchical trees of visual components with unconstrained granularity. It includes over 21K images and 1.8M structural nodes, with 3.5K unique labels, and uses the Open Tree Quality (OTQ) metric for scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22068"],"document_share":0.0002723311546840959,"domain":"General AI","name":"COCOTree","organization_count":1,"organizations":["claire_radar"],"rank":649,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22068"},{"aliases":[],"benchmark_id":"claire-radar-2606-02603","caveat":"COD10K-C evaluates camouflaged object detection models under 8 corruption types at 5 severity levels, yielding 40 conditions and 81,040 image pairs based on COD10K. Models are scored on Dice and other standard metrics for segmentation robustness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02603"],"document_share":0.0002723311546840959,"domain":"Robustness","name":"COD10K-C","organization_count":1,"organizations":["claire_radar"],"rank":650,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.02603"},{"aliases":[],"benchmark_id":"claire-radar-2606-15300","caveat":"CODA-Bench is a benchmark for evaluating AI agents on data-intensive analytical tasks in a Linux sandbox with 1,009 tasks across 31 communities, requiring data discovery, code generation, and correct answers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15300"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"CODA-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":651,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2606.15300"},{"aliases":["code-taste-bench"],"benchmark_id":"claire-radar-github-dymoo-code-taste-bench","caveat":"Evaluates code-generating models on code taste across five rubric dimensions and pairwise duels, producing Taste Elo and per-dimension diagnostics.","document_count":1,"document_ids":["claire_radar:https://github.com/dymoo/code-taste-bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Code Taste Bench","organization_count":1,"organizations":["claire_radar"],"rank":652,"released":"2026-09-22","source":"claire_radar","url":"https://github.com/dymoo/code-taste-bench"},{"aliases":["Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA"],"benchmark_id":"claire-radar-2605-29277","caveat":"An open-source framework for generating repository-level question-answering benchmarks under closed-book, code-only, and documented conditions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29277"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Code-QA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":653,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29277"},{"aliases":[],"benchmark_id":"claire-radar-2609-36777","caveat":"Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely control scene state remains unclear.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36777"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Code4Scene","organization_count":1,"organizations":["claire_radar"],"rank":654,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36777"},{"aliases":[],"benchmark_id":"claire-radar-2609-33807","caveat":"How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code?","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33807"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"CodeActionBench","organization_count":1,"organizations":["claire_radar"],"rank":655,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33807"},{"aliases":[],"benchmark_id":"claire-radar-2608-03535","caveat":"CodeAssay is a benchmark of 185 Python tasks across ten software-engineering categories with audited ground truth, public and hidden tests, and code-property measures. It evaluates LLM code generation correctness and other code properties.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03535"],"document_share":0.0002723311546840959,"domain":"Code","name":"CodeAssay","organization_count":1,"organizations":["claire_radar"],"rank":656,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03535"},{"aliases":[],"benchmark_id":"opencompass-1546-codecriticbench","caveat":"CodeCriticBench assesses LLMs' critiquing ability in code generation and QA tasks. Covering 10 criteria, it features a 4.3k-samples dataset with three difficulty levels and balanced distribution. CodeCriticBench ，旨在系统地评估LLMs在代码生成和代码问答任务中的批评能力。其涵盖 10 个不同的标准，数据集根据难度分为三个等级，共包含4.3k个样本，确保了难度级别的平衡分布。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CodeCriticBench"],"document_share":0.0002723311546840959,"domain":"代码","name":"CodeCriticBench","organization_count":1,"organizations":["opencompass_hub"],"rank":657,"released":"2025-02-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CodeCriticBench"},{"aliases":[],"benchmark_id":"opencompass-1624-codeelo","caveat":"Researchers introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible. CodeElo，这是一个标准化的竞赛级代码生成基准测试，有效解决了所有这些挑战。CodeElo 基准测试主要基于官方 CodeForces 平台，并尽可能与该平台保持一致。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CodeElo"],"document_share":0.0002723311546840959,"domain":"代码","name":"CodeElo","organization_count":1,"organizations":["opencompass_hub"],"rank":658,"released":"2025-01-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CodeElo"},{"aliases":[],"benchmark_id":"llm-stats-codeforces","caveat":"A competitive programming benchmark using problems from the CodeForces platform. The benchmark evaluates code generation capabilities of LLMs on algorithmic problems with difficulty ratings ranging from 800 to 2400. Problems cover diverse algorithmic categories including dynamic programming, graph algorithms, data structures, and mathematical problems with standardized evaluation through direct platform submission.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/codeforces?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"CodeForces","organization_count":1,"organizations":["llm_stats"],"rank":659,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/codeforces?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-04023","caveat":"CodegenBench evaluates LLM-generated parallel code across x86_64, Sunway, and Kunpeng architectures using BLAS routines and specialized kernels. Scoring measures efficiency on each platform.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04023"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"CodegenBench","organization_count":1,"organizations":["claire_radar"],"rank":660,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.04023"},{"aliases":[],"benchmark_id":"claire-radar-2605-30394","caveat":"CodeGolf Bench evaluates concise code generation across 60 programming languages, using code golf platform problems and human performance baselines for scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30394"],"document_share":0.0002723311546840959,"domain":"Code","name":"CodeGolf Bench","organization_count":1,"organizations":["claire_radar"],"rank":661,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30394"},{"aliases":[],"benchmark_id":"llm-stats-codegolf-v2-2","caveat":"Codegolf v2.2 benchmark","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/codegolf-v2.2?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"Codegolf v2.2","organization_count":1,"organizations":["llm_stats"],"rank":662,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/codegolf-v2.2?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1582-codemmlu","caveat":"CodeMMLU is a comprehensive benchmark designed to evaluate the capabilities of large language models (LLMs) in coding and software knowledge. It builds upon the structure of multiple-choice question answering (MCQA) to cover a wide range of programming tasks and domains. CodeMMLU 是一个旨在评估大型语言模型（LLMs）在编码和软件知识方面能力的全面基准。它基于多项选择题回答（MCQA）的结构，涵盖了广泛的编程任务和领域，包括代码生成、缺陷检测、软件工程原则等。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CodeMMLU"],"document_share":0.0002723311546840959,"domain":"代码","name":"CodeMMLU","organization_count":1,"organizations":["opencompass_hub"],"rank":663,"released":"2024-06-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CodeMMLU"},{"aliases":[],"benchmark_id":"claire-radar-2606-22902","caveat":"CodeRouterBench is an evaluation environment for comparing model routers on streaming coding tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22902"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CodeRouterBench","organization_count":1,"organizations":["claire_radar"],"rank":664,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.22902"},{"aliases":[],"benchmark_id":"claire-radar-github-sipratt-p-coding-agent-bench","caveat":"Nine Python implementation tasks covering data structures, concurrency, parsing and regex work, scored by executing hidden test programs against extracted model answers, with a separate text-only code-quality heuristic and an agent-harness arena.","document_count":1,"document_ids":["claire_radar:https://github.com/sipratt-p/coding-agent-bench"],"document_share":0.0002723311546840959,"domain":"Agents","name":"coding-agent-bench","organization_count":1,"organizations":["claire_radar"],"rank":665,"released":"2026-09-15","source":"claire_radar","url":"https://github.com/sipratt-p/coding-agent-bench"},{"aliases":["Codoku: Renewable Program-Reasoning Challenges for Frontier Coding Agents"],"benchmark_id":"claire-radar-2609-34661","caveat":"Evaluates coding agents on solving Codoku puzzles, where typed cells in partial programs must be filled to satisfy global static and dynamic constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34661"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Codoku","organization_count":1,"organizations":["claire_radar"],"rank":666,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34661"},{"aliases":[],"benchmark_id":"claire-radar-2609-34749","caveat":"Real-world driving is inherently multi-agent, yet most existing driving world models generate observations from a single ego vehicle.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34749"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CoDrive-Bench","organization_count":1,"organizations":["claire_radar"],"rank":667,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34749"},{"aliases":[],"benchmark_id":"claire-radar-2606-16613","caveat":"CoffeeBench evaluates LLM agents as a coffee roaster in a 90-day multi-agent economy with fixed reference agents, measuring cumulative net income through autonomous communication, negotiation, and transactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16613"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CoffeeBench","organization_count":1,"organizations":["claire_radar"],"rank":668,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16613"},{"aliases":[],"benchmark_id":"claire-radar-2606-15867","caveat":"CogCanvas is a benchmark for multi-subject reference-based image generation, with 1,952 curated reference images, 1,361 compositional prompts, and a unified six-axis evaluation protocol. It includes tasks for reference-based generation, text-to-image composition, and reference retrieval, with metrics BG-Sim and Attr-VQA.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15867"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CogCanvas","organization_count":1,"organizations":["claire_radar"],"rank":669,"released":"2026-06-14","source":"claire_radar","url":"https://arxiv.org/abs/2606.15867"},{"aliases":[],"benchmark_id":"claire-radar-2606-06099","caveat":"Evaluates 15 manipulative behavior strategies in 1,000 multi-turn LLM interaction scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06099"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"CogManip","organization_count":1,"organizations":["claire_radar"],"rank":670,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06099"},{"aliases":[],"benchmark_id":"llm-stats-cohere-agentic-question-answering","caveat":"Cohere's internal North evaluation for measuring how well a model answers enterprise questions using MCP-connected cloud file systems. Scores are reported with LLM-as-a-judge techniques.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cohere-agentic-question-answering?top_n=500"],"document_share":0.0002723311546840959,"domain":"question_answering","name":"Cohere Agentic Question Answering","organization_count":1,"organizations":["llm_stats"],"rank":671,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-agentic-question-answering?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cohere-data-analysis","caveat":"Cohere's internal North evaluation for measuring a model's ability to perform data science tasks over uploaded spreadsheets. Scores are reported with LLM-as-a-judge techniques.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cohere-data-analysis?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Cohere Data Analysis","organization_count":1,"organizations":["llm_stats"],"rank":672,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-data-analysis?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cohere-memory-usage-quality","caveat":"Cohere's internal North evaluation for measuring how well an agent uses information from North's memory system across sessions. Scores are reported with LLM-as-a-judge techniques.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cohere-memory-usage-quality?top_n=500"],"document_share":0.0002723311546840959,"domain":"memory","name":"Cohere Memory Usage Quality","organization_count":1,"organizations":["llm_stats"],"rank":673,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cohere-memory-usage-quality?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-34782","caveat":"Evaluates multi-humanoid collaboration policies in simulation under egocentric visual observations across 10 tasks with two or three humanoids.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34782"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration","organization_count":1,"organizations":["claire_radar"],"rank":674,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34782"},{"aliases":[],"benchmark_id":"claire-radar-2608-17566","caveat":"CoinVE-Bench provides 361 multi-instruction test cases for compositional video editing, with four-dimensional evaluation metrics covering instruction following, editing accuracy, visual quality, and temporal consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17566"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CoinVE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":675,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17566"},{"aliases":[],"benchmark_id":"claire-radar-2610-05590","caveat":"Cold-start drug-drug interaction (DDI) prediction tests whether models can identify clinically significant interactions for drugs without training-time interaction history.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05590"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"ColdDDI","organization_count":1,"organizations":["claire_radar"],"rank":676,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05590"},{"aliases":[],"benchmark_id":"claire-radar-2606-05793","caveat":"Evaluates collaborative ability of LLM agents in cooperative game environments with diverse player profiles and proactive engagement.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05793"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CollabBench","organization_count":1,"organizations":["claire_radar"],"rank":677,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05793"},{"aliases":["COLLIE"],"benchmark_id":"collie","caveat":"Systematic construction of constrained text generation tasks; evaluates rich, compositional constraints across diverse generation levels.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"COLLIE","organization_count":1,"organizations":["ByteDance"],"rank":678,"released":"2023-07-17","source":"model_reports","url":"https://collie-benchmark.github.io/"},{"aliases":[],"benchmark_id":"llm-stats-collie","caveat":"COLLIE is a grammar-based framework for systematic construction of constrained text generation tasks. It allows specification of rich, compositional constraints across diverse generation levels and modeling challenges including language understanding, logical reasoning, and semantic planning. The COLLIE-v1 dataset contains 2,080 instances across 13 constraint structures.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/collie?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"COLLIE","organization_count":1,"organizations":["llm_stats"],"rank":679,"released":"2023-07-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/collie?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1250-collie","caveat":"COLLIE is a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage) and modeling challenges (e.g.,language understanding, logical reasoning, counting, semantic planning). COLLIE用于评估大模型在约束性文本生成任务中的表现，可指定具有不同生成级别（单词、句子、段落、段落）和建模挑战（例如，语言理解、逻辑推理、计数、语义规划）的丰富组合。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Collie"],"document_share":0.0002723311546840959,"domain":"强推理","name":"Collie","organization_count":1,"organizations":["opencompass_hub"],"rank":680,"released":"2023-07-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Collie"},{"aliases":[],"benchmark_id":"opencompass-1777-colorbench","caveat":"ColorBench, an innovative benchmark meticulously crafted to assess the capabilities of VLMs in color understanding, including color perception, reasoning, and robustness. ColorBench，这是一个创新且精心设计的基准测试，旨在评估VLMs在颜色理解方面的能力，包括颜色感知、推理和鲁棒性。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ColorBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ColorBench","organization_count":1,"organizations":["opencompass_hub"],"rank":681,"released":"2025-04-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ColorBench"},{"aliases":[],"benchmark_id":"claire-radar-2605-27759","caveat":"Simulation benchmark for VLA generalization with 28 tasks across 13 categories and two robot morphologies, using ManiSkill.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27759"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"Colosseum V2","organization_count":1,"organizations":["claire_radar"],"rank":682,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27759"},{"aliases":[],"benchmark_id":"claire-radar-2606-10479","caveat":"ComBench evaluates large language models on 100 human-annotated Olympiad-level combinatorics problems, split into 50 analysis-centric and 50 construction-centric tasks. Scoring combines rubric-guided proof grading with deterministic verification of construction outputs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10479"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"ComBench","organization_count":1,"organizations":["claire_radar"],"rank":683,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.10479"},{"aliases":[],"benchmark_id":"claire-radar-2606-19788","caveat":"CombEval evaluates combinatorial counting abilities of large language models using problems generated from typed Cofola specifications, with solver-verified answers. It supports systematic variation of object types, entity scales, constraint counts, and reasoning depth.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19788"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CombEval","organization_count":1,"organizations":["claire_radar"],"rank":684,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.19788"},{"aliases":[],"benchmark_id":"opencompass-1943-combibench","caveat":"We introduce CombiBench, a comprehensive benchmark comprising 100 combinatorial problems, each formalized in Lean4 and paired with its corresponding informal statement. The problem set covers a wide spectrum of difficulty levels, ranging from middle school to IMO and university level. CombiBench是一个包含100个组合问题的综合基准测试，每个问题都用Lean4进行了形式化，并附有其对应的非形式化表述。这些问题涵盖了从中学生到国际数学奥林匹克竞赛（IMO）以及大学水平的广泛难度范围。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CombiBench"],"document_share":0.0002723311546840959,"domain":"数学","name":"CombiBench","organization_count":1,"organizations":["opencompass_hub"],"rank":685,"released":"2025-04-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CombiBench"},{"aliases":[],"benchmark_id":"claire-radar-2608-09282","caveat":"ComboShoppingBench is a benchmark for agentic basket shopping with coupons, evaluating LLM agents on constructing feasible baskets with budget and coupon constraints in a simulated environment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09282"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"ComboShoppingBench","organization_count":1,"organizations":["claire_radar"],"rank":686,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09282"},{"aliases":[],"benchmark_id":"claire-radar-2608-12805","caveat":"CoMedBench evaluates synthetic medical data fidelity and downstream utility across 37 dataset-task pairs from seven public sources, using a common clinical-validity framework and shared training and evaluation engine.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12805"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"CoMedBench","organization_count":1,"organizations":["claire_radar"],"rank":687,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.12805"},{"aliases":[],"benchmark_id":"claire-radar-2609-32192","caveat":"Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information must remain isolated.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32192"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CoMemBench","organization_count":1,"organizations":["claire_radar"],"rank":688,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32192"},{"aliases":[],"benchmark_id":"claire-radar-2606-15320","caveat":"CoMET-Bench is a benchmark for conditional multi-event temporal grounding in long-form video, with 2,789 queries over 600 videos and a unified evaluation protocol including counting, grounding, and negative-query recognition.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15320"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CoMET-Bench","organization_count":1,"organizations":["claire_radar"],"rank":689,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2606.15320"},{"aliases":[],"benchmark_id":"claire-radar-2608-07570","caveat":"COMEX is a benchmark for explainable aesthetic image cropping containing 33,161 quadruples of expanded image, crop box, composition category, and composition-grounded explanation. It supports joint evaluation of crop localization, composition understanding, and explanation generation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07570"],"document_share":0.0002723311546840959,"domain":"General AI","name":"COMEX","organization_count":1,"organizations":["claire_radar"],"rank":690,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.07570"},{"aliases":[],"benchmark_id":"llm-stats-common-voice-15","caveat":"Common Voice is a massively-multilingual collection of transcribed speech intended for speech technology research and development. Version 15.0 contains 28,750 recorded hours across 114 languages, consisting of crowdsourced voice recordings with corresponding transcriptions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/common-voice-15?top_n=500"],"document_share":0.0002723311546840959,"domain":"speech_to_text","name":"Common Voice 15","organization_count":1,"organizations":["llm_stats"],"rank":691,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/common-voice-15?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-commonsenseqa","caveat":"CommonSenseQA is a multiple-choice question answering dataset that requires different types of commonsense knowledge to predict correct answers. It contains 12,102 questions with one correct answer and four distractors, designed to test semantic reasoning and conceptual relationships. Questions are created based on ConceptNet concepts and require prior world knowledge for accurate reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/commonsenseqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CommonSenseQA","organization_count":1,"organizations":["llm_stats"],"rank":692,"released":"2018-11-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/commonsenseqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-511-commonsenseqa","caveat":"CommonsenseQA is a multiple-choice question answering dataset that requires different types of commonsense knowledge to predict the correct answers . It contains 12,102 questions with one correct answer and four distractor answers. CommonsenseQA是一个选择题数据集，它需要不同类型的常识知识来预测正确答案。它包含12,102个问题，有一个正确答案和四个干扰答案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CommonSenseQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"CommonSenseQA","organization_count":1,"organizations":["opencompass_hub"],"rank":693,"released":"2018-11-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CommonSenseQA"},{"aliases":[],"benchmark_id":"claire-radar-2605-30241","caveat":"CommunityFact evaluates misinformation detection on 15,992 claims across five languages and two domains, using accuracy against human Community Notes ratings as the scoring metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30241"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CommunityFact","organization_count":1,"organizations":["claire_radar"],"rank":694,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30241"},{"aliases":[],"benchmark_id":"claire-radar-2608-02046","caveat":"CompanionBench is an interactive bilingual benchmark for AI emotional companionship, grounding scenarios and a user simulator in de-identified real-world data. It evaluates ten capabilities derived from 25 theories, using a rubric and deterministic disclosure measure.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02046"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CompanionBench","organization_count":1,"organizations":["claire_radar"],"rank":695,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02046"},{"aliases":[],"benchmark_id":"claire-radar-github-0xboyu-company-mental-model-benchmark","caveat":"Compares three evidence-to-thesis treatments for LLM-based company analysis across eight companies, scored by blind reviewers against reference models on identifying dominant compounding mechanisms.","document_count":1,"document_ids":["claire_radar:https://github.com/0xboyu/company-mental-model-benchmark"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Company Mental Model Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":696,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/0xboyu/company-mental-model-benchmark"},{"aliases":[],"benchmark_id":"opencompass-1336-compbench","caveat":"CompBench is designed to evaluate the comparative reasoning capability of multimodal large language models, including a collection of around 40K image pairs and visually oriented questions covering 8 dimensions CompBench旨在评估多模态大模型的比较推理能力，包含约4万个图像对及8个维度的视觉配对问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CompBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"CompBench","organization_count":1,"organizations":["opencompass_hub"],"rank":697,"released":"2024-07-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CompBench"},{"aliases":["Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction"],"benchmark_id":"claire-radar-2608-11242","caveat":"COMPINT measures how well context compactors preserve user-issued session constraints across chat, agentic, and research scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11242"],"document_share":0.0002723311546840959,"domain":"General AI","name":"COMPINT","organization_count":1,"organizations":["claire_radar"],"rank":698,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.11242"},{"aliases":["ComplexConstraints and Beyond: Expert Rubrics for RLVR"],"benchmark_id":"claire-radar-2606-09118","caveat":"An expert-curated instruction-following benchmark containing 75 public prompts and 1,559 atomic rubric criteria.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09118"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ComplexConstraints","organization_count":1,"organizations":["claire_radar"],"rank":699,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09118"},{"aliases":[],"benchmark_id":"llm-stats-complexfuncbench","caveat":"ComplexFuncBench is a benchmark designed to evaluate large language models' capabilities in handling complex function calling scenarios. It encompasses multi-step and constrained function calling tasks that require long-parameter filling, parameter value reasoning, and managing contexts up to 128k tokens. The benchmark includes 1,000 samples across five real-world scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/complexfuncbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"ComplexFuncBench","organization_count":1,"organizations":["llm_stats"],"rank":700,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/complexfuncbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-07584","caveat":"ComplexityWorld is a benchmark of 390 visual decision-making tasks across 39 worlds, scored by an executable verifier. It evaluates VLMs on tasks requiring global constraint satisfaction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07584"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ComplexityWorld","organization_count":1,"organizations":["claire_radar"],"rank":701,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.07584"},{"aliases":[],"benchmark_id":"claire-radar-2606-18051","caveat":"CompSkillBench evaluates compositional skill routing for LLM agents: given a user query and a library of 2,209 real MCP server skills across 24 categories, systems must decompose the query into sub-tasks, retrieve a skill per sub-task, and produce an executable plan. Scoring uses step-level category recall and decomposition accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18051"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CompSkillBench","organization_count":1,"organizations":["claire_radar"],"rank":702,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18051"},{"aliases":[],"benchmark_id":"claire-radar-2608-16812","caveat":"A granular evaluation suite for image editing across over 1,000 fine-grained edit concepts, designed to diagnose model capabilities across real-world scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16812"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ConceptEdit-Bench","organization_count":1,"organizations":["claire_radar"],"rank":703,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16812"},{"aliases":["If, Then, Otherwise: Diagnosing Conditional Branching in Vision-Language Navigation"],"benchmark_id":"claire-radar-2608-17318","caveat":"CondVLN evaluates vision-language navigation agents on 11,500 generated conditional instructions across four simulators, using standard VLN metrics plus Branch Selection Accuracy and Conditional Success Rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17318"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"CondVLN","organization_count":1,"organizations":["claire_radar"],"rank":704,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17318"},{"aliases":[],"benchmark_id":"claire-radar-2608-01792","caveat":"ConfBench is a calibration-specific benchmark for key information extraction from documents. It applies 20 degradation pipelines to create 1,346 variants and over 70K entity-level evaluations, spanning the accuracy spectrum for evaluating confidence estimates of VLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01792"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ConfBench","organization_count":1,"organizations":["claire_radar"],"rank":705,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01792"},{"aliases":[],"benchmark_id":"claire-radar-2607-20526","caveat":"A calibration benchmark evaluating verbalized confidence estimates in frontier LLMs using Brier scores across 200 multiple-choice questions in four categories. Scores are elicited via prompting without logits, applicable to closed and open models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20526"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ConfidenceBench","organization_count":1,"organizations":["claire_radar"],"rank":706,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.20526"},{"aliases":[],"benchmark_id":"claire-radar-2606-26437","caveat":"ConflictBench is a benchmark with ConflictScore metric to quantify how well models acknowledge conflicting evidence in grounding documents, decomposing responses into claims and labeling them against documents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26437"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ConflictBench","organization_count":1,"organizations":["claire_radar"],"rank":707,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.26437"},{"aliases":[],"benchmark_id":"claire-radar-2608-03505","caveat":"ConlangBench evaluates large language models on translation and vocabulary learning across 21 constructed languages, with a corpus of over 21 million conlang-English parallel sentence pairs and 321K vocabulary entries. The benchmark includes bidirectional translation tasks and learning-curve analysis for models trained on eight conlangs with sufficient parallel data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03505"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"ConlangBench","organization_count":1,"organizations":["claire_radar"],"rank":708,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03505"},{"aliases":["ConnectomeBench"],"benchmark_id":"claire-radar-2606-21116","caveat":"ConnectomeBench2 evaluates automated proofreading of connectomic segmentation errors across mouse, human, zebrafish, and fly data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21116"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ConnectomeBench2","organization_count":1,"organizations":["claire_radar"],"rank":709,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21116"},{"aliases":[],"benchmark_id":"llm-stats-openai-connectors","caveat":"Connectors is an OpenAI internal production benchmark measuring reliable use of connector-based tools in agentic workflows, reported as a pass rate.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-connectors?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Connectors","organization_count":1,"organizations":["llm_stats"],"rank":710,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-connectors?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-13877","caveat":"ContactWorld is a benchmark and empirical study for vision-tactile world models in contact-rich manipulation, spanning 12 tasks including insertion and screwing, evaluating planning success rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13877"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"ContactWorld","organization_count":1,"organizations":["claire_radar"],"rank":711,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13877"},{"aliases":["When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models"],"benchmark_id":"claire-radar-2605-27851","caveat":"Context-flip evaluation tests whether aligned language models adjust their actions when a situational change reverses which action is safe.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27851"],"document_share":0.0002723311546840959,"domain":"General AI","name":"context-flip evaluation","organization_count":1,"organizations":["claire_radar"],"rank":712,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27851"},{"aliases":["ContextBias: Controlled Evaluation of Bias Persistence Under Context Shift in Text-to-Image Models"],"benchmark_id":"claire-radar-2608-29847","caveat":"Evaluates text-to-image models on 92 roles and 1,656 controlled prompts across context-free, related, and unrelated conditions, measuring bias persistence in visual attributes through metrics like Bias Intensity and Context Consistency Score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29847"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ContextBench","organization_count":1,"organizations":["claire_radar"],"rank":713,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29847"},{"aliases":[],"benchmark_id":"claire-radar-2605-24279","caveat":"ContextEcho measures persona drift in long agentic-coding sessions, combining a 25-probe identity suite, snapshot-then-probe protocol, and three anonymized Claude Code sessions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24279"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ContextEcho","organization_count":1,"organizations":["claire_radar"],"rank":714,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.24279"},{"aliases":[],"benchmark_id":"claire-radar-2606-09495","caveat":"ContextShift is a controlled benchmark that manipulates object-context relationships in COCO images to isolate context as an independent variable for object detection evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09495"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ContextShift","organization_count":1,"organizations":["claire_radar"],"rank":715,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09495"},{"aliases":[],"benchmark_id":"opencompass-1679-contextualjudgebench","caveat":"ContextualJudgeBench is a pairwise benchmark with 2,000 samples for evaluating LLM-as-judge models in two contextual settings: Contextual QA and summarization. We propose a pairwise evaluation hierarchy and generate splits for our proposed hierarchy. ContextualJudgeBench 是一个包含 2,000 个样本的成对基准，用于评估在两个上下文环境下的LLM-as-judge 模型：上下文问答和摘要。我们提出一个成对评估层次结构，并为我们的层次结构生成分割。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ContextualJudgeBench"],"document_share":0.0002723311546840959,"domain":"理解","name":"ContextualJudgeBench","organization_count":1,"organizations":["opencompass_hub"],"rank":716,"released":"2025-03-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ContextualJudgeBench"},{"aliases":[],"benchmark_id":"claire-radar-2608-04830","caveat":"ContextWeave is a longitudinal benchmark for evaluating memory in office workflows, with 1,005 executable tasks from 14 participants. It measures workspace quality and preference alignment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04830"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ContextWeave","organization_count":1,"organizations":["claire_radar"],"rank":717,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04830"},{"aliases":[],"benchmark_id":"claire-radar-2606-05661","caveat":"Evaluates continual learning in LLM-based systems across six expert-validated domains with stateful tasks sharing learnable latent structure.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05661"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Continual Learning Bench","organization_count":1,"organizations":["claire_radar"],"rank":718,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05661"},{"aliases":[],"benchmark_id":"claire-radar-2607-15899","caveat":"ContinuityBench evaluates stateful failover in multi-provider LLM routing. It measures Continuity Preservation Rate (CPR) and Continuity Latency Overhead (CLO) using synthetic conversation graphs under injected provider failures, with an LLM-as-a-judge scoring context preservation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15899"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ContinuityBench","organization_count":1,"organizations":["claire_radar"],"rank":719,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.15899"},{"aliases":[],"benchmark_id":"claire-radar-2606-01849","caveat":"ContinuousBench evaluates differentially private synthetic text by measuring capability gain on QA sets derived from a new quarterly corpus (Geminon procedural data or News articles), with standardized training and evaluation harness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01849"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ContinuousBench","organization_count":1,"organizations":["claire_radar"],"rank":720,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01849"},{"aliases":[],"benchmark_id":"llm-stats-contphy","caveat":"ContPhy is a continuum physical-reasoning benchmark evaluating understanding of physical dynamics in video.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/contphy?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ContPhy","organization_count":1,"organizations":["llm_stats"],"rank":721,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/contphy?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-20204","caveat":"Evaluates LLMs on legal contract scrubbing tasks using hand-crafted contracts covering error categories like defined term misuse and inconsistent language.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.20204"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ContractScrub","organization_count":1,"organizations":["claire_radar"],"rank":722,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.20204"},{"aliases":[],"benchmark_id":"claire-radar-2607-21722","caveat":"ConVBench is a vision-centric reasoning benchmark where each image is paired with two logically equivalent questions across six categories (action/state, complex counting, spatial reasoning, causal/intent, commonsense, temporal perception), with metrics for logical consistency and robust accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21722"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ConVBench","organization_count":1,"organizations":["claire_radar"],"rank":723,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21722"},{"aliases":[],"benchmark_id":"opencompass-1269-convbench","caveat":"ConvBench is a novel multi-turn conversation evaluation benchmark tailored for Large Vision-Language Models (LVLMs). It comprises 577 meticulously curated multi-turn conversations encompassing 215 tasks reflective of real-world demands. ConvBench是专用于大型视觉语言模型 （LVLM）的新型多轮对话评估基准，由基于215 个反映实际需求的任务的577个多轮次对话组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ConvBench"],"document_share":0.0002723311546840959,"domain":"推理","name":"ConvBench","organization_count":1,"organizations":["opencompass_hub"],"rank":724,"released":"2024-03-29","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ConvBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-25454","caveat":"A proposed evaluation suite for geographic implicit neural representations across multiple spatial scales.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25454"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CoordBench","organization_count":1,"organizations":["claire_radar"],"rank":725,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.25454"},{"aliases":[],"benchmark_id":"opencompass-529-copa","caveat":"COPA is a causal inference task, which requires to select the correct causal relation based on the given premise. COPA是一个因果推断任务，要求根据给定的前提，选择正确的因果关系。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/COPA"],"document_share":0.0002723311546840959,"domain":"推理","name":"COPA","organization_count":1,"organizations":["opencompass_hub"],"rank":726,"released":null,"source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/COPA"},{"aliases":[],"benchmark_id":"claire-radar-2610-04329","caveat":"Reliable language models should resist unsupported user pressure while effectively using objective contextual information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.04329"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CoPE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":727,"released":"2026-10-03","source":"claire_radar","url":"https://arxiv.org/abs/2610.04329"},{"aliases":["CF-YOLO: Context-Aware Feature Refinement for Camouflaged Industrial Micro-Defect Detection"],"benchmark_id":"claire-radar-2608-28070","caveat":"Evaluates object detection on 1,847 copper tube images with 4,898 bounding-box defect instances for identifying surface micro-defects.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.28070"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Copper Tube Defect Dataset","organization_count":1,"organizations":["claire_radar"],"rank":728,"released":"2026-08-28","source":"claire_radar","url":"https://arxiv.org/abs/2608.28070"},{"aliases":[],"benchmark_id":"claire-radar-2607-21799","caveat":"Copyright-Bench evaluates LLM agents' compliance with copyright law through realistic commercial tasks—website development, merchandise design, and pitch deck production—where agents choose between public-domain and copyrighted content, with prompt variations and time pressure.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21799"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Copyright-Bench","organization_count":1,"organizations":["claire_radar"],"rank":729,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21799"},{"aliases":[],"benchmark_id":"claire-radar-2609-01161","caveat":"Compares copyright defenses in LLMs across three intervention levels using controlled memorization over five public-domain books and a shared protocol measuring leakage, utility, and degeneracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01161"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CopyShield","organization_count":1,"organizations":["claire_radar"],"rank":730,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01161"},{"aliases":[],"benchmark_id":"claire-radar-2608-30768","caveat":"Evaluates neuron reconstruction from light microscopy at block and whole-brain scales, with structure-aware fiber prediction metrics for topological accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30768"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CORAL","organization_count":1,"organizations":["claire_radar"],"rank":731,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30768"},{"aliases":[],"benchmark_id":"opencompass-1622-coral","caveat":"Researchers present a large-scale conversational RAG benchmark named CORAL and propose a unified framework for standardizing and evaluating various conversational RAG baselines. CORAL 是一个大规模对话 RAG 基准，包含一个统一框架，用于标准化和评估各种对话 RAG 基线。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CORAL"],"document_share":0.0002723311546840959,"domain":"其他","name":"CORAL","organization_count":1,"organizations":["opencompass_hub"],"rank":732,"released":"2024-10-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CORAL"},{"aliases":[],"benchmark_id":"claire-radar-2609-01600","caveat":"Evaluates language models on reasoning about component lifecycle changes in dynamic agent harnesses via 1,200 questions covering localization, prediction, conditions, and reconfiguration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01600"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CordisBench","organization_count":1,"organizations":["claire_radar"],"rank":733,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01600"},{"aliases":[],"benchmark_id":"claire-radar-2608-27391","caveat":"LLMs are increasingly able to answer complex questions about enterprise-scale document collections.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27391"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CorporateBench","organization_count":1,"organizations":["claire_radar"],"rank":734,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27391"},{"aliases":[],"benchmark_id":"llm-stats-corpusqa","caveat":"CorpusQA is a multi-document, free-form long-context question answering benchmark in which a model must retrieve and reason over information distributed across a large corpus to produce open-ended answers that are scored by an LLM judge.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/corpusqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"CorpusQA","organization_count":1,"organizations":["llm_stats"],"rank":735,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-corpusqa-1m","caveat":"CorpusQA 1M is a long-context question answering benchmark designed to evaluate models at approximately 1 million token contexts. Models are scored on accuracy when retrieving and reasoning over information distributed across an extremely long input corpus.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/corpusqa-1m?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"CorpusQA 1M","organization_count":1,"organizations":["llm_stats"],"rank":736,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/corpusqa-1m?top_n=500"},{"aliases":["CoSec: Benchmarking Agent Security in Communities"],"benchmark_id":"claire-radar-2609-34790","caveat":"CoSec evaluates privacy and authorization enforcement in LLM agent systems across 208 scenarios involving community boundaries, protected information, and attacks via dialogue, environment, memory, and workflows.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34790"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CoSec","organization_count":1,"organizations":["claire_radar"],"rank":737,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34790"},{"aliases":[],"benchmark_id":"llm-stats-countbench","caveat":"CountBench evaluates object counting capabilities in visual understanding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/countbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CountBench","organization_count":1,"organizations":["llm_stats"],"rank":738,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/countbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-02316","caveat":"Evaluates defenses against generative engine optimization using 247 queries with rewrites, scoring attack success rate, false positive rate, and answer quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02316"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Counter-GEO-Bench","organization_count":1,"organizations":["claire_radar"],"rank":739,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02316"},{"aliases":["Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning"],"benchmark_id":"claire-radar-2606-24548","caveat":"A three-level benchmark evaluating whether text-to-image models can follow explicit and implicit counterfactual rules rather than defaulting to real-world visual priors.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24548"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Counterfactual-World (CF-World)","organization_count":1,"organizations":["claire_radar"],"rank":740,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24548"},{"aliases":[],"benchmark_id":"llm-stats-countqa","caveat":"CountQA is a benchmark for visual object counting and quantity reasoning over images.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/countqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"CountQA","organization_count":1,"organizations":["llm_stats"],"rank":741,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/countqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-08415","caveat":"CoVEBench evaluates compositional video editing with 416 source videos, 626 multi-point instructions, and 9,990 checklist items, using MLLM judges and objective metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08415"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CoVEBench","organization_count":1,"organizations":["claire_radar"],"rank":742,"released":"2026-06-07","source":"claire_radar","url":"https://arxiv.org/abs/2606.08415"},{"aliases":[],"benchmark_id":"llm-stats-covost2","caveat":"CoVoST 2 is a large-scale multilingual speech translation corpus derived from Common Voice, covering translations from 21 languages into English and from English into 15 languages. The dataset contains 2,880 hours of speech with 78K speakers for speech translation research.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/covost2?top_n=500"],"document_share":0.0002723311546840959,"domain":"speech_to_text","name":"CoVoST2","organization_count":1,"organizations":["llm_stats"],"rank":743,"released":"2020-07-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-covost2-en-zh","caveat":"CoVoST 2 English-to-Chinese subset is part of the large-scale multilingual speech translation corpus derived from Common Voice. This subset focuses specifically on English to Chinese speech translation tasks within the broader CoVoST 2 dataset.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/covost2-en-zh?top_n=500"],"document_share":0.0002723311546840959,"domain":"speech_to_text","name":"CoVoST2 en-zh","organization_count":1,"organizations":["llm_stats"],"rank":744,"released":"2020-07-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/covost2-en-zh?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-coworkbench","caveat":"CoWorkBench is Qwen's internal cowork benchmark for evaluating long-horizon office and productivity agent tasks across domains such as computer science, finance, law, and medicine.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/coworkbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"productivity","name":"CoWorkBench","organization_count":1,"organizations":["llm_stats"],"rank":745,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/coworkbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-14546","caveat":"CPI-Bench evaluates image editing models across general, practical, and intelligent tasks with VLM-as-Judge scoring, including multi-image and reasoning-based editing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14546"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CPI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":746,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14546"},{"aliases":[],"benchmark_id":"opencompass-1972-cpret","caveat":"Programming contests are long used to evaluate algorithmic thinking and coding skills, and have recently become benchmarks for assessing large language models (LLMs). However, the rapid expansion of problem sets has led to a surge in duplicate or highly similar problems, compromising fairness ... 编程竞赛长期用于评估算法与编程能力，近年来也被用于大语言模型的评测。但随着题库扩展，重复或相似题激增，影响竞赛公平性与模型评测效果。为此，本文提出“相似题目检索”任务，并构建统一检索基准数据集 CPRet，涵盖题目与代码的四类检索任务，包含自动爬取和人工标注的数据样本。同时，设计并训练了两种检索模型 CPRetriever-Code 与 CPRetriever-Prob，显著提升检索效果。实验还发现相似题会提高模型得分、减小模型差异，强调了评测中引入“相似性感知”的必要性。我们还发布了开源检索平台，支持重复题检测与相似题推荐。项目地址：https://github.com/coldchair/","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CPRet"],"document_share":0.0002723311546840959,"domain":"代码","name":"CPRet","organization_count":1,"organizations":["opencompass_hub"],"rank":747,"released":"2025-05-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CPRet"},{"aliases":[],"benchmark_id":"claire-radar-2609-26029","caveat":"Assesses LLM-assisted ontology generation from competency questions using gold OWL ontologies with provenance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.26029"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CQ4OE","organization_count":1,"organizations":["claire_radar"],"rank":748,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.26029"},{"aliases":[],"benchmark_id":"claire-radar-2607-19361","caveat":"A session-layer framework for tracking conversational risk accumulation in multi-turn LLM systems, including CRA-Bench datasets and trajectory-native evaluation protocols.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19361"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CRA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":749,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.19361"},{"aliases":[],"benchmark_id":"claire-radar-2606-01815","caveat":"CRAB-Bench evaluates LLM agents on tasks generated via a constraint graph over multiple interdependent entities, using the RUSE user simulator with imperfect behavior. Scoring is pass@1 against valid solutions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01815"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CRAB-Bench","organization_count":1,"organizations":["claire_radar"],"rank":750,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01815"},{"aliases":[],"benchmark_id":"claire-radar-2607-19396","caveat":"Evaluates hidden prompt injection detection in PDFs through classification and paired ranking tasks, using 29,322 generated PDFs from 4,983 base documents. Includes frozen splits, features, and metrics for reproducibility.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19396"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CrackedPDFs","organization_count":1,"organizations":["claire_radar"],"rank":751,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.19396"},{"aliases":[],"benchmark_id":"claire-radar-2605-30611","caveat":"CraftBench evaluates scientific figure generation across three figure types and four input conditions, with human-drawn targets and a referenced VLM judge for scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30611"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CraftBench","organization_count":1,"organizations":["claire_radar"],"rank":752,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30611"},{"aliases":[],"benchmark_id":"llm-stats-crag","caveat":"CRAG (Comprehensive RAG Benchmark) is a factual question answering benchmark consisting of 4,409 question-answer pairs across 5 domains (finance, sports, music, movie, open domain) and 8 question categories. The benchmark includes mock APIs to simulate web and Knowledge Graph search, designed to represent the diverse and dynamic nature of real-world QA tasks with temporal dynamism ranging from years to seconds. It evaluates retrieval-augmented generation systems for trustworthy question answering.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/crag?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CRAG","organization_count":1,"organizations":["llm_stats"],"rank":753,"released":"2024-06-07","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/crag?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1626-crag","caveat":"The Comprehensive RAG Benchmark (CRAG) is a rich and comprehensive factual question answering benchmark designed to advance research in RAG. Besides question-answer pairs, CRAG provides mock APIs to simulate web and knowledge graph search. CRAG是一个丰富且全面的基于事实的问题回答基准，旨在推进 RAG 研究。除了问答对之外，CRAG 还提供了模拟网页和知识图谱搜索的模拟 API。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CRAG"],"document_share":0.0002723311546840959,"domain":"其他","name":"CRAG","organization_count":1,"organizations":["opencompass_hub"],"rank":754,"released":"2024-06-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CRAG"},{"aliases":[],"benchmark_id":"claire-radar-2607-21155","caveat":"CRAG-MM-Diagnostics is a diagnostic benchmark for knowledge-intensive visual question answering (KI-VQA) with stage-wise annotations to isolate visual grounding, object identification, and knowledge retrieval/reasoning, including metadata like target ROIs and visual complexity scores.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21155"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CRAG-MM-Diagnostics","organization_count":1,"organizations":["claire_radar"],"rank":755,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21155"},{"aliases":[],"benchmark_id":"claire-radar-2610-13668","caveat":"With the advances of coding agents, software developers are spending more time reviewing code than writing code.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13668"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"CRBugBench","organization_count":1,"organizations":["claire_radar"],"rank":756,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13668"},{"aliases":[],"benchmark_id":"opencompass-1643-creation-mmbench","caveat":"A multimodal benchmark specifically designed to evaluate the creative capabilities of MLLMs. It features three main aspects: 1. Comprehensive Creation Benchmark for MLLM and LLM. 2. Robust Evaluation Methodology. 3. Attractive Experiment Insight. 专为评估 多模态大模型 的创作能力而设计的多模态基准。采用两个不同指标对模型的基础感知能力和深层次视觉创作能力进行评估，采用GPT-4o作为评判模型进行评估。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Creation-MMBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"Creation-MMBench","organization_count":1,"organizations":["opencompass_hub"],"rank":757,"released":"2025-03-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Creation-MMBench"},{"aliases":[],"benchmark_id":"llm-stats-creative-writing-v3","caveat":"EQ-Bench Creative Writing v3 is an LLM-judged creative writing benchmark that evaluates models across 32 writing prompts with 3 iterations per prompt. Uses a hybrid scoring system combining rubric assessment and Elo ratings through pairwise comparisons. Challenges models in areas like humor, romance, spatial awareness, and unique perspectives to assess emotional intelligence and creative writing capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/creative-writing-v3?top_n=500"],"document_share":0.0002723311546840959,"domain":"creativity","name":"Creative Writing v3","organization_count":1,"organizations":["llm_stats"],"rank":758,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/creative-writing-v3?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-creativework","caveat":"CreativeWork evaluates agents on open-ended creative production tasks within realistic tool and application environments, measuring the quality and completeness of generated deliverables.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/creativework?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CreativeWork","organization_count":1,"organizations":["llm_stats"],"rank":759,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/creativework?top_n=500"},{"aliases":["credit-risk-tabpfn-benchmark"],"benchmark_id":"claire-radar-github-mehkh-analysis-credit-risk-tabpfn-benchmark","caveat":"The repository compares TabPFN with traditional machine learning models on a credit risk dataset, but lacks a formally declared benchmark name or explicit public submission path.","document_count":1,"document_ids":["claire_radar:https://github.com/MehKh-Analysis/credit-risk-tabpfn-benchmark"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"Credit Risk Benchmark: TabPFN vs. Traditional ML","organization_count":1,"organizations":["claire_radar"],"rank":760,"released":"2026-09-24","source":"claire_radar","url":"https://github.com/MehKh-Analysis/credit-risk-tabpfn-benchmark"},{"aliases":[],"benchmark_id":"opencompass-2046-crew-wildfire","caveat":"CREW-WILDFIRE is a benchmark designed to evaluate large language model-based multi-agent systems’ collaboration capabilities in complex, dynamic tasks, targeting agentic frameworks with perception, planning, and execution abilities. CREW-WILDFIRE 是一个用于评估基于大语言模型的多智能体系统在复杂动态任务中协作能力的基准，面向具备感知、规划与执行能力的智能体框架。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CREW-WILDFIRE"],"document_share":0.0002723311546840959,"domain":"智能体","name":"CREW-WILDFIRE","organization_count":1,"organizations":["opencompass_hub"],"rank":761,"released":"2025-07-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CREW-WILDFIRE"},{"aliases":[],"benchmark_id":"claire-radar-2608-11154","caveat":"CriticalSCM-Bench v1 is a synthetic supply-chain benchmark with causal ground truth, paired factual/counterfactual rollouts, and an explicit net-value objective. It evaluates intervention ranking policies across semiconductor, critical-material, and digital-infrastructure archetypes under partial and delayed information, with stress tests on fidelity, timing, cost, and held-out disruptions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11154"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CriticalSCM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":762,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.11154"},{"aliases":[],"benchmark_id":"opencompass-568-criticbench","caveat":"CriticBench, a novel benchmark designed to comprehensively and reliably evaluate four key critique ability dimensions of LLMs: feedback, comparison, refinement and meta-feedback. CriticBench encompasses nine diverse tasks, each assessing the LLMs' ability to critique responses at varying levels of quality granularity. CriticBench是一个新颖的基准，旨在全面可靠地评估LLM的四个关键批判能力维度。CriticBench包括九项不同的任务，每项任务都评估LLM在不同质量粒度水平上对响应进行批评的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CriticBench"],"document_share":0.0002723311546840959,"domain":"理解","name":"CriticBench","organization_count":1,"organizations":["opencompass_hub"],"rank":763,"released":"2024-02-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CriticBench"},{"aliases":[],"benchmark_id":"artificial-analysis-critpt","caveat":"Physics reasoning","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/critpt"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"CritPt","organization_count":1,"organizations":["artificial_analysis"],"rank":764,"released":"2025-11-21","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/critpt"},{"aliases":[],"benchmark_id":"llm-stats-critpt","caveat":"CritPT is a challenging reasoning benchmark reported by Qwen for evaluating frontier mathematical and critical problem-solving capability.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/critpt?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"CritPT","organization_count":1,"organizations":["llm_stats"],"rank":765,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/critpt?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-37216","caveat":"Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37216"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"CRJudgeBench","organization_count":1,"organizations":["claire_radar"],"rank":766,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37216"},{"aliases":[],"benchmark_id":"claire-radar-2605-23699","caveat":"CRONOS is an intervention-based benchmark for evaluating counterfactual physical consistency in video prediction models. It provides a photorealistic Unreal Engine environment with controlled videos of physical events (collision, occlusion, fall) while intervening on viewpoint, scene, object category, and object appearance. The evaluation protocol defines six metrics computed via video segmentation, tracking, 3D reconstruction, and VLM-based task performance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23699"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"CRONOS","organization_count":1,"organizations":["claire_radar"],"rank":767,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23699"},{"aliases":[],"benchmark_id":"claire-radar-2610-09550","caveat":"Vision-language models increasingly reason through crops, regions, and tool-produced observations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.09550"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CROSS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":768,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.09550"},{"aliases":[],"benchmark_id":"claire-radar-github-tiago369-cross-engine-humanoid-benchmark","caveat":"Compares MuJoCo and PyBullet on a scripted squat-and-recover task for the Unitree G1 humanoid, measuring joint tracking RMSE, contact forces, CoM deviation, and realtime factor, with PSO-based calibration.","document_count":1,"document_ids":["claire_radar:https://github.com/tiago369/cross-engine-humanoid-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Cross-Engine Humanoid Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":769,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/tiago369/cross-engine-humanoid-benchmark"},{"aliases":["Socialized UAV Cross-Task Learning: Towards Cross-Granularity Collaboration through Hierarchical Interaction"],"benchmark_id":"claire-radar-2609-19867","caveat":"UAV benchmark for joint object detection and instance segmentation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19867"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CrossUAV","organization_count":1,"organizations":["claire_radar"],"rank":770,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.19867"},{"aliases":[],"benchmark_id":"llm-stats-crossvid","caveat":"CrossVid evaluates cross-video reasoning, requiring models to integrate information across multiple videos.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/crossvid?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"CrossVid","organization_count":1,"organizations":["llm_stats"],"rank":771,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/crossvid?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-15539","caveat":"CrossView is a multi-camera video question-answering benchmark spanning autonomous driving, security surveillance, egocentric/exocentric video, and robotics. It evaluates vision-language models on tasks that require joint reasoning across multiple simultaneous camera views, including resolving occlusions and integrating evidence across perspectives.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15539"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"CrossView","organization_count":1,"organizations":["claire_radar"],"rank":772,"released":"2026-08-16","source":"claire_radar","url":"https://arxiv.org/abs/2608.15539"},{"aliases":[],"benchmark_id":"opencompass-1739-crosswordbench","caveat":"CrossWordBench, a benchmark designed to evaluate the reasoning capabilities of both LLMs and LVLMs through the medium of crossword puzzles. CrossWordBench，这是一个基准测试，旨在通过填字游戏的方式来评估LLMs和LVLMs的推理能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CrossWordBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"CrossWordBench","organization_count":1,"organizations":["opencompass_hub"],"rank":773,"released":"2025-03-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CrossWordBench"},{"aliases":[],"benchmark_id":"opencompass-1123-crows-pairs","caveat":"CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping. CrowS-Pairs 包含 1508 个示例，涵盖与九种偏见类型相关的刻板印象，例如种族、宗教和年龄。在 CrowS-Pairs 中，模型会接收到两句话：一句是更具刻板印象的，另一句则是较少刻板印象的。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Crows-Pairs"],"document_share":0.0002723311546840959,"domain":"安全","name":"Crows-Pairs","organization_count":1,"organizations":["opencompass_hub"],"rank":774,"released":"2020-09-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Crows-Pairs"},{"aliases":[],"benchmark_id":"opencompass-1500-crpe","caveat":"CRPE is a benchmark designed to quantitatively evaluate the object recognition and relation comprehension ability of models. It consists of four splits, and the evaluation is formulated as single-choice questions. CRPE用于定量评估多模态大模型的对象识别和关系理解能力，分为四部分，以单选题形式呈现。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CRPE"],"document_share":0.0002723311546840959,"domain":"多模态","name":"CRPE","organization_count":1,"organizations":["opencompass_hub"],"rank":775,"released":"2024-02-29","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CRPE"},{"aliases":[],"benchmark_id":"llm-stats-crperelation","caveat":"Clinical reasoning problems evaluation benchmark for assessing diagnostic reasoning and medical knowledge application capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/crperelation?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CRPErelation","organization_count":1,"organizations":["llm_stats"],"rank":776,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/crperelation?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-22059","caveat":"Evaluates 15 pretrained image encoder families on ISIC 2019, APTOS 2019, and CheXpert using discrimination, calibration, label efficiency, and robustness, summarized by the Clinical Reliability Score.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22059v1"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"CRS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":777,"released":"2026-08-22","source":"claire_radar","url":"http://arxiv.org/abs/2608.22059v1"},{"aliases":[],"benchmark_id":"claire-radar-2607-19380","caveat":"CruiseBench evaluates remaining useful life prediction for aircraft engines using a fixed cruise-stage protocol derived from N-CMAPSS. It provides a reproducible sub-benchmark with datasets and baseline results for LSTM, GRU, TCN, and TSMixer models under specific settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19380"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CruiseBench","organization_count":1,"organizations":["claire_radar"],"rank":778,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.19380"},{"aliases":[],"benchmark_id":"llm-stats-crux-o","caveat":"CRUXEval-O (output prediction) is part of the CRUXEval benchmark consisting of 800 Python functions (3-13 lines) designed to evaluate AI models' capabilities in code reasoning, understanding, and execution. The benchmark tests models' ability to predict correct function outputs given function code and inputs, focusing on short problems that a good human programmer should be able to solve in a minute.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/crux-o?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CRUX-O","organization_count":1,"organizations":["llm_stats"],"rank":779,"released":"2024-01-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/crux-o?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cruxeval-input-cot","caveat":"CRUXEval input prediction task with Chain of Thought (CoT) prompting. Part of the CRUXEval benchmark for code reasoning, understanding, and execution evaluation. Given a Python function and its expected output, the task is to predict the appropriate input using chain-of-thought reasoning. Consists of 800 Python functions (3-13 lines) designed to evaluate code comprehension and reasoning capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-input-cot?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CRUXEval-Input-CoT","organization_count":1,"organizations":["llm_stats"],"rank":780,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-input-cot?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cruxeval-o","caveat":"CruxEval-O is the output prediction task of the CRUXEval benchmark, designed to evaluate code reasoning, understanding, and execution capabilities. It consists of 800 Python functions (3-13 lines) where models must predict the output given a function and input. The benchmark tests fundamental code execution reasoning abilities and goes beyond simple code generation to assess deeper understanding of program behavior.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-o?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CruxEval-O","organization_count":1,"organizations":["llm_stats"],"rank":781,"released":"2024-01-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-o?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cruxeval-output-cot","caveat":"CRUXEval-O (output prediction) with Chain-of-Thought prompting. Part of the CRUXEval benchmark consisting of 800 Python functions (3-13 lines) designed to evaluate code reasoning, understanding, and execution capabilities. The output prediction task requires models to predict the output of a given Python function with specific inputs, evaluated using chain-of-thought reasoning methodology.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-output-cot?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"CRUXEval-Output-CoT","organization_count":1,"organizations":["llm_stats"],"rank":782,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cruxeval-output-cot?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-14811","caveat":"Crypto Accounting Bench tests whether language models can reproduce the exact journal entry recorded for 118 crypto-asset transactions, using transaction evidence and each organization's full chart of accounts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.14811"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Crypto Accounting Bench","organization_count":1,"organizations":["claire_radar"],"rank":783,"released":"2026-09-13","source":"claire_radar","url":"https://arxiv.org/abs/2609.14811"},{"aliases":[],"benchmark_id":"claire-radar-2605-29446","caveat":"CrystalXRD-Bench evaluates vision-language models on XRD peak indexing, requiring the model to identify HKL indices from rendered XRD images and CIF text across 250 samples from 10 databases.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29446"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CrystalXRD-Bench","organization_count":1,"organizations":["claire_radar"],"rank":784,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29446"},{"aliases":[],"benchmark_id":"opencompass-924-cs-bench","caveat":"CS-Bench, the first bilingual (Chinese-English) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 5K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. CS-Bench 第一个专门用于评估LLMs在计算机科学中表现的双语（中英文）基准。CS-Bench包括约5,000个精心策划的测试样本，涵盖了计算机科学4个关键领域中的26个子领域，并包括各种任务形式和知识推理的划分。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CS-Bench"],"document_share":0.0002723311546840959,"domain":"推理","name":"CS-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":785,"released":"2024-06-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CS-Bench"},{"aliases":[],"benchmark_id":"opencompass-1219-cs-eval","caveat":"CS-Eval is a large language model cybersecurity capability evaluation suite jointly established by Alibaba Security, Fudan University, and the University of Chinese Academy of Sciences. The dataset encompasses 11 major cybersecurity categories and 42 subcategories, offering comprehensive assessment CS-Eval 是由阿里安全、复旦大学和中国科学院大学联合建立的大模型网络安全能力评测集。数据集覆盖11个网络安全大类领域、42个子类领域，提供知识型和实战型的综合评估任务，支持用户自主评测，同时为大模型落地网络安全提供参考和启发。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CS-Eval"],"document_share":0.0002723311546840959,"domain":"学科","name":"CS-Eval","organization_count":1,"organizations":["opencompass_hub"],"rank":786,"released":"2024-05-31","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CS-Eval"},{"aliases":[],"benchmark_id":"llm-stats-csimpleqa","caveat":"Chinese SimpleQA is the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions. It contains 3,000 high-quality questions spanning 6 major topics with 99 diverse subtopics, designed to assess Chinese factual knowledge across humanities, science, engineering, culture, and society.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/csimpleqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"CSimpleQA","organization_count":1,"organizations":["llm_stats"],"rank":787,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/csimpleqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-519-csl","caveat":"CSL is a large-scale Chinese Scientific Literature dataset, which contains the titles, abstracts, keywords and academic fields of 396k papers. CSL是一个大规模的中文科技文献数据集，包含 39.6 万篇论文的标题、摘要、关键词和学术领域信息。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CSL"],"document_share":0.0002723311546840959,"domain":"理解","name":"CSL","organization_count":1,"organizations":["opencompass_hub"],"rank":788,"released":"2022-09-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CSL"},{"aliases":[],"benchmark_id":"opencompass-1833-csts","caveat":"CSTS is a synthetic benchmark for evaluating correlation structure discovery in multivariate time series. It features 23 distinct correlation structures with systematic data variations (distribution shifts, sparsification, downsampling) and provides ground truth labels for validation. CSTS is a synthetic benchmark for evaluating correlation structure discovery in multivariate time series. It features 23 distinct correlation structures with systematic data variations (distribution shifts, sparsification, downsampling) and provides ground truth labels for validation.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CSTS"],"document_share":0.0002723311546840959,"domain":"其他","name":"CSTS","organization_count":1,"organizations":["opencompass_hub"],"rank":789,"released":"2025-05-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CSTS"},{"aliases":[],"benchmark_id":"claire-radar-2607-05571","caveat":"CSTutorBench evaluates small language models as tutors in VEX VR block-based programming, with 17 scenario-based questions scored via a rubric and LLM-as-judge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05571"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"CSTutorBench","organization_count":1,"organizations":["claire_radar"],"rank":790,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.05571"},{"aliases":[],"benchmark_id":"claire-radar-2605-21974","caveat":"CSVFidelity-Bench evaluates knowledge graph construction from statistical tables, focusing on the effect of extraction schema and format coupling on fidelity. It includes 15 datasets, 11 Type-II and 4 Type-III tables, with 1,892 gold standard facts across 6 domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.21974"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CSVFidelity-Bench","organization_count":1,"organizations":["claire_radar"],"rank":791,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.21974"},{"aliases":[],"benchmark_id":"claire-radar-2608-12002","caveat":"Evaluates AI agents on telecom network troubleshooting tasks, focusing on root cause analysis and path restoration, using expert-grounded metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12002"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CTBench","organization_count":1,"organizations":["claire_radar"],"rank":792,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.12002"},{"aliases":["No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow"],"benchmark_id":"claire-radar-2609-29245","caveat":"CTC-Bench is a 22-task suite released alongside code and data to study corpus task complexity as corpora grow, per the abstract.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.29245"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CTC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":793,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.29245"},{"aliases":[],"benchmark_id":"opencompass-1268-ctibench","caveat":"CTIBench is a benchmark designed to assess LLMs' performance in CTI (Cyber threat intelligence) applications. It includes multiple datasets focused on evaluating knowledge acquired by LLMs in the cyber-threat landscape. CTIBench旨在评估LLM在CTI（网络安全情报）场景下的能力，包含多个数据集，专注于评测LLM在网络威胁环境中获得的知识。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CTIBench"],"document_share":0.0002723311546840959,"domain":"安全","name":"CTIBench","organization_count":1,"organizations":["opencompass_hub"],"rank":794,"released":"2024-06-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CTIBench"},{"aliases":["Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming"],"benchmark_id":"claire-radar-2606-05233","caveat":"A public 793-episode benchmark for evaluating prompt-injection attacks against computer-using agents across web tasks, attack templates, attack families, and system-prompt configurations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05233"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CUA-HandCrafted","organization_count":1,"organizations":["claire_radar"],"rank":795,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05233"},{"aliases":[],"benchmark_id":"claire-radar-2609-32600","caveat":"Software development requires more than editing code: developers repeatedly run software, interact with its interfaces, visually inspect its behavior, and use these observations to decide what to change next and whether a change works.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32600"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"CUA-SWE","organization_count":1,"organizations":["claire_radar"],"rank":796,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32600"},{"aliases":[],"benchmark_id":"claire-radar-2609-32574","caveat":"Long-term memory is essential for multimodal agents that interact with users across sustained conversations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32574"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CUE-Mem","organization_count":1,"organizations":["claire_radar"],"rank":797,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32574"},{"aliases":["CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage"],"benchmark_id":"claire-radar-2606-21618","caveat":"CulMind evaluates multimodal understanding across Chinese cultural-heritage tasks, while CulMind-R and ReaScore assess reasoning processes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21618"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CulMind","organization_count":1,"organizations":["claire_radar"],"rank":798,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21618"},{"aliases":[],"benchmark_id":"claire-radar-2608-23065","caveat":"Evaluates video cultural reasoning and grounding on 306 expert-curated concepts from Southeast Asia, scoring naming, visual recognition, and temporal localization.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23065v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Cultural Moment Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":799,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23065v1"},{"aliases":[],"benchmark_id":"claire-radar-2609-03526","caveat":"Evaluates multimodal models on culinary reasoning across 4,870 items in 10 languages and 18 regions, with tasks from dish recognition to cultural attribution using images and procedural text.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03526"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CulturalMenuBench","organization_count":1,"organizations":["claire_radar"],"rank":800,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03526"},{"aliases":["Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC"],"benchmark_id":"claire-radar-2605-25626","caveat":"CULTURE-MT evaluates social-media translation across cultural transmission, emotion resonance, expression accuracy, and cultural adaptability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25626"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CULTURE-MT","organization_count":1,"organizations":["claire_radar"],"rank":801,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25626"},{"aliases":[],"benchmark_id":"claire-radar-2606-01879","caveat":"CultureForest is a benchmark for cultural norm grounded reasoning, featuring 5,378 examples across 8 domains and 53 countries/regions, with progressive evaluation from multiple-choice to open-ended generation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01879"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CultureForest","organization_count":1,"organizations":["claire_radar"],"rank":802,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01879"},{"aliases":[],"benchmark_id":"claire-radar-2607-21016","caveat":"CultureTalk-ID is a dialogue-based benchmark for cultural commonsense in Indonesian and local languages, containing 4,496 culturally grounded dialogues across 11 languages and 13 topics, with three tasks: dialogue-based multiple-choice reasoning, culturally faithful translation, and language steering.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21016"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CultureTalk-ID","organization_count":1,"organizations":["claire_radar"],"rank":803,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21016"},{"aliases":[],"benchmark_id":"claire-radar-2608-01942","caveat":"CultureVidBench evaluates cultural understanding in text-to-video generation with 1,000 curated prompts covering 12 countries, 6 continents, and 14 cultural aspects. It assesses cultural faithfulness, multimodal cultural rendering, semantic adherence, and perceptual quality using human studies and MLLM-based automatic assessment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01942"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CultureVidBench","organization_count":1,"organizations":["claire_radar"],"rank":804,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01942"},{"aliases":[],"benchmark_id":"llm-stats-cursorbench-3-2","caveat":"CursorBench v3.2 evaluates coding agents on interactive software engineering tasks in the Cursor environment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cursorbench-3.2?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"CursorBench v3.2","organization_count":1,"organizations":["llm_stats"],"rank":805,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cursorbench-3.2?top_n=500"},{"aliases":["Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation"],"benchmark_id":"claire-radar-2609-08275","caveat":"CutCraft evaluates editing-technique execution in multi-shot audio-video generation, covering shot structure, transitions, audio-video cuts, and montage.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08275"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CutCraft","organization_count":1,"organizations":["claire_radar"],"rank":806,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08275"},{"aliases":[],"benchmark_id":"claire-radar-2606-00931","caveat":"CV-Arena is a benchmark of 12K high-resolution image instruction pairs across 16 task types, with a human-AI collaborative preference protocol (Active Elo) for evaluating instructional computer vision problem solving.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00931"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CV-Arena","organization_count":1,"organizations":["claire_radar"],"rank":807,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00931"},{"aliases":["SeqLoc: Beyond the Single Frame for Cross-View Geo-Localization in Feature-Sparse Scenes"],"benchmark_id":"claire-radar-2608-07835","caveat":"A benchmark for cross-view geo-localization in feature-sparse scenes, pairing sequential panoramas from rural regions with aligned OpenStreetMap maps and evaluating position and orientation recall.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07835"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CV-FSS","organization_count":1,"organizations":["claire_radar"],"rank":808,"released":"2026-08-08","source":"claire_radar","url":"https://arxiv.org/abs/2608.07835"},{"aliases":[],"benchmark_id":"opencompass-1985-cvdp","caveat":"CVDP is a next-generation benchmark for evaluating large language models (LLMs) and agents in hardware design and verification, comprising 783 problems across 13 task categories, including RTL generation, verification, debugging, specification alignment, and technical Q&A. CVDP 是一个面向大型语言模型（LLM）和智能体的下一代硬件设计与验证评测基准，涵盖 13 类任务共 783 个问题，涉及 RTL 生成、验证、调试、规范对齐和技术问答等。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CVDP"],"document_share":0.0002723311546840959,"domain":"智能体","name":"CVDP","organization_count":1,"organizations":["opencompass_hub"],"rank":809,"released":"2025-06-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CVDP"},{"aliases":["CVE-Bench","CVEBench"],"benchmark_id":"cvebench","caveat":"Real CVE exploitation in sandboxes. Vendors report it as a safety ceiling rather than a capability to maximize.","document_count":1,"document_ids":["model_reports:openai_gpt_5_6_system_card"],"document_share":0.0002723311546840959,"domain":"security","name":"CVE-Bench","organization_count":1,"organizations":["OpenAI"],"rank":810,"released":"2025-03-21","source":"model_reports","url":"https://github.com/uiuc-kang-lab/cve-bench"},{"aliases":[],"benchmark_id":"opencompass-1239-cvqa","caveat":"CVQA is a new culturally-diverse multilingual Visual Question Answering benchmark, designed to cover a rich set of languages and cultures. It includes culturally-driven images and 10k questions from across 30 countries on 4 continents. CVQA是一种新的文化多元化多语言视觉问答基准，旨在涵盖丰富的语言和文化，包括来自四大洲30个国家/地区的文化向图像和10000个问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CVQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"CVQA","organization_count":1,"organizations":["opencompass_hub"],"rank":811,"released":"2024-06-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CVQA"},{"aliases":[],"benchmark_id":"claire-radar-2606-22476","caveat":"CVSBench evaluates cross-view spatial reasoning in vision-language models using satellite-street image pairs. It includes tasks for cross-view VQA, grounding, and viewpoint identification, with 3,297 image groups, 9,468 object-level annotations, and 40,679 QA pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22476"],"document_share":0.0002723311546840959,"domain":"General AI","name":"CVSBench","organization_count":1,"organizations":["claire_radar"],"rank":812,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22476"},{"aliases":[],"benchmark_id":"llm-stats-cvtg-2k","caveat":"CVTG-2K (Chinese Visual Text Generation 2K) is a benchmark for evaluating text-to-image models on their ability to accurately render text within generated images. It measures Word Accuracy, Normalized Edit Distance (NED), and CLIPScore across 2,000 prompts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cvtg-2k?top_n=500"],"document_share":0.0002723311546840959,"domain":"image-generation","name":"CVTG-2K","organization_count":1,"organizations":["llm_stats"],"rank":813,"released":"2025-03-30","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cvtg-2k?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-27779","caveat":"CXR-Retrieve is a benchmark for compositional chest X-ray text-to-image retrieval, with 5,159 MIMIC-CXR-JPG test images and 145 queries over single and conjunction findings with positive and negative assertions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27779"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"CXR-Retrieve","organization_count":1,"organizations":["claire_radar"],"rank":814,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27779"},{"aliases":[],"benchmark_id":"llm-stats-cybench","caveat":"CyBench is a suite of Capture-the-Flag (CTF) challenges measuring agentic cyber attack capabilities. It evaluates dual-use cybersecurity knowledge and measures the 'unguided success rate', where agents complete tasks end-to-end without guidance on appropriate subtasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cybench?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"CyBench","organization_count":1,"organizations":["llm_stats"],"rank":815,"released":"2024-08-15","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cybench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-26216","caveat":"CyberChainBench evaluates LLM-based agents on smart contract security across vulnerability detection, exploit generation, and patch synthesis, using 541 real exploit incidents with on-chain evaluation on historical forks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26216"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"CyberChainBench","organization_count":1,"organizations":["claire_radar"],"rank":816,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.26216"},{"aliases":[],"benchmark_id":"claire-radar-2609-32424","caveat":"Large language model agents have demonstrated promising capabilities in cybersecurity tasks, yet their ability to reconstruct complete Advanced Persistent Threat attack campaigns from complex security logs remains largely unexplored.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32424"],"document_share":0.0002723311546840959,"domain":"Agents","name":"CyberClear","organization_count":1,"organizations":["claire_radar"],"rank":817,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32424"},{"aliases":[],"benchmark_id":"llm-stats-cybergym","caveat":"CyberGym is a benchmark for evaluating AI agents on cybersecurity tasks, testing their ability to identify vulnerabilities, perform security analysis, and complete security-related challenges in a controlled environment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cybergym?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"CyberGym","organization_count":1,"organizations":["llm_stats"],"rank":818,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cybergym?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-04460","caveat":"CyberGym-E2E evaluates AI agents on end-to-end cybersecurity tasks, covering vulnerability discovery, proof-of-concept generation, and patch generation, using 920 real-world vulnerabilities from 139 open-source projects.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04460"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"CyberGym-E2E","organization_count":1,"organizations":["claire_radar"],"rank":819,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04460"},{"aliases":[],"benchmark_id":"claire-radar-2605-24765","caveat":"The evaluation object is a dataset for privacy-aware cybersecurity question answering, covering key security domains with private entity labels. The main capability evaluated is QA accuracy and masking performance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24765"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"CyberMaskQA","organization_count":1,"organizations":["claire_radar"],"rank":820,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24765"},{"aliases":[],"benchmark_id":"opencompass-1349-cyberseceval","caveat":"CyberSecEval is a comprehensive benchmark developed to help bolster the cybersecurity of LLMs. It provides a thorough evaluation in two crucial security domains: the propensity to generate insecure code and the level of compliance when asked to assist in cyberattacks. CyberSecEval旨在评估LLM的安全性，聚焦于大模型生成不安全代码的倾向以及当被要求协助网络攻击时的合规性水平。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/CyberSecEval"],"document_share":0.0002723311546840959,"domain":"安全","name":"CyberSecEval","organization_count":1,"organizations":["opencompass_hub"],"rank":821,"released":"2023-12-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/CyberSecEval"},{"aliases":[],"benchmark_id":"llm-stats-cyberseceval-4","caveat":"CyberSecEval 4 is an evaluation suite covering cybersecurity-related capabilities and risks of large language models. The insecure-code-generation tracks measure whether a model produces vulnerable code: the Instruct track presents coding requests designed to elicit known insecure patterns, while the Autocomplete track prompts the model with code context leading up to a known insecure pattern, with vulnerabilities detected via static analysis.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cyberseceval-4?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"CyberSecEval 4","organization_count":1,"organizations":["llm_stats"],"rank":822,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cyberseceval-4?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-cybersecurity-ctfs","caveat":"Cybersecurity Capture the Flag (CTF) benchmark for evaluating LLMs in offensive security challenges. Contains diverse cybersecurity tasks including cryptography, web exploitation, binary analysis, and forensics to assess AI capabilities in cybersecurity problem-solving.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/cybersecurity-ctfs?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"Cybersecurity CTFs","organization_count":1,"organizations":["llm_stats"],"rank":823,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/cybersecurity-ctfs?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-36987","caveat":"Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the multi-turn sessions through which analysts actually work.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36987"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"CypherTurn","organization_count":1,"organizations":["claire_radar"],"rank":824,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36987"},{"aliases":[],"benchmark_id":"claire-radar-2610-03226","caveat":"GPU kernels generated by large language model (LLM) agents can remain less efficient than expert implementations, but runtime alone does not reveal how the gap relates to design discovery and implementation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03226"],"document_share":0.0002723311546840959,"domain":"General AI","name":"D2K-Bench","organization_count":1,"organizations":["claire_radar"],"rank":825,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03226"},{"aliases":[],"benchmark_id":"claire-radar-2607-19834","caveat":"D2VBench evaluates LLM value alignment using 10,000 daily dilemma scenarios covering 158 fine-grained value concepts, with a hybrid paradigm of multiple-choice and open-ended questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19834"],"document_share":0.0002723311546840959,"domain":"General AI","name":"D2VBench","organization_count":1,"organizations":["claire_radar"],"rank":826,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.19834"},{"aliases":["OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses"],"benchmark_id":"claire-radar-2608-24160","caveat":"D3-Omni diagnoses multimodal judge models across T2I, T2V, and TTS tasks with 53 balanced orthogonal binary dimensions and 10,671 samples that isolate specific capability failures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24160"],"document_share":0.0002723311546840959,"domain":"General AI","name":"D3-Omni","organization_count":1,"organizations":["claire_radar"],"rank":827,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24160"},{"aliases":[],"benchmark_id":"opencompass-2030-dabstep","caveat":"DABstep is a benchmark for evaluating AI agents’ multi-step reasoning and planning abilities in realistic data analysis tasks, targeting code-executing language model agents. DABstep 是一个用于评估 AI 智能体在现实多步骤数据分析任务中推理与规划能力的基准，面向具备代码执行能力的语言模型代理。该基准包含450多个任务，源自金融分析平台，涵盖结构化数据处理、非结构化文档理解和跨源信息整合。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DABstep"],"document_share":0.0002723311546840959,"domain":"智能体","name":"DABstep","organization_count":1,"organizations":["opencompass_hub"],"rank":828,"released":"2025-06-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DABstep"},{"aliases":[],"benchmark_id":"claire-radar-2607-24016","caveat":"DailyBench is a unified benchmark for evaluating AI-generated image detectors on modern full-image synthesis and object-level manipulation. It comprises FakeBench, with images from recent open-source and commercial generative models, and ManipulationBench, with subtle edits to real images. Detectors are scored by balanced accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24016"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"DailyBench","organization_count":1,"organizations":["claire_radar"],"rank":829,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24016"},{"aliases":[],"benchmark_id":"llm-stats-dailyomni","caveat":"DailyOmni evaluates multimodal models on daily-life video understanding tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dailyomni?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"DailyOmni","organization_count":1,"organizations":["llm_stats"],"rank":830,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/dailyomni?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-12871","caveat":"DailyReport is a benchmark with 150 open-ended daily search tasks and 3,546 rubrics, evaluating search agents on information-seeking tasks. Tasks are decomposed into subtasks with cascade rubrics across dimensions, yielding interpretable scores and a user preference score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12871"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DailyReport","organization_count":1,"organizations":["claire_radar"],"rank":831,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.12871"},{"aliases":[],"benchmark_id":"claire-radar-2608-18034","caveat":"DAS-Bench provides 30 topics for evaluating academic survey generation, with DAS-Eval scoring 16 criteria across citation quality, taxonomic synthesis, hierarchical discourse, and manuscript reliability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.18034"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DAS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":832,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.18034"},{"aliases":["Leveraging Fine-grained Error Correction in Korean Speech Recognition for Consultation Services"],"benchmark_id":"claire-radar-2609-09889","caveat":"Dialogue-level ASR error correction benchmark dataset with 1,974 dialogues and 115,460 utterances from Korean call center interactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09889"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DasanCallDial","organization_count":1,"organizations":["claire_radar"],"rank":833,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09889"},{"aliases":[],"benchmark_id":"claire-radar-2607-28033","caveat":"DataClawEval evaluates autonomous data-engineering agents across 100 end-to-end tasks spanning PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL, with deterministic rule-based grading in isolated sandboxes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28033"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"DataClawEval","organization_count":1,"organizations":["claire_radar"],"rank":834,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28033"},{"aliases":["DataComp-VLM: Improved Open Datasets for Vision-Language Models"],"benchmark_id":"claire-radar-2606-28551","caveat":"A controlled benchmark for comparing vision-language training-data curation strategies across specified model scales and token budgets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.28551"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DataComp for VLMs (DCVLM)","organization_count":1,"organizations":["claire_radar"],"rank":835,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.28551"},{"aliases":[],"benchmark_id":"claire-radar-2607-06482","caveat":"DataGovBench evaluates LLMs on real-world data analysis using government open data. It comprises Table QA (complex decomposable questions with textual or visual answers) and Table Insight (exploratory data analysis producing expert-level findings). Scoring uses unambiguous reference-based metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06482"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DataGovBench","organization_count":1,"organizations":["claire_radar"],"rank":836,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06482"},{"aliases":[],"benchmark_id":"claire-radar-2608-25061","caveat":"Evaluates LLM-generated CUDA or Triton kernels for database query operations on GPUs, measuring execution speedup over torch.compile on TPC-H workloads.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25061"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DataKernelBench","organization_count":1,"organizations":["claire_radar"],"rank":837,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.25061"},{"aliases":[],"benchmark_id":"claire-radar-2609-37233","caveat":"Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37233"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"DatalogBench","organization_count":1,"organizations":["claire_radar"],"rank":838,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37233"},{"aliases":[],"benchmark_id":"claire-radar-2610-12190","caveat":"As claims about recursive self-improvement (RSI) and artificial general intelligence (AGI) proliferate, we ask a simple question: do frontier AI models have a sense of data, i.e., can they reliably select the right data for training?","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12190"],"document_share":0.0002723311546840959,"domain":"AI Scientist","name":"DataSense-Bench","organization_count":1,"organizations":["claire_radar"],"rank":839,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.12190"},{"aliases":[],"benchmark_id":"claire-radar-2608-03451","caveat":"DataSpace evaluates data agents on analytical questions over heterogeneous workspaces containing CSV, JSON, SQLite, Markdown, PDF, and video artifacts. Agents must discover and integrate relevant evidence and return complete tabular results, scored by a deterministic evaluator using header-invariant column alignment and type-aware comparison.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03451"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DataSpace","organization_count":1,"organizations":["claire_radar"],"rank":840,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03451"},{"aliases":[],"benchmark_id":"claire-radar-2610-01306","caveat":"Professional work often starts with a brief request that leaves the professional to work out what is needed, which documents matter, and whether the request's premise holds.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.01306"],"document_share":0.0002723311546840959,"domain":"Finance","name":"DAYJOB","organization_count":1,"organizations":["claire_radar"],"rank":841,"released":"2026-10-01","source":"claire_radar","url":"https://arxiv.org/abs/2610.01306"},{"aliases":[],"benchmark_id":"claire-radar-2607-22165","caveat":"A production-fidelity benchmark for LLM-based database operations agents, using instrumented PostgreSQL environments with active workloads and outcome-first evaluation across 106 scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22165"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DBA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":842,"released":"2026-07-24","source":"claire_radar","url":"https://arxiv.org/abs/2607.22165"},{"aliases":[],"benchmark_id":"claire-radar-2608-03794","caveat":"Evaluates LLMs across five database lifecycle phases using tasks like schema design, SQL implementation, and debugging, with a focus on holistic database management capabilities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03794"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DBLifeBench","organization_count":1,"organizations":["claire_radar"],"rank":843,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03794"},{"aliases":["Evaluating Agentic Code Repair Capabilities in Distributed Systems"],"benchmark_id":"claire-radar-2608-14863","caveat":"DDBench is a code-repair benchmark with 60 historical bugs from 13 open-source distributed systems, evaluated under symptom-only and context-augmented conditions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14863"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"DDBench","organization_count":1,"organizations":["claire_radar"],"rank":844,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14863"},{"aliases":[],"benchmark_id":"claire-radar-2605-23629","caveat":"DDX-TRACE evaluates medical diagnostic trajectories in multimodal neuroradiology over 211 physician-adjudicated cases, where models request imaging studies, update differential diagnoses, and stop with a localized final diagnosis under hidden evidence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23629"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"DDX-TRACE","organization_count":1,"organizations":["claire_radar"],"rank":845,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23629"},{"aliases":[],"benchmark_id":"claire-radar-2607-25218","caveat":"DebtBench evaluates negotiation dialogue systems in debt collection with persona-enriched user profiles to reflect behavioral heterogeneity. It uses a dataset of negotiation scenarios with multiple user personas, scoring financial recovery and interaction experience metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25218"],"document_share":0.0002723311546840959,"domain":"Finance","name":"DebtBench","organization_count":1,"organizations":["claire_radar"],"rank":846,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25218"},{"aliases":[],"benchmark_id":"opencompass-1078-debugbench","caveat":"DebugBench is an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python. DebugBench 是一个包含 4,253 个实例的 LLM 调试基准。它涵盖了 C++、Java 和 Python 中的四个主要错误类别和 18 种次要类型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DebugBench"],"document_share":0.0002723311546840959,"domain":"代码","name":"DebugBench","organization_count":1,"organizations":["opencompass_hub"],"rank":847,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DebugBench"},{"aliases":[],"benchmark_id":"claire-radar-2610-07967","caveat":"As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their reliable deployment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.07967"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DecepEval","organization_count":1,"organizations":["claire_radar"],"rank":848,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.07967"},{"aliases":["decision-layer-benchmark"],"benchmark_id":"claire-radar-github-gohaike-decision-layer-benchmark","caveat":"A token-cost model for an AI agent's decision layer: estimating monthly spend for five decision questions across naive, cached, and tuned OpenAI, Anthropic, and Google baselines versus Jev.","document_count":1,"document_ids":["claire_radar:https://github.com/GOhaike/decision-layer-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Decision Layer Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":849,"released":"2026-10-12","source":"claire_radar","url":"https://github.com/GOhaike/decision-layer-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-github-supreethbare-decision-model-bench","caveat":"The repository compares typed decision models on several enterprise tasks, but it is a one-off harness without a formal benchmark name or clear external submission mechanism.","document_count":1,"document_ids":["claire_radar:https://github.com/supreethbare/decision-model-bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"decision-model-bench","organization_count":1,"organizations":["claire_radar"],"rank":850,"released":"2026-09-23","source":"claire_radar","url":"https://github.com/supreethbare/decision-model-bench"},{"aliases":[],"benchmark_id":"llm-stats-deck-bench","caveat":"DECK-Bench is Moonshot AI's internal evaluation of agents on creating and reasoning over presentation-style knowledge-work artifacts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deck-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"productivity","name":"DECK-Bench","organization_count":1,"organizations":["llm_stats"],"rank":851,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/deck-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-15879","caveat":"DECODEM provides benchmark datasets for evaluating automated extraction of corporate governance variables from organizational documents, with human-annotated charters and bylaws.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15879"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DECODEM","organization_count":1,"organizations":["claire_radar"],"rank":852,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.15879"},{"aliases":[],"benchmark_id":"claire-radar-2606-13994","caveat":"DeCompBench evaluates agent safety against decomposition attacks, where harmful tasks are broken into benign subtasks; it measures refusal rates and objective fulfillment on decomposed variants.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13994"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DeCompBench","organization_count":1,"organizations":["claire_radar"],"rank":853,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.13994"},{"aliases":[],"benchmark_id":"claire-radar-2607-12868","caveat":"Deep4ge is a dataset of 14,227 training runs from 59 DNN programs with documented faults, providing per-epoch features for fault detection and diagnosis tasks, including binary detection, multi-class diagnosis, and early prediction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.12868"],"document_share":0.0002723311546840959,"domain":"Software & AI Compute","name":"Deep4ge","organization_count":1,"organizations":["claire_radar"],"rank":854,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.12868"},{"aliases":[],"benchmark_id":"claire-radar-2607-11228","caveat":"DeepBiasBench is a benchmark for in-depth probing of social biases in LVLMs, built using an adaptive framework with dynamic test data generation and iterative rewriting.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11228"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DeepBias","organization_count":1,"organizations":["claire_radar"],"rank":855,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11228"},{"aliases":[],"benchmark_id":"claire-radar-2608-26757","caveat":"Faithful chart generation in real-world data-science workflows requires grounding visualizations in scattered evidence, computing chart-ready quantities, and rendering them accurately.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26757"],"document_share":0.0002723311546840959,"domain":"Finance","name":"DEEPCHART","organization_count":1,"organizations":["claire_radar"],"rank":856,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26757"},{"aliases":[],"benchmark_id":"opencompass-1605-deepfake-eval-2024","caveat":"Deepfake-Eval-2024 is an in-the-wild deepfake dataset. Deepfake-Eval-2024 contains 44 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing contemporary manipulation technologies, diverse media content, 88 different website sources, and 52 different languages. Deepfake-Eval-2024 是一个真实场景的深度伪造数据集。该数据集包含44小时的视频、56.5小时的音频和1,975张图像，涵盖当代篡改技术、多样化的媒体内容、来自88个不同网站来源的素材以及52种不同语言。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Deepfake-Eval-2024"],"document_share":0.0002723311546840959,"domain":"其他","name":"Deepfake-Eval-2024","organization_count":1,"organizations":["opencompass_hub"],"rank":857,"released":"2025-03-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Deepfake-Eval-2024"},{"aliases":[],"benchmark_id":"llm-stats-deep-planning","caveat":"DeepPlanning evaluates LLMs on complex multi-step planning tasks requiring long-horizon reasoning, goal decomposition, and strategic decision-making.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deep-planning?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"DeepPlanning","organization_count":1,"organizations":["llm_stats"],"rank":858,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/deep-planning?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1987-deepresearchbench","caveat":"DeepResearch Bench is a comprehensive benchmark designed to evaluate large language model (LLM) agents on complex research tasks. DeepResearch Bench 是一个面向大型语言模型（LLM）智能体的综合性评测基准，专为评估其在复杂研究任务中的表现而设计。 该基准包含 100 个由 22 个领域的专家精心设计的博士级研究任务，涵盖多领域的深度研究需求。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DeepResearchBench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"DeepResearchBench","organization_count":1,"organizations":["opencompass_hub"],"rank":859,"released":"2025-06-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DeepResearchBench"},{"aliases":[],"benchmark_id":"llm-stats-deepsearchqa","caveat":"DeepSearchQA is a benchmark for evaluating deep search and question-answering capabilities, testing models' ability to perform multi-hop reasoning and information retrieval across complex knowledge domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deepsearchqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"DeepSearchQA","organization_count":1,"organizations":["llm_stats"],"rank":860,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/deepsearchqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-07946","caveat":"Evaluates coding agents on 113 original, long-horizon software engineering tasks across 91 open-source repositories in five languages. Tasks are written from scratch, with hand-written verifiers that check requested functionality and accept any correct implementation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.07946"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"DeepSWE","organization_count":1,"organizations":["claire_radar"],"rank":861,"released":"2026-05-18","source":"claire_radar","url":"https://arxiv.org/abs/2607.07946"},{"aliases":[],"benchmark_id":"llm-stats-deepswe","caveat":"DeepSWE is a software engineering agent benchmark evaluated with the mini-swe-agent harness, where each task is solved in an isolated container with no internet access. It measures an agent's ability to autonomously resolve real-world coding issues end to end.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deepswe?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"DeepSWE","organization_count":1,"organizations":["llm_stats"],"rank":862,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-deepswe-1-0","caveat":"DeepSWE 1.0 pass@1 benchmark as reported by Artificial Analysis using provider-specific harness runs.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.0?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"DeepSWE 1.0","organization_count":1,"organizations":["llm_stats"],"rank":863,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.0?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-deepswe-1-1","caveat":"DeepSWE 1.1 evaluates software engineering agents using the mini-swe-agent harness.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.1?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"DeepSWE 1.1","organization_count":1,"organizations":["llm_stats"],"rank":864,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.1?top_n=500"},{"aliases":["DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models"],"benchmark_id":"claire-radar-2606-18557","caveat":"A formally grounded benchmark and generation pipeline for evaluating defeasible abduction through automatically verifiable hypothesis construction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18557"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"DeFAb","organization_count":1,"organizations":["claire_radar"],"rank":865,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18557"},{"aliases":[],"benchmark_id":"claire-radar-2609-11504","caveat":"Benchmark of 207 prompts for synthesizing DeFi workflows from natural language.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11504"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DeFiFlowBench","organization_count":1,"organizations":["claire_radar"],"rank":866,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11504"},{"aliases":[],"benchmark_id":"claire-radar-2606-04067","caveat":"DelegateCI-Bench evaluates privacy-conscious query rewriting for LLM delegation, with 3,167 samples combining synthetic data across 20 task types, real user queries from WildChat, and a medical challenge set. Systems rewrite queries to suppress non-essential sensitive information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04067"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"DelegateCI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":867,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.04067"},{"aliases":[],"benchmark_id":"claire-radar-2610-05532","caveat":"AI agents that send emails, edit files, and make purchases must decide when to act on their own and when to check with the user first.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05532"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DelegationBench","organization_count":1,"organizations":["claire_radar"],"rank":868,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05532"},{"aliases":[],"benchmark_id":"claire-radar-2608-22770","caveat":"Evaluates search-enabled LLMs on reconstructing security-level delisting events from public sources, using a 1,200-record benchmark with joint accuracy metrics.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22770v1"],"document_share":0.0002723311546840959,"domain":"Finance","name":"DelistBench","organization_count":1,"organizations":["claire_radar"],"rank":869,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.22770v1"},{"aliases":[],"benchmark_id":"claire-radar-2608-19653","caveat":"DeltaML-Bench contains 48 tasks from research papers requiring agents to improve published baselines within open-source repositories, evaluated under compute constraints with fixed grading and integrity checks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.19653"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DeltaML-Bench","organization_count":1,"organizations":["claire_radar"],"rank":870,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.19653"},{"aliases":[],"benchmark_id":"claire-radar-github-snyderconsulting-delverl","caveat":"Evaluates local game-playing agents on a deterministic, procedurally generated, partially observed roguelike where agents must secure a key and reach the exit on each floor, scored by floor depth under a 1,500-turn cap.","document_count":1,"document_ids":["claire_radar:https://github.com/SnyderConsulting/DelveRL"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"DelveRL","organization_count":1,"organizations":["claire_radar"],"rank":871,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/SnyderConsulting/DelveRL"},{"aliases":[],"benchmark_id":"claire-radar-2606-20634","caveat":"A cross-regime benchmark for testing whether agent-runtime records are sufficient to reconstruct decision-level governance properties under deterministic degradation conditions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20634"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DEMM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":872,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.20634"},{"aliases":["Genesis: A Generative Engine for Hierarchical Satellite Image Synthesis","Genesis"],"benchmark_id":"claire-radar-2609-02683","caveat":"Provides a multi-scale pyramid dataset with diverse geographic regions for evaluating hierarchical satellite image synthesis, along with pyramid-level metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02683"],"document_share":0.0002723311546840959,"domain":"General AI","name":"dense500","organization_count":1,"organizations":["claire_radar"],"rank":873,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02683"},{"aliases":[],"benchmark_id":"claire-radar-2608-30300","caveat":"Evaluates coding agents on 203 real-world dependency-upgrade tasks requiring source adaptation, scored by task completion rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30300"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"DEPBENCH","organization_count":1,"organizations":["claire_radar"],"rank":874,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30300"},{"aliases":[],"benchmark_id":"claire-radar-2606-05238","caveat":"DeployBench evaluates LLM agents on 51 research-artifact deployment tasks across AI/ML, systems, and scientific computing. Tasks require setting up environments with multi-language toolchains and system-level dependencies, verified by hidden pipelines that execute the paper's experiments and check outputs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05238"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"DeployBench","organization_count":1,"organizations":["claire_radar"],"rank":875,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05238"},{"aliases":[],"benchmark_id":"llm-stats-dermmcqa","caveat":"Dermatology multiple choice question assessment benchmark for evaluating medical knowledge and diagnostic reasoning in dermatological conditions and treatments.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dermmcqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"healthcare","name":"DermMCQA","organization_count":1,"organizations":["llm_stats"],"rank":876,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/dermmcqa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-design2code","caveat":"Design2Code evaluates the ability to generate code (HTML/CSS/JS) from visual designs.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/design2code?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Design2Code","organization_count":1,"organizations":["llm_stats"],"rank":877,"released":"2024-03-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/design2code?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-03103","caveat":"DeskCraft evaluates desktop GUI agents on long-horizon professional workflows and human-in-the-loop collaboration across 538 executable tasks in live Ubuntu desktop environments. Covers design, video, audio, and 3D creation software, with execution-based verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03103"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DeskCraft","organization_count":1,"organizations":["claire_radar"],"rank":878,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03103"},{"aliases":[],"benchmark_id":"claire-radar-2607-26041","caveat":"Desktop-Delta Bench (DDB) evaluates computer-use models on step-level GUI transition understanding through two tasks: temporal ordering of 3-frame observations and before-after pair classification with five action types, covering 2,013 human-verified instances across ~15 applications.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26041"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Desktop-Delta Bench","organization_count":1,"organizations":["claire_radar"],"rank":879,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.26041"},{"aliases":["Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval"],"benchmark_id":"claire-radar-2606-30473","caveat":"DevDataBench evaluates structured-metadata retrieval using grounded, facet-targeted queries across 15 languages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30473"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DevDataBench","organization_count":1,"organizations":["claire_radar"],"rank":880,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30473"},{"aliases":[],"benchmark_id":"claire-radar-2607-13465","caveat":"DevicesWorld is an executable benchmark for cross-device agent evaluation, comprising 6,140 tasks across Android, Linux, and SmartHome environments, with rule-based verifiers for automatic scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.13465"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DevicesWorld","organization_count":1,"organizations":["claire_radar"],"rank":881,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.13465"},{"aliases":["devin-agent-benchmark"],"benchmark_id":"claire-radar-github-majorminorstudio-devin-agent-benchmark","caveat":"A paired software-repair evaluation comparing Devin's SWE-2 Medium and Max modes on 30 unique bugs across three difficulty tiers, with hidden tests and a clean regression suite defining success.","document_count":1,"document_ids":["claire_radar:https://github.com/MAJORminorStudio/devin-agent-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Devin + SWE-2 Reasoning Effort Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":882,"released":"2026-09-16","source":"claire_radar","url":"https://github.com/MAJORminorStudio/devin-agent-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2610-03278","caveat":"As dexterous hands proliferate, collecting data and training policies separately for every morphology becomes increasingly impractical.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03278"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"DexJoCo-X","organization_count":1,"organizations":["claire_radar"],"rank":883,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03278"},{"aliases":[],"benchmark_id":"claire-radar-2606-19501","caveat":"A six-axis evaluation harness for regulator-aligned assessment of supervisory tickets produced in decentralized-finance risk monitoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19501"],"document_share":0.0002723311546840959,"domain":"Finance","name":"DeXposure-Bench","organization_count":1,"organizations":["claire_radar"],"rank":884,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19501"},{"aliases":[],"benchmark_id":"claire-radar-2607-08751","caveat":"Evaluates dexterous manipulation across 100 tasks, multiple embodiments, and visual variations, with 3,180 demonstrations and a VR teleoperation interface.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08751"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"DexVerse","organization_count":1,"organizations":["claire_radar"],"rank":885,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08751"},{"aliases":[],"benchmark_id":"claire-radar-2607-20491","caveat":"DFAH-Bench measures whether financial agents reproduce decisions and observable execution paths across qualified replay groups.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20491"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DFAH-Bench","organization_count":1,"organizations":["claire_radar"],"rank":886,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.20491"},{"aliases":[],"benchmark_id":"claire-radar-2609-34913","caveat":"Evaluates tool-using LLM governance agents reviewing evidence, focusing on resistance to planted deception.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34913"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DGF-Bench","organization_count":1,"organizations":["claire_radar"],"rank":887,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34913"},{"aliases":[],"benchmark_id":"claire-radar-github-luongnv89-dgx-spark-llm-lab","caveat":"This repository contains tools for benchmarking LLM configurations (model, quantization, serving flags, thinking mode) on local hardware, with hidden executable tests and leaderboard-like reports.","document_count":1,"document_ids":["claire_radar:https://github.com/luongnv89/dgx-spark-llm-lab"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"dgx-spark-llm-lab","organization_count":1,"organizations":["claire_radar"],"rank":888,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/luongnv89/dgx-spark-llm-lab"},{"aliases":[],"benchmark_id":"claire-radar-2608-03591","caveat":"DiagChain evaluates LLM agents on evidence-grounded attack chain reconstruction from heterogeneous telemetry. It provides 69 scenarios and five metrics for stage-wise assessment of reasoning steps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03591"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"DiagChain","organization_count":1,"organizations":["claire_radar"],"rank":889,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03591"},{"aliases":[],"benchmark_id":"claire-radar-2608-12262","caveat":"Diagram-MMU evaluates multimodal language models on parsing scientific diagrams into LaTeX TikZ code, editing diagram code, and answering questions about diagrams, using 3.7k diagrams and 18.3k questions across six domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12262"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Diagram-MMU","organization_count":1,"organizations":["claire_radar"],"rank":890,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.12262"},{"aliases":[],"benchmark_id":"opencompass-2032-dice-bench","caveat":"DICE-BENCH is a benchmark for evaluating large language models’ tool-use capabilities in multi-round, multi-party dialogues, focusing on function selection, parameter filling, and dialogue context comprehension. DICE-BENCH 是一个评估大型语言模型在多轮、多方对话中工具调用能力的基准，涵盖函数选择、参数填充和对话上下文理解等维度。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DICE-BENCH"],"document_share":0.0002723311546840959,"domain":"智能体","name":"DICE-BENCH","organization_count":1,"organizations":["opencompass_hub"],"rank":891,"released":"2025-06-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DICE-BENCH"},{"aliases":[],"benchmark_id":"claire-radar-2606-26602","caveat":"DiCoBench evaluates multimodal LLMs on multi-image fine-grained perception using high-resolution (up to 2K) image pairs. It includes 765 multiple-choice questions across two tracks: differential and commonality visual cues, covering 8 perception tasks, with exact-match scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26602"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DiCoBench","organization_count":1,"organizations":["claire_radar"],"rank":892,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.26602"},{"aliases":[],"benchmark_id":"claire-radar-2608-12593","caveat":"Evaluates AI agents on discovery of unknown transformation rules across 70 independent games with seven difficulty tiers, requiring experimentation and rule generalization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12593"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DiG-bench","organization_count":1,"organizations":["claire_radar"],"rank":893,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.12593"},{"aliases":[],"benchmark_id":"claire-radar-2605-30090","caveat":"DirectorBench evaluates long-form video generation across 5 dimensions (script, visual, audio, cross-modal, stability) using 80 metadata entries, 7 user profiles, and 40 checkpoint criteria.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30090"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DirectorBench","organization_count":1,"organizations":["claire_radar"],"rank":894,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30090"},{"aliases":[],"benchmark_id":"claire-radar-2605-27957","caveat":"DisasterBench is a benchmark for evaluating structured multi-agent planning over disaster-response tools. It includes 233 expert-verified tasks, 26 agents, 81 typed compatibility edges, and 5 planning paradigms. It uses First-Point-of-Failure (FPoF) for step-level failure attribution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27957"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DisasterBench","organization_count":1,"organizations":["claire_radar"],"rank":895,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27957"},{"aliases":[],"benchmark_id":"claire-radar-2606-06217","caveat":"Evaluates multimodal reasoning for UAV-based disaster response across 14 scene types and 9 tasks spanning pre-, during-, and post-disaster stages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06217"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DisasterBench","organization_count":1,"organizations":["claire_radar"],"rank":896,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06217"},{"aliases":["DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?"],"benchmark_id":"claire-radar-2609-33357","caveat":"Evaluates AI agents on evidence-grounded scientific reasoning across 203 tasks in eight life-science tracks, covering data integrity, analysis verification, and hypothesis generation under adversarial review.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33357"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DISCERN","organization_count":1,"organizations":["claire_radar"],"rank":897,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33357"},{"aliases":[],"benchmark_id":"claire-radar-2606-27669","caveat":"DiscoBench evaluates search agents on clarification-aware deep search, covering 211 samples and 463 ambiguity instances across 11 domains, with four ambiguity types, measuring task utility, ambiguity detection, interaction strategy, and cost efficiency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27669"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DiscoBench","organization_count":1,"organizations":["claire_radar"],"rank":898,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.27669"},{"aliases":[],"benchmark_id":"claire-radar-2605-26087","caveat":"DiscoverPhysics is an interactive benchmark for LLM agents to discover laws of motion in simulated worlds with altered physics, scoring trajectory MSE and explanation quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26087"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DiscoverPhysics","organization_count":1,"organizations":["claire_radar"],"rank":899,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26087"},{"aliases":["distilbert-onnx-quantization-benchmark"],"benchmark_id":"claire-radar-github-parth-udawant-distilbert-onnx-quantization-benchmark","caveat":"An end-to-end ONNX quantization benchmark for a DistilBERT sentiment classifier, comparing PyTorch FP32, ONNX FP32, dynamic INT8, and static INT8 variants on accuracy, latency, and model size using a fixed 200-sample SST-2 validation set.","document_count":1,"document_ids":["claire_radar:https://github.com/Parth-Udawant/distilbert-onnx-quantization-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DistilBERT ONNX Quantization Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":900,"released":"2026-09-06","source":"claire_radar","url":"https://github.com/Parth-Udawant/distilbert-onnx-quantization-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2606-08894","caveat":"Distract-Bench evaluates robustness of vision-language models to semantic visual distractions, which are meaningful but task-irrelevant cues that preserve the ground-truth answer.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08894"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Distract-Bench","organization_count":1,"organizations":["claire_radar"],"rank":901,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.08894"},{"aliases":[],"benchmark_id":"claire-radar-2606-13931","caveat":"DLawBench evaluates multi-turn legal consultation in Chinese and U.S. law across four client personas. It scores LLMs on information gathering, legal reasoning, and claim support using 461 cases, 5,532 paired fact entries, inquiry and issue rubrics, and a public leaderboard.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13931"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DLawBench","organization_count":1,"organizations":["claire_radar"],"rank":902,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13931"},{"aliases":[],"benchmark_id":"opencompass-1648-dme","caveat":"VLB provides a robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. Based on LlavaBench and MMvet, we have curated two more challenging versions of the datasets: LlavaBench_hard and MMvet_hard. These are the hardest multimodal combinations. VLB 为 LVLMs 提供了一种稳健且全面的评估，降低了数据污染并具有灵活的复杂性。基于 LlavaBench 和 MMvet，我们精心制作了两个更具挑战性的数据集版本：LlavaBench_hard 和 MMvet_hard。这些是我们动态策略中最具挑战性的多模态组合（V1+L4）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DME"],"document_share":0.0002723311546840959,"domain":"多模态","name":"DME","organization_count":1,"organizations":["opencompass_hub"],"rank":903,"released":"2024-10-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DME"},{"aliases":[],"benchmark_id":"claire-radar-2606-27499","caveat":"DMV-Bench evaluates visual memory of multimodal agents in an interactive shopping environment with 1,000 product variants. Agents run autonomous sessions and must recall cued product images via exact URL match, with text leakage controlled.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27499"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DMV-Bench","organization_count":1,"organizations":["claire_radar"],"rank":904,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.27499"},{"aliases":["Can LLMs Normalize Databases? A Benchmark and Multi-Agent Framework for Schema Normalization"],"benchmark_id":"claire-radar-2609-11141","caveat":"A benchmark of 3,275 samples for evaluating LLM-driven database normalization from 1NF to BCNF.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11141"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DNBENCH","organization_count":1,"organizations":["claire_radar"],"rank":905,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11141"},{"aliases":[],"benchmark_id":"claire-radar-2609-12446","caveat":"A belief-shift evaluation in Werewolf measures whether models update beliefs in response to accusers or accusations, using belief updating as a signal of communication skills.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12446"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf","organization_count":1,"organizations":["claire_radar"],"rank":906,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12446"},{"aliases":[],"benchmark_id":"claire-radar-2608-01451","caveat":"DOC2CI evaluates LLM-generated CI/CD configuration YAML against reference configurations from four CI services, measuring exact match and schema validity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01451"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Doc2CI","organization_count":1,"organizations":["claire_radar"],"rank":907,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.01451"},{"aliases":[],"benchmark_id":"claire-radar-2608-08459","caveat":"Doc2DB-Bench evaluates document-to-database construction, converting long heterogeneous documents into normalized relational databases with entity identities, keys, cross-table links, and integrity constraints. It includes 203 document instances across 42 schemas and 7 domains, with fine-grained capability annotations for intra-table extraction and inter-table reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08459"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Doc2DB-Bench","organization_count":1,"organizations":["claire_radar"],"rank":908,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08459"},{"aliases":[],"benchmark_id":"claire-radar-2609-20574","caveat":"Evaluates source attribution in Document VQA by grounding answers to layout elements such as text blocks, tables, and images.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.20574"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DocAttriBench","organization_count":1,"organizations":["claire_radar"],"rank":909,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.20574"},{"aliases":[],"benchmark_id":"claire-radar-2606-01936","caveat":"DocFormBench evaluates content-aware document formatting for LLMs and multimodal models, using accuracy and efficiency metrics, with a workflow method DocFormFlow for target localization and modification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01936"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DocFormBench","organization_count":1,"organizations":["claire_radar"],"rank":910,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01936"},{"aliases":[],"benchmark_id":"claire-radar-2609-02059","caveat":"Evaluates integrated chart-context reasoning in document-style images.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02059"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DocHop","organization_count":1,"organizations":["claire_radar"],"rank":911,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02059"},{"aliases":[],"benchmark_id":"claire-radar-2608-12911","caveat":"DocPrivacyBench evaluates susceptibility of document understanding MLLMs to relational privacy leakage when visual evidence is absent or minimal, using KIE tasks on identity documents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12911"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DocPrivacyBench","organization_count":1,"organizations":["claire_radar"],"rank":912,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.12911"},{"aliases":["documentvision-multimodal-benchmark"],"benchmark_id":"claire-radar-github-master72o-documentvision-multimodal-benchmark","caveat":"Evaluates OCR pipelines and vision-language models on dense tables, key-value extraction, charts, and layout grounding across high-resolution enterprise document images.","document_count":1,"document_ids":["claire_radar:https://github.com/master72o/documentvision-multimodal-benchmark"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"DocumentVision","organization_count":1,"organizations":["claire_radar"],"rank":913,"released":"2026-10-10","source":"claire_radar","url":"https://github.com/master72o/documentvision-multimodal-benchmark"},{"aliases":[],"benchmark_id":"llm-stats-docvqa","caveat":"A dataset for Visual Question Answering on document images containing 50,000 questions defined on 12,000+ document images. The benchmark tests AI's ability to understand document structure and content, requiring models to comprehend document layout and perform information retrieval to answer questions about document images.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/docvqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"DocVQA","organization_count":1,"organizations":["llm_stats"],"rank":914,"released":"2020-07-01","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-docvqatest","caveat":"DocVQA is a Visual Question Answering benchmark on document images containing 50,000 questions defined on 12,000+ document images. The benchmark focuses on understanding document structure and content to answer questions about various document types including letters, memos, notes, and reports from the UCSF Industry Documents Library.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/docvqatest?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"DocVQAtest","organization_count":1,"organizations":["llm_stats"],"rank":915,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/docvqatest?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-02396","caveat":"A controlled evaluation of 13 attribution methods across CNN and Vision Transformer architectures using five quality and cost dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02396"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs","organization_count":1,"organizations":["claire_radar"],"rank":916,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02396"},{"aliases":[],"benchmark_id":"claire-radar-2609-39909","caveat":"We introduce DoGBENCH (Documentation Generation Benchmark), to our knowledge, the first benchmark for generating and maintaining real user-facing software documentation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39909"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"DoGBench","organization_count":1,"organizations":["claire_radar"],"rank":917,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39909"},{"aliases":[],"benchmark_id":"claire-radar-2609-24971","caveat":"Evaluates agent memory through task completion.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24971"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DolphinBench","organization_count":1,"organizations":["claire_radar"],"rank":918,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24971"},{"aliases":[],"benchmark_id":"claire-radar-2606-04262","caveat":"DOSEBENCH evaluates LLM decision-making on over-the-counter dosing questions, with 81 curated scenarios for adult acetaminophen and ibuprofen use. Correct answers require tracking dose timing, computing rolling 24-hour intake, and following product-label constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04262"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"DOSEBENCH","organization_count":1,"organizations":["claire_radar"],"rank":919,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.04262"},{"aliases":[],"benchmark_id":"llm-stats-doubao-multi-turn-bench","caveat":"Doubao Multi-Turn Bench evaluates models on multi-turn conversational tasks, measuring context retention, instruction following, and coherent reasoning across extended dialogues.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/doubao-multi-turn-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"instruction_following","name":"Doubao Multi-Turn Bench","organization_count":1,"organizations":["llm_stats"],"rank":920,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/doubao-multi-turn-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1731-dove","caveat":"DOVE (Dataset Of Variation Evaluation) is a large-scale dataset containing prompt perturbations of various evaluation benchmarks. DOVE（变异评估数据集），这是一个大规模数据集，包含了各种评估基准的提示扰动。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DOVE"],"document_share":0.0002723311546840959,"domain":"其他","name":"DOVE","organization_count":1,"organizations":["opencompass_hub"],"rank":921,"released":"2025-03-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DOVE"},{"aliases":[],"benchmark_id":"claire-radar-2605-27904","caveat":"Dr-CiK is a benchmark for evaluating whether agents can retrieve forecasting-relevant supporting context from a document corpus, filter distractors, distill evidence, and generate forecasts. It provides context ablations and evaluates deep research and forecasting methods.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27904"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Dr-CiK","organization_count":1,"organizations":["claire_radar"],"rank":922,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27904"},{"aliases":[],"benchmark_id":"draco","caveat":"Deep-research style long-context agent tasks; the retrieval and tool stack is part of the score.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"long_context","name":"DRACO","organization_count":1,"organizations":["Tencent"],"rank":923,"released":null,"source":"model_reports","url":"https://huggingface.co/datasets/perplexity-ai/draco"},{"aliases":[],"benchmark_id":"llm-stats-draco","caveat":"DRACO is a deep research benchmark that evaluates an agent's ability to gather, synthesize, and reason over information to answer complex research questions. Scores are based on official rubrics per question, with the final score being the average across all questions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/draco?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"DRACO","organization_count":1,"organizations":["llm_stats"],"rank":924,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/draco?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-06322","caveat":"DragOn is a drag grounding benchmark and training dataset for GUI agents across four interaction types: text highlighting, cell selection, element resizing, and slider manipulation, with 3.5M training tasks and 2,000 evaluation examples.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06322"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"DragOn","organization_count":1,"organizations":["claire_radar"],"rank":925,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06322"},{"aliases":[],"benchmark_id":"opencompass-2045-dragon","caveat":"DRAGON 是一个用于评估检索增强生成（RAG）系统在俄语新闻语境中事实性与检索能力的动态基准，支持对检索器与生成器组件的全面评估。 DRAGON is a dynamic benchmark for evaluating Retrieval-Augmented Generation (RAG) systems in Russian news contexts, supporting comprehensive assessment of both retriever and generator components.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DRAGON"],"document_share":0.0002723311546840959,"domain":"知识","name":"DRAGON","organization_count":1,"organizations":["opencompass_hub"],"rank":926,"released":"2025-07-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DRAGON"},{"aliases":[],"benchmark_id":"claire-radar-2609-00646","caveat":"Evaluates every stage of short-drama production, from script to finished drama, across five axes and 63 leaf dimensions using professional annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00646"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DramaChain Bench","organization_count":1,"organizations":["claire_radar"],"rank":927,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.00646"},{"aliases":["DrawAI: Agentic Benchmark and Workflow for Making Raster Images Editable"],"benchmark_id":"claire-radar-2608-00548","caveat":"DrawAI-Bench evaluates systems that reconstruct editable, structured artifacts from raster images across figures, slides, posters, and diagrams.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00548"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DrawAI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":928,"released":"2026-08-01","source":"claire_radar","url":"https://arxiv.org/abs/2608.00548"},{"aliases":[],"benchmark_id":"claire-radar-2607-15418","caveat":"The evaluation object is unclear from the provided information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15418"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DrawingVQA","organization_count":1,"organizations":["claire_radar"],"rank":929,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.15418"},{"aliases":[],"benchmark_id":"claire-radar-2606-18191","caveat":"DRFLOW evaluates an agent's ability to predict personalized workflows, sequences of action-steps, from heterogeneous sources across five domains. It includes 100 tasks with reference workflow steps and multiple diagnostic metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18191"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DRFLOW","organization_count":1,"organizations":["claire_radar"],"rank":930,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18191"},{"aliases":[],"benchmark_id":"claire-radar-2607-19772","caveat":"DRGBT-1K is a large-scale benchmark for dynamic RGBT tracking with 1,045 real-world sequences, 795K frame pairs, dense annotations, and a unified evaluation protocol across 20 trackers, plus an online leaderboard.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19772"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DRGBT-1K","organization_count":1,"organizations":["claire_radar"],"rank":931,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.19772"},{"aliases":[],"benchmark_id":"claire-radar-2609-38604","caveat":"Modern LLM agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing benchmarks generally assume that users always accurately and sufficiently communicate a fixed intent.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38604"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Drift-Bench","organization_count":1,"organizations":["claire_radar"],"rank":932,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.38604"},{"aliases":[],"benchmark_id":"claire-radar-2610-01204","caveat":"Autoregressive modelling has achieved remarkable success in language and sequence tasks by learning to predict future states from previous observation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.01204"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DrillBench","organization_count":1,"organizations":["claire_radar"],"rank":933,"released":"2026-10-01","source":"claire_radar","url":"https://arxiv.org/abs/2610.01204"},{"aliases":[],"benchmark_id":"claire-radar-2605-24267","caveat":"The evaluation targets conversational implicature in question utterances, using a semi-automated pipeline to generate question-context-interpretation instances with controlled variation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24267"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DRInQ","organization_count":1,"organizations":["claire_radar"],"rank":934,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.24267"},{"aliases":[],"benchmark_id":"claire-radar-2609-08117","caveat":"DriveMotion provides 393 hours of driver motion sequences with synchronized driving context for continuous driver motion forecasting.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08117"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DriveMotion","organization_count":1,"organizations":["claire_radar"],"rank":935,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08117"},{"aliases":[],"benchmark_id":"claire-radar-2605-23176","caveat":"DriveSpatial is a benchmark of 15.6K human-verified QA pairs across 20 tasks from five AD datasets, evaluating VLMs on cognitive scene construction, multi-view relational understanding, temporal reasoning, and generalization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23176"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"DRIVESPATIAL","organization_count":1,"organizations":["claire_radar"],"rank":936,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23176"},{"aliases":[],"benchmark_id":"claire-radar-2609-38948","caveat":"Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38948"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"DrivingBench","organization_count":1,"organizations":["claire_radar"],"rank":937,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.38948"},{"aliases":[],"benchmark_id":"llm-stats-drop","caveat":"DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring discrete reasoning over paragraph content. It contains crowdsourced, adversarially-created questions that require resolving references and performing discrete operations like addition, counting, or sorting, demanding comprehensive paragraph understanding beyond paraphrase-and-entity-typing shortcuts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/drop?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"DROP","organization_count":1,"organizations":["llm_stats"],"rank":938,"released":"2019-03-01","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/drop?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-536-drop","caveat":"DROP is a QA dataset which tests comprehensive understanding of paragraphs. In this crowdsourced, adversarially-created, 96k question-answering benchmark, a system must resolve multiple references in a question, map them onto a paragraph, and perform discrete operations over them (such as addition, counting, or sorting). DROP 是一个测试段落综合理解能力的 QA 数据集。在这个众包、对抗性创建的 96K 问题解答基准中，系统必须解析问题中的多个引用，将它们映射到段落中，并对它们执行离散操作（如加法、计数或排序）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DROP"],"document_share":0.0002723311546840959,"domain":"推理","name":"DROP","organization_count":1,"organizations":["opencompass_hub"],"rank":939,"released":"2019-03-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DROP"},{"aliases":[],"benchmark_id":"claire-radar-2606-20663","caveat":"DrugBench evaluates AI control protocols for mitigating medication-related harm using 3,671 medical conversations and FDA drug labels, covering drug interactions, contraindications, dosing constraints, and patient action restrictions. Introduces severity-based monitoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20663"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"DrugBench","organization_count":1,"organizations":["claire_radar"],"rank":940,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.20663"},{"aliases":[],"benchmark_id":"claire-radar-2610-13116","caveat":"As large language models (LLMs) acquire advanced biological design capabilities, they offer substantial research benefits but simultaneously introduce severe dual-use risks, such as engineering enhanced pathogens or drug-resistant bacteria.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13116"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DRYLAB-Bench","organization_count":1,"organizations":["claire_radar"],"rank":941,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13116"},{"aliases":[],"benchmark_id":"opencompass-544-ds-1000","caveat":"DS-1000 is a code generation benchmark with a thousand data science questions spanning seven Python libraries that (1) reflects diverse, realistic, and practical use cases, (2) has a reliable metric, (3) defends against memorization by perturbing questions. DS-1000 是一个代码生成基准测试，包含一千个数据科学问题，涵盖七个Python库，其特点是（1）反映多样化、现实且实用的用例，（2）具有可靠的度量标准，（3）通过扰乱问题来防止记忆化。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DS-1000"],"document_share":0.0002723311546840959,"domain":"代码","name":"DS-1000","organization_count":1,"organizations":["opencompass_hub"],"rank":942,"released":"2022-11-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DS-1000"},{"aliases":[],"benchmark_id":"llm-stats-ds-arena-code","caveat":"Data Science Arena Code benchmark for evaluating LLMs on realistic data science code generation tasks. Tests capabilities in complex data processing, analysis, and programming across popular Python libraries used in data science workflows.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ds-arena-code?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"DS-Arena-Code","organization_count":1,"organizations":["llm_stats"],"rank":943,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-arena-code?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ds-fim-eval","caveat":"DeepSeek's internal Fill-in-the-Middle evaluation dataset for measuring code completion performance improvements in data science contexts","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ds-fim-eval?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"DS-FIM-Eval","organization_count":1,"organizations":["llm_stats"],"rank":944,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ds-fim-eval?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-10366","caveat":"DSAgentBench evaluates language agents on end-to-end data-science workflows in real computer environments. It comprises 275 tasks spanning data wrangling, exploration, modeling, visualization, and validation, with deterministic evaluators verifying analytical correctness, visual outputs, and model performance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10366"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DSAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":945,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10366"},{"aliases":[],"benchmark_id":"llm-stats-dsbench-fullstack","caveat":"DSBench-FullStack is DeepSeek's internal full-stack development test set for evaluating coding agents on end-to-end software engineering tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dsbench-fullstack?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"DSBench-FullStack","organization_count":1,"organizations":["llm_stats"],"rank":946,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-fullstack?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-dsbench-hard","caveat":"DSBench-Hard is DeepSeek's internal test set of difficult coding-agent problems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dsbench-hard?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"DSBench-Hard","organization_count":1,"organizations":["llm_stats"],"rank":947,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/dsbench-hard?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-30248","caveat":"Evaluates execution efficiency of LLM-generated data science code across 1,000 tasks using correctness and efficiency scores from stress-testing harnesses.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30248"],"document_share":0.0002723311546840959,"domain":"Code","name":"DSEffi-Bench","organization_count":1,"organizations":["claire_radar"],"rank":948,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30248"},{"aliases":[],"benchmark_id":"claire-radar-2610-08102","caveat":"Conversational MLLM agents are increasingly expected to assist in professional workflows, from AI research and engineering design to product management and business operations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.08102"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DSV-Mem","organization_count":1,"organizations":["claire_radar"],"rank":949,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.08102"},{"aliases":[],"benchmark_id":"claire-radar-2610-09374","caveat":"Urban diagnosis integrates heterogeneous observations to identify urban problems, localize affected areas, and investigate contributing factors, informing evidence-based urban planning and management.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.09374"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DUDA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":950,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.09374"},{"aliases":[],"benchmark_id":"llm-stats-dude","caveat":"DUDE (Document Understanding Dataset and Evaluation) tests multi-page, multi-domain document understanding and reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dude?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"DUDE","organization_count":1,"organizations":["llm_stats"],"rank":951,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/dude?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-24662","caveat":"Evaluates LLM agent security under dual-control interaction where both agent and user influence shared environment state.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24662"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DUMA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":952,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24662"},{"aliases":[],"benchmark_id":"claire-radar-2608-26546","caveat":"Autonomous agents are increasingly adopted to complete complex, multi-tool workflows in real-world settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26546"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DuMateBench","organization_count":1,"organizations":["claire_radar"],"rank":953,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26546"},{"aliases":[],"benchmark_id":"claire-radar-2607-29577","caveat":"DungeonBench evaluates tactical reasoning in Dungeons & Dragons combat with two tracks: Encounter for single fights and Day for linked encounters with persistent resources, using a shared decision stream of complete tactical observations and legal options.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29577"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DungeonBench","organization_count":1,"organizations":["claire_radar"],"rank":954,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29577"},{"aliases":[],"benchmark_id":"claire-radar-2608-01456","caveat":"DunphyBench evaluates long-horizon embodied decision-making in housing environments, requiring agents to navigate and choose options aligned with multi-dimensional human preferences under partial observations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01456"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DunphyBench","organization_count":1,"organizations":["claire_radar"],"rank":955,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.01456"},{"aliases":[],"benchmark_id":"claire-radar-2606-11901","caveat":"DuoBench is a benchmarking framework for bimanual manipulation policies on the FR3 Duo platform, with eleven tasks across four coordination categories, simulated and partially real-world, plus human-teleoperated datasets. It proposes stage-based evaluation beyond binary success.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11901"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"DuoBench","organization_count":1,"organizations":["claire_radar"],"rank":956,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.11901"},{"aliases":[],"benchmark_id":"claire-radar-2608-29711","caveat":"Evaluates multimodal large language models on data videos that combine dynamic charts and structured narratives.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29711"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DVBench","organization_count":1,"organizations":["claire_radar"],"rank":957,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29711"},{"aliases":[],"benchmark_id":"claire-radar-2610-03020","caveat":"Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history evolves.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03020"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DyadMem","organization_count":1,"organizations":["claire_radar"],"rank":958,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03020"},{"aliases":[],"benchmark_id":"opencompass-1971-dycodeeval","caveat":"DyCodeEval introduces methods to generate dynamic evaluation dataset and metric. Using multi-agent cooperation to rewrite benchmarks at evaluation time, it generates semantically equivalent, diverse, and non-deterministic problems—reducing data contamination and enabling more trustworthy evaluation. DyCodeEval 提出了一种动态生成评测数据集和评测指标的方法。通过多智能体协作，在评测时对基准题目进行重写，生成语义等价、多样化且非确定性的问题，从而减少数据污染，实现更可信的评测。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DyCodeEval"],"document_share":0.0002723311546840959,"domain":"强推理","name":"DyCodeEval","organization_count":1,"organizations":["opencompass_hub"],"rank":959,"released":"2025-06-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DyCodeEval"},{"aliases":[],"benchmark_id":"claire-radar-2609-01059","caveat":"Evaluates vision-language models on ego-motion understanding using counterfactual 3D rendering to separate visual appearance from kinematic properties.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01059"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Dyn-3D","organization_count":1,"organizations":["claire_radar"],"rank":960,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01059"},{"aliases":[],"benchmark_id":"claire-radar-2607-22119","caveat":"A benchmark for robot manipulation in dynamic environments, including dynamic manipulation and bimanual coordination tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22119"],"document_share":0.0002723311546840959,"domain":"Agents","name":"DynaBench","organization_count":1,"organizations":["claire_radar"],"rank":961,"released":"2026-07-24","source":"claire_radar","url":"https://arxiv.org/abs/2607.22119"},{"aliases":[],"benchmark_id":"llm-stats-dynamath","caveat":"A multimodal mathematics and reasoning benchmark focused on dynamic visual problem solving.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/dynamath?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"DynaMath","organization_count":1,"organizations":["llm_stats"],"rank":962,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/dynamath?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1374-dynamath","caveat":"DynaMath is a dynamic visual math benchmark designed for in-depth assessment of VLMs. It includes 501 high-quality, multi-topic seed questions, each represented as a Python program enabling the automatic generation of a much larger set of concrete questions. DynaMath用于评估多模态大模型的数学能力，包括501个高质量、多主题的种子问题，每个问题都以Python程序表示，能够自动生成大量具体的多样化问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/DynaMath"],"document_share":0.0002723311546840959,"domain":"多模态","name":"DynaMath","organization_count":1,"organizations":["opencompass_hub"],"rank":963,"released":"2024-10-29","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/DynaMath"},{"aliases":[],"benchmark_id":"claire-radar-2606-22877","caveat":"DynamicMem evaluates long-horizon memory in LLM agents through a synthetic benchmark with 15 months of multi-app activity per user across 16 applications, including attributes, habits, and preferences that evolve and must be inferred from scattered evidence. Scoring occurs at quarterly checkpoints, tracking performance as history grows.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22877"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DynamicMem","organization_count":1,"organizations":["claire_radar"],"rank":964,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.22877"},{"aliases":[],"benchmark_id":"claire-radar-2609-39198","caveat":"While Vision-Language-Action (VLA) models excel in static tasks, they struggle in dynamic environments where objects are in motion (e.g., conveyor belt manipulation).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39198"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"DynBench","organization_count":1,"organizations":["claire_radar"],"rank":965,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39198"},{"aliases":[],"benchmark_id":"claire-radar-2609-33980","caveat":"Dynamic graph anomaly detection requires repeated decisions as graph structure and class prevalence drift, yet detector benchmarks usually score a fixed pipeline after current labels are known.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33980"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DynGraphAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":966,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33980"},{"aliases":["DynoFluxBench: Benchmarking Kinodynamic Space-Time Planners in Dynamic Environments"],"benchmark_id":"claire-radar-2609-18549","caveat":"A benchmark framework for comparing kinodynamic planners in known dynamic environments with unbounded arrival time, facilitating systematic evaluation of planning algorithms.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18549"],"document_share":0.0002723311546840959,"domain":"General AI","name":"DynoFluxBench","organization_count":1,"organizations":["claire_radar"],"rank":967,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18549"},{"aliases":[],"benchmark_id":"claire-radar-2607-23722","caveat":"E-Bench evaluates multi-step tool-use agents in synthetic state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. It requires agents to discover hidden information and compose multiple tool calls before changing state, with deterministic grading by database-state diffs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23722"],"document_share":0.0002723311546840959,"domain":"General AI","name":"E-Bench","organization_count":1,"organizations":["claire_radar"],"rank":968,"released":"2026-07-26","source":"claire_radar","url":"https://arxiv.org/abs/2607.23722"},{"aliases":[],"benchmark_id":"claire-radar-2608-30730","caveat":"Evaluates LLM agents over a deterministic 365-day simulated e-commerce operation, scoring asset growth, negotiation, fraud avoidance, efficiency, and learning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30730"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"E-Commerce Bench","organization_count":1,"organizations":["claire_radar"],"rank":969,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30730"},{"aliases":[],"benchmark_id":"opencompass-1080-e-eval","caveat":"E-EVAL is the first comprehensive evaluation benchmark specifically tailored for Chinese K-12 education. E-EVAL comprises 4,351 multiple-choice questions spanning primary, middle, and high school levels, covering a diverse array of subjects. E-EVAL 是首个专门针对中国 K-12 教育的综合评估基准。E-EVAL 包含 4,351 道选择题，涵盖小学、初中和高中各个年级，涉及多种学科。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/E-EVAL"],"document_share":0.0002723311546840959,"domain":"学科","name":"E-EVAL","organization_count":1,"organizations":["opencompass_hub"],"rank":970,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/E-EVAL"},{"aliases":[],"benchmark_id":"claire-radar-2609-14302","caveat":"Evaluates financial vision-language models on evidence-to-action reliability across 969 queries using deterministic evidence anchors and metrics for grounding, consistency, calibration, and coverage.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.14302"],"document_share":0.0002723311546840959,"domain":"Finance","name":"E2A-Bench","organization_count":1,"organizations":["claire_radar"],"rank":971,"released":"2026-09-13","source":"claire_radar","url":"https://arxiv.org/abs/2609.14302"},{"aliases":[],"benchmark_id":"claire-radar-2609-38335","caveat":"Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38335"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"E2E-SWE","organization_count":1,"organizations":["claire_radar"],"rank":972,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.38335"},{"aliases":[],"benchmark_id":"claire-radar-2609-26175","caveat":"An advanced evaluation benchmark for LLMs on AI compliance using a knowledge graph and legal experts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.26175"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"EADC","organization_count":1,"organizations":["claire_radar"],"rank":973,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.26175"},{"aliases":[],"benchmark_id":"claire-radar-2607-28788","caveat":"EarlyDx is a benchmark for open-ended early diagnosis from emergency department encounters in MIMIC-IV, using admission-time records and LLM-auditor-supervised free-text labels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28788"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"EarlyDx","organization_count":1,"organizations":["claire_radar"],"rank":974,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28788"},{"aliases":[],"benchmark_id":"claire-radar-2607-23813","caveat":"Earnings25 is a finance-domain benchmark for evaluating automatic speech recognition (ASR) on English-language earnings calls. It includes testset-full (498 hours of S&P 500 earnings calls from Q4 2025) and testset-segmented (46 hours of 290 segments from 2025 U.S. earnings calls), with aligned transcripts and metadata like speaker roles and industry labels, enabling speaker- and industry-aware evaluation beyond WER.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23813"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Earnings25","organization_count":1,"organizations":["claire_radar"],"rank":975,"released":"2026-07-26","source":"claire_radar","url":"https://arxiv.org/abs/2607.23813"},{"aliases":["EarthShift: a benchmark for measuring robustness to real-world distribution shifts in Earth observation"],"benchmark_id":"claire-radar-2605-29330","caveat":"A public testbed that measures how Earth observation models perform in- versus out-of-distribution across shifts in source, time, geography, and sensor, covering 11 tasks and 5 shift types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29330"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EarthShift","organization_count":1,"organizations":["claire_radar"],"rank":976,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29330"},{"aliases":[],"benchmark_id":"claire-radar-2608-23525","caveat":"Evaluates scientific agents on 405 reproducible tasks across 199 documented natural hazard events, scoring fine-grained answer units and task-specific rubrics.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23525v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EarthVerse","organization_count":1,"organizations":["claire_radar"],"rank":977,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23525v1"},{"aliases":["Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents"],"benchmark_id":"claire-radar-2608-25417","caveat":"Evaluates dexterous visual tool use through reference-guided painting, semantic annotation, handwriting, and path planning tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25417"],"document_share":0.0002723311546840959,"domain":"Agents","name":"EASEL","organization_count":1,"organizations":["claire_radar"],"rank":978,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25417"},{"aliases":[],"benchmark_id":"claire-radar-2606-18239","caveat":"EBench is a simulation benchmark for diagnosing generalist mobile manipulation policies. It comprises 26 manipulation tasks annotated along five capability dimensions (scene, atomic skill, horizon, precision, mobility) and four generalization dimensions (object, background, instruction, mixed), with strict train/test splits and held-out online evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18239"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"EBench","organization_count":1,"organizations":["claire_radar"],"rank":979,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18239"},{"aliases":[],"benchmark_id":"claire-radar-2607-26397","caveat":"EC-Reason-Bench is a training-free diagnostic protocol for LLM enzyme classification, isolating four reasoning levers: output structure, external knowledge, reasoning structure, and robustness, to measure why LLMs fail on EC number prediction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26397"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"EC-Reason-Bench","organization_count":1,"organizations":["claire_radar"],"rank":980,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.26397"},{"aliases":[],"benchmark_id":"claire-radar-2609-00103","caveat":"Evaluates vision-language model memory along efficiency, compression, and calibration axes beyond raw accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00103"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ECCBench","organization_count":1,"organizations":["claire_radar"],"rank":981,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2609.00103"},{"aliases":[],"benchmark_id":"claire-radar-2607-27404","caveat":"ECG-InterpBench evaluates interpretability of ECG foundation models using sparse autoencoders with matched capacity, measuring reconstruction fidelity, clinical feature accessibility, and cross-seed reproducibility across 450 cells.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27404"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ECG-InterpBench","organization_count":1,"organizations":["claire_radar"],"rank":982,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27404"},{"aliases":[],"benchmark_id":"claire-radar-2608-30893","caveat":"Evaluates contextual ECG knowledge via 10,904 True/False questions from ECG references, scored by zero-shot and fine-tuned accuracy on a held-out test set.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30893"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ECGQuest","organization_count":1,"organizations":["claire_radar"],"rank":983,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30893"},{"aliases":["Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale"],"benchmark_id":"claire-radar-2607-28074","caveat":"Echoverse releases stateful application environments for evaluating computer-use agents against application-grounded task outcomes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28074"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Echoverse","organization_count":1,"organizations":["claire_radar"],"rank":984,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28074"},{"aliases":[],"benchmark_id":"llm-stats-eclektic","caveat":"A multilingual closed-book question answering dataset that evaluates cross-lingual knowledge transfer in large language models across 12 languages, using knowledge-seeking questions based on Wikipedia articles that exist only in one language","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/eclektic?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ECLeKTic","organization_count":1,"organizations":["llm_stats"],"rank":985,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/eclektic?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-05519","caveat":"EcoAgent-Bench evaluates LLM agents' economic decision-making under budget constraints across 304 tasks in five families, with priced actions and metrics for micro accuracy and economic consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05519"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EcoAgent-Bench","organization_count":1,"organizations":["claire_radar"],"rank":986,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05519"},{"aliases":[],"benchmark_id":"claire-radar-2609-26375","caveat":"A benchmark covering 11 commercial editing tasks with task-specific visual evaluation and CTR-based ranking.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.26375"],"document_share":0.0002723311546840959,"domain":"Advertising & Pricing","name":"Ecom-Bench","organization_count":1,"organizations":["claire_radar"],"rank":987,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.26375"},{"aliases":[],"benchmark_id":"claire-radar-2606-17698","caveat":"EComAgentBench evaluates LLM-based shopping agents on 662 long-horizon product selection tasks built from real Amazon data. Each task requires uncovering hidden requirements spread across an explicit query, tool-gated profile, and scripted clarification, verifying candidates, and committing to one product within 100 tool calls. Scoring uses typed, source-tagged rubrics for each requirement.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17698"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EComAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":988,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.17698"},{"aliases":["ESAC"],"benchmark_id":"claire-radar-github-ecowestern-esac","caveat":"Compact benchmark program with ESAC-GI released (42 items, 75 points) measuring general intelligence across logic, math, factual recall, reading comprehension, abstraction, instruction-following, writing quality, and response-depth calibration; ESAC-AG partially implemented.","document_count":1,"document_ids":["claire_radar:https://github.com/EcoWestern/ESAC"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"EcoWestern Short and Cheap Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":989,"released":"2026-09-22","source":"claire_radar","url":"https://github.com/EcoWestern/ESAC"},{"aliases":["How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards"],"benchmark_id":"claire-radar-2609-18908","caveat":"Predicting non-pecuniary damage awards at the European Court of Human Rights from case information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18908"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"ECtHR-NPD","organization_count":1,"organizations":["claire_radar"],"rank":990,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18908"},{"aliases":["edge-model-compression-benchmark"],"benchmark_id":"claire-radar-github-suresh4330-edge-model-compression-benchmark","caveat":"EDGE-BENCH evaluates four ResNet-18 compression approaches—baseline, INT8 quantization, pruning, and knowledge distillation—on CIFAR-100 via accuracy, size, parameters, and CPU latency.","document_count":1,"document_ids":["claire_radar:https://github.com/suresh4330/edge-model-compression-benchmark"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"EDGE-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":991,"released":"2026-09-09","source":"claire_radar","url":"https://github.com/suresh4330/edge-model-compression-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2607-05155","caveat":"EdgeBench evaluates autonomous agents on 134 real-world tasks across scientific discovery, software engineering, optimization, knowledge work, formal mathematics, and games. Each task requires 12+ hours of continuous interaction with multi-level feedback. Scoring tracks agent performance over time (at 2,4,6,8,10,12 hours). A public leaderboard is maintained; 51 tasks and evaluation framework are open-sourced.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05155"],"document_share":0.0002723311546840959,"domain":"Agents","name":"EdgeBench","organization_count":1,"organizations":["claire_radar"],"rank":992,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.05155"},{"aliases":[],"benchmark_id":"claire-radar-github-michailfragkiskos-edgemosaic","caveat":"Defines a manifest-driven on-device AI evaluation lab that aggregates latency, memory, and thermal metrics across model, delegate, and configuration combinations.","document_count":1,"document_ids":["claire_radar:https://github.com/michailfragkiskos/EdgeMosaic"],"document_share":0.0002723311546840959,"domain":"Finance","name":"EdgeMosaic","organization_count":1,"organizations":["claire_radar"],"rank":993,"released":"2026-09-04","source":"claire_radar","url":"https://github.com/michailfragkiskos/EdgeMosaic"},{"aliases":[],"benchmark_id":"claire-radar-2608-13441","caveat":"Edit2TikZ evaluates instruction-guided scientific figure editing with TikZ code, featuring 1,548 samples with textual or visual localization requests and multi-step edits, using a human-aligned evaluation framework to measure edit completion and content preservation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13441"],"document_share":0.0002723311546840959,"domain":"Software & AI Compute","name":"Edit2TikZ","organization_count":1,"organizations":["claire_radar"],"rank":994,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13441"},{"aliases":[],"benchmark_id":"claire-radar-2607-22705","caveat":"EditCLEVR is a paired-scene intervention benchmark for object-centric representations, with before/after CLEVR renders and a known attribute change. Includes metrics like SGIA and Delta-SGIA for semantic faithfulness, with probe-free diagnostics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22705"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EditCLEVR","organization_count":1,"organizations":["claire_radar"],"rank":995,"released":"2026-07-19","source":"claire_radar","url":"https://arxiv.org/abs/2607.22705"},{"aliases":[],"benchmark_id":"opencompass-1970-editinspector","caveat":"EditInspector is a novel benchmark for evaluation of text-guided image edits, based on human annotations collected using an extensive template for edit verification. We leverage EditInspector to evaluate the performance of state-of-the-art (SoTA) vision and language models. EditInspector 评估最先进（SoTA）视觉和语言模型在多个维度上评估编辑的性能，包括准确性、瑕疵检测、视觉质量、与图像场景的无缝融合、遵循常识以及描述编辑引起变化的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EditInspector"],"document_share":0.0002723311546840959,"domain":"创作","name":"EditInspector","organization_count":1,"organizations":["opencompass_hub"],"rank":996,"released":"2025-06-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/EditInspector"},{"aliases":[],"benchmark_id":"claire-radar-2607-02007","caveat":"EduArt evaluates art-historical knowledge and visual reasoning in multimodal LLMs using 871 human-authored questions in Italian and English, covering multiple formats and languages. Scoring is based on accuracy and psychometric properties.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02007"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"EduArt","organization_count":1,"organizations":["claire_radar"],"rank":997,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02007"},{"aliases":[],"benchmark_id":"claire-radar-2608-03206","caveat":"EduClaw-Bench evaluates pedagogical LLM agents in a simulated 30-day tutoring relationship with a knowledge-tracing-based simulated learner, scoring learning gain, responsiveness, helpfulness, and curriculum design across 55 scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03206"],"document_share":0.0002723311546840959,"domain":"Agents","name":"EduClaw-Bench","organization_count":1,"organizations":["claire_radar"],"rank":998,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03206"},{"aliases":[],"benchmark_id":"claire-radar-2608-00739","caveat":"EduPluginBench is an executable benchmark for evaluating code-generation models on producing plugins that meet governed ecosystem requirements including least privilege, telemetry consent, provenance, and bounded failure. It uses 1,440 mutants and 120 clean references with staged admission levels P0-P4.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00739"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"EduPluginBench","organization_count":1,"organizations":["claire_radar"],"rank":999,"released":"2026-08-01","source":"claire_radar","url":"https://arxiv.org/abs/2608.00739"},{"aliases":[],"benchmark_id":"claire-radar-2605-26918","caveat":"Benchmark for educational validity of video generation models, grounded in the Knowledge-Skills-Attitude framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26918"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"EduVideoBench","organization_count":1,"organizations":["claire_radar"],"rank":1000,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26918"},{"aliases":[],"benchmark_id":"claire-radar-2607-27857","caveat":"EEG-EditBench evaluates EEG-to-image retrieval models using 2,137 controlled edits of 200 THINGS-EEG2 test images, covering object identity, attributes, background, and presence, with metrics like 200-way accuracy and 2AFC accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27857"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EEG-EditBench","organization_count":1,"organizations":["claire_radar"],"rank":1001,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27857"},{"aliases":[],"benchmark_id":"claire-radar-2606-09547","caveat":"Ego-MC-Bench is a benchmark for evaluating reactive step-by-step task guidance in cooking scenarios, focusing on timely interventions when mistakes occur.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09547"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Ego-MC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1002,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09547"},{"aliases":[],"benchmark_id":"claire-radar-2606-02246","caveat":"Ego-METAS evaluates online temporal action segmentation in egocentric video, where models must select sensor modalities (RGB, audio, gaze, IMU, monochrome) per timestep to maximize accuracy under hardware-representative energy budgets. It includes 100+ hours of untrimmed video from multiple datasets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02246"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"Ego-METAS","organization_count":1,"organizations":["claire_radar"],"rank":1003,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.02246"},{"aliases":[],"benchmark_id":"claire-radar-2610-01092","caveat":"Video generation models are increasingly being explored as world simulators for embodied planning and learning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.01092"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"Ego2Act","organization_count":1,"organizations":["claire_radar"],"rank":1004,"released":"2026-10-01","source":"claire_radar","url":"https://arxiv.org/abs/2610.01092"},{"aliases":[],"benchmark_id":"claire-radar-2610-02715","caveat":"Egocentric videos capture how people carry out everyday activities, yet testing an agent requires evaluating the consequences of actions it chooses itself.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.02715"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Ego2World","organization_count":1,"organizations":["claire_radar"],"rank":1005,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.02715"},{"aliases":[],"benchmark_id":"claire-radar-2608-04533","caveat":"EgoAfford is a benchmark for egocentric referring segmentation with task-oriented affordance grounding, comprising 15.5k images and 102 real images. It includes EgoLens, a 3B MLLM reference model.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04533"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EgoAfford","organization_count":1,"organizations":["claire_radar"],"rank":1006,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04533"},{"aliases":[],"benchmark_id":"claire-radar-2605-27820","caveat":"EgoBench is an interactive multimodal benchmark for tool-using agents. It comprises 1,045 egocentric-video-grounded tasks across four daily scenarios, with a user-agent-tool interactive environment. It assesses multimodal perception, tool invocation with multi-hop reasoning, and dynamic user interaction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27820"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"EgoBench","organization_count":1,"organizations":["claire_radar"],"rank":1007,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27820"},{"aliases":[],"benchmark_id":"claire-radar-2610-04506","caveat":"Vision-language models (VLMs) are increasingly evaluated for egocentric and cross-view video reasoning, yet existing benchmarks largely focus on semantic event understanding, temporal relations, or correspondence between already observed v…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.04506"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EgoExo-Next","organization_count":1,"organizations":["claire_radar"],"rank":1008,"released":"2026-10-03","source":"claire_radar","url":"https://arxiv.org/abs/2610.04506"},{"aliases":[],"benchmark_id":"claire-radar-2607-00547","caveat":"EgoGapBench evaluates egocentric action selection in multi-agent scenes, isolating the ability to choose actions from the agent's perspective when other agents are present. The benchmark includes training and test splits with human performance as reference.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00547"],"document_share":0.0002723311546840959,"domain":"Agents","name":"EgoGapBench","organization_count":1,"organizations":["claire_radar"],"rank":1009,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.00547"},{"aliases":["Beyond Visual Quality: Evaluating Physical Consistency under Ego-Motion with EgoGenEval"],"benchmark_id":"claire-radar-2609-11172","caveat":"EgoGenEval offers 1,400 cases and 2,360 target views to assess physical consistency of visual generators under ego-motion, measuring Camera Motion Grounding and Scene State Preservation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11172"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"EgoGenEval","organization_count":1,"organizations":["claire_radar"],"rank":1010,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11172"},{"aliases":["EgoInertia-MI: A Multimodal Egocentric Vision and IMU Benchmark for Motor Impairment Assessment"],"benchmark_id":"claire-radar-2607-03934","caveat":"A multimodal benchmark of synchronized egocentric video and wearable IMU signals covering 19 activities, with action recognition and motor impairment severity estimation tasks scored by Macro-F1.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03934"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"EgoInertia-MI","organization_count":1,"organizations":["claire_radar"],"rank":1011,"released":"2026-07-04","source":"claire_radar","url":"https://arxiv.org/abs/2607.03934"},{"aliases":[],"benchmark_id":"claire-radar-2609-12350","caveat":"EgoMaize evaluates first-person maize instance segmentation, requiring ownership-consistent plant masks and plant-owned stem and tassel cues from close-range field images with severe same-class overlap.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12350"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EgoMaize","organization_count":1,"organizations":["claire_radar"],"rank":1012,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12350"},{"aliases":[],"benchmark_id":"claire-radar-2608-13113","caveat":"EgoMonth benchmarks month-level egocentric video understanding with 300+ hours from 20 participants over 20-120 days, 1,443 QA pairs, and a 14-task framework across schema consolidation, episodic indexing, and cascading reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13113"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EgoMonth","organization_count":1,"organizations":["claire_radar"],"rank":1013,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13113"},{"aliases":[],"benchmark_id":"opencompass-1604-egonormia","caveat":"EgoNormia is a challenging QA benchmark that tests VLMs' ability to reason over norms in context. The datset consists of 1,853 physically grounded egocentric interaction clips from Ego4D and corresponding five-way multiple-choice questions tasks for each. EgoNormia 是一个具有挑战性的问答基准，用于测试 VLMs 在上下文中推理规范的能力。该数据集包含来自 Ego4D 的 1,853 个物理基础化的以自我为中心的交互剪辑，以及每个剪辑对应的五选一多项选择题任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EgoNormia"],"document_share":0.0002723311546840959,"domain":"学科","name":"EgoNormia","organization_count":1,"organizations":["opencompass_hub"],"rank":1014,"released":"2025-02-27","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/EgoNormia"},{"aliases":[],"benchmark_id":"claire-radar-2605-24456","caveat":"EgoProx evaluates multimodal large language models on egocentric 3D proximity reasoning, with tasks organized along a cognitive hierarchy covering intention, exploration, exploitation, and chain-of-actions reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24456"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EgoProx","organization_count":1,"organizations":["claire_radar"],"rank":1015,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24456"},{"aliases":[],"benchmark_id":"claire-radar-2607-26518","caveat":"EgoSafe-Bench evaluates visual safety understanding in first-person video, using 12,000 QA samples from 3,000 clips under the Hierarchical Reasoning Evaluation (HRE) protocol, which requires reasoning from feature anchoring to intent inference.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26518"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EgoSafe-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1016,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.26518"},{"aliases":[],"benchmark_id":"claire-radar-2607-00218","caveat":"EgoSafetyBench is a diagnostic egocentric video benchmark of 1,200 robot-view scenarios to evaluate vision-language models as runtime safety guards. It assesses situational awareness across routine, suspicious, obvious, and contextual hazards, and visual-channel robustness against misleading in-scene text.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00218"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"EgoSafetyBench","organization_count":1,"organizations":["claire_radar"],"rank":1017,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.00218"},{"aliases":[],"benchmark_id":"claire-radar-2606-24422","caveat":"EgoSAT evaluates vision-language models on egocentric video reasoning in streaming settings. It contains 1,997 videos (165 hours) and about 4,800 QA pairs covering retrospective, online, and prospective reasoning tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24422"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EgoSAT","organization_count":1,"organizations":["claire_radar"],"rank":1018,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24422"},{"aliases":[],"benchmark_id":"llm-stats-egoschema","caveat":"A diagnostic benchmark for very long-form video language understanding consisting of over 5000 human curated multiple choice questions based on 3-minute video clips from Ego4D, covering a broad range of natural human activities and behaviors","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/egoschema?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"EgoSchema","organization_count":1,"organizations":["llm_stats"],"rank":1019,"released":"2023-08-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/egoschema?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-31557","caveat":"Egostream evaluates streaming episodic memory in egocentric vision through 2,250 curated questions across seven cognitive dimensions, expanded into 8,528 recall-conditioned evaluations using the Answer Validity Window to test recall from instant to ultra-long-term.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31557"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EGOSTREAM","organization_count":1,"organizations":["claire_radar"],"rank":1020,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31557"},{"aliases":[],"benchmark_id":"claire-radar-2606-09243","caveat":"EgoTactile is a dataset pairing egocentric video with full-hand pressure supervision for everyday objects, including a bare-hand transfer subset.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09243"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"EgoTactile","organization_count":1,"organizations":["claire_radar"],"rank":1021,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09243"},{"aliases":[],"benchmark_id":"claire-radar-2606-23301","caveat":"EHR-Complex evaluates clinical agent performance on interactive reasoning over MIMIC-IV electronic health records. It consists of about 52K tasks across six clinical intents, requiring agents to execute SQL or Python in a sandboxed environment to answer patient- and population-level queries. Scoring is based on exact-match accuracy against expected outcomes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23301"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"EHR-Complex","organization_count":1,"organizations":["claire_radar"],"rank":1022,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23301"},{"aliases":["ELICITED: EHR-grounded Longitudinal Interactive Conversations for Information-seeking Triage Evaluation and Decision-making"],"benchmark_id":"claire-radar-2608-09024","caveat":"EHR2Dial-Triage evaluates triage conversations grounded in EHR data across information elicitation, evidence use, five-level Emergency Severity Index prediction, and patient-facing communication.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09024"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"EHR2Dial-Triage","organization_count":1,"organizations":["claire_radar"],"rank":1023,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09024"},{"aliases":[],"benchmark_id":"claire-radar-2605-30637","caveat":"EHRBench evaluates LLM-based clinical decision-making using nearly 1M QA items generated from real EHR trajectories, covering diagnosis, treatment, and prognosis tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30637"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"EHRBench","organization_count":1,"organizations":["claire_radar"],"rank":1024,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30637"},{"aliases":[],"benchmark_id":"claire-radar-2606-15735","caveat":"EHRNote-ChatQA is a benchmark for evidence-grounded multi-turn clinical QA over longitudinal discharge summaries. Built from MIMIC-IV, it includes 967 patient-level samples and 16,072 expert-verified QA pairs across eight clinical categories, with evidence-grounding QA pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15735"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"EHRNote-ChatQA","organization_count":1,"organizations":["claire_radar"],"rank":1025,"released":"2026-06-14","source":"claire_radar","url":"https://arxiv.org/abs/2606.15735"},{"aliases":[],"benchmark_id":"opencompass-1327-ehrnoteqa","caveat":"EHRNoteQA evaluates LLM's ability to assist clinical decision-making based on electronic health records, comprising 962 different QA pairs each linked to distinct patients' discharge summaries. EHRNoteQA用于评估LLM基于电子健康记录辅助临床决策的能力，由962个问答对组成，每个问答对都与不同患者的出院总结相关联。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EHRNoteQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"EHRNoteQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1026,"released":"2024-02-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/EHRNoteQA"},{"aliases":[],"benchmark_id":"claire-radar-2606-15532","caveat":"EIBench is a simulator-based benchmark for interactive emotion management, containing 2,222 scenarios across a 2x2 taxonomy of Support, Defense, Repair, and Charm. It evaluates LLM agents on multi-turn dialogue where a user simulator updates emotion-relation states and provides anchor-based scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15532"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EIBench","organization_count":1,"organizations":["claire_radar"],"rank":1027,"released":"2026-06-14","source":"claire_radar","url":"https://arxiv.org/abs/2606.15532"},{"aliases":[],"benchmark_id":"claire-radar-2608-09548","caveat":"ELBench is a benchmark for education-facing large language models, evaluating General Capability, Safety and Trustworthiness, Basic Education, and High-Level Cultivation under a common protocol with 2,939 items.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09548"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ELBench","organization_count":1,"organizations":["claire_radar"],"rank":1028,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09548"},{"aliases":[],"benchmark_id":"opencompass-2571-elbench","caveat":"ELBench is a multidimensional benchmark for education-facing large language models. It covers safety and trustworthiness, high-level educational cultivation, and  general capabilities, combining objective evaluation with LLM-as-a-Judge. ELBench 是面向教育场景大语言模型的多维评测集，覆盖安全可信、高阶育人和通用能力。公开任务包含安全回答、教育开放任务、选择题、数学与推理任务，并结合客观规则和大模型判决进行评测。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ELBench"],"document_share":0.0002723311546840959,"domain":"推理","name":"ELBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1029,"released":"2026-06-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ELBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-04850","caveat":"ElderBench is a benchmark for evaluating mobile GUI agents on tasks elicited from older adults, focusing on natural language patterns like indirect speech and ambiguity. It assesses agent performance on 249 tasks across 20 apps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04850"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"ElderBench","organization_count":1,"organizations":["claire_radar"],"rank":1030,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.04850"},{"aliases":["election-bias-benchmark"],"benchmark_id":"claire-radar-github-kilojoules-election-bias-benchmark","caveat":"Investigates whether a candidate's AGI-regulation stance influences LLM outputs on unrelated property tax positions, using factorial candidate set and two readouts (blind rubric judge and logit forced-choice).","document_count":1,"document_ids":["claire_radar:https://github.com/kilojoules/election-bias-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Election: issue-contingent bias benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1031,"released":"2026-09-26","source":"claire_radar","url":"https://github.com/kilojoules/election-bias-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2608-28478","caveat":"Closed-book knowledge probe with 1,094 source-traceable questions that measure whether language models recall multiple verified accounts of long-tail facts, scored via complete, partial, or failed recall rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.28478"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"ElephantBench","organization_count":1,"organizations":["claire_radar"],"rank":1032,"released":"2026-08-28","source":"claire_radar","url":"https://arxiv.org/abs/2608.28478"},{"aliases":[],"benchmark_id":"claire-radar-2610-05013","caveat":"Lifelong physical agents must reason over extended interactions where past events continue to shape the world long after they disappear from view.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05013"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"EMBER-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1033,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05013"},{"aliases":["Embodied-Bench"],"benchmark_id":"claire-radar-2609-13082","caveat":"Embodied-BenchForge is an agentic workflow that transforms user-specified evaluation intents into embodied benchmark artifacts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.13082"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"Embodied-BenchForge","organization_count":1,"organizations":["claire_radar"],"rank":1034,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.13082"},{"aliases":[],"benchmark_id":"claire-radar-2605-29074","caveat":"Embodied3DBench is a robot-centric benchmark for low-level spatial intelligence in embodied 3D environments. It includes 6 task categories: Grounding, Spatial Relation Prediction, Multi-view Correspondence, Affordance Prediction, Grasp Point Prediction, and Trajectory Prediction, with 21k QA pairs across 12 subcategories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29074"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"Embodied3DBench","organization_count":1,"organizations":["claire_radar"],"rank":1035,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.29074"},{"aliases":[],"benchmark_id":"opencompass-1243-embodiedagentinterface","caveat":"Embodied Agent Interface supports the formalization of various types of tasks and input-output specifications of LLM-based modules, offering a comprehensive assessment of LLMs' performance for different subtasks and pinpointing the strengths and weaknesses in LLM-powered embodied AI systems. Embodied Agent Interface支持各种类型的任务和基于LLM模块的输入输出规范的形式化，对LLM在不同子任务中的性能进行了全面评估，指出了基于LLM的具身AI系统的优势和劣势。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EmbodiedAgentInterface"],"document_share":0.0002723311546840959,"domain":"推理","name":"EmbodiedAgentInterface","organization_count":1,"organizations":["opencompass_hub"],"rank":1036,"released":"2024-06-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/EmbodiedAgentInterface"},{"aliases":[],"benchmark_id":"opencompass-1564-embodiedbench","caveat":"EmbodiedBench is a comprehensive benchmark designed to evaluate Multi-modal Large Language Models (MLLMs) as embodied agents. EmbodiedBench，这是一个旨在评估多模态大型语言模型作为具身智能体的全面基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EmbodiedBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"EmbodiedBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1037,"released":"2025-02-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/EmbodiedBench"},{"aliases":[],"benchmark_id":"claire-radar-2610-13021","caveat":"Continual learning requires models to accumulate past experience and transfer it to new tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13021"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"EmbodiedCLBench","organization_count":1,"organizations":["claire_radar"],"rank":1038,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13021"},{"aliases":[],"benchmark_id":"claire-radar-2609-28236","caveat":"Benchmark for embodied memory in long-horizon embodied tasks with 2,554 interactive episodes across four task families.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.28236"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"EmbodiedMemory-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1039,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.28236"},{"aliases":[],"benchmark_id":"claire-radar-2609-27308","caveat":"Simulation benchmark for coding agents spanning contact-rich manipulation, deformable objects, and long-horizon tasks up to half an hour.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27308"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"EMBODIEDSWE-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":1040,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27308"},{"aliases":[],"benchmark_id":"claire-radar-2610-03248","caveat":"Panoramic images provide a complete 360-degree field of view, enabling comprehensive scene understanding for embodied perception.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03248"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"EmbPASS","organization_count":1,"organizations":["claire_radar"],"rank":1041,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03248"},{"aliases":[],"benchmark_id":"llm-stats-embspatialbench","caveat":"EmbSpatialBench evaluates embodied spatial understanding and reasoning capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/embspatialbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"spatial_reasoning","name":"EmbSpatialBench","organization_count":1,"organizations":["llm_stats"],"rank":1042,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/embspatialbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-03374","caveat":"eMEM-Bench v1 evaluates embodied memory systems through 988 probes across eight cognitive-psychology paradigms (DRM lures, pattern separation, pattern completion, source monitoring, context-dependent retrieval, long-horizon interference, serial position, foil retention curve) in ProcTHOR-10K scenes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03374"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"eMEM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1043,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03374"},{"aliases":[],"benchmark_id":"claire-radar-2610-06450","caveat":"Foundation models (FMs) are increasingly being developed for general time series and physiological signals, yet their transferability to downstream physiological tasks remains poorly understood.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.06450"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EMG-FM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1044,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.06450"},{"aliases":["greencalculus-benchmark"],"benchmark_id":"claire-radar-github-greencalculus-greencalculus-benchmark","caveat":"Benchmark of 467 emission-factor questions scored for numeric accuracy and citation correctness across five models without tools.","document_count":1,"document_ids":["claire_radar:https://github.com/greencalculus/greencalculus-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Emission-factor accuracy benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1045,"released":"2026-09-10","source":"claire_radar","url":"https://github.com/greencalculus/greencalculus-benchmark"},{"aliases":[],"benchmark_id":"llm-stats-emma","caveat":"EMMA (Enhanced MultiModal reAsoning) is a benchmark for organic multimodal reasoning across mathematics, physics, chemistry, and coding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/emma?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"EMMA","organization_count":1,"organizations":["llm_stats"],"rank":1046,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/emma?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1640-emma","caveat":"EMMA is composed of 2,788 problems, of which 1,796 are newly constructed, across four domains. Within each subject, we further provide fine-grained labels for each question based on the specific skills it measures. EMMA 由 2,788 个问题组成，其中 1,796 个是新构建的，涵盖四个领域。在每个主题中，我们根据所测量的具体技能为每个问题提供细粒度标签。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EMMA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"EMMA","organization_count":1,"organizations":["opencompass_hub"],"rank":1047,"released":"2025-01-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/EMMA"},{"aliases":[],"benchmark_id":"claire-radar-2608-09189","caveat":"EmoSBench evaluates emotional intelligence in spoken language models across four theoretical branches: perceiving, understanding, using, and managing emotion, with ten subtasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09189"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EmoSBench","organization_count":1,"organizations":["claire_radar"],"rank":1048,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09189"},{"aliases":[],"benchmark_id":"claire-radar-2610-15452","caveat":"EmoTrace-Bench evaluates text-to-video generation on emotional expression by controlling prompt cues and intensity, and scoring videos on target emotion, perceived intensity, and support across ten audiovisual cue channels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.15452"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EmoTrace-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1049,"released":"2026-10-11","source":"claire_radar","url":"https://arxiv.org/abs/2610.15452"},{"aliases":[],"benchmark_id":"claire-radar-2606-30256","caveat":"EMPATH evaluates safety of emotional-support chatbots via auditor-generated multi-turn conversations scored on 19 metrics across crisis handling, therapeutic quality, conversational integrity, emotional safety, and cultural adaptation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30256"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EMPATH","organization_count":1,"organizations":["claire_radar"],"rank":1050,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30256"},{"aliases":["On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos"],"benchmark_id":"claire-radar-2608-14262","caveat":"Evaluates temporal vision-language models on surgical endoscopy video understanding under six realistic corruptions, using public videos and standardized prompts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14262"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Endo-C6","organization_count":1,"organizations":["claire_radar"],"rank":1051,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14262"},{"aliases":["EnergyBridge: Benchmarking Household Energy Management, User Participation, and Grid Flexibility"],"benchmark_id":"claire-radar-2608-08691","caveat":"EnergyBridge evaluates household energy management by connecting capacity reporting, event-specific household authorization simulation, and physical execution in EnergyPlus environments for Tianjin and Berlin, scoring authorization, energy use, and capacity commitment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08691"],"document_share":0.0002723311546840959,"domain":"Agents","name":"EnergyBridge","organization_count":1,"organizations":["claire_radar"],"rank":1052,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08691"},{"aliases":["EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning"],"benchmark_id":"claire-radar-2609-33261","caveat":"Evaluates multimodal engineering state understanding and design intervention reasoning through 3,229 questions across seven domains, spanning state grounding, relational reasoning, diagnosis, and intervention tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33261"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EngIntervene","organization_count":1,"organizations":["claire_radar"],"rank":1053,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33261"},{"aliases":["Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams"],"benchmark_id":"claire-radar-2606-03410","caveat":"Enginuity evaluates vision-language models on structured parts-table extraction and visual question answering over complex engineering diagrams.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03410"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Enginuity","organization_count":1,"organizations":["claire_radar"],"rank":1054,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03410"},{"aliases":[],"benchmark_id":"claire-radar-2609-37686","caveat":"Autonomous agents have made rapid progress in general-purpose computer use, but reliable automation of professional industrial engineering remains out of reach, as engineering workflows demand reasoning over geometric and physical constrai…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37686"],"document_share":0.0002723311546840959,"domain":"CAD","name":"EngiWorld","organization_count":1,"organizations":["claire_radar"],"rank":1055,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37686"},{"aliases":[],"benchmark_id":"claire-radar-2609-39284","caveat":"While large language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained reasoning, complex state management, and continuous cross-domain abstraction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39284"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"EngramBench","organization_count":1,"organizations":["claire_radar"],"rank":1056,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39284"},{"aliases":[],"benchmark_id":"claire-radar-2605-27240","caveat":"Benchmark for emotional need-aware proactive memory retrieval in support agents, with 1,800+ dialogues mapped to Maslow's hierarchy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27240"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ENPMR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1057,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27240"},{"aliases":[],"benchmark_id":"claire-radar-2609-37658","caveat":"LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-term trade-offs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37658"],"document_share":0.0002723311546840959,"domain":"Finance","name":"EnterpriseBench","organization_count":1,"organizations":["claire_radar"],"rank":1058,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37658"},{"aliases":[],"benchmark_id":"claire-radar-2606-23654","caveat":"Evaluates coding agents on realistic enterprise workflows using 852 reproducible tasks with recovered fixtures, prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Scoring covers artifact delivery, visual quality, cost, runtime, and skill transfer.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23654"],"document_share":0.0002723311546840959,"domain":"Agents","name":"EnterpriseClawBench","organization_count":1,"organizations":["claire_radar"],"rank":1059,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23654"},{"aliases":[],"benchmark_id":"claire-radar-2605-26394","caveat":"EnterpriseMem-Bench is a multi-turn Text-to-SQL benchmark with 300 sessions and 1,400 turns across three enterprise domains, featuring deterministic ground truth and per-turn memory-critical annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26394"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"EnterpriseMem-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1060,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26394"},{"aliases":[],"benchmark_id":"artificial-analysis-enterpriseops-gym-aa","caveat":"Agentic business operations","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/enterprise-ops-gym-aa"],"document_share":0.0002723311546840959,"domain":"agentic","name":"EnterpriseOps-Gym-AA","organization_count":1,"organizations":["artificial_analysis"],"rank":1061,"released":null,"source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/enterprise-ops-gym-aa"},{"aliases":[],"benchmark_id":"claire-radar-2608-11584","caveat":"The benchmark evaluates LLM instruction adherence and robustness in enterprise retrieval scenarios, using 983 expert-validated samples across six domains, simulating retrieval noise, knowledge gaps, and factual conflicts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11584"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"EnterpriseRAG","organization_count":1,"organizations":["claire_radar"],"rank":1062,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.11584"},{"aliases":["ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering"],"benchmark_id":"claire-radar-2608-10679","caveat":"Evaluates enterprise question answering over 2,341 documents and 907 questions spanning explicit lookup, cross-source composition, and latent organizational reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10679"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ENTLORE","organization_count":1,"organizations":["claire_radar"],"rank":1063,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10679"},{"aliases":[],"benchmark_id":"claire-radar-2606-03363","caveat":"EntSQL evaluates text-to-SQL systems on enterprise knowledge grounding, with 1,066 Chinese-English examples across five business domains requiring private business knowledge. Systems generate SQL from questions and schema, with some provided long-form documents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03363"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"EntSQL","organization_count":1,"organizations":["claire_radar"],"rank":1064,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03363"},{"aliases":[],"benchmark_id":"claire-radar-2606-15240","caveat":"EnvShip is a unified multi-region framework for context-aware and cross-region vessel trajectory forecasting, with standardized tracks and evaluation protocols using public AIS data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15240"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EnvShip","organization_count":1,"organizations":["claire_radar"],"rank":1065,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2606.15240"},{"aliases":["Benchmarking_MLIPs"],"benchmark_id":"claire-radar-github-soham0605-benchmarking-mlips","caveat":"Evaluates MLIP equation-of-state predictions against DFT for volume-scaled structures, reporting percent errors in equilibrium volume and bulk modulus.","document_count":1,"document_ids":["claire_radar:https://github.com/Soham0605/Benchmarking_MLIPs"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"eos-bench","organization_count":1,"organizations":["claire_radar"],"rank":1066,"released":"2026-09-17","source":"claire_radar","url":"https://github.com/Soham0605/Benchmarking_MLIPs"},{"aliases":[],"benchmark_id":"claire-radar-2606-13602","caveat":"EpiBench evaluates AI agents on short-horizon epigenomics analysis tasks across CUT&Tag/CUT&RUN, ATAC-seq, ChIP-seq, and DNA methylation workflows, with deterministically gradable answers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13602"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EpiBench","organization_count":1,"organizations":["claire_radar"],"rank":1067,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13602"},{"aliases":[],"benchmark_id":"claire-radar-2608-06022","caveat":"EpiBench evaluates LLMs' epitope reasoning from antibody and antigen sequences across five tasks: targetable region discovery, epitope identification, binning, functional assessment, and escape assessment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06022"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"EpiBench","organization_count":1,"organizations":["claire_radar"],"rank":1068,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06022"},{"aliases":[],"benchmark_id":"opencompass-522-eprstmt","caveat":"(E-commerce Product Review Dataset for Sentiment Analysis), also known as EPRSTMT, is a binary sentiment analysis dataset based on product reviews on e-commerce platform. Each sample is labelled as Positive or Negative. It collect by ICIP Lab of Beijing Normal University. EPRSTMT，也称作电子商务产品评论情感分析数据集，是一个基于电子商务平台上的产品评论的二元情感分析数据集。每个样本都被标记为积极或消极。该数据集由北京师范大学 ICIP 实验室收集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EPRSTMT"],"document_share":0.0002723311546840959,"domain":"理解","name":"EPRSTMT","organization_count":1,"organizations":["opencompass_hub"],"rank":1069,"released":"2021-07-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/EPRSTMT"},{"aliases":[],"benchmark_id":"llm-stats-eq-bench","caveat":"EQ-Bench is an LLM-judged test evaluating active emotional intelligence abilities, understanding, insight, empathy, and interpersonal skills. The test set contains 45 challenging roleplay scenarios, most of which constitute pre-written prompts spanning 3 turns. The benchmark evaluates the performance of models by validating responses against several criteria and conducts pairwise comparisons to report a normalized Elo computation for each model.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/eq-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"EQ-Bench","organization_count":1,"organizations":["llm_stats"],"rank":1070,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/eq-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1892-er-reason","caveat":"ER-Reason is a large-scale benchmark suite for evaluating the clinical reasoning capabilities of large language models (LLMs) in the emergency room (ER) — a high-stakes environment where clinicians make rapid, life-critical decisions. ER-Reason 是一个大规模基准套件，用于评估大语言模型（LLMs）在急诊室（ER）中的临床推理能力。急诊室是一个高风险环境，临床医生需要快速做出关乎生命的关键决策。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ER-Reason"],"document_share":0.0002723311546840959,"domain":"推理","name":"ER-Reason","organization_count":1,"organizations":["opencompass_hub"],"rank":1071,"released":"2025-05-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ER-Reason"},{"aliases":[],"benchmark_id":"claire-radar-2606-09276","caveat":"ERBench is an evaluation framework for equation discovery algorithms, assessing recovery of groundtruth formulas under varying dimensionality, sampling size, distribution, and domain.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09276"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ERBench","organization_count":1,"organizations":["claire_radar"],"rank":1072,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09276"},{"aliases":[],"benchmark_id":"claire-radar-2605-31251","caveat":"ERGeoBench evaluates vision-driven embodied geo-localization in MLLMs with 2,207 globally distributed street-view panoramas under single-view, panorama-view, and embodied-view settings. It measures foundational perception, spatial awareness, common sense reasoning, and geo-localization reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31251"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"ERGeoBench","organization_count":1,"organizations":["claire_radar"],"rank":1073,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31251"},{"aliases":["Anchor: Mitigating Artifact Drift in Agent Benchmark Generation"],"benchmark_id":"claire-radar-2605-26321","caveat":"ERP-Bench evaluates long-horizon procurement and manufacturing tasks in a production-grade ERP system using state-based verification and solver-certified solutions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26321"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ERP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1074,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26321"},{"aliases":[],"benchmark_id":"claire-radar-2609-04667","caveat":"ERPBench evaluates LLM agents in a six-round ERP simulation with pricing, production, procurement, inventory, finance, and shared-market competition. It includes 100 fixed problems tested in Solo (vs rule-based opponents) and Arena (six agents competing) ecologies, scoring based on firm valuation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04667"],"document_share":0.0002723311546840959,"domain":"Finance","name":"ERPBench","organization_count":1,"organizations":["claire_radar"],"rank":1075,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.04667"},{"aliases":[],"benchmark_id":"claire-radar-2609-17885","caveat":"ERPBench evaluates screenshot-only agents on a live ERP system with scoring against database ground-truth values.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17885"],"document_share":0.0002723311546840959,"domain":"Finance","name":"ERPBench","organization_count":1,"organizations":["claire_radar"],"rank":1076,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17885"},{"aliases":[],"benchmark_id":"llm-stats-erqa","caveat":"Embodied Reasoning Question Answering benchmark consisting of 400 multiple-choice visual questions across spatial reasoning, trajectory reasoning, action reasoning, state estimation, and multi-view reasoning for evaluating AI capabilities in physical world interactions","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/erqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ERQA","organization_count":1,"organizations":["llm_stats"],"rank":1077,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/erqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-17639","caveat":"ERQA-Plus evaluates embodied reasoning in AI systems with 1,766 question-answer instances grounded in 711 robot-centric images, organized by a taxonomy covering perceptual, action-centric, social-interaction, navigation-environmental, and commonsense reasoning. Scoring uses overall accuracy and SBERT similarity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17639"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"ERQA-Plus","organization_count":1,"organizations":["claire_radar"],"rank":1078,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.17639"},{"aliases":[],"benchmark_id":"claire-radar-2607-24707","caveat":"ERUnderstand evaluates vision-language models on structured understanding of entity-relationship diagrams. The benchmark contains 2,960 diagrams across curated educational sources, real-world schemas, and synthetic data, with standardized machine-readable JSON annotations. Scoring uses F1, BLEU, and graph edit distance to measure how accurately models recover schema elements such as entities, relationships, attributes, and extended ER constructs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24707"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ERUnderstand","organization_count":1,"organizations":["claire_radar"],"rank":1079,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24707"},{"aliases":[],"benchmark_id":"claire-radar-2607-17812","caveat":"ESCUCHA is a Spanish speech understanding benchmark evaluating LALMs across heterogeneous acoustic conditions and reasoning abilities. It includes 1,000 human-curated audio-question pairs spanning perceptual and reasoning categories, with multiple accents and non-normative speech.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.17812"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ESCUCHA","organization_count":1,"organizations":["claire_radar"],"rank":1080,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.17812"},{"aliases":[],"benchmark_id":"claire-radar-2607-23326","caveat":"ESF-Bench evaluates slot filling in enterprise contexts, covering 810 multi-turn dialogues and 6,530 slots across 8 domains, with a taxonomy of 57 challenging scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23326"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ESF-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1081,"released":"2026-07-25","source":"claire_radar","url":"https://arxiv.org/abs/2607.23326"},{"aliases":[],"benchmark_id":"claire-radar-2608-27371","caveat":"This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27371"],"document_share":0.0002723311546840959,"domain":"Planning","name":"ESRP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1082,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27371"},{"aliases":[],"benchmark_id":"claire-radar-2609-23659","caveat":"A benchmark for evaluating Ethereum DeFi representations under various distribution shifts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23659"],"document_share":0.0002723311546840959,"domain":"Finance","name":"ETH-TraceBench","organization_count":1,"organizations":["claire_radar"],"rank":1083,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23659"},{"aliases":[],"benchmark_id":"claire-radar-2608-04549","caveat":"EuroExec is an expert-authored benchmark of 413 open-ended European executive tasks evaluated by human experts. It uses a multi-attribute rubric and preference ranking to compute a Solve Rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04549"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EuroExec","organization_count":1,"organizations":["claire_radar"],"rank":1084,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04549"},{"aliases":["EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages"],"benchmark_id":"claire-radar-2609-05043","caveat":"A multilingual benchmark for verifiable instruction following, derived from IFEval and localized to 50 European languages and regional varieties via task-preserving translation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05043"],"document_share":0.0002723311546840959,"domain":"General AI","name":"European-IFEval","organization_count":1,"organizations":["claire_radar"],"rank":1085,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05043"},{"aliases":[],"benchmark_id":"claire-radar-github-gorkafs13-euskal-agent-bench","caveat":"Euskal-agent-bench is a project evaluating LLM agents in Basque, Spanish, and English across 30 tasks with direct prompting and tool-using agent architectures, measuring capability and agentic-stack degradation.","document_count":1,"document_ids":["claire_radar:https://github.com/GorkaFS13/euskal-agent-bench"],"document_share":0.0002723311546840959,"domain":"Agents","name":"euskal-agent-bench","organization_count":1,"organizations":["claire_radar"],"rank":1086,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/GorkaFS13/euskal-agent-bench"},{"aliases":[],"benchmark_id":"claire-radar-2605-23055","caveat":"EvalAwareBench is a factor-controlled benchmark of 100 paired safety-capability tasks, each toggling one of eight evaluation-awareness triggers while holding the underlying request fixed. It measures recognition and behavioral change in language models under evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23055"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EvalAwareBench","organization_count":1,"organizations":["claire_radar"],"rank":1087,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.23055"},{"aliases":[],"benchmark_id":"claire-radar-github-bagheri365-evallab","caveat":"Provides a frozen 120-task Python coding benchmark with paired comparisons and uncertainty analysis focused on ranking stability.","document_count":1,"document_ids":["claire_radar:https://github.com/bagheri365/evallab"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"EvalLab","organization_count":1,"organizations":["claire_radar"],"rank":1088,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/bagheri365/evallab"},{"aliases":[],"benchmark_id":"llm-stats-evalplus","caveat":"A rigorous code synthesis evaluation framework that augments existing datasets with extensive test cases generated by LLM and mutation-based strategies to better assess functional correctness of generated code, including HumanEval+ with 80x more test cases","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/evalplus?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"EvalPlus","organization_count":1,"organizations":["llm_stats"],"rank":1089,"released":"2023-05-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/evalplus?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-30217","caveat":"A benchmark of 50 diverse task-policy pairs where task completion requires an operation prohibited by a runtime monitor.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30217"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EvasionBench","organization_count":1,"organizations":["claire_radar"],"rank":1090,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.30217"},{"aliases":[],"benchmark_id":"claire-radar-2609-28149","caveat":"Power-system protection benchmark on EMT waveform data with multiple grids, protocols, and tasks, including fault detection, classification, and location.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.28149"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EvEMTBench","organization_count":1,"organizations":["claire_radar"],"rank":1091,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.28149"},{"aliases":[],"benchmark_id":"claire-radar-2608-01948","caveat":"Event ActivityNet is a large-scale simulated-event benchmark for untrimmed action understanding, derived from ActivityNet videos. It includes 3,263 videos, 200 action classes, event-voxel representations, temporal annotations, and supports recognition, event-language alignment, and online temporal localization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01948"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Event ActivityNet","organization_count":1,"organizations":["claire_radar"],"rank":1092,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01948"},{"aliases":[],"benchmark_id":"claire-radar-2606-04098","caveat":"EVID-Bench is a benchmark for search-grounded video misinformation detection, covering 222 videos and multiple manipulation types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04098"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"EVID-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1093,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.04098"},{"aliases":["Evidence-Bounded Mental Health Reasoning from Heterogeneous Speech Protocols"],"benchmark_id":"claire-radar-2608-31014","caveat":"Evaluates evidence-bounded mental health reasoning over 1,870 packages from six heterogeneous speech-text sources, scoring diagnosis accuracy and claim violations under modality masks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.31014"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Evidence Package Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1094,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.31014"},{"aliases":[],"benchmark_id":"claire-radar-2608-09096","caveat":"Evo-Bench is a benchmark for evaluating whether language models can autonomously improve their agent harness, with 608 harness-sensitive tasks across Search, Office, and General domains, fixed policy model, and resource budget.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09096"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Evo-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1095,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09096"},{"aliases":[],"benchmark_id":"claire-radar-2607-05202","caveat":"EvoAgentBench evaluates agent self-evolution via ability transfer across web research, algorithmic reasoning, software engineering, and knowledge work, using trace-grounded abilities and domain graphs with train/test splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05202"],"document_share":0.0002723311546840959,"domain":"Agents","name":"EvoAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":1096,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.05202"},{"aliases":[],"benchmark_id":"claire-radar-2606-13681","caveat":"EvoArena is a benchmark suite for LLM agents in dynamic environments, covering terminal workflows, software repositories, and social preferences. It evaluates step and chain accuracy under progressive environment updates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13681"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"EvoArena","organization_count":1,"organizations":["claire_radar"],"rank":1097,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13681"},{"aliases":[],"benchmark_id":"claire-radar-2606-13120","caveat":"EvoBrowseComp evaluates search agents on evolving knowledge with 800 contamination-free questions synthesized from live web traversal. The benchmark is designed to be regularly updated to prevent contamination.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13120"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"EvoBrowseComp","organization_count":1,"organizations":["claire_radar"],"rank":1098,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13120"},{"aliases":[],"benchmark_id":"claire-radar-2607-09711","caveat":"EvoClawBench evaluates whether an agent runtime can convert evidence from its own runs into reusable skills that improve fresh executions. It covers 100 tasks and 502 sub-problems across coding, data, office, security, operations, and domain-document workflows, comparing direct execution, pre-authored skills, and post-run skill summarization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09711"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EvoClawBench","organization_count":1,"organizations":["claire_radar"],"rank":1099,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.09711"},{"aliases":[],"benchmark_id":"claire-radar-2605-24110","caveat":"EvoCode-Bench evaluates coding agents in multi-turn iterative interactions, with 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, specifies requirements via observable behavior, and uses cumulative executable tests to verify new and prior requirements. Scoring uses MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24110"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EvoCode-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1100,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.24110"},{"aliases":[],"benchmark_id":"claire-radar-2608-29387","caveat":"Evaluates large language models as multi-turn generative UI assistants on 150 five-turn tasks with browser-executed artifacts, using turn-level, episode-level, and Adjacent Pass Retention metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29387"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EvoGenUI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1101,"released":"2026-08-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.29387"},{"aliases":["EvoHarmBench: Breaking Content Moderation with Iterative Human-Like Evasion"],"benchmark_id":"claire-radar-2608-27844","caveat":"Evaluates content moderation systems against iteratively evolved adversarial samples across 229 semantic sub-clusters and five violation categories, with scoring based on attack success rate under readability constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27844"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"EvoHarmBench","organization_count":1,"organizations":["claire_radar"],"rank":1102,"released":"2026-08-28","source":"claire_radar","url":"https://arxiv.org/abs/2608.27844"},{"aliases":[],"benchmark_id":"claire-radar-2609-08292","caveat":"EvoNav-Bench evaluates lifelong navigation agents in evolving environments built on ProcTHOR, with environment modifications between tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08292"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"EvoNav-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1103,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08292"},{"aliases":[],"benchmark_id":"claire-radar-2609-24663","caveat":"A benchmark proposed to track individual capabilities during artifact-level self-evolution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24663"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EvoPathBench","organization_count":1,"organizations":["claire_radar"],"rank":1104,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24663"},{"aliases":[],"benchmark_id":"claire-radar-2607-02440","caveat":"EvoPolicyGym evaluates autonomous policy evolution in interactive RL environments. A harness-model agent iteratively edits an executable policy under a fixed interaction budget. The benchmark records trajectories of programs, submissions, feedback, and selection, and scores agents on held-out episodes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02440"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"EvoPolicyGym","organization_count":1,"organizations":["claire_radar"],"rank":1105,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02440"},{"aliases":[],"benchmark_id":"claire-radar-2610-03153","caveat":"Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03153"],"document_share":0.0002723311546840959,"domain":"Agents","name":"EvoRiskBench","organization_count":1,"organizations":["claire_radar"],"rank":1106,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03153"},{"aliases":[],"benchmark_id":"claire-radar-2608-23903","caveat":"Evaluates continual visual learning under evolving semantic concept shift using ImageNet, iNaturalist, CUB-200-2011, and DomainNet with semantic transitions and metrics such as Rewrite Accuracy, Preservation Accuracy, Obsolete Retention, and Selective Revision Score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.23903"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EvoShift-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1107,"released":"2026-08-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.23903"},{"aliases":[],"benchmark_id":"opencompass-1843-ewmbench","caveat":"EWMBench is a benchmark for evaluating Embodied World Models, covering aspects such as scene consistency, motion correctness, and semantic alignment. It includes a diverse dataset and multi-dimensional metrics tailored for embodied manipulation tasks. EWMBench 是一个用于评估具身世界模型的基准，涵盖“场景一致性”、“动作正确性”和“语义对齐”等方面。它包含多样化的数据集和面向具身任务的多维评估指标，能够揭示现有模型的局限，并为生成具物理基础、任务导向的视频提供评价标准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/EWMBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"EWMBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1108,"released":"2025-05-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/EWMBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-37871","caveat":"Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37871"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"ExceptionDrive","organization_count":1,"organizations":["claire_radar"],"rank":1109,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37871"},{"aliases":[],"benchmark_id":"claire-radar-2607-24177","caveat":"EXE-Bench evaluates AI-based Windows malware detectors on performance, temporal robustness, adversarial robustness, and computational overhead, aggregated into a single score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24177"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"EXE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1110,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24177"},{"aliases":["WaveTLM: Reliable Time-Series Language Modeling through Task Compilation"],"benchmark_id":"claire-radar-2609-18812","caveat":"A benchmark spanning time-series forecasting, imputation, classification, anomaly detection, and waveform analysis across contract-grounded tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18812"],"document_share":0.0002723311546840959,"domain":"Software & AI Compute","name":"ExecTS-QA","organization_count":1,"organizations":["claire_radar"],"rank":1111,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18812"},{"aliases":["ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecasting"],"benchmark_id":"claire-radar-2608-20009","caveat":"24,000 simulated scenes for multi-object trajectory forecasting with explicit labels for mass, friction, and restitution, including ID and OOD splits over physical parameters and initial states.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.20009"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ExPhy","organization_count":1,"organizations":["claire_radar"],"rank":1112,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.20009"},{"aliases":[],"benchmark_id":"claire-radar-2607-26451","caveat":"Evaluates code explanations from agents by checking whether explanations enable an LLM to correctly answer questions about intended behavior and patch effects, using a question-based suite derived from agent patches.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26451"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"ExplainBench","organization_count":1,"organizations":["claire_radar"],"rank":1113,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.26451"},{"aliases":[],"benchmark_id":"claire-radar-2608-07062","caveat":"Evaluates explanation stability of test-time adaptation methods across histopathology datasets and architectures, measuring drift in attribution maps along with accuracy and calibration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07062"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Explanation Stability of Test-Time Adaptation in Computational Pathology: A Large-Scale Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1114,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07062"},{"aliases":[],"benchmark_id":"llm-stats-exploitbench","caveat":"ExploitBench is a cybersecurity benchmark that evaluates a model's ability to discover and exploit software vulnerabilities, reported as the fraction of challenges where the model captures the target (Cap%).","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/exploitbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"ExploitBench","organization_count":1,"organizations":["llm_stats"],"rank":1115,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitbench?top_n=500"},{"aliases":["ExploitGym"],"benchmark_id":"exploitgym","caveat":"Exploitation capability gym; different vulnerability classes have very different difficulty profiles.","document_count":1,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report"],"document_share":0.0002723311546840959,"domain":"security","name":"ExploitGym","organization_count":1,"organizations":["DeepSeek"],"rank":1116,"released":null,"source":"model_reports","url":"https://github.com/sunblaze-ucb/exploitgym"},{"aliases":[],"benchmark_id":"llm-stats-exploitgym","caveat":"ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel. Given a vulnerability and a proof-of-vulnerability input, agents must craft a working end-to-end exploit that achieves unauthorized code execution.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/exploitgym?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"ExploitGym","organization_count":1,"organizations":["llm_stats"],"rank":1117,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/exploitgym?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-30199","caveat":"A framework built on verifiable Alien Worlds where rules are executable and every answer can be checked exactly.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30199"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"ExplorationBench","organization_count":1,"organizations":["claire_radar"],"rank":1118,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.30199"},{"aliases":[],"benchmark_id":"claire-radar-2607-03110","caveat":"ExpoMotion is a large-scale benchmark for multi-exposure fusion with dynamic scenes, containing 1,738 sequences and 10,909 images across diverse environments. It provides high-fidelity ground truth for reference-based evaluation and a separate set for no-reference evaluation. The benchmark includes training and testing splits with controlled and real-world motions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03110"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ExpoMotion","organization_count":1,"organizations":["claire_radar"],"rank":1119,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.03110"},{"aliases":[],"benchmark_id":"claire-radar-2607-29677","caveat":"ExtractBench evaluates schema-guided extraction from enterprise documents. Given a document and a user-defined JSON schema, systems must return schema-valid JSON with correct values, include every record of repeated structures, mark missing fields as null, and provide source evidence. The benchmark includes 4,869 pages across 370 documents, 8 business domains, and 67 document types, with tags for challenge, perception, table structure, length, and domain. Scoring uses unified value F1 for value accuracy and word- and page-level F1 for grounding.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29677"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"ExtractBench","organization_count":1,"organizations":["claire_radar"],"rank":1120,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29677"},{"aliases":[],"benchmark_id":"claire-radar-2606-12300","caveat":"ExtremeWhenBench evaluates natural-language temporal grounding in hour-long videos, with 2,273 open-form queries over 194 videos. Metrics include mIoU and R@k for predicted time intervals, with strict parse-failure handling.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12300"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ExtremeWhenBench","organization_count":1,"organizations":["claire_radar"],"rank":1121,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12300"},{"aliases":[],"benchmark_id":"claire-radar-2609-32352","caveat":"Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insufficiently characterized.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32352"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"EyeVQA","organization_count":1,"organizations":["claire_radar"],"rank":1122,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32352"},{"aliases":[],"benchmark_id":"claire-radar-2607-10428","caveat":"EYT-Bench evaluates multi-turn dialogue capabilities of LLMs using a decoupled three-party setup with user simulation, target modeling, and judging.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10428"],"document_share":0.0002723311546840959,"domain":"General AI","name":"EYT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1123,"released":"2026-07-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.10428"},{"aliases":["Measuring in-context algorithmic reasoning in language models against an exact Bayes-optimal standard"],"benchmark_id":"claire-radar-2608-01575","caveat":"F-ICL evaluates in-context algorithmic inference against an exactly computed Bayes-optimal posterior under a bounded universal prior.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01575"],"document_share":0.0002723311546840959,"domain":"General AI","name":"F-ICL","organization_count":1,"organizations":["claire_radar"],"rank":1124,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01575"},{"aliases":[],"benchmark_id":"llm-stats-facts-grounding","caveat":"A benchmark evaluating language models' ability to generate factually accurate and well-grounded responses based on long-form input context, comprising 1,719 examples with documents up to 32k tokens requiring detailed responses that are fully grounded in provided documents","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/facts-grounding?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FACTS Grounding","organization_count":1,"organizations":["llm_stats"],"rank":1125,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/facts-grounding?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-factscore","caveat":"A fine-grained atomic evaluation metric for factual precision in long-form text generation that breaks generated text into atomic facts and computes the percentage supported by reliable knowledge sources, with automated assessment using retrieval and language models","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/factscore?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FActScore","organization_count":1,"organizations":["llm_stats"],"rank":1126,"released":"2023-05-23","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/factscore?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-03611","caveat":"A robot failure detection benchmark with 2,197 manipulation attempts from real and simulated sources.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03611"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FailBench","organization_count":1,"organizations":["claire_radar"],"rank":1127,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03611"},{"aliases":["failure-transparent-agents"],"benchmark_id":"claire-radar-github-junru-zhu-failure-transparent-agents","caveat":"Measures whether LLM agents admit tool failures instead of claiming success across 100 tasks covering web, attachment, execution, permission, and stale-data failures under varying pressure and instruction conditions.","document_count":1,"document_ids":["claire_radar:https://github.com/junru-zhu/failure-transparent-agents"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Failure-Transparent Agents","organization_count":1,"organizations":["claire_radar"],"rank":1128,"released":"2026-09-10","source":"claire_radar","url":"https://github.com/junru-zhu/failure-transparent-agents"},{"aliases":[],"benchmark_id":"claire-radar-2607-28934","caveat":"A benchmark that audits distributive bias in LLM resource allocation, varying task type, comparison context, and audit transparency across 600 financial-assistance requests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28934"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FairFund-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1129,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.28934"},{"aliases":["FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making"],"benchmark_id":"claire-radar-2609-01691","caveat":"A benchmark and evaluation framework measuring fairness and validity of VLM responses in hiring, legal, and healthcare using over 100K image-question pairs per model across demographic groups.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01691"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FAIRLENS","organization_count":1,"organizations":["claire_radar"],"rank":1130,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01691"},{"aliases":[],"benchmark_id":"claire-radar-2610-05790","caveat":"Remote sensing foundation models (RSFMs) are commonly evaluated using aggregate metrics, which can hide systematic performance disparities across ecological regions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05790"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FairRSFM","organization_count":1,"organizations":["claire_radar"],"rank":1131,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.05790"},{"aliases":[],"benchmark_id":"claire-radar-2608-10916","caveat":"A benchmark for evaluating faithfulness of mathematical chain-of-thought autoformalisation, using perturbed reasoning steps to test validity and invalidity preservation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10916"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FaithformBench","organization_count":1,"organizations":["claire_radar"],"rank":1132,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10916"},{"aliases":[],"benchmark_id":"claire-radar-2609-27603","caveat":"Benchmark with 3,500 cases of pseudoscientific claims across seven domains, scored by a deterministic whole-answer judge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27603"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FakeContextBench","organization_count":1,"organizations":["claire_radar"],"rank":1133,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27603"},{"aliases":[],"benchmark_id":"claire-radar-2608-03096","caveat":"FakeI2V-Bench is a benchmark for evaluating image-level and video-level deepfake detectors in video detection scenarios. It comprises 97,548 videos generated by recent generation models and covers multiple content categories, with detectors scored by AUC on detection tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03096"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FakeI2V-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1134,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03096"},{"aliases":[],"benchmark_id":"opencompass-1980-falsereject","caveat":"Safety alignment approaches in large language models (LLMs) often lead\nto the over-refusal of benign queries, significantly diminishing their utility\nin sensitive scenarios. To address this challenge, we introduce FalseReject,\na comprehensive resource containing 16k seemingly toxic queries accompani FalseReject 构建了包含 16 000 条表面“有毒”但实为良性的查询样本，覆盖 44 个安全相关类别，并提出一种基于图信息的对抗多智能体交互框架，用以生成多样且复杂的提示–响应对，并在响应中引入显式推理链，帮助模型更准确地区分安全与不安全上下文；该工作还为标准指令调优模型和推理导向模型分别准备了专项训练集，并附带人工标注的基准测试集，针对 29 款最先进 LLM 进行了大规模评估，结果表明经 FalseReject 监督微调后，模型在显著减少对良性查询的过度拒绝的同时，不仅未损失整体安全性，也保持了语言生成能力，为敏感场景下提升 LLM 可用性提供了首个系统化、可复现的资源与方法框","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FalseReject"],"document_share":0.0002723311546840959,"domain":"推理","name":"FalseReject","organization_count":1,"organizations":["opencompass_hub"],"rank":1135,"released":"2025-05-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/FalseReject"},{"aliases":[],"benchmark_id":"claire-radar-2605-31410","caveat":"FAM-Bench evaluates multimodal language and vision-language models on Food-as-Medicine reasoning. It includes 2500 expert-verified instances across 13 health conditions, with two tasks: dish-level suitability assessment (judging if a dish is suitable for a condition) and comparative dish analysis (ranking four dishes by suitability).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31410"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"FAM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1136,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31410"},{"aliases":[],"benchmark_id":"claire-radar-2609-39649","caveat":"Video super-resolution (VSR) is normally judged by PSNR and SSIM on clips that were downsampled bicubically, although in surveillance its purpose is to make faces and licence plates \\emph{recognisable}.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39649"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FANVIDv2","organization_count":1,"organizations":["claire_radar"],"rank":1137,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39649"},{"aliases":[],"benchmark_id":"claire-radar-2605-23215","caveat":"FastKernels evaluates GPU kernel generation for production inference across 46 representative architectures spanning 8 categories. It measures correctness and speedup of candidate kernels against reference implementations at multiple abstraction levels, from single-kernel to end-to-end serving.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23215"],"document_share":0.0002723311546840959,"domain":"Code","name":"FastKernels","organization_count":1,"organizations":["claire_radar"],"rank":1138,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23215"},{"aliases":[],"benchmark_id":"claire-radar-2608-27021","caveat":"A benchmark of 200 troubleshooting scenarios across eight network topologies evaluating network troubleshooting LLM agents under unreliable user tickets, including false fault reports and incorrect device attribution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27021"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FaulT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1139,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27021"},{"aliases":[],"benchmark_id":"claire-radar-2605-31349","caveat":"FBHM evaluates vision-language models on hateful meme detection across 25 rhetorical functionalities and 10 target communities, with 5,000 memes. Performance is measured by Macro-F1 score on this curated dataset.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31349"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"FBHM","organization_count":1,"organizations":["claire_radar"],"rank":1140,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31349"},{"aliases":[],"benchmark_id":"claire-radar-2609-27571","caveat":"136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27571"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"FDE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1141,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27571"},{"aliases":[],"benchmark_id":"opencompass-1733-feabench","caveat":"FEABench is a benchmark to evaluate the ability of large language models (LLMs) and LLM agents to simulate and solve physics, mathematics and engineering problems using finite element analysis (FEA). FEABench，一个用于评估大型语言模型和LLM代理使用有限元分析（FEA）模拟和解决物理、数学及工程问题能力的基准测试。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FEABench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"FEABench","organization_count":1,"organizations":["opencompass_hub"],"rank":1142,"released":"2025-04-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/FEABench"},{"aliases":[],"benchmark_id":"claire-radar-2608-26433","caveat":"Data-driven prognostics and health management has emerged as a key enabler for Industry 4.0, yet the development of robust remaining useful life (RUL) estimation models is often limited by the scarcity of run-to-failure data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26433"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FedCMAPSS","organization_count":1,"organizations":["claire_radar"],"rank":1143,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.26433"},{"aliases":[],"benchmark_id":"claire-radar-2609-15625","caveat":"This entry describes a benchmark for federated long-tail learning, but no task dataset, scoring protocol, or public code release is evidenced.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15625"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"FedLTLib","organization_count":1,"organizations":["claire_radar"],"rank":1144,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15625"},{"aliases":[],"benchmark_id":"opencompass-1618-fedmabench","caveat":"FedMABench is an open-source benchmark for federated training and evaluation of mobile agents, specifically designed for heterogeneous scenarios. FedMABench 是一个开源的联邦训练和评估移动代理的基准，特别为异构场景设计。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FedMABench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"FedMABench","organization_count":1,"organizations":["opencompass_hub"],"rank":1145,"released":"2025-03-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/FedMABench"},{"aliases":[],"benchmark_id":"claire-radar-2606-05949","caveat":"FEPBench evaluates text-to-image models on natural-science illustration generation using fine-grained atom set annotations, assessing instruction faithfulness, reasoning enrichment, and semantic precision across disciplines.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05949"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FEPBench","organization_count":1,"organizations":["claire_radar"],"rank":1146,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05949"},{"aliases":[],"benchmark_id":"claire-radar-2608-16055","caveat":"Evaluates governability of financial agents across 626 episodes and 100 KYC/AML task variants, measuring escalation, abstention, and audit trail completeness under different agent architectures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16055"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Fiducia-bench","organization_count":1,"organizations":["claire_radar"],"rank":1147,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16055"},{"aliases":[],"benchmark_id":"llm-stats-figqa","caveat":"FigQA is a multiple-choice benchmark on interpreting scientific figures from biology papers. It evaluates dual-use biological knowledge and multimodal reasoning relevant to bioweapons development.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/figqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"FigQA","organization_count":1,"organizations":["llm_stats"],"rank":1148,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/figqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-24241","caveat":"FilmBench evaluates text-to-video and reference-to-video generation using a Cinematic Language taxonomy with 1,169 prompts and 35 sub-metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24241"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FilmBench","organization_count":1,"organizations":["claire_radar"],"rank":1149,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24241"},{"aliases":[],"benchmark_id":"opencompass-895-fin-eva","caveat":"Ant Group and Shanghai University of Finance and Economics jointly launched the financial benchmark，Fin-Eva Version 1.0, covering multiple financial scenarios and subjects such as wealth management, insurance, investment research. The number of this benchmark's questions reaches 13,000+. 蚂蚁集团、上海财经大学联合推出金融评测集Fin-Eva Version 1.0，覆盖财富管理、保险、投资研究等多个金融场景以及金融专业主题学科，总评测题数目达到13,000+。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Fin-Eva"],"document_share":0.0002723311546840959,"domain":"知识","name":"Fin-Eva","organization_count":1,"organizations":["opencompass_hub"],"rank":1150,"released":"2023-12-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Fin-Eva"},{"aliases":[],"benchmark_id":"claire-radar-2609-04706","caveat":"FinalityBench evaluates agent decisions in financial exception scenarios where payment systems have inconsistent views due to message faults. It uses a hidden canonical event log and derives system states via faulted delivery. Scoring measures executed monetary effects against a reference outcome.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04706"],"document_share":0.0002723311546840959,"domain":"Agents","name":"FinalityBench","organization_count":1,"organizations":["claire_radar"],"rank":1151,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.04706"},{"aliases":[],"benchmark_id":"llm-stats-finance-agent","caveat":"Finance Agent is a benchmark for evaluating AI models on agentic financial analysis tasks, testing their ability to process financial data, perform calculations, and generate accurate analyses across various financial domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finance-agent?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Finance Agent","organization_count":1,"organizations":["llm_stats"],"rank":1152,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-finance-agent-v1-1","caveat":"Finance Agent v1.1 is an agentic financial-analysis benchmark that evaluates models on real-world finance workflows, including retrieving and reasoning over financial documents and performing multi-step calculations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v1.1?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Finance Agent v1.1","organization_count":1,"organizations":["llm_stats"],"rank":1153,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v1.1?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-finance-agent-v2","caveat":"Finance Agent v2 is an agentic financial-analysis benchmark from Vals that evaluates models on real-world finance workflows, measuring their ability to retrieve and reason over financial documents, perform multi-step calculations, and produce accurate analyses.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Finance Agent v2","organization_count":1,"organizations":["llm_stats"],"rank":1154,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/finance-agent-v2?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-github-gauthamrohan-financial-sentiment-benchmark","caveat":"Evaluates financial news sentiment classification comparing TF-IDF, DistilBERT fine-tuned, QLoRA, and prompted GPT-5-nano on a 300-sentence test subset using macro-F1.","document_count":1,"document_ids":["claire_radar:https://github.com/gauthamRohan/financial-sentiment-benchmark"],"document_share":0.0002723311546840959,"domain":"Finance","name":"financial-sentiment-benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1155,"released":"2026-08-21","source":"claire_radar","url":"https://github.com/gauthamRohan/financial-sentiment-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-32835","caveat":"As AI agents are becoming widely adopted in the financial services industry, careful measurement is essential to understand where they can be reliably deployed and where oversight and professional review remain necessary.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32835"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinancialAuditBench","organization_count":1,"organizations":["claire_radar"],"rank":1156,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32835"},{"aliases":[],"benchmark_id":"claire-radar-2606-15949","caveat":"FinBalance is a multi-document accounting reconciliation benchmark built from 710 source-document bundles across eight industries, three period types, and five difficulty levels. It evaluates models on producing journal entries, balance sheets, and inconsistency labels, with a deterministic ledger for ground truth.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15949"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinBalance","organization_count":1,"organizations":["claire_radar"],"rank":1157,"released":"2026-06-14","source":"claire_radar","url":"https://arxiv.org/abs/2606.15949"},{"aliases":[],"benchmark_id":"claire-radar-2607-16229","caveat":"FinBench is a benchmark for evaluating calibration and uncertainty in financial forecasting with time-gated tasks, requiring probability of positive return and 80% prediction interval, scored with Brier and Winkler scores.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.16229"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinBench","organization_count":1,"organizations":["claire_radar"],"rank":1158,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2607.16229"},{"aliases":[],"benchmark_id":"claire-radar-2605-27896","caveat":"FinBoardBench is an evaluation suite based on three financial board games: Cashflow, Acquire, and Monopoly. It assesses financial skills including personal cash flow management, corporate investment and acquisition forecasting, and competitive trade negotiations with asset auctions, using game simulations as the environment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27896"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinBoardBench","organization_count":1,"organizations":["claire_radar"],"rank":1159,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27896"},{"aliases":[],"benchmark_id":"claire-radar-2609-07603","caveat":"Evaluates construction of dynamic financial computer-using agent (CUA) tasks across 576 requests spanning 24 workflows and runtime variations, with qualification based on execution tests and quality checks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07603"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"FinCUABuildBench","organization_count":1,"organizations":["claire_radar"],"rank":1160,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07603"},{"aliases":[],"benchmark_id":"claire-radar-2608-00764","caveat":"FinDeepIndicator evaluates deep research agents on end-to-end financial indicator construction across four stages: formula specification, data collection, indicator calculation, and answer generation. It includes 3,350 QA pairs from U.S. and Chinese markets, 10 years of historical data, and 800 listed companies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00764"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinDeepIndicator","organization_count":1,"organizations":["claire_radar"],"rank":1161,"released":"2026-08-01","source":"claire_radar","url":"https://arxiv.org/abs/2608.00764"},{"aliases":[],"benchmark_id":"claire-radar-2608-12342","caveat":"FinED-Bench evaluates error detection in financial documents across nine real-world financial scenarios, covering three cognitive complexity levels. It includes over 900 documents from 2025, with data and code publicly available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12342"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinED-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1162,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.12342"},{"aliases":[],"benchmark_id":"claire-radar-2606-07861","caveat":"FineSightBench probes fine-scale visual perception in VLMs across scales from 4 to 48 pixels, separating perception and reasoning tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07861"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FineSightBench","organization_count":1,"organizations":["claire_radar"],"rank":1163,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07861"},{"aliases":[],"benchmark_id":"claire-radar-2608-06144","caveat":"FinEvo-Bench evaluates self-evolving agents on 120 real-case-grounded tasks across 20 business scenes in six financial domains, with institution-provided procedures and rubrics for quality and compliance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06144"],"document_share":0.0002723311546840959,"domain":"Agents","name":"FinEvo-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1164,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06144"},{"aliases":[],"benchmark_id":"claire-radar-2606-06960","caveat":"FinEvolveBench evaluates self-evolving agents on low-repetition financial prediction tasks with implicit rewards, measuring utility updates over delayed returns.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06960"],"document_share":0.0002723311546840959,"domain":"Agents","name":"FinEvolveBench","organization_count":1,"organizations":["claire_radar"],"rank":1165,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.06960"},{"aliases":["FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing"],"benchmark_id":"claire-radar-2609-25298","caveat":"A large-scale annotated dataset adding culture-language-region labels to existing web corpora.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25298"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FineWeb-CLaR","organization_count":1,"organizations":["claire_radar"],"rank":1166,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.25298"},{"aliases":["FinExam-10K: When Retrieval Helps Financial Reasoning?"],"benchmark_id":"claire-radar-2608-28155","caveat":"Evaluates financial reasoning with 10,198 expert-reannotated questions across CFA and FRM exams, including a public set of 5,110 questions and a sequestered set of 5,088 for a maintained leaderboard, with tracks for full coverage and context-complete reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.28155"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"FinExam-10K","organization_count":1,"organizations":["claire_radar"],"rank":1167,"released":"2026-08-28","source":"claire_radar","url":"https://arxiv.org/abs/2608.28155"},{"aliases":["FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability"],"benchmark_id":"claire-radar-2609-25192","caveat":"An open benchmark of 123 financial research tasks evaluated with atomic rubrics across information acquisition, source verification, and computation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25192"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinFIRST","organization_count":1,"organizations":["claire_radar"],"rank":1168,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.25192"},{"aliases":[],"benchmark_id":"claire-radar-2608-15177","caveat":"FinFraudBench is a heterogeneous graph benchmark for financial fraud detection. It contains two datasets (CreditCard-Fraud and BankTrans-Fraud) with up to 8.99M nodes and 89.23M directed typed edges, preserving six financial entity types and fourteen edge types. The evaluation protocol covers ranking and imbalance-sensitive classification metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15177"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinFraudBench","organization_count":1,"organizations":["claire_radar"],"rank":1169,"released":"2026-08-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.15177"},{"aliases":[],"benchmark_id":"claire-radar-2605-29427","caveat":"The evaluation focuses on financial regulatory non-compliance detection in LLM interactions. It involves a benchmark with expert-annotated labels at query and response levels, but the exact task, environment, or scoring setup is not detailed.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29427"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"FinGuard-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1170,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29427"},{"aliases":[],"benchmark_id":"claire-radar-2608-00909","caveat":"A benchmark of 33 financial FPGA computing tasks that tests LLM hardware generation, system-level configuration tuning, and adaptation to specification changes, scoring both functional correctness and latency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00909"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinHardBench","organization_count":1,"organizations":["claire_radar"],"rank":1171,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.00909"},{"aliases":["FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering"],"benchmark_id":"claire-radar-2609-24002","caveat":"The benchmark evaluates ambiguous financial question answering with clarification and intent integration across a five-category ambiguity taxonomy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24002"],"document_share":0.0002723311546840959,"domain":"Agents","name":"FinInteract","organization_count":1,"organizations":["claire_radar"],"rank":1172,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24002"},{"aliases":[],"benchmark_id":"claire-radar-2608-22842","caveat":"A financial document parsing evaluation suite covering diverse document scenarios, but release details and public access are unclear.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22842v1"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinixDocBench","organization_count":1,"organizations":["claire_radar"],"rank":1173,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.22842v1"},{"aliases":[],"benchmark_id":"claire-radar-2609-01198","caveat":"Two tasks over cumulative longitudinal banking dialogue: reconstructing every life-event instance with its first-establishing session and reconstructing a complete 34-path financial state at consecutive checkpoints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01198"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinLifeBench","organization_count":1,"organizations":["claire_radar"],"rank":1174,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01198"},{"aliases":[],"benchmark_id":"claire-radar-2608-04095","caveat":"FinPerMA evaluates personalized memory in LLM agents using frozen longitudinal investor trajectories with theory-informed impact rules. It includes a Post-Shock checkpoint to assess integration of material events into persistent user models, with 2,994 questions from 276 personas.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04095"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FinPerMA","organization_count":1,"organizations":["claire_radar"],"rank":1175,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.04095"},{"aliases":[],"benchmark_id":"claire-radar-2606-31522","caveat":"Evaluates the longitudinal stability of behavioral mandates in LLM-based financial agents using a synthetic market simulation that decouples observable price from hidden fundamental value, scoring mandate adherence across calm, crash, and bubble market regimes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31522"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"FinPersona-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1176,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31522"},{"aliases":[],"benchmark_id":"claire-radar-2608-04077","caveat":"FinProBench evaluates financial AI agents on professional tasks using rubrics derived from practitioner deliverables. It includes 1,723 curated deliverables across 57 occupations and an evaluation set of 20 tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04077"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinProBench","organization_count":1,"organizations":["claire_radar"],"rank":1177,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.04077"},{"aliases":[],"benchmark_id":"llm-stats-finqa","caveat":"A large-scale dataset for numerical reasoning over financial data with question-answering pairs written by financial experts, featuring complex numerical reasoning and understanding of heterogeneous representations with annotated gold reasoning programs for full explainability","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"FinQA","organization_count":1,"organizations":["llm_stats"],"rank":1178,"released":"2021-09-01","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/finqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-07400","caveat":"FinRank evaluates passage retrieval, reranking, and hard-negative discrimination over SEC filings, with 1185 question-answer records and curated hard negatives, scoring recall and accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07400"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinRank","organization_count":1,"organizations":["claire_radar"],"rank":1179,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07400"},{"aliases":[],"benchmark_id":"claire-radar-2608-18534","caveat":"FinRCA-Bench is a synthetic benchmark for financial reconciliation root-cause analysis, with 2,250 cases across 14 tables and 15 failure categories, providing evidence retrieval and reasoning evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.18534"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinRCA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1180,"released":"2026-08-19","source":"claire_radar","url":"https://arxiv.org/abs/2608.18534"},{"aliases":[],"benchmark_id":"claire-radar-2608-04374","caveat":"FinReportBench evaluates institution-grade financial report generation using 244 bilingual tasks sourced from 10,000 financial research records. It uses a 35-item rubric covering deliverability, report identity, and institutional completeness, with three judge families.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04374"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinReportBench","organization_count":1,"organizations":["claire_radar"],"rank":1181,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04374"},{"aliases":[],"benchmark_id":"claire-radar-2608-25325","caveat":"Evaluates Chinese financial LLMs on static operation execution across 9,742 instances and evidence-state control via FinRisk-Ask using pre-action trajectory states.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25325"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinRiskAtlas","organization_count":1,"organizations":["claire_radar"],"rank":1182,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25325"},{"aliases":[],"benchmark_id":"llm-stats-finsearchcomp-t2-t3","caveat":"FinSearchComp T2&T3 is a combined benchmark for evaluating financial search and reasoning capabilities on Tier 2 and Tier 3 tasks, testing models' ability to retrieve and analyze complex financial information using tools.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t2-t3?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FinSearchComp T2&T3","organization_count":1,"organizations":["llm_stats"],"rank":1183,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t2-t3?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-finsearchcomp-t3","caveat":"FinSearchComp-T3 is a benchmark for evaluating financial search and reasoning capabilities, testing models' ability to retrieve and analyze financial information using tools.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t3?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FinSearchComp-T3","organization_count":1,"organizations":["llm_stats"],"rank":1184,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/finsearchcomp-t3?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-03564","caveat":"Financial AI agents must do more than retrieve facts: investment workflows require correct quantitative execution, reliable use of procedural resources, and auditable structured outputs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03564"],"document_share":0.0002723311546840959,"domain":"Agents","name":"FinSkillBench","organization_count":1,"organizations":["claire_radar"],"rank":1185,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03564"},{"aliases":[],"benchmark_id":"claire-radar-2606-03184","caveat":"FinStressTS is a synthetic benchmark for financial time-series forecasting with controllable mechanisms and diagnostic environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03184"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinStressTS","organization_count":1,"organizations":["claire_radar"],"rank":1186,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03184"},{"aliases":[],"benchmark_id":"claire-radar-2605-29586","caveat":"FinVerBench evaluates financial statement verification in LLMs, using SEC 10-K XBRL filings from 43 S&P 500 companies with a four-category error taxonomy (arithmetic, cross-statement linkage, year-over-year, magnitude perturbations).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29586"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinVerBench","organization_count":1,"organizations":["claire_radar"],"rank":1187,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29586"},{"aliases":[],"benchmark_id":"claire-radar-2608-03259","caveat":"FinVerse is a finance-domain time-series forecasting benchmark with 116,897 series (171.1M observations), selecting 60,232 as evaluation targets. It defines 11 metric families (78 metrics) tailored to each series' economic meaning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03259"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FinVerse","organization_count":1,"organizations":["claire_radar"],"rank":1188,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03259"},{"aliases":[],"benchmark_id":"claire-radar-2609-23064","caveat":"Evaluates complex physical world intelligence in multimodal large language models and agents through coupled-field fire dynamics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23064"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FireWorldBench","organization_count":1,"organizations":["claire_radar"],"rank":1189,"released":"2026-09-19","source":"claire_radar","url":"https://arxiv.org/abs/2609.23064"},{"aliases":[],"benchmark_id":"claire-radar-2608-21839","caveat":"FIRM-Video-Bench is a benchmark of 250 text-to-video generations with 750 point-wise human annotations covering Instruction Following, World Coherence, and Perceptual Quality.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.21839v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FIRM-Video-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1190,"released":"2026-08-22","source":"claire_radar","url":"http://arxiv.org/abs/2608.21839v1"},{"aliases":["FirmCORe: A Benchmark for Structured Reasoning about Inter-Firm Collaboration Opportunities"],"benchmark_id":"claire-radar-2609-17128","caveat":"FirmCORe is a human-annotated benchmark of 2,805 firm pairs for reasoning about inter-firm collaboration opportunities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17128"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FirmCORe","organization_count":1,"organizations":["claire_radar"],"rank":1191,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17128"},{"aliases":[],"benchmark_id":"claire-radar-2608-08736","caveat":"FitAQA evaluates fitness action quality assessment in MLLMs across perception, judgement, and temporal grounding tasks, using a unified taxonomy of 38 form errors in six dimensions over 30 exercises.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08736"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FitAQA","organization_count":1,"organizations":["claire_radar"],"rank":1192,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08736"},{"aliases":[],"benchmark_id":"llm-stats-flame-vlm-code","caveat":"Flame-VLM-Code evaluates multimodal models on visual code generation tasks, measuring ability to generate code from visual inputs such as UI mockups and design specifications.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/flame-vlm-code?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Flame-VLM-Code","organization_count":1,"organizations":["llm_stats"],"rank":1193,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/flame-vlm-code?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-945-flames","caveat":"Flames is a highly adversarial benchmark in Chinese for LLM's value alignment evaluation developed by Shanghai AI Lab and Fudan NLP Group. Flames meticulously designs a dataset of 2,251 highly adversarial, manually crafted prompts, each tailored to probe a specific value dimension (i.e., Fairness, Safety, Morality, Legality, Data protection). Currently,  Flames releases 1,000 prompts for public use (Flames_1k_Chinese). Flames 是上海人工智能实验室和复旦大学 NLP团队开发的 LLM 价值对齐方向的中文高度对抗性基准。Flames 精心设计了一个由 2,251 个高度对抗性、人工创建的提示词成的评测集，每个提示词都经过精心设计，以探究特定的价值维度（即公平、安全、道德、合法、数据保护）。目前，Flames 发布了 1,000 个提示词供公众使用（Flames_1k_Chinese）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Flames"],"document_share":0.0002723311546840959,"domain":"安全","name":"Flames","organization_count":1,"organizations":["opencompass_hub"],"rank":1194,"released":"2024-03-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Flames"},{"aliases":[],"benchmark_id":"claire-radar-2606-27128","caveat":"FlameVQA is a multiple-choice visual question answering benchmark for UAV-based wildfire monitoring, built on FLAME 3 with paired RGB and radiometric thermal images. It includes 34 questions per image across six capability groups, covering detection, localization, coverage estimation, cross-modal reasoning, and flight planning. Labels are generated via MLLM assistance, deterministic thermal rules, and human auditing. The dataset and code are open-source.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27128"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FlameVQA","organization_count":1,"organizations":["claire_radar"],"rank":1195,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.27128"},{"aliases":[],"benchmark_id":"claire-radar-2609-38719","caveat":"Experimental investigation and modeling of flapping-wing aerial vehicles are limited by the scarcity of large-scale records that temporally align wing kinematics, aerodynamic responses, and flight states.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38719"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FlapKAD","organization_count":1,"organizations":["claire_radar"],"rank":1196,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.38719"},{"aliases":[],"benchmark_id":"claire-radar-2608-14049","caveat":"FlatLab is a simulation-based benchmark for robotic manipulation of flat objects, providing physical simulation, data collection, and standardized task definitions and evaluation protocols.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14049"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"FlatLab","organization_count":1,"organizations":["claire_radar"],"rank":1197,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14049"},{"aliases":[],"benchmark_id":"llm-stats-flenqa","caveat":"Flexible Length Question Answering dataset for evaluating the impact of input length on reasoning performance of language models, featuring True/False questions embedded in contexts of varying lengths (250-3000 tokens) across three reasoning tasks: Monotone Relations, People In Rooms, and simplified Ruletaker","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/flenqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"FlenQA","organization_count":1,"organizations":["llm_stats"],"rank":1198,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/flenqa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-fleurs","caveat":"Few-shot Learning Evaluation of Universal Representations of Speech - a parallel speech dataset in 102 languages built on FLoRes-101 with approximately 12 hours of speech supervision per language for tasks including ASR, speech language identification, translation and retrieval. Scores are shown as speech recognition accuracy (1 - word error rate), so higher is better.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/fleurs?top_n=500"],"document_share":0.0002723311546840959,"domain":"speech_to_text","name":"FLEURS","organization_count":1,"organizations":["llm_stats"],"rank":1199,"released":"2022-05-25","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/fleurs?top_n=500"},{"aliases":["Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning"],"benchmark_id":"claire-radar-2608-01437","caveat":"FLEX evaluates expert routing across a 34-task long-horizon multimodal continual instruction-tuning sequence with reduced task-identifying textual fingerprints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01437"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FLEX","organization_count":1,"organizations":["claire_radar"],"rank":1200,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.01437"},{"aliases":[],"benchmark_id":"claire-radar-2605-25717","caveat":"FLOATBench is a tabular benchmark for surrogate modeling of tower fatigue in 22 MW floating offshore wind turbines. It provides 582,120 per-section fatigue damage labels from OpenFAST simulations across three tower geometries, with regime-aware splits and three evaluation protocols (random, within-tower regime-aware, cross-tower transfer).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25717"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FLOATBench","organization_count":1,"organizations":["claire_radar"],"rank":1201,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25717"},{"aliases":[],"benchmark_id":"claire-radar-2608-15410","caveat":"FloodReasonBench is a benchmark for vision-language model reasoning segmentation in flood response scenarios. It introduces FloodResponseSeg, a flood-specific dataset, and evaluates pipelines under lightweight visual encoding, split inference, and compressed representations. It also measures accuracy, latency, energy, and communication tradeoffs on an embedded platform.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15410"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"FloodReasonBench","organization_count":1,"organizations":["claire_radar"],"rank":1202,"released":"2026-08-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.15410"},{"aliases":[],"benchmark_id":"opencompass-509-flores","caveat":"Flores is a benchmark dataset for machine translation between English and low-resource languages, which consists of sentences translated from Wikipedia, involving English and four low-resource languages, namely Nepali, Sinhala, Khmer and Pashto. Flores has two versions, we use the Flores-101 version here, which is the second version including 101 languages besides english. Flores是一个用于评估低资源语言机器翻译的基准数据集，它包含了从维基百科翻译的句子，涉及英语和四种低资源语言，分别是尼泊尔语、僧伽罗语、高棉语和普什图语。Flores有两个版本，我们这里使用的是第一个版本Flores-101，它包含有除英语外的101种语言。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Flores"],"document_share":0.0002723311546840959,"domain":"语言","name":"Flores","organization_count":1,"organizations":["opencompass_hub"],"rank":1203,"released":"2021-06-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Flores"},{"aliases":[],"benchmark_id":"opencompass-1324-flub","caveat":"FLUB evaluates the reasoning and understanding abilities of LLMs. It includes three tasks with increasing difficulty, consisting of the tricky, humorous, and misleading texts collected from the real internet environment. FLUB用于评估LLM的推理和理解能力，其中包含3个难度递进的任务，由从真实互联网环境中收集的狡猾、幽默和误导性的文本构成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FLUB"],"document_share":0.0002723311546840959,"domain":"推理","name":"FLUB","organization_count":1,"organizations":["opencompass_hub"],"rank":1204,"released":"2024-02-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/FLUB"},{"aliases":[],"benchmark_id":"claire-radar-2608-18423","caveat":"FM-Bench evaluates LLM agents on long-horizon football club management through 20 in-game years with 26 tools, measuring managerial decision quality via a deterministic engine scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.18423"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1205,"released":"2026-08-19","source":"claire_radar","url":"https://arxiv.org/abs/2608.18423"},{"aliases":[],"benchmark_id":"claire-radar-2608-12324","caveat":"FMG-Bench evaluates English-language Christian theological triage and pastoral guidance in LLMs. It scores whether model responses match expected behaviors for triage levels: primary doctrine, secondary doctrine, prudential, and pastoral, with safety-related escalation appropriate to each scenario. The corpus includes 120 base scenarios plus 37 perturbation variants.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12324"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"FMG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1206,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.12324"},{"aliases":[],"benchmark_id":"claire-radar-2609-33158","caveat":"Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33158"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"FOCUS","organization_count":1,"organizations":["claire_radar"],"rank":1207,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33158"},{"aliases":[],"benchmark_id":"claire-radar-2609-10243","caveat":"A physical-world benchmark for long-horizon robotic manipulation of deformable objects using real-robot data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10243"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"FolDeX","organization_count":1,"organizations":["claire_radar"],"rank":1208,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10243"},{"aliases":["llm-food-benchmark-2026-09"],"benchmark_id":"claire-radar-github-tim2000s-llm-food-benchmark-2026-09","caveat":"A repository for a September 2026 rerun of an existing carbohydrate-estimation benchmark for LLMs on 13 food photographs.","document_count":1,"document_ids":["claire_radar:https://github.com/tim2000s/llm-food-benchmark-2026-09"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Food photograph benchmark, September 2026 rerun","organization_count":1,"organizations":["claire_radar"],"rank":1209,"released":"2026-09-21","source":"claire_radar","url":"https://github.com/tim2000s/llm-food-benchmark-2026-09"},{"aliases":[],"benchmark_id":"claire-radar-2605-24503","caveat":"Evaluates multimodal large language models on explainable compliance analysis in commercial kitchen surveillance, using video clips with dual-channel annotations and a composite metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24503"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FoodMonitor","organization_count":1,"organizations":["claire_radar"],"rank":1210,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24503"},{"aliases":[],"benchmark_id":"claire-radar-2607-19409","caveat":"FORCE-Bench evaluates agentic AI systems in enterprise finance across three task types: financial obligation research, financial entity performance research, and business brief generation. It includes 251 expert-annotated queries and a rubric-based scoring framework across eight dimensions: accuracy, citations, clarity, depth, groundedness, recency, relevance, and structure.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19409"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FORCE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1211,"released":"2026-07-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.19409"},{"aliases":[],"benchmark_id":"claire-radar-2605-28044","caveat":"FORCEBENCH is a contrastive stress test for evidence-force calibration in cited RAG. It pairs fixed cited passages with evidence-calibrated claims and force-raised variants across five axes: relation, modality, scope, temporal validity, and numeric specificity. Evaluation is a fixed, locality-filtered set of 198 pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28044"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FORCEBENCH","organization_count":1,"organizations":["claire_radar"],"rank":1212,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28044"},{"aliases":[],"benchmark_id":"claire-radar-2609-27385","caveat":"Evaluates language-model decisions with budgeted forecast tools, but no distinct reusable evaluation protocol is introduced.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27385"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Forecast Workflow Bench","organization_count":1,"organizations":["claire_radar"],"rank":1213,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27385"},{"aliases":[],"benchmark_id":"claire-radar-2606-18686","caveat":"A simulated-world forecasting benchmark built on Freeciv game rollouts. Evaluates probabilistic reasoning by scoring forecasts about hidden future game states, with continuous or binary questions, paired intervention worlds, and artifacts for release.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18686"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ForecastBench-Sim","organization_count":1,"organizations":["claire_radar"],"rank":1214,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18686"},{"aliases":[],"benchmark_id":"opencompass-1671-forensics-bench","caveat":"A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Forensics-bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"Forensics-bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1215,"released":"2025-03-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Forensics-bench"},{"aliases":[],"benchmark_id":"claire-radar-2606-00644","caveat":"Temporally controlled benchmark with 500 tasks across AI domains for forward-looking research judgment; includes offline knowledge bases and validation protocols.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00644"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ForeSci","organization_count":1,"organizations":["claire_radar"],"rank":1216,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00644"},{"aliases":[],"benchmark_id":"claire-radar-2606-27079","caveat":"A diagnostic safety benchmark for vision-language-action models, evaluating policies across physical interaction, instruction, and perception safety with cumulative cost and risk exposure metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27079"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"ForesightSafety-VLA","organization_count":1,"organizations":["claire_radar"],"rank":1217,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.27079"},{"aliases":[],"benchmark_id":"claire-radar-2608-08605","caveat":"ForestBench evaluates multi-agent collaboration by representing MAS traces as unified collaboration graphs, comparing them against query-specific reference forests of successful graphs, and scoring structural similarity with a metric panel.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08605"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ForestBench","organization_count":1,"organizations":["claire_radar"],"rank":1218,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08605"},{"aliases":["One Polluted Page Is Enough: Evaluating Web Content Pollution in Generative Recommenders"],"benchmark_id":"claire-radar-2606-13610","caveat":"FORGE evaluates whether search-augmented language models recommend fake products introduced through controlled modifications to retrieved web pages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13610"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"FORGE","organization_count":1,"organizations":["claire_radar"],"rank":1219,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13610"},{"aliases":[],"benchmark_id":"claire-radar-2607-26455","caveat":"Proposes a benchmark for evaluating forgetting dynamics in language models under continual knowledge editing, with concept-based and scenario-based QA, but no public artifacts are available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26455"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"ForgetBench","organization_count":1,"organizations":["claire_radar"],"rank":1220,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.26455"},{"aliases":["FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models"],"benchmark_id":"claire-radar-2608-20153","caveat":"175 expert-validated instances from STOC, FOCS, SODA, and COLT papers (2025-2026) for evaluating LLMs on end-to-end theoretical computer science research, including autoformalization and proof tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.20153"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"FormalTCS","organization_count":1,"organizations":["claire_radar"],"rank":1221,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.20153"},{"aliases":[],"benchmark_id":"claire-radar-2608-10396","caveat":"The paper introduces FormStruct-Bench, an evaluation dataset for table-form document structure recognition, but no official links to the benchmark artifacts are provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10396"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FormStruct-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1222,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10396"},{"aliases":[],"benchmark_id":"opencompass-1737-fortisavqa","caveat":"FortisAVQA is the first dataset designed to assess the robustness of AVQA models. FortisAVQA，这是首个用于评估AVQA模型鲁棒性的数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FortisAVQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"FortisAVQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1223,"released":"2025-04-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/FortisAVQA"},{"aliases":[],"benchmark_id":"claire-radar-2609-03331","caveat":"A benchmark with 1,080 images and 10,800 question turns across 10-turn dialogues, evaluating correction and cooperation when false visual premises recur, scored by CorrTP@K and two detectors.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03331"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FPCO-Dialog","organization_count":1,"organizations":["claire_radar"],"rank":1224,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03331"},{"aliases":[],"benchmark_id":"claire-radar-2609-03370","caveat":"A multiple-choice benchmark for English and Japanese that tests whether models distinguish semantic frames evoked by the same verb across contexts, using FrameNet resources and generation-verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03370"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FrameBench","organization_count":1,"organizations":["claire_radar"],"rank":1225,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03370"},{"aliases":[],"benchmark_id":"llm-stats-frames","caveat":"Factuality, Retrieval, And reasoning MEasurement Set - a unified evaluation dataset of 824 challenging multi-hop questions for testing retrieval-augmented generation systems across factuality, retrieval accuracy, and reasoning capabilities, requiring integration of 2-15 Wikipedia articles per question","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frames?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FRAMES","organization_count":1,"organizations":["llm_stats"],"rank":1226,"released":"2024-09-19","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frames?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-24551","caveat":"A protocol-sensitive benchmark for adversarial robustness in financial fraud and credit risk, evaluating the same model-attack-defence setting under three protocols: unconstrained, post-hoc filtering, and deployment-aware constraint-integrated attacks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24551"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FraudBench","organization_count":1,"organizations":["claire_radar"],"rank":1227,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24551"},{"aliases":[],"benchmark_id":"opencompass-1145-freb-tqa","caveat":"FREB-TQA is a Fine-grained Robustness Evaluation Benchmark for Table Question Answering. FREB-TQA 是一个细粒度的稳健性评估基准，专注于表格问答（TQA）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/FREB-TQA"],"document_share":0.0002723311546840959,"domain":"理解","name":"FREB-TQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1228,"released":"2024-06-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/FREB-TQA"},{"aliases":[],"benchmark_id":"claire-radar-2607-07343","caveat":"FreightBidBench is a public-calibrated, closed-loop benchmark for real-time truckload bid acceptance with explicit operational feasibility and economics, including pickup reach, appointment windows, hours-of-service, and yard delays. It provides two full-horizon hindsight ceilings for evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.07343"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FreightBidBench","organization_count":1,"organizations":["claire_radar"],"rank":1229,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.07343"},{"aliases":[],"benchmark_id":"llm-stats-french-mmlu","caveat":"French version of MMLU-Pro, a multilingual benchmark for evaluating language models' cross-lingual reasoning capabilities across 14 diverse domains including mathematics, physics, chemistry, law, engineering, psychology, and health.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/french-mmlu?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"French MMLU","organization_count":1,"organizations":["llm_stats"],"rank":1230,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/french-mmlu?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-04281","caveat":"FreshCache-Bench provides 8,072 base queries across five freshness classes with ground truth staleness labels from web snapshots at 1, 12, 24 hours, and 7 days, expanded to 31,201 queries via paraphrase generation, for evaluating semantic caching in retrieval-augmented LLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04281"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FreshCache-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1231,"released":"2026-07-05","source":"claire_radar","url":"https://arxiv.org/abs/2607.04281"},{"aliases":[],"benchmark_id":"claire-radar-2607-29602","caveat":"FriendBench evaluates dyadic familiarity inference in humans and models using 96 dyads, comparing accuracy across modalities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29602"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FriendBench","organization_count":1,"organizations":["claire_radar"],"rank":1232,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29602"},{"aliases":[],"benchmark_id":"claire-radar-github-chirindaopensource-from-financial-sentiment-classification-to-return-predictability","caveat":"Evaluates LLMs on financial sentiment classification and downstream return predictability using a reproducible QLoRA fine-tuning and evaluation pipeline.","document_count":1,"document_ids":["claire_radar:https://github.com/chirindaopensource/from_financial_sentiment_classification_to_return_predictability"],"document_share":0.0002723311546840959,"domain":"Finance","name":"From Financial Sentiment Classification to Return Predictability","organization_count":1,"organizations":["claire_radar"],"rank":1233,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/chirindaopensource/from_financial_sentiment_classification_to_return_predictability"},{"aliases":[],"benchmark_id":"claire-radar-2609-34710","caveat":"A deterministic football-management benchmark evaluating autonomous agents across configurable factors using a headless simulation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34710"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"FromPitch2Board","organization_count":1,"organizations":["claire_radar"],"rank":1234,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34710"},{"aliases":[],"benchmark_id":"claire-radar-github-shubhraj5575-frontier-agent-benchmark","caveat":"Evaluates autonomous AI engineering agents on engineering quality dimensions such as testing, architecture, and maintainability, using observed telemetry with provenance labels.","document_count":1,"document_ids":["claire_radar:https://github.com/shubhraj5575/frontier-agent-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Frontier Agent Benchmark (FAB)","organization_count":1,"organizations":["claire_radar"],"rank":1235,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/shubhraj5575/frontier-agent-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-36739","caveat":"Multi-agent LLM systems are increasingly structured like organizations, with roles, critics and shared memory, yet they are evaluated on tasks that last minutes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36739"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Frontier Autolab","organization_count":1,"organizations":["claire_radar"],"rank":1236,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36739"},{"aliases":[],"benchmark_id":"claire-radar-2607-20645","caveat":"Frontier Financial Judgement assesses agents' ability to identify valuation-relevant financial information from 656 synthetic and real news items, matching expert labels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20645"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Frontier Financial Judgement","organization_count":1,"organizations":["claire_radar"],"rank":1237,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.20645"},{"aliases":[],"benchmark_id":"llm-stats-frontier-science","caveat":"Frontier Science is a benchmark of exceptionally challenging scientific reasoning problems spanning advanced natural-science domains, designed to test expert-level scientific understanding and multi-step reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontier-science?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Frontier Science","organization_count":1,"organizations":["llm_stats"],"rank":1238,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-science?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-frontier-bench-v0-1","caveat":"Frontier-Bench v0.1 evaluates agentic terminal coding. Anthropic reports results using the mini-SWE-agent harness and a GKE backend, measured as mean reward across five attempts per task.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontier-bench-v0.1?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Frontier-Bench v0.1","organization_count":1,"organizations":["llm_stats"],"rank":1239,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-bench-v0.1?top_n=500"},{"aliases":["Frontier-Eng","Frontier-Engineering Bench"],"benchmark_id":"frontier_engineering","caveat":"Medal Score uses peer-relative thresholds, not solve rate. September 2026 corrected results combine rescoring and shorter reruns; separate v1 (47 tasks), v1-lite (10 tasks), and historical average ranks.","document_count":1,"document_ids":["model_reports:einsia_frontier_engineering_leaderboard_2026_09_15"],"document_share":0.0002723311546840959,"domain":"agents","name":"Frontier-Engineering","organization_count":1,"organizations":["Navers Lab / Einsia"],"rank":1240,"released":"2026-04-14","source":"model_reports","url":"https://lab.einsia.ai/frontier-eng/"},{"aliases":[],"benchmark_id":"claire-radar-2608-24979","caveat":"Evaluates scientific agents on 97 released end-to-end workflows across six domains, using pass rate and average score to measure full delivery of required scientific deliverables.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24979"],"document_share":0.0002723311546840959,"domain":"General AI","name":"FrontierChallenge","organization_count":1,"organizations":["claire_radar"],"rank":1241,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24979"},{"aliases":["FrontierChallenge","Frontier Challenge"],"benchmark_id":"frontier_challenge","caveat":"Pass-rate results are model-scaffold measurements over the 97 released tasks with one trajectory per system-task pair, so a reported percentage is not a model-only number and does not estimate reliability beyond this release or under a different scaffold.","document_count":1,"document_ids":["model_reports:frontier_challenge_leaderboard_2026_08_25"],"document_share":0.0002723311546840959,"domain":"scientific_agent","name":"FrontierChallenge","organization_count":1,"organizations":["ApodexAI"],"rank":1242,"released":"2026-08-25","source":"model_reports","url":"https://apodexai.github.io/FrontierAgent/benchmarks/FrontierChallenge/"},{"aliases":["FrontierCode","FrontierCode (Diamond)"],"benchmark_id":"frontiercode","caveat":"Cognition's production-standard coding evaluation, asking whether models write *good* code rather than merely correct code. Unrelated to Terminal-Bench's Frontier-Bench despite the shared prefix. Reported per reasoning-effort setting (the Diamond subset at xhigh), so the effort level has to travel with the number. First-party to a competing coding agent's vendor, and revised as 1.1 on 2026-07-07, so the methodology version has to travel with the number too.","document_count":1,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5"],"document_share":0.0002723311546840959,"domain":"coding_agent","name":"FrontierCode","organization_count":1,"organizations":["Anthropic"],"rank":1243,"released":"2026-06-08","source":"model_reports","url":"https://cognition.com/blog/frontier-code"},{"aliases":[],"benchmark_id":"llm-stats-frontiercode","caveat":"FrontierCode is Cognition's coding evaluation that tests whether models can pass difficult coding tasks while meeting the standards of high-quality production codebases. The Diamond subset contains the hardest problems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiercode?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FrontierCode","organization_count":1,"organizations":["llm_stats"],"rank":1244,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-frontiercode-1-1","caveat":"FrontierCode 1.1 evaluates whether coding-agent changes are mergeable, using unit tests, maintainer-defined rubrics, and verifiers. Runs flagged for unfair internet use receive a zero score.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiercode-1.1?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FrontierCode 1.1","organization_count":1,"organizations":["llm_stats"],"rank":1245,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercode-1.1?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-frontiercs","caveat":"FrontierCS is a benchmark of frontier computer-science problems requiring deep theoretical understanding and rigorous multi-step reasoning at the edge of the field.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiercs?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FrontierCS","organization_count":1,"organizations":["llm_stats"],"rank":1246,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiercs?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-11683","caveat":"Comprises 220 expert-crafted queries with 11,543 source-attributed rubrics across six finance use cases, evaluating agents on open-ended analyst-style answers via rubric-based grading.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11683"],"document_share":0.0002723311546840959,"domain":"Finance","name":"FrontierFinance","organization_count":1,"organizations":["claire_radar"],"rank":1247,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.11683"},{"aliases":[],"benchmark_id":"llm-stats-frontiermath","caveat":"A benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians, covering most major branches of modern mathematics from number theory and real analysis to algebraic geometry and category theory.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiermath?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"FrontierMath","organization_count":1,"organizations":["llm_stats"],"rank":1248,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-frontiermath-tier-4-v2","caveat":"FrontierMath Tier 4 subset from the v2 evaluation release.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontiermath-tier-4-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"FrontierMath Tier 4 (v2)","organization_count":1,"organizations":["llm_stats"],"rank":1249,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontiermath-tier-4-v2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-frontierscience-olympiad","caveat":"FrontierScience Olympiad is a benchmark of olympiad-level scientific reasoning problems, testing expert understanding and multi-step reasoning across advanced natural-science domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-olympiad?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"FrontierScience Olympiad","organization_count":1,"organizations":["llm_stats"],"rank":1250,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-olympiad?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-frontierscience-research","caveat":"FrontierScience Research is a benchmark evaluating AI models on cutting-edge scientific research questions requiring deep domain expertise, multi-step reasoning, and synthesis of complex scientific concepts across disciplines.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-research?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FrontierScience Research","organization_count":1,"organizations":["llm_stats"],"rank":1251,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierscience-research?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-frontierswe","caveat":"FrontierSWE measures whether an agent can complete open-ended technical projects at the scale of hours to tens of hours, spanning systems optimization, large-scale code construction, and applied ML research. Performance is reported as a dominance score, where higher is better.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontierswe?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"FrontierSWE","organization_count":1,"organizations":["llm_stats"],"rank":1252,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontierswe?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-frontier-swe-impl","caveat":"FrontierSWE (Impl.) evaluates software engineering implementation ability and reports model ranking on implementation tasks. Lower rank is better.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/frontier-swe-impl?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"FrontierSWE (Impl.)","organization_count":1,"organizations":["llm_stats"],"rank":1253,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/frontier-swe-impl?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-fullstackbench-en","caveat":"English subset of FullStackBench for evaluating end-to-end software engineering and full-stack development capability.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-en?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FullStackBench en","organization_count":1,"organizations":["llm_stats"],"rank":1254,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-en?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-fullstackbench-zh","caveat":"Chinese subset of FullStackBench for evaluating end-to-end software engineering and full-stack development capability.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-zh?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"FullStackBench zh","organization_count":1,"organizations":["llm_stats"],"rank":1255,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/fullstackbench-zh?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-functionalmath","caveat":"A functional variant of the MATH benchmark that tests language models' ability to generalize reasoning patterns across different problem instances, revealing the reasoning gap between static and functional performance.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/functionalmath?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"FunctionalMATH","organization_count":1,"organizations":["llm_stats"],"rank":1256,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/functionalmath?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-25915","caveat":"FunPiQ is a benchmark for pixel-level fundus image quality assessment with pixel-level annotations for anatomical visibility, using a three-class scheme (good, usable, bad). It also introduces EFIQA-CP, an explainable method trained with pseudo-labels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25915"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"FunPiQ","organization_count":1,"organizations":["claire_radar"],"rank":1257,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25915"},{"aliases":["Future Rendering $\\neq$ Future Surface: A Benchmark and Dataset for Dynamic Surface Reconstruction Beyond the Observed Window"],"benchmark_id":"claire-radar-2607-21471","caveat":"A diagnostic benchmark for reconstructing dynamic surfaces beyond the observed time window. Methods train on 75% of a sequence and are scored by Chamfer distance on the held-out future frames.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21471"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"FutureSurf","organization_count":1,"organizations":["claire_radar"],"rank":1258,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21471"},{"aliases":[],"benchmark_id":"claire-radar-2608-25158","caveat":"Evaluates LLMs on discovering distinct crashes in 77 open-source software challenges using sanitizer-instrumented Docker harnesses and deterministic scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25158"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"FuzzingBrain-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1259,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.25158"},{"aliases":[],"benchmark_id":"claire-radar-2606-18989","caveat":"G-IdiomAlign is a gloss-pivoted benchmark for cross-lingual idiom alignment, with protocols for multiple-choice equivalence and gloss-contrastive generation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18989"],"document_share":0.0002723311546840959,"domain":"General AI","name":"G-IdiomAlign","organization_count":1,"organizations":["claire_radar"],"rank":1260,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18989"},{"aliases":[],"benchmark_id":"claire-radar-2608-01684","caveat":"GABench evaluates LLM agents on graph analysis tasks across three graph types and four task categories: graph retrieval, graph theory, graph machine learning, and graph open-ended QA. It provides 84 executable tools and 10,400 tasks with verifiable ground truth.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01684"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GABench","organization_count":1,"organizations":["claire_radar"],"rank":1261,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01684"},{"aliases":[],"benchmark_id":"opencompass-1599-gaia","caveat":"GAIA is a benchmark which aims at evaluating next-generation LLMs (LLMs with augmented capabilities due to added tooling, efficient prompting, access to search, etc), mading of more than 450 non-trivial question with an unambiguous answer, requiring different levels of tooling and autonomy to solve. GAIA 是一个旨在评估下一代LLMs（由于增加了工具、高效的提示、访问搜索等功能而具有增强能力的LLMs）的基准，由超过 450 个非平凡问题组成，这些问题有明确的答案，需要不同层次的工具和自主性来解决。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GAIA"],"document_share":0.0002723311546840959,"domain":"智能体","name":"GAIA","organization_count":1,"organizations":["opencompass_hub"],"rank":1262,"released":"2023-11-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GAIA"},{"aliases":[],"benchmark_id":"llm-stats-gaia2","caveat":"GAIA2 evaluates general-purpose AI agents on real-world, multi-step questions that require reasoning, tool use, and information retrieval.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gaia2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"GAIA2","organization_count":1,"organizations":["llm_stats"],"rank":1263,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gaia2?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-14068","caveat":"GAMA-Bench evaluates LLMs on gender-asymmetric moral framing across 1,298 paired conflict scenarios, measuring response differences in punitive, therapeutic, and blame dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14068"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GAMA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1264,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14068"},{"aliases":[],"benchmark_id":"claire-radar-2609-21293","caveat":"Evaluates coding agents on generating complete browser games from natural-language requirements with static and behavioral tests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21293"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GameASG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1265,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21293"},{"aliases":[],"benchmark_id":"claire-radar-2610-11129","caveat":"Game commentary is an open-ended generation task requiring multimodal perception, strategic reasoning, and contextual knowledge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11129"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GameCommBench","organization_count":1,"organizations":["claire_radar"],"rank":1266,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11129"},{"aliases":[],"benchmark_id":"claire-radar-2606-17861","caveat":"A benchmark of 140 Godot engine tasks across 15 game families. Coding agents must build playable games end-to-end, and the games are scored through replayed demonstrations and rubric-guided multimodal judging.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17861"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GameCraft-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1267,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.17861"},{"aliases":[],"benchmark_id":"claire-radar-2607-03525","caveat":"GameEngineBench evaluates coding agents on scoped C++ implementation tasks within Unreal Engine 5 projects, built from nine real-world game repositories. The 110 tasks span gameplay, multiplayer, AI, animation, UI, and other areas, requiring native C++ changes that compile and pass behavioral tests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03525"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"GameEngineBench","organization_count":1,"organizations":["claire_radar"],"rank":1268,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.03525"},{"aliases":[],"benchmark_id":"claire-radar-2609-25001","caveat":"A planned evaluation suite for gameplay across offline multiple-choice questions and online task environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25001"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GameHorizon-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1269,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.25001"},{"aliases":[],"benchmark_id":"claire-radar-2609-21562","caveat":"Evaluates coding agents on 72 gameplay-logic tasks in Godot projects using black-box behavioral assertions under deterministic simulation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21562"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"GameLogicBench","organization_count":1,"organizations":["claire_radar"],"rank":1270,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21562"},{"aliases":[],"benchmark_id":"llm-stats-gameworld","caveat":"GameWorld evaluates agents on interactive game environments, testing perception, planning, and sequential decision-making to accomplish in-game objectives.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gameworld?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"GameWorld","organization_count":1,"organizations":["llm_stats"],"rank":1271,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gameworld?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-21833","caveat":"Evaluates coding agents across three game development lifecycle tracks: generation, bug diagnosis and repair, and multi-turn optimization, using live interaction, behavioral tests, and product criteria.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.21833v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GameXpert-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1272,"released":"2026-08-22","source":"claire_radar","url":"http://arxiv.org/abs/2608.21833v1"},{"aliases":[],"benchmark_id":"opencompass-500-gaokao-bench","caveat":"GAOKAO-bench is an evaluation framework that utilizes Chinese high school entrance examination (GAOKAO) questions as a dataset to evaluate the language understanding and logical reasoning abilities of large language models. GAOKAO-bench是一个以中国高考题目为数据集，旨在提供和人类对齐的，直观，高效地测评大模型语言理解能力、逻辑推理能力的测评框架","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GAOKAO-Bench"],"document_share":0.0002723311546840959,"domain":"学科","name":"GAOKAO-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1273,"released":"2023-05-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GAOKAO-Bench"},{"aliases":[],"benchmark_id":"opencompass-1083-gaokao-mm","caveat":"GAOKAO-MM is a multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO), comprising of 8 subjects and 12 types of images, such as diagrams, function graphs, maps and photos. GAOKAO-MM 是一个基于中国高考的多模态基准，包含 8 个科目和 12 种图像类型，例如图表、函数图、地图和照片。GAOKAO-MM 源自本土中文语境，并对模型的能力设置了人类水平的要求，包括感知、理解、知识和推理。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GAOKAO-MM"],"document_share":0.0002723311546840959,"domain":"学科","name":"GAOKAO-MM","organization_count":1,"organizations":["opencompass_hub"],"rank":1274,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GAOKAO-MM"},{"aliases":["gas-sensor-drift-ml-benchmark"],"benchmark_id":"claire-radar-github-pesquisadoug-gas-sensor-drift-ml-benchmark","caveat":"A multi-task benchmark on the UCI Gas Sensor Array Drift dataset covering gas classification, concentration regression, clustering, and later-batch temporal evaluation. It fixes random hold-out and batches 1-5 to 6-10 split protocols, compares dummy, tree, SVM, and MLP baselines, and reports classification, regression, and clustering metrics.","document_count":1,"document_ids":["claire_radar:https://github.com/PesquisaDoug/gas-sensor-drift-ml-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Gas Sensor Drift ML Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1275,"released":"2026-09-17","source":"claire_radar","url":"https://github.com/PesquisaDoug/gas-sensor-drift-ml-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2606-18829","caveat":"GateMem evaluates memory governance in multi-principal shared-memory agents across medical, office, education, and household domains. It measures utility, access control, and active forgetting via checkpoints and a composite score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18829"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"GateMem","organization_count":1,"organizations":["claire_radar"],"rank":1276,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18829"},{"aliases":[],"benchmark_id":"claire-radar-2607-24889","caveat":"GAUGE evaluates agent-built financial valuation models against observed analyst practice using 56 facets, eight validity gates, and a failure-aware score over 196 tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24889"],"document_share":0.0002723311546840959,"domain":"Agents","name":"GAUGE","organization_count":1,"organizations":["claire_radar"],"rank":1277,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24889"},{"aliases":[],"benchmark_id":"claire-radar-2608-05948","caveat":"GAUGE evaluates physical fidelity of simulation engines and video world models using 22 task families covering rigid bodies, cables, textiles, and deformable objects, with real-world trajectories and calibrated metadata.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05948"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"GAUGE","organization_count":1,"organizations":["claire_radar"],"rank":1278,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05948"},{"aliases":[],"benchmark_id":"opencompass-2574-gauge","caveat":"GAUGE: A Physical‑Realism Evaluation Benchmark Leveraging real‑world replicated experimental data as ground‑truth references, it delivers systematic evaluation for world models and physics engines across rigid bodies, ropes, fabrics, and 3D soft bodies. 物理真实度评测基准GAUGE，以真实重复实验数据为对照标准，为世界模型与物理引擎提供跨刚体、绳索、织物和三维软体的系统化评测。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GAUGE"],"document_share":0.0002723311546840959,"domain":"其他","name":"GAUGE","organization_count":1,"organizations":["opencompass_hub"],"rank":1279,"released":"2026-08-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GAUGE"},{"aliases":[],"benchmark_id":"claire-radar-2606-14397","caveat":"GauntletBench evaluates agent generalisation across five professional web applications with 100 vision-intensive tasks, probing temporal perception, graphical understanding, and 3D reasoning via automated objective scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14397"],"document_share":0.0002723311546840959,"domain":"Agents","name":"GauntletBench","organization_count":1,"organizations":["claire_radar"],"rank":1280,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14397"},{"aliases":["Gaze Target Estimation Anywhere with Concepts"],"benchmark_id":"claire-radar-2608-11367","caveat":"Evaluates promptable gaze target estimation: given an image and a text or visual prompt identifying a subject, models output the subject's location, in/out-of-frame status, and a gaze target heatmap.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11367"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Gaze-Co","organization_count":1,"organizations":["claire_radar"],"rank":1281,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.11367"},{"aliases":["Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents"],"benchmark_id":"claire-radar-2608-06312","caveat":"Evaluates LLM capability for rule-intensive review of Chinese GB/T national standard documents, including error location, dimension, and type identification across 25 diagnostic error types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06312"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GB/T-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1282,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06312"},{"aliases":["Urban Boundaries, Social Barriers: A Benchmark and Vision-Centric Framework for Mapping Gated Communities and Equity Implications"],"benchmark_id":"claire-radar-2609-03804","caveat":"Evaluates multimodal recognition of gated versus open residential communities in China's Greater Bay Area using satellite imagery, text metadata, and structured attributes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03804"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GBA-GCs","organization_count":1,"organizations":["claire_radar"],"rank":1283,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03804"},{"aliases":[],"benchmark_id":"claire-radar-2608-14389","caveat":"GBU-Palm is a large-scale multimodal video dataset for palm presentation attack detection, containing 21,326 videos from 105 subjects across six acquisition environments, including bona fide, Print, and Replay attacks, with 6,310 synchronized RGB-NIR samples. It evaluates video architectures under environment-matched and held-out-environment protocols, using metrics such as true accept, true reject, false accept, and false reject rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14389"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GBU-Palm","organization_count":1,"organizations":["claire_radar"],"rank":1284,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14389"},{"aliases":[],"benchmark_id":"claire-radar-2607-14341","caveat":"GCA-Bench evaluates robotic grasping in scenarios requiring scene-level reasoning and semantic constraints, comparing large foundation models on complex action tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14341"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"GCA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1285,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.14341"},{"aliases":["GDP.pdf"],"benchmark_id":"gdp_pdf","caveat":"Vision-based knowledge work over 100 PDFs across 10 domains, reported no-tools. Published by Surge AI with a public dataset (huggingface.co/datasets/surgeai/GDP.pdf), harness (github.com/surge-ai/gdp-pdf) and paper (arXiv:2607.11192). The name resembles GDPval but the two are separate instruments and must not be merged.","document_count":1,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"GDP.pdf","organization_count":1,"organizations":["Anthropic"],"rank":1286,"released":"2026-04-14","source":"model_reports","url":"https://surgehq.ai/blog/gdp-pdf-can-100b-ai-models-master-the-documents-that-run-the-world"},{"aliases":[],"benchmark_id":"llm-stats-gdp-pdf","caveat":"GDP.pdf is a knowledge-work vision benchmark that evaluates models on economically valuable professional tasks presented as visual documents (PDFs), testing document-based reasoning, chart and table interpretation, and problem solving without tools.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdp-pdf?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"GDP.pdf","organization_count":1,"organizations":["llm_stats"],"rank":1287,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gdp-pdf?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-03764","caveat":"GDPevo evaluates agent self-evolution on real business tasks across 24 groups (240 tasks) in domains like CRM, ERP, finance, and healthcare. It uses rule hybridization to attribute test-time gains to training experience, with held-out test tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03764"],"document_share":0.0002723311546840959,"domain":"Agents","name":"GDPevo","organization_count":1,"organizations":["claire_radar"],"rank":1288,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03764"},{"aliases":[],"benchmark_id":"llm-stats-gdpval","caveat":"GDPval is an OpenAI benchmark evaluating AI models on economically valuable, real-world knowledge-work tasks spanning many professional occupations and industries.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdpval?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"GDPval","organization_count":1,"organizations":["llm_stats"],"rank":1289,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-gdpval-aa","caveat":"GDPval-AA evaluates AI agents on economically valuable professional knowledge-work tasks and reports performance as an Elo score.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdpval-aa?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"GDPval-AA","organization_count":1,"organizations":["llm_stats"],"rank":1290,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-aa?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-gdpval-aa-v2-raw-elo","caveat":"Agentic real-world work tasks","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/gdpval-aa"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"GDPval-AA v2 (Elo)","organization_count":1,"organizations":["artificial_analysis"],"rank":1291,"released":"2026-06-15","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/gdpval-aa"},{"aliases":[],"benchmark_id":"artificial-analysis-gdpval-aa-v2-normalized-score","caveat":"Agentic real-world work tasks","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/gdpval-aa"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"GDPval-AA v2 (normalized score)","organization_count":1,"organizations":["artificial_analysis"],"rank":1292,"released":"2026-06-15","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/gdpval-aa"},{"aliases":[],"benchmark_id":"llm-stats-gdpval-mm","caveat":"GDPval-MM is the multimodal variant of the GDPval benchmark, evaluating AI model performance on real-world economically valuable tasks that require processing and generating multimodal content including documents, slides, diagrams, spreadsheets, images, and other professional deliverables across diverse industries.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdpval-mm?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"GDPval-MM","organization_count":1,"organizations":["llm_stats"],"rank":1293,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-mm?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-gdpval-rubrics","caveat":"GDPval-Rubrics evaluates AI model performance on economically valuable knowledge work tasks drawn from the public GDPval dataset. It uses pointwise scoring based on public rubrics, with the environment aligned to the GDPval-AA scaffolding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gdpval-rubrics?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"GDPval-Rubrics","organization_count":1,"organizations":["llm_stats"],"rank":1294,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gdpval-rubrics?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-04111","caveat":"GEB-Bench evaluates models on abstract structural motifs (e.g., self-reference, strange loops) presented in multiple modalities (natural scenes, folk stories, math theorems, programmatic skeletons) with tasks probing cross-modal mapping.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04111"],"document_share":0.0002723311546840959,"domain":"Mathematics & Formal Science","name":"GEB-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1295,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.04111"},{"aliases":[],"benchmark_id":"claire-radar-2609-34271","caveat":"Although recent 3D generative models produce increasingly realistic assets, controllable 3D asset editing remains challenging.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34271"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GEdit3D-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1296,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34271"},{"aliases":[],"benchmark_id":"opencompass-1376-genai-bench","caveat":"GenAI-Bench is a benchmark designed to benchmark MLLMs’s ability in judging the quality of AI generative contents, containing over 40,000 human ratings to evaluate the performance of MLLMs on aligning with human preferences. GenAI-Bench用于衡量MLLM判断AI生成内容质量的能力，包含超过40000个人工评分用以评估大模型与人类偏好的一致性。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GenAI-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"GenAI-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1297,"released":"2024-06-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GenAI-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2606-04525","caveat":"GENEB evaluates frozen representations from 40 genomic foundation models across 100 DNA classification tasks in 13 functional categories, using a unified linear probing protocol with full-data, 10-shot, and 1-shot regimes. Primary metric is Matthews correlation coefficient, with rankings at overall, category, and task levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04525"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"GENEB","organization_count":1,"organizations":["claire_radar"],"rank":1298,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04525"},{"aliases":[],"benchmark_id":"llm-stats-genebench","caveat":"GeneBench is an evaluation focused on multi-stage scientific data analysis in genetics and quantitative biology. Tasks require reasoning about ambiguous or noisy data with minimal supervisory guidance, addressing realistic obstacles such as hidden confounders or QC failures, and correctly implementing and interpreting modern statistical methods.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/genebench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"GeneBench","organization_count":1,"organizations":["llm_stats"],"rank":1299,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-genebench-pro","caveat":"GeneBench-Pro is a research-level benchmark of 129 multi-stage computational-biology problems spanning genomics, quantitative biology, and translational biomedicine. Each problem gives the agent a messy dataset, brief context, and a target estimand, and requires navigating dependent inferential decision points to reach a verifiable answer.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/genebench-pro?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"GeneBench-Pro","organization_count":1,"organizations":["llm_stats"],"rank":1300,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/genebench-pro?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2081-general-bench","caveat":"General Bench is a set of universal evaluation benchmarks for multimodal large models, covering language, image, video, audio, and 3D five modalities, with a total of 145 skills, over 700 tasks, and 325800 samples. General-Bench 是一套面向多模态大模型的通用评测基准，涵盖语言、图像、视频、音频和 3D 五大模态，共计 145 项技能、700 余个任务，包含 325 800 条样本。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/General-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"General-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1301,"released":"2025-05-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/General-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2608-06972","caveat":"GEB evaluates embeddings by measuring answer-relevant content recoverable by a decoder, using a visual question-answering dataset with development and test splits, and scoring answer quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06972"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Generative Embedding Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1302,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.06972"},{"aliases":[],"benchmark_id":"claire-radar-2608-28021","caveat":"Evaluates LLM-generated Infrastructure-as-Code security using 100 scenarios across Terraform, CloudFormation, and Kubernetes, scanned by Checkov, Trivy, and KICS, with metrics including vulnerability density, schema validity, and LLM-judge review.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.28021v1"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"GenIaC-SecBench","organization_count":1,"organizations":["claire_radar"],"rank":1303,"released":"2026-08-28","source":"claire_radar","url":"http://arxiv.org/abs/2608.28021v1"},{"aliases":[],"benchmark_id":"claire-radar-2609-00525","caveat":"Evaluates relative object scale in image generation and editing tasks using a benchmark and evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00525"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GenScale","organization_count":1,"organizations":["claire_radar"],"rank":1304,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.00525"},{"aliases":[],"benchmark_id":"claire-radar-2605-29107","caveat":"GEO-Bench evaluates ranking manipulation attacks in generative engine optimization under a unified protocol, measuring effectiveness and stealth across five datasets against a fixed ranker, with open-source attack implementations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29107"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"GEO-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1305,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.29107"},{"aliases":["GeoAgent: Evaluating VLM Geolocalization Through Embodied Navigation"],"benchmark_id":"claire-radar-2608-29483","caveat":"GeoAgent evaluates VLM geolocalization performance through embodied navigation in Street View environments, comparing agentic exploration to static image baselines.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29483"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"GeoAgent","organization_count":1,"organizations":["claire_radar"],"rank":1306,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29483"},{"aliases":["GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks"],"benchmark_id":"claire-radar-2608-07411","caveat":"Evaluates LLMs on 17 geo-related subdatasets across 8 tasks covering knowledge, reasoning, and application, with inference and evaluation scripts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07411"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GeoBenchLLM","organization_count":1,"organizations":["claire_radar"],"rank":1307,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07411"},{"aliases":[],"benchmark_id":"claire-radar-2606-17246","caveat":"GeoDisaster is an operational geospatial disaster reasoning benchmark with 2,921 instances across 43 question types and five task families, integrating EO/GIS evidence and grounding answers in executable geospatial workflows.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17246"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GeoDisaster","organization_count":1,"organizations":["claire_radar"],"rank":1308,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.17246"},{"aliases":[],"benchmark_id":"claire-radar-2606-23669","caveat":"GeoFidelity-Bench evaluates segment-level geographic fidelity in text-to-image street-view generation using 7,117 Mapillary images across 109 road segments in 25 cities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23669"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GeoFidelity-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1309,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23669"},{"aliases":[],"benchmark_id":"claire-radar-2608-02315","caveat":"GEOID-Flood is a large-scale multi-modal benchmark for flood segmentation with over 14,000 tiles, co-registered SAR and optical data, and validated labels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02315"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GEOID-Flood","organization_count":1,"organizations":["claire_radar"],"rank":1310,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02315"},{"aliases":[],"benchmark_id":"claire-radar-2606-12821","caveat":"GeoNatureAgent Benchmark evaluates LLM agents on environmental geospatial analysis through structured tool calls to a self-hostable API. It includes 93 tasks across 18 categories, with metrics for accuracy and cost.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12821"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GeoNatureAgent Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1311,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.12821"},{"aliases":[],"benchmark_id":"claire-radar-2609-36082","caveat":"We introduce GeoOutageBench, a benchmark for assessing LLM-based geospatiotemporal KGQA for multimodal outage and resilience analysis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36082"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GeoOutageBench","organization_count":1,"organizations":["claire_radar"],"rank":1312,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.36082"},{"aliases":[],"benchmark_id":"claire-radar-2610-12881","caveat":"Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12881"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"GeoSpatialBench","organization_count":1,"organizations":["claire_radar"],"rank":1313,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.12881"},{"aliases":["GeoStore: Finding Small Storefronts in Large Scenes -- A Fine-Grained POI Localization Benchmark with Global-to-Local Asymmetric Matching"],"benchmark_id":"claire-radar-2609-02012","caveat":"Focuses on finding small storefronts in large scenes through a fine-grained POI localization benchmark with global-to-local asymmetric matching.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02012"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"GeoStore","organization_count":1,"organizations":["claire_radar"],"rank":1314,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02012"},{"aliases":[],"benchmark_id":"claire-radar-2606-24829","caveat":"GeoT2V-Bench evaluates 3D consistency in camera-prompted text-to-video models via 3D reconstruction, using metrics like static rendering error and flow agreement.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24829"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GeoT2V-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1315,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24829"},{"aliases":[],"benchmark_id":"claire-radar-2609-12606","caveat":"GeoVAD-Bench diagnoses visual chain-of-thought for geometry problem solving along five dimensions: perception, auxiliary quality, utilization, deductive reasoning, and final correctness, with controlled No-Aux, Auto-Aux, and GT-Aux intervention settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12606"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"GeoVAD-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1316,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12606"},{"aliases":[],"benchmark_id":"llm-stats-giantsteps-tempo","caveat":"A dataset for tempo estimation in electronic dance music containing 664 2-minute audio previews from Beatport, annotated from user corrections for evaluating automatic tempo estimation algorithms.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/giantsteps-tempo?top_n=500"],"document_share":0.0002723311546840959,"domain":"audio","name":"GiantSteps Tempo","organization_count":1,"organizations":["llm_stats"],"rank":1317,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/giantsteps-tempo?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-01778","caveat":"Reliable evaluation of image forgery localization (IFL) requires assessing models under diverse distribution changes, yet existing benchmarks often cover limited manipulation conditions or entangle multiple factors in cross-dataset evaluat…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.01778"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GIFTBench","organization_count":1,"organizations":["claire_radar"],"rank":1318,"released":"2026-10-01","source":"claire_radar","url":"https://arxiv.org/abs/2610.01778"},{"aliases":[],"benchmark_id":"claire-radar-2608-01645","caveat":"GISAgentBench evaluates LLM agents on multi-step GIS tasks from practitioner sources, with 349 tasks and executable reference trajectories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01645"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GISAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":1319,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01645"},{"aliases":[],"benchmark_id":"claire-radar-2606-08036","caveat":"GIScholarBench evaluates LLM overconfidence in GIS research across three tasks: metadata retrieval, literature linking, and research direction generation, using 10,865 papers from 25 GIScience journals.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08036"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GIScholarBench","organization_count":1,"organizations":["claire_radar"],"rank":1320,"released":"2026-06-06","source":"claire_radar","url":"https://arxiv.org/abs/2606.08036"},{"aliases":[],"benchmark_id":"opencompass-1865-gitgoodbench","caveat":"GitGoodBench Lite is a subset of 900 samples for evaluating the performance of AI agents in resolving git tasks (see Supported Scenarios). GitGoodBench Lite是一个包含900个样本的子集，用于评估AI智能体在解决git任务方面的性能（参见支持的场景）。数据集中的样本在编程语言Python、Java和Kotlin以及样本类型合并冲突解决和文件提交语法之间均匀分布。因此，该数据集包含每种样本类型和编程语言各150个样本。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GitGoodBench"],"document_share":0.0002723311546840959,"domain":"代码","name":"GitGoodBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1321,"released":"2025-05-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GitGoodBench"},{"aliases":[],"benchmark_id":"llm-stats-global-piqa","caveat":"Global PIQA is a multilingual commonsense reasoning benchmark that evaluates physical interaction knowledge across 100 languages and cultures. It tests AI systems' understanding of physical world knowledge in diverse cultural contexts through multiple choice questions about everyday situations requiring physical commonsense.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/global-piqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"physics","name":"Global PIQA","organization_count":1,"organizations":["llm_stats"],"rank":1322,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/global-piqa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-global-mmlu","caveat":"A comprehensive multilingual benchmark covering 42 languages that addresses cultural and linguistic biases in evaluation, with improved translation quality and culturally sensitive question subsets.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Global-MMLU","organization_count":1,"organizations":["llm_stats"],"rank":1323,"released":"2024-12-04","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-global-mmlu-lite","caveat":"A lightweight version of Global MMLU benchmark that evaluates language models across multiple languages while addressing cultural and linguistic biases in multilingual evaluation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu-lite?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Global-MMLU-Lite","organization_count":1,"organizations":["llm_stats"],"rank":1324,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/global-mmlu-lite?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-24636","caveat":"GlobalDentBench evaluates LLM clinical reasoning in dentistry with 8,978 expert-validated questions across 14 specialties and 88 countries, covering multiple-choice, short-answer, and case-based formats at three reasoning levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24636"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"GlobalDentBench","organization_count":1,"organizations":["claire_radar"],"rank":1325,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24636"},{"aliases":[],"benchmark_id":"claire-radar-2606-08194","caveat":"GlobeAudio evaluates audio-language models on naturalistic audio understanding across six languages. It includes 5,637 multiple-choice questions with naturally occurring audio, testing auditory reasoning and cultural interpretation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08194"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GlobeAudio","organization_count":1,"organizations":["claire_radar"],"rank":1326,"released":"2026-06-06","source":"claire_radar","url":"https://arxiv.org/abs/2606.08194"},{"aliases":[],"benchmark_id":"claire-radar-2606-06881","caveat":"GlucoFM-Bench evaluates time-series foundation models for blood glucose forecasting across 15 datasets, protocols including zero-shot, few-shot, and full-shot, with metrics like RMSE.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06881"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GlucoFM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1327,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.06881"},{"aliases":[],"benchmark_id":"opencompass-1927-gmaimmbench","caveat":"GMAI-MMBench is the most comprehensive and structured benchmark developed to evaluate Large Vision-Language Models (LVLMs) in general medical artificial intelligence (GMAI) applications. It addresses the limitations of existing benchmarks that typically focus on narrow domains and lack perceptual di GMAI-MMBench 是目前最全面、结构化最完善的通用医学人工智能（GMAI）基准数据集，专为评估大规模视觉语言模型（LVLMs）在医疗领域中的表现而设计。针对现有医学基准通常局限于特定学科、感知粒度单一的问题，GMAI-MMBench 从285个真实医学数据集构建而成，涵盖39种医学影像模态、18个临床任务、18个医学科室，以及4种不同的感知粒度。所有任务采用视觉问答（VQA）形式组织，具备良好的交互性和通用性。其独特的词汇树结构允许用户针对具体研究需求灵活定制评估路径，支持多样化的评测场景。我们对50个主流LVLMs进行了系统测试，发现即使是目前最先进的 GPT-4o，其准确率也仅为5","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GMAIMMBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"GMAIMMBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1328,"released":"2024-08-31","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GMAIMMBench"},{"aliases":[],"benchmark_id":"claire-radar-2610-05387","caveat":"Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05387"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GNN-CB","organization_count":1,"organizations":["claire_radar"],"rank":1329,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05387"},{"aliases":[],"benchmark_id":"opencompass-1128-gorilla","caveat":"Gorilla enables LLMs to use tools by invoking APIs. Given a natural language query, Gorilla comes up with the semantically- and syntactically- correct API to invoke. Gorilla 使大语言模型能够通过调用 API 使用工具。针对自然语言查询，Gorilla 能够生成语义和语法上正确的 API 调用。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Gorilla"],"document_share":0.0002723311546840959,"domain":"智能体","name":"Gorilla","organization_count":1,"organizations":["opencompass_hub"],"rank":1330,"released":"2023-05-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Gorilla"},{"aliases":[],"benchmark_id":"llm-stats-gorilla-benchmark-api-bench","caveat":"APIBench, a comprehensive dataset of over 11,000 instruction-API pairs from HuggingFace, TorchHub, and TensorHub APIs for evaluating language models' ability to generate accurate API calls.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gorilla-benchmark-api-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Gorilla Benchmark API Bench","organization_count":1,"organizations":["llm_stats"],"rank":1331,"released":"2023-05-24","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gorilla-benchmark-api-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-govreport","caveat":"A long document summarization dataset consisting of reports from government research agencies including Congressional Research Service and U.S. Government Accountability Office, with significantly longer documents and summaries than other datasets.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/govreport?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"GovReport","organization_count":1,"organizations":["llm_stats"],"rank":1332,"released":"2021-04-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/govreport?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-gpqa","caveat":"A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gpqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"physics","name":"GPQA","organization_count":1,"organizations":["llm_stats"],"rank":1333,"released":"2023-11-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1135-gpqa","caveat":"GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. GPQA 包含 448 道由生物学、物理学和化学领域专家撰写的多项选择题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GPQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"GPQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1334,"released":"2023-11-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GPQA"},{"aliases":[],"benchmark_id":"llm-stats-gpqa-biology","caveat":"Biology subset of GPQA, containing challenging multiple-choice questions written by domain experts in biology. These Google-proof questions require graduate-level knowledge and reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gpqa-biology?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"GPQA Biology","organization_count":1,"organizations":["llm_stats"],"rank":1335,"released":"2023-11-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-biology?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-gpqa-chemistry","caveat":"Chemistry subset of GPQA, containing challenging multiple-choice questions written by domain experts in chemistry. These Google-proof questions require graduate-level knowledge and reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gpqa-chemistry?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"GPQA Chemistry","organization_count":1,"organizations":["llm_stats"],"rank":1336,"released":"2023-11-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-chemistry?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-gpqa-diamond","caveat":"Scientific reasoning","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/gpqa-diamond"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"GPQA Diamond","organization_count":1,"organizations":["artificial_analysis"],"rank":1337,"released":"2023-11-20","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/gpqa-diamond"},{"aliases":[],"benchmark_id":"llm-stats-gpqa-physics","caveat":"Physics subset of GPQA, containing challenging multiple-choice questions written by domain experts in physics. These Google-proof questions require graduate-level knowledge and reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gpqa-physics?top_n=500"],"document_share":0.0002723311546840959,"domain":"physics","name":"GPQA Physics","organization_count":1,"organizations":["llm_stats"],"rank":1338,"released":"2023-11-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gpqa-physics?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1740-gpt-imgeval","caveat":"GPT-ImgEval, quantitatively and qualitatively diagnoses GPT-4o's performance across three critical dimensions: (1) generation quality, (2) editing proficiency, and (3) world knowledge-informed semantic synthesis. GPT-ImgEval，从三个关键维度对GPT-4o的性能进行定量和定性诊断：（1）生成质量，（2）编辑能力，以及（3）基于世界知识的语义合成能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GPT-ImgEval"],"document_share":0.0002723311546840959,"domain":"多模态","name":"GPT-ImgEval","organization_count":1,"organizations":["opencompass_hub"],"rank":1339,"released":"2025-04-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GPT-ImgEval"},{"aliases":[],"benchmark_id":"claire-radar-2606-28514","caveat":"GPTNT evaluates multimodal agents on real-time collaborative bomb defusal in the game Keep Talking and Nobody Explodes, requiring asynchronous communication under time pressure and information asymmetry. Success is measured by defusing procedurally generated bombs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.28514"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GPTNT","organization_count":1,"organizations":["claire_radar"],"rank":1340,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.28514"},{"aliases":[],"benchmark_id":"claire-radar-github-jyotydivya-gpu-accelerated-ml-inference-benchmark","caveat":"Compares inference latency, throughput, and GPU utilization for PyTorch CPU, PyTorch CUDA, and TensorRT across batch sizes.","document_count":1,"document_ids":["claire_radar:https://github.com/jyotydivya/GPU-Accelerated-ML-Inference-Benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GPU-Accelerated ML Inference Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1341,"released":"2026-08-23","source":"claire_radar","url":"https://github.com/jyotydivya/GPU-Accelerated-ML-Inference-Benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-35639","caveat":"Writing fast GPU code for physical simulation is difficult: implementations must preserve numerical accuracy while handling irregular data access, synchronization, and iterative solvers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35639"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GPUPhysBench","organization_count":1,"organizations":["claire_radar"],"rank":1342,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35639"},{"aliases":[],"benchmark_id":"claire-radar-2606-16151","caveat":"GRACE evaluates step-level faithfulness of chain-of-thought reasoning in context-grounded tasks, with human annotations and a taxonomy of error categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16151"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GRACE","organization_count":1,"organizations":["claire_radar"],"rank":1343,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16151"},{"aliases":["baoyan-agent-benchmark"],"benchmark_id":"claire-radar-github-yeyuxin777-baoyan-agent-benchmark","caveat":"Evaluates Chinese-language agent products on graduate admissions advising, covering recommendation, planning, and two-stage information-update tasks with per-task rubrics.","document_count":1,"document_ids":["claire_radar:https://github.com/yeyuxin777/baoyan-agent-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Graduate Admissions Agent Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1344,"released":"2026-10-12","source":"claire_radar","url":"https://github.com/yeyuxin777/baoyan-agent-benchmark"},{"aliases":[],"benchmark_id":"opencompass-1098-grailqa","caveat":"GrailQA is a new large-scale, high-quality dataset for question answering on knowledge bases (KBQA) on Freebase with 64,331 questions annotated with both answers and corresponding logical forms in different syntax (i.e., SPARQL, S-expression, etc.). GrailQA 是一个大规模高质量数据集，用于知识库问答，包含 64,331 个问题，并附有答案和不同语法的相应逻辑形式。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GrailQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"GrailQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1345,"released":"2021-02-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GrailQA"},{"aliases":["Reliability-Prioritized Fine-Grained Generation in Multimodal Large"],"benchmark_id":"claire-radar-2606-29573","caveat":"A granularity-aware evaluation of whether multimodal models produce visually correct descriptions at reliably specific category levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29573"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GranFact","organization_count":1,"organizations":["claire_radar"],"rank":1346,"released":"2026-06-28","source":"claire_radar","url":"https://arxiv.org/abs/2606.29573"},{"aliases":[],"benchmark_id":"claire-radar-2605-31031","caveat":"GraphARC is a benchmark for abstract reasoning on graphs, generalizing ARC few-shot learning. No artifacts are provided in this article.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31031"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GraphARC","organization_count":1,"organizations":["claire_radar"],"rank":1347,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31031"},{"aliases":[],"benchmark_id":"claire-radar-2606-11562","caveat":"GraphInfer-Bench evaluates LLMs on graph inference tasks where answers reside in no single node or path, covering five task types over six real-world graphs with 42,000 samples. Tasks include masked-node prediction, edge inference, theme description, outlier detection, and community partition.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11562"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GraphInfer-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1348,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.11562"},{"aliases":[],"benchmark_id":"claire-radar-2609-39777","caveat":"LLM-based multi-agent systems coordinate specialized reasoning through aggregation, interaction, and adaptive control, yet their potential for graph learning remains unexplored.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39777"],"document_share":0.0002723311546840959,"domain":"Agents","name":"GraphMAS","organization_count":1,"organizations":["claire_radar"],"rank":1349,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39777"},{"aliases":[],"benchmark_id":"claire-radar-2607-24878","caveat":"GraphRareBench evaluates phenotype-driven rare-disease ranking with 2,365 cases, 18,093 target-confounder pairs, and auditable evidence records.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24878"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GraphRareBench","organization_count":1,"organizations":["claire_radar"],"rank":1350,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24878"},{"aliases":[],"benchmark_id":"claire-radar-2608-06769","caveat":"GraphVerse evaluates multimodal large language models on visual graph reasoning, covering perception, reasoning, and text-based graph reasoning in single and paired image settings, with process-sensitive scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06769"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GraphVerse","organization_count":1,"organizations":["claire_radar"],"rank":1351,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.06769"},{"aliases":[],"benchmark_id":"llm-stats-graphwalks","caveat":"GraphWalks is a synthetic multi-hop long-context reasoning benchmark in which a model is given an edge-list representation of a graph and must traverse it to find neighboring nodes (via breadth-first search) or parent nodes for a given start node. Performance is reported as F1 of the model-predicted answer set versus the ground truth.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"GraphWalks","organization_count":1,"organizations":["llm_stats"],"rank":1352,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-graphwalks-bfs-1m","caveat":"GraphWalks BFS variant evaluated on 1M-token contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-1m?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"Graphwalks BFS 1M","organization_count":1,"organizations":["llm_stats"],"rank":1353,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-1m?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-graphwalks-bfs-128k","caveat":"A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length under 128k tokens, returning nodes reachable at specified depths.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3C128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Graphwalks BFS <128k","organization_count":1,"organizations":["llm_stats"],"rank":1354,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3C128k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-graphwalks-bfs-128k-2","caveat":"A graph reasoning benchmark that evaluates language models' ability to perform breadth-first search (BFS) operations on graphs with context length over 128k tokens, testing long-context reasoning capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3E128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"Graphwalks BFS >128k","organization_count":1,"organizations":["llm_stats"],"rank":1355,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-bfs-%3E128k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-graphwalks-parents-128k","caveat":"A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length under 128k tokens, requiring understanding of graph structure and edge relationships.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3C128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Graphwalks parents <128k","organization_count":1,"organizations":["llm_stats"],"rank":1356,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3C128k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-graphwalks-parents-128k-2","caveat":"A graph reasoning benchmark that evaluates language models' ability to find parent nodes in graphs with context length over 128k tokens, testing long-context reasoning and graph structure understanding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3E128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"Graphwalks parents >128k","organization_count":1,"organizations":["llm_stats"],"rank":1357,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/graphwalks-parents-%3E128k?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-18752","caveat":"GreekBarRetrieval is a retrieval benchmark for Greek statutory articles, comprising bar-exam questions with case facts and candidate articles.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.18752"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GreekBarRetrieval","organization_count":1,"organizations":["claire_radar"],"rank":1358,"released":"2026-08-19","source":"claire_radar","url":"https://arxiv.org/abs/2608.18752"},{"aliases":[],"benchmark_id":"claire-radar-github-guptabhishekumar-grit-benchmark","caveat":"Evaluates LLM planners on robot skill execution under turbulence, scoring recovery from disruptions across tasks, tiers, and seeds.","document_count":1,"document_ids":["claire_radar:https://github.com/guptabhishekumar/grit-benchmark"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"GRIT","organization_count":1,"organizations":["claire_radar"],"rank":1359,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/guptabhishekumar/grit-benchmark"},{"aliases":[],"benchmark_id":"opencompass-2027-groundingsuite","caveat":"GroundingSuite is designed to test the localization capabilities of multimodal models. It created 3,720 pixel-level data entries based on COCO Unlabeled images. This dataset covers four dimensions: Stuff Class Object, Multi-Object, Part-Level Object, and Single Object. GroundingSuite 用来测试多模态模型的定位能力。它通过半自动标注和人工筛选在COCO Unlabel的图片基础上创建了3720条pixel-level的数据，覆盖Stuff Class Object, Multi Object, Part Level Object, Single Object四个维度，是一个全面评测多模态模型定位能力的测试集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GroundingSuite"],"document_share":0.0002723311546840959,"domain":"多模态","name":"GroundingSuite","organization_count":1,"organizations":["opencompass_hub"],"rank":1360,"released":"2025-07-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GroundingSuite"},{"aliases":[],"benchmark_id":"claire-radar-huggingface-eigenformai-groundtruth-dynamic-benchmarking","caveat":"Geology question sets and grading rubrics for evaluating LLMs on real-world geological reasoning, with 153 questions across four configs, each grounded in source corpora with evidence locators and a harness for generation and scoring.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Groundtruth Dynamic Benchmarking","organization_count":1,"organizations":["claire_radar"],"rank":1361,"released":"2026-08-22","source":"claire_radar","url":"https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking"},{"aliases":[],"benchmark_id":"llm-stats-groundui-1k","caveat":"A subset of GroundUI-18K for UI grounding evaluation, where models must predict action coordinates on screenshots based on single-step instructions across web, desktop, and mobile platforms.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/groundui-1k?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"GroundUI-1K","organization_count":1,"organizations":["llm_stats"],"rank":1362,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/groundui-1k?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-04184","caveat":"GroupToM-Bench is a multimodal benchmark for evaluating group-level theory of mind through a seven-level cognitive audit framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04184"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GroupToM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1363,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.04184"},{"aliases":[],"benchmark_id":"claire-radar-2605-25200","caveat":"A benchmark for multi-user, multi-turn travel planning with 650 tasks and a synchronous group-chat sandbox, evaluating elicitation, coordination, and fairness-aware planning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25200"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GroupTravelBench","organization_count":1,"organizations":["claire_radar"],"rank":1364,"released":"2026-05-24","source":"claire_radar","url":"https://arxiv.org/abs/2605.25200"},{"aliases":[],"benchmark_id":"llm-stats-gsm-8k-cot","caveat":"Grade School Math 8K with Chain-of-Thought prompting, featuring 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gsm-8k-%28cot%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"GSM-8K (CoT)","organization_count":1,"organizations":["llm_stats"],"rank":1365,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm-8k-%28cot%29?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1272-gsm1k","caveat":"GSM1k is meant for evaluating LLM's math reasoning ability. It mirrors the style and complexity of the established GSM8k benchmark while consider the problem of data-leaking and overfitting GSM1k可用于评估LLM的数学推理能力；它与GSM8k保持了风格和复杂性的一致，同时考虑了数据泄露和过拟合的问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GSM1k"],"document_share":0.0002723311546840959,"domain":"数学","name":"GSM1k","organization_count":1,"organizations":["opencompass_hub"],"rank":1366,"released":"2024-05-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GSM1k"},{"aliases":[],"benchmark_id":"llm-stats-gsm8k","caveat":"Grade School Math 8K, a dataset of 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gsm8k?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"GSM8k","organization_count":1,"organizations":["llm_stats"],"rank":1367,"released":"2021-04-01","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-535-gsm8k","caveat":"GSM8K is a dataset of 8,500 high quality linguistically diverse grade school math word problems created by human problem writers. The dataset is segmented into 7,500 training problems and 1,000 test problems. These problems take between 2 and 8 steps to solve, and solutions primarily involve performing a sequence of elementary calculations using basic arithmetic operations (+ − × ÷) to reach the final answer. GSM8K 是一个包含 8,500 个高质量、语言多样化的小学数学单词问题的数据集，由人类问题编写者创建。该数据集分为 7,500 个训练问题和 1,000 个测试问题。这些问题的解题步骤在 2 到 8 步之间，解题过程主要涉及使用基本算术运算（+ - × ÷）进行一连串的基本计算，从而得出最终答案。一个聪明的初中生应该能够解决每一个问题。它可用于多步数学推理。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GSM8K"],"document_share":0.0002723311546840959,"domain":"数学","name":"GSM8K","organization_count":1,"organizations":["opencompass_hub"],"rank":1368,"released":"2021-04-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GSM8K"},{"aliases":[],"benchmark_id":"llm-stats-gsm8k-chat","caveat":"Grade School Math 8K adapted for chat format evaluation, featuring 8.5K high-quality linguistically diverse grade school math word problems requiring multi-step reasoning and elementary arithmetic operations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/gsm8k-chat?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"GSM8K Chat","organization_count":1,"organizations":["llm_stats"],"rank":1369,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/gsm8k-chat?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2233-gsm8k-v","caveat":"GSM8K-V is a multi-image, purely visual mathematical reasoning benchmark built by rendering GSM8K textual problems into visual scenes. It exposes substantial gaps in current vision-language models’ visual reasoning despite their near-saturated performance on text-based GSM8K. GSM8K-V 是一个多图视觉数学推理基准，通过将 GSM8K 的文本题系统性转换为多场景图像构建而成。它揭示了当前 VLMs 在视觉数学推理上的表现与其在文本数学推理上的表现之间存在显著差距。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GSM8K-V"],"document_share":0.0002723311546840959,"domain":"多模态","name":"GSM8K-V","organization_count":1,"organizations":["opencompass_hub"],"rank":1370,"released":"2025-09-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GSM8K-V"},{"aliases":[],"benchmark_id":"claire-radar-2609-12408","caveat":"A large-scale industrial vision benchmark for standard operating procedure understanding in petrochemical unloading, built from 50,325 images across 64 stations with 321,432 bounding boxes. It links nine macroscopic procedural steps and 15 microscopic operational states across a core joint step-state detection task and a diagnostic frame-level step classification task, using a strict cross-site split.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12408"],"document_share":0.0002723311546840959,"domain":"Logistics & Operations","name":"GSO-Net","organization_count":1,"organizations":["claire_radar"],"rank":1371,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12408"},{"aliases":[],"benchmark_id":"claire-radar-2608-05747","caveat":"GST-Bench is a VQA benchmark for global spatial intelligence in video understanding, covering synthetic videos and human-verified questions. It evaluates VLMs' ability to infer spatial relations from novel viewpoints and map egocentric observations to top-down views.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05747"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"GST-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1372,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05747"},{"aliases":[],"benchmark_id":"opencompass-1328-gta","caveat":"GTA is meant for LLM's tool-use evaluations under real-world scenarios, including 229 real-world tasks and executable tool chains. GTA用于评估LLM调用工具解决实际问题的能力，由229个真实任务和可执行工具链组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/GTA"],"document_share":0.0002723311546840959,"domain":"智能体","name":"GTA","organization_count":1,"organizations":["opencompass_hub"],"rank":1373,"released":"2024-07-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/GTA"},{"aliases":[],"benchmark_id":"claire-radar-2606-03144","caveat":"GTBench is a curriculum-grounded benchmark for evaluating LLMs as mathematical research assistants in graph theory, with problems across three difficulty groups.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03144"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GTBench","organization_count":1,"organizations":["claire_radar"],"rank":1374,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03144"},{"aliases":[],"benchmark_id":"claire-radar-2608-08994","caveat":"Guardian Crawler is a retrieval-first testbed for knowledge discovery over synthetic web-like corpora, combining BM25 retrieval with reranking and constrained generation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08994"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Guardian Crawler","organization_count":1,"organizations":["claire_radar"],"rank":1375,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.08994"},{"aliases":[],"benchmark_id":"claire-radar-2607-20982","caveat":"GuardianAgentBench (GABench) is a benchmark of 580 agent scenarios across six domains, evaluated on three production frameworks (LangChain, LlamaIndex, Vectara), with multi-stage validation and five adversarial attack modes to assess tool-use correctness and safety.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20982"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GuardianAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":1376,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.20982"},{"aliases":[],"benchmark_id":"claire-radar-2608-21928","caveat":"3,024 instruction-scene examples organized as same-scene Safe/Unsafe contrastive pairs across hazard categories, based on international safety standards. Evaluates VLM safety reasoning under latent contextual risk.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.21928v1"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"GuardianBench","organization_count":1,"organizations":["claire_radar"],"rank":1377,"released":"2026-08-22","source":"claire_radar","url":"http://arxiv.org/abs/2608.21928v1"},{"aliases":[],"benchmark_id":"claire-radar-2609-00048","caveat":"GUI-CC evaluates whether GUI world models maintain contextual consistency in multi-step agent interactions using offline reference trajectories and online probing agents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00048"],"document_share":0.0002723311546840959,"domain":"Agents","name":"GUI-CC","organization_count":1,"organizations":["claire_radar"],"rank":1378,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.00048"},{"aliases":["GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding"],"benchmark_id":"claire-radar-2608-21832","caveat":"994-item benchmark of contrastive instruction pairs over seven spatial relations in GUIs, isolating whether vision-language models bind relational language to UI elements correctly.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.21832v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GUI-Primitives","organization_count":1,"organizations":["claire_radar"],"rank":1379,"released":"2026-08-22","source":"claire_radar","url":"http://arxiv.org/abs/2608.21832v1"},{"aliases":[],"benchmark_id":"claire-radar-2607-02991","caveat":"GuideMe is a benchmark for evaluating multimodal large language models on streaming video task guidance. It includes 2,458 videos (223.7 hours) with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance. Assessment uses temporal-semantic matching, behavioral classification, and LLM-as-a-Judge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02991"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GuideMe","organization_count":1,"organizations":["claire_radar"],"rank":1380,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.02991"},{"aliases":[],"benchmark_id":"claire-radar-2605-29532","caveat":"Evaluates exploratory GUI testing agents on 61 Android apps with 508 preset defects, using an open-set evaluator that decomposes trajectories into independently diagnosable capabilities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29532"],"document_share":0.0002723311546840959,"domain":"Agents","name":"GUITestScape","organization_count":1,"organizations":["claire_radar"],"rank":1381,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29532"},{"aliases":[],"benchmark_id":"opencompass-2035-gym4real","caveat":"Gym4ReaL is a benchmark suite designed to evaluate reinforcement learning algorithms in real-world scenarios, addressing challenges such as non-stationarity, partial observability, and large state-action spaces. Gym4ReaL 是一个用于评估强化学习算法在真实世界场景中表现的基准套件，涵盖非平稳性、部分可观测性和大状态-动作空间等挑战。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Gym4ReaL"],"document_share":0.0002723311546840959,"domain":"其他","name":"Gym4ReaL","organization_count":1,"organizations":["opencompass_hub"],"rank":1382,"released":"2025-06-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Gym4ReaL"},{"aliases":["GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models"],"benchmark_id":"claire-radar-2609-18384","caveat":"An evaluation of large language models' cultural value orientations across the two Inglehart-Welzel axes, covering several domains and roles, to quantify their alignment along traditional vs secular-rational and survival vs self-expression values.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18384"],"document_share":0.0002723311546840959,"domain":"General AI","name":"GYROval","organization_count":1,"organizations":["claire_radar"],"rank":1383,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18384"},{"aliases":[],"benchmark_id":"claire-radar-2606-09461","caveat":"H2HMem is a benchmark for evaluating memory capabilities of agents in human-human multimodal interactions, covering dyadic and multi-party conversations with tasks in memory recall, reasoning, and application.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09461"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"H2HMem","organization_count":1,"organizations":["claire_radar"],"rank":1384,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09461"},{"aliases":[],"benchmark_id":"claire-radar-2608-13049","caveat":"H2R-Bench evaluates cross-embodiment human-to-robot manipulation video generation. Models convert egocentric human demonstrations into robot manipulation videos under specified target embodiments. Scoring covers five dimensions: goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and general video quality, aggregated into H2RCore.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13049"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"H2R-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1385,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13049"},{"aliases":[],"benchmark_id":"claire-radar-2609-24778","caveat":"A benchmark proposed for evaluating human-to-robot transfer methods in a real-to-sim setting.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24778"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"H2RBench","organization_count":1,"organizations":["claire_radar"],"rank":1386,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24778"},{"aliases":[],"benchmark_id":"claire-radar-2608-22103","caveat":"Adapts Terminal Bench with hack-verifiable environments to automatically detect reward hacking in agent trajectories.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22103v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Hack-Verifiable Terminal Bench","organization_count":1,"organizations":["claire_radar"],"rank":1387,"released":"2026-08-22","source":"claire_radar","url":"http://arxiv.org/abs/2608.22103v1"},{"aliases":[],"benchmark_id":"claire-radar-2606-22778","caveat":"Evaluates retrieval architectures and efficiency settings (dimensionality reduction, quantization, reranking) across 35 benchmarks and 551 tasks in 43 languages, with unified conditions and metrics for BM25, dense, sparse, late interaction, and reranker models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22778"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HAKARI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1388,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.22778"},{"aliases":[],"benchmark_id":"claire-radar-2606-01132","caveat":"HakushoBench is a Japanese chart and table VQA benchmark built from 33 governmental white papers, containing 2,053 images across over 10 image types with manually annotated QA pairs. It evaluates vision-language models on deep holistic understanding of charts and tables.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01132"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HakushoBench","organization_count":1,"organizations":["claire_radar"],"rank":1389,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01132"},{"aliases":[],"benchmark_id":"llm-stats-hallusion-bench","caveat":"A comprehensive benchmark designed to evaluate image-context reasoning in large visual-language models (LVLMs) by challenging models with 346 images and 1,129 carefully crafted questions to assess language hallucination and visual illusion","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hallusion-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Hallusion Bench","organization_count":1,"organizations":["llm_stats"],"rank":1390,"released":"2023-10-23","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hallusion-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1355-hallusionbench","caveat":"HallusionBench is a comprehensive benchmark designed for the evaluation of image-context reasoning, comprising 346 images paired with 1129 questions. HallusionBench是一个专为评估图像上下文推理而设计的综合基准测试，包括346张图像和1129个问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HallusionBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"HallusionBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1391,"released":"2023-10-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HallusionBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-20219","caveat":"HalluTruthQA is a fine-grained benchmark for Arabic QA hallucination detection, localization, and explanation, containing 2,400 expert-curated examples across four knowledge domains with span-level and explanation annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20219"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HalluTruthQA","organization_count":1,"organizations":["claire_radar"],"rank":1392,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.20219"},{"aliases":[],"benchmark_id":"opencompass-1121-halueval","caveat":"HaluEval evaluates the performance of LLMs in recognizing hallucination. It includes 5,000 general user queries with ChatGPT responses and 30,000 task-specific examples from three tasks, i.e., question answering, knowledge-grounded dialogue, and text summarization. HaluEval用于评估大语言模型识别幻觉的能力，包含 5,000 条普通用户查询及 ChatGPT 的回答，以及来自三个任务的 30,000 个特定任务示例，即问答、基于知识的对话和文本摘要。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HaluEval"],"document_share":0.0002723311546840959,"domain":"安全","name":"HaluEval","organization_count":1,"organizations":["opencompass_hub"],"rank":1393,"released":"2023-10-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HaluEval"},{"aliases":[],"benchmark_id":"claire-radar-2607-04845","caveat":"HamQASBench evaluates Quantum Architecture Search methods across 11 molecules organized in five structural tiers, using energy accuracy, per-qubit entanglement, and pairwise state fidelity to diagnose structural failures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04845"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"HamQASBench","organization_count":1,"organizations":["claire_radar"],"rank":1394,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.04845"},{"aliases":["HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following"],"benchmark_id":"claire-radar-2607-25398","caveat":"HANDBOOK.md evaluates whether tool-using agents consistently obey lengthy standing procedures while completing workplace tasks in self-contained company environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25398"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"HANDBOOK.md","organization_count":1,"organizations":["claire_radar"],"rank":1395,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25398"},{"aliases":[],"benchmark_id":"claire-radar-2609-24308","caveat":"A benchmark that evaluates whether generated worlds remain reliable as agents interact with them.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24308"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"HappyWorld-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1396,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24308"},{"aliases":["Harbor-Index","Harbor Index"],"benchmark_id":"harbor_index","caveat":"Compact high-signal frontier-agent evaluation; the environment and harness define the score.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"coding_agent","name":"Harbor-Index","organization_count":1,"organizations":["Tencent"],"rank":1397,"released":null,"source":"model_reports","url":"https://github.com/harbor-framework/harbor-index"},{"aliases":[],"benchmark_id":"opencompass-1842-hardmath2","caveat":"HARDMath2 is a benchmark for applied mathematics created by students in a graduate class at Harvard University, featuring 211 original problems covering core topics such as boundary-layer analysis, WKB methods, asymptotic solutions of nonlinear partial differential equations, and the asymptotics. HARDMath2是由哈佛大学研究生课程的学生创建的一项应用数学基准测试，包含211道原创问题，涵盖边界层分析、WKB方法、非线性偏微分方程的渐近解以及振荡积分的渐近性等核心主题。该基准通过一种创新的协作方式构建，学生不仅设计并改进符合课程大纲的高难度问题，还对解决方案进行同行验证，同时测试不同模型的表现。最终，LLM生成的解答会与学生的答案以及数值真值进行自动对比，以评估模型的准确性和能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HARDMath2"],"document_share":0.0002723311546840959,"domain":"推理","name":"HARDMath2","organization_count":1,"organizations":["opencompass_hub"],"rank":1398,"released":"2025-05-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HARDMath2"},{"aliases":[],"benchmark_id":"claire-radar-2605-28315","caveat":"HardMTBench is a difficulty-aware diagnostic benchmark for Chinese-English domain translation, covering 12 domains with 20,000 directional test items and annotated hardness scores based on domain knowledge, translation difficulty, and terminology load.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28315"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"HardMTBench","organization_count":1,"organizations":["claire_radar"],"rank":1399,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28315"},{"aliases":[],"benchmark_id":"claire-radar-2605-29224","caveat":"AgentREVEAL is a diagnostic framework that evaluates safety alignment degradation in LLM agents when web retrieval is integrated. It assesses the impact of retrieval integration and content properties on harmful compliance, using a set of harmful behaviors and retrieval sources.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29224"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"HarmURLBench","organization_count":1,"organizations":["claire_radar"],"rank":1400,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29224"},{"aliases":[],"benchmark_id":"claire-radar-2606-27187","caveat":"Evaluates harmful video understanding in large multimodal models using 1,379 videos and 4,137 multiple-choice questions across three hierarchical dimensions: Observable Evidence, Clip-Internal Meaning, and Beyond-Clip Reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27187"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HarmVideoBench","organization_count":1,"organizations":["claire_radar"],"rank":1401,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.27187"},{"aliases":[],"benchmark_id":"claire-radar-2605-27922","caveat":"Harness-Bench evaluates configuration-level harness effects in agent workflows with 106 sandboxed tasks, measuring completion, process quality, efficiency, and failure behavior across model-harness pairings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27922"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Harness-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1402,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27922"},{"aliases":[],"benchmark_id":"claire-radar-2609-01437","caveat":"HarnessDev evaluates LLMs on creating and evolving runnable agent harnesses, shifting evaluation from task outputs to the infrastructure capabilities themselves.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01437"],"document_share":0.0002723311546840959,"domain":"Agents","name":"HarnessDev","organization_count":1,"organizations":["claire_radar"],"rank":1403,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01437"},{"aliases":[],"benchmark_id":"claire-radar-2608-06301","caveat":"Evaluates LLMs optimizing a target agent's harness (prompts, tools, control flow) under budgeted, stochastic evaluation. Scoring is normalized gain over seed on a held-out test partition, with trusted execution environment enforcing evaluation boundary.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06301"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HarnessOpt-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1404,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06301"},{"aliases":[],"benchmark_id":"claire-radar-2608-17597","caveat":"HarnessRisk evaluates agent harness safety across six lifecycle phases with 128 sandboxed cases. It measures Utility, Attack Success Rate, Persistence, and Detection for each trajectory.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17597"],"document_share":0.0002723311546840959,"domain":"Agent safety","name":"HarnessRisk","organization_count":1,"organizations":["claire_radar"],"rank":1405,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17597"},{"aliases":[],"benchmark_id":"claire-radar-2608-06984","caveat":"HarnessSafe evaluates safety of agent harnesses through 328 executable cases across seven persistent-carrier families, using trace-based evaluation to track attack chains from entry to potential violation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06984"],"document_share":0.0002723311546840959,"domain":"Agents","name":"HarnessSafe","organization_count":1,"organizations":["claire_radar"],"rank":1406,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.06984"},{"aliases":[],"benchmark_id":"claire-radar-2610-07639","caveat":"Coding agent harnesses mediate tool use and authorize actions, yet their security mechanisms and runtime effects remain incompletely characterized.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.07639"],"document_share":0.0002723311546840959,"domain":"Agents","name":"HarnessSecurity-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1407,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.07639"},{"aliases":[],"benchmark_id":"llm-stats-harvey-lab","caveat":"Harvey LAB (Vals) is a professional legal-work evaluation of AI systems on complex law-firm style tasks, reported by Vals.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab?top_n=500"],"document_share":0.0002723311546840959,"domain":"knowledge","name":"Harvey LAB (Vals)","organization_count":1,"organizations":["llm_stats"],"rank":1408,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-harvey-lab-aa","caveat":"Legal agentic work, criterion pass rate","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/harvey-lab-aa"],"document_share":0.0002723311546840959,"domain":"agentic","name":"Harvey LAB-AA","organization_count":1,"organizations":["artificial_analysis"],"rank":1409,"released":null,"source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/harvey-lab-aa"},{"aliases":[],"benchmark_id":"llm-stats-harvey-lab-aa","caveat":"Harvey LAB-AA evaluates model performance on complex legal workflows.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab-aa?top_n=500"],"document_share":0.0002723311546840959,"domain":"knowledge","name":"Harvey LAB-AA","organization_count":1,"organizations":["llm_stats"],"rank":1410,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/harvey-lab-aa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-16863","caveat":"HawkesNest is a synthetic benchmark for spatiotemporal point process models, providing controlled generators and complexity ladders across four axes: space-time entanglement, background heterogeneity, cross-type interaction, and domain topology. It includes simulation, export, and visualization tools.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16863"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HawkesNest","organization_count":1,"organizations":["claire_radar"],"rank":1411,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16863"},{"aliases":[],"benchmark_id":"claire-radar-2608-19981","caveat":"Evaluates multilingual medical LLMs on 1,000 examples per language across nine languages, covering MCQA, NLI, and open-ended QA tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.19981"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"HealMed","organization_count":1,"organizations":["claire_radar"],"rank":1412,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.19981"},{"aliases":[],"benchmark_id":"claire-radar-2606-31179","caveat":"Evaluates AI agents on 54 realistic healthcare tasks across 7 categories, including medical imaging, EHR analysis, and clinical trial matching. Agents operate in terminal environments with task-specific verifiers and a final task success rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31179"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"HealthAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":1413,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31179"},{"aliases":[],"benchmark_id":"llm-stats-healthbench","caveat":"An open-source benchmark for measuring performance and safety of large language models in healthcare, consisting of 5,000 multi-turn conversations evaluated by 262 physicians using 48,562 unique rubric criteria across health contexts and behavioral dimensions","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/healthbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"healthcare","name":"HealthBench","organization_count":1,"organizations":["llm_stats"],"rank":1414,"released":"2025-05-12","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-healthbench-consensus","caveat":"HealthBench Consensus is a HealthBench subset focused on questions where physician-created rubric criteria have especially high agreement, measuring healthcare performance and safety on consensus-evaluable conversations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/healthbench-consensus?top_n=500"],"document_share":0.0002723311546840959,"domain":"healthcare","name":"HealthBench Consensus","organization_count":1,"organizations":["llm_stats"],"rank":1415,"released":"2025-05-12","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-consensus?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-healthbench-hard","caveat":"A challenging variation of HealthBench that evaluates large language models' performance and safety in healthcare through 5,000 multi-turn conversations with particularly rigorous evaluation criteria validated by 262 physicians from 60 countries","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/healthbench-hard?top_n=500"],"document_share":0.0002723311546840959,"domain":"healthcare","name":"HealthBench Hard","organization_count":1,"organizations":["llm_stats"],"rank":1416,"released":"2025-05-12","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-hard?top_n=500"},{"aliases":["HealthBench Professional"],"benchmark_id":"healthbench_professional","caveat":"A distinct, harder split from the HealthBench variants recorded separately in this registry, so its scores are not comparable to them. Published as an OpenAI paper (525 examples, 3 clinician use cases) with a public dataset; released 2026-04-22 per PDF metadata. Graded by an LLM against physician-written rubrics.","document_count":1,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5"],"document_share":0.0002723311546840959,"domain":"health","name":"HealthBench Professional","organization_count":1,"organizations":["Anthropic"],"rank":1417,"released":"2026-04-22","source":"model_reports","url":"https://cdn.openai.com/dd128428-0184-4e25-b155-3a7686c7d744/HealthBench-Professional.pdf"},{"aliases":[],"benchmark_id":"llm-stats-healthbench-professional","caveat":"HealthBench Professional evaluates model capability and safety for clinician use cases using real clinician-style chats and physician-authored grading rubrics.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/healthbench-professional?top_n=500"],"document_share":0.0002723311546840959,"domain":"healthcare","name":"HealthBench Professional","organization_count":1,"organizations":["llm_stats"],"rank":1418,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/healthbench-professional?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-25071","caveat":"Evaluates LLMs on 610 mental-health conversations from HealthBench using a three-judge panel and physician rubrics, with a harder 119-conversation subset.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25071"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"HealthBench-Psych","organization_count":1,"organizations":["claire_radar"],"rank":1419,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.25071"},{"aliases":[],"benchmark_id":"claire-radar-2609-06976","caveat":"HealthLoopQA evaluates LLM reasoning over continuous diabetes monitoring data through 127 tasks and 1,500 QA instances covering process mining, anomaly detection, and prediction over 30-day horizons, with a fault-injected simulation testbed for safety scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06976"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"HealthLoopQA","organization_count":1,"organizations":["claire_radar"],"rank":1420,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.06976"},{"aliases":[],"benchmark_id":"claire-radar-2609-18680","caveat":"A benchmark for implicit context in speech recognition, pairing Mandarin-English synthetic speech with assistant replies that support varying interpretations, to test models' handling of contextual ambiguity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18680"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HearInContext","organization_count":1,"organizations":["claire_radar"],"rank":1421,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18680"},{"aliases":[],"benchmark_id":"claire-radar-2605-30058","caveat":"HEART-Bench evaluates whether LLM agents can simulate coherent, human-like psychology. It provides 11 fictional characters with raw episodic memories, 64 decision-making scenarios based on the DIAMONDS taxonomy, and 673 multiple-choice questions with expert-annotated ground truth. Two evaluation tracks: MCQ and open-ended consciousness-narrative.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30058"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HEART-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1422,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30058"},{"aliases":[],"benchmark_id":"claire-radar-2608-07640","caveat":"HeatCast is a benchmark for monthly Land Surface Temperature forecasting at 30m resolution across 124 U.S. cities. It provides Landsat-based tiles, fixed temporal split, LCZ-stratified metrics, and a reference evaluation harness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07640"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HeatCast","organization_count":1,"organizations":["claire_radar"],"rank":1423,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07640"},{"aliases":[],"benchmark_id":"claire-radar-2606-03918","caveat":"Hedge-Bench 1.0 evaluates agents on 102 financial reasoning tasks with deterministic grading against expert reasoning traces.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03918"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Hedge-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1424,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03918"},{"aliases":[],"benchmark_id":"claire-radar-2607-13155","caveat":"HEDGEHOG is a six-stage filtration benchmark for evaluating molecular generators on medicinal plausibility, including physicochemical, structural, synthesis, docking, and 3D pose checks, applied to 23 generators and 230,000 molecules.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.13155"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"HEDGEHOG","organization_count":1,"organizations":["claire_radar"],"rank":1425,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.13155"},{"aliases":[],"benchmark_id":"llm-stats-hellaswag","caveat":"A challenging commonsense natural language inference dataset that uses Adversarial Filtering to create questions trivial for humans (>95% accuracy) but difficult for state-of-the-art models, requiring completion of sentence endings based on physical situations and everyday activities","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hellaswag?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"HellaSwag","organization_count":1,"organizations":["llm_stats"],"rank":1426,"released":"2019-05-19","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hellaswag?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-531-hellaswag","caveat":"HellaSwag is a challenge dataset for evaluating commonsense natural language inference, which is specially hard for state-of-the-art models, though its questions are trivial for humans (>95% accuracy). It consists of 70k multiple choice questions, each with a scenario and four possible endings, which requires to select the most reasonable ending. These questions come from two domains:activitynet and wikihow, involving video and text scenarios respectively. The correct answers of these questions are the real sentences for the next event, while the incorrect answers are adversarially generated and human verified, so as to fool machines but not humans. HellaSwag 是一个用于评估常识性自然语言推理的数据集，HellaSwag的问题对于最先进的模型来说是特别困难的，尽管它的问题对于人类来说非常轻松就能回答的（> 95% 的准确率）。它由7万多道多项选择题组成，每道题都有一个场景和四种可能的答案，需要选择最合理的答案。这些问题来自两个领域：activitynet和wikihow，分别涉及视频和文本场景。这些问题的正确答案是下一个事件的真实句子，而错误答案是通过对抗技术生成的并经过人类验证，这些答案可以欺骗机器但不能欺骗人类。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HellaSwag"],"document_share":0.0002723311546840959,"domain":"推理","name":"HellaSwag","organization_count":1,"organizations":["opencompass_hub"],"rank":1427,"released":"2019-05-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HellaSwag"},{"aliases":[],"benchmark_id":"opencompass-1097-hellobench","caveat":"HelloBench is a hierarchical long text generation benchmark to evaluate LLMs' performance in generating long text.  Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation. HelloBench为长文本生成基准，这是一个全面的、开放式的基准，用于评估LLM在生成长文本方面的性能。基于Bloom的分类法，HelloBench将长文本生成任务分为五个子任务：开放式QA、摘要、聊天、文本完成和启发式文本生成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HelloBench"],"document_share":0.0002723311546840959,"domain":"长文本","name":"HelloBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1428,"released":"2024-09-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HelloBench"},{"aliases":["Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains"],"benchmark_id":"claire-radar-2608-05138","caveat":"HERA evaluates retrieval-augmented generation for Modern Greek across legal, energy, financial, and medical specialist domains using mined corpora and expert-judged answers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05138"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HERA","organization_count":1,"organizations":["claire_radar"],"rank":1429,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.05138"},{"aliases":[],"benchmark_id":"claire-radar-2606-20487","caveat":"HeraBench is a fault-injected benchmark for multi-device agent workflows on Linux and Android, evaluating hierarchical replanning under injected strategy- and device-level failures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20487"],"document_share":0.0002723311546840959,"domain":"Agents","name":"HeraBench","organization_count":1,"organizations":["claire_radar"],"rank":1430,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.20487"},{"aliases":[],"benchmark_id":"opencompass-2031-herb","caveat":"HERB comprises 39,190 artifacts—including documents, meeting transcripts, Slack messages, GitHub content, and URLs—simulating business workflows across product planning, development, and support stages, with noisy, multi-hop QA tasks featuring both answerable and unanswerable queries. HERB基准包含39,190个企业文档、会议记录、Slack消息、GitHub内容和网页链接，模拟产品规划、开发与支持等业务流程，生成包含噪声的多跳问答任务，涵盖可回答与不可回答查询。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HERB"],"document_share":0.0002723311546840959,"domain":"其他","name":"HERB","organization_count":1,"organizations":["opencompass_hub"],"rank":1431,"released":"2025-06-29","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HERB"},{"aliases":["hermes-code-agent-benchmark"],"benchmark_id":"claire-radar-github-flappearlabs-hermes-code-agent-benchmark","caveat":"Evaluates software engineering task completion across 20 selected SWE-bench Verified, SWE-bench Pro V2, and Terminal-Bench instances using official verifiers.","document_count":1,"document_ids":["claire_radar:https://github.com/FlapPearLabs/hermes-code-agent-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Hermes Code Agent Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1432,"released":"2026-09-27","source":"claire_radar","url":"https://github.com/FlapPearLabs/hermes-code-agent-benchmark"},{"aliases":["HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning"],"benchmark_id":"claire-radar-2607-08784","caveat":"HERO is a heterogeneity-aware benchmark library for federated continual learning that separates task splits, client data splits, and client task sequences.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08784"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HERO","organization_count":1,"organizations":["claire_radar"],"rank":1433,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.08784"},{"aliases":[],"benchmark_id":"claire-radar-2606-02556","caveat":"Tests rule induction in goal-directed episodic tasks through text games, covering eight tasks across attribute and procedural induction families with controllable lexical grounding and identifiability conditions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02556"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HERO'S JOURNEY","organization_count":1,"organizations":["claire_radar"],"rank":1434,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02556"},{"aliases":[],"benchmark_id":"claire-radar-2606-25491","caveat":"HG-Bench evaluates page-aware, two-level answer-region grounding: given multi-page handwritten homework images, models must localize complete answer regions and ordered step-level subregions, with question- and step-level boxes under a hierarchical constraint.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25491"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1435,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25491"},{"aliases":["Escaping Python Dependency Hell: A Hybrid Replay-and-Repair Pipeline for Python Dependency Resolution"],"benchmark_id":"claire-radar-2609-26952","caveat":"Evaluates a Python dependency resolution pipeline on dependency-failing snippets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.26952"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HG2.9K","organization_count":1,"organizations":["claire_radar"],"rank":1436,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.26952"},{"aliases":[],"benchmark_id":"llm-stats-hiddenmath","caveat":"Google DeepMind's internal mathematical reasoning benchmark that introduces novel problems not encountered during model training to evaluate true mathematical reasoning capabilities rather than memorization","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hiddenmath?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"HiddenMath","organization_count":1,"organizations":["llm_stats"],"rank":1437,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hiddenmath?top_n=500"},{"aliases":["HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification"],"benchmark_id":"claire-radar-2606-13706","caveat":"HierSVA-B evaluates LLM-generated SystemVerilog assertions for hierarchical hardware formal verification across six quality axes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13706"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"HierSVA-B","organization_count":1,"organizations":["claire_radar"],"rank":1438,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.13706"},{"aliases":[],"benchmark_id":"claire-radar-2606-04126","caveat":"HighTide is an AI-assisted VLSI benchmark suite with open-source designs, Bazel-based compilation, and agent skills for design curation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04126"],"document_share":0.0002723311546840959,"domain":"Agents","name":"HighTide","organization_count":1,"organizations":["claire_radar"],"rank":1439,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.04126"},{"aliases":["Beam-Wise Statistical Background Subtraction for Static Roadside LiDAR: A Cross-Sensor Benchmark Study"],"benchmark_id":"claire-radar-2608-14868","caveat":"HighwayScene is a multi-LiDAR dataset recorded in a static roadside setup for evaluating background subtraction, with public annotations and implementations released.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14868"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HighwayScene","organization_count":1,"organizations":["claire_radar"],"rank":1440,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14868"},{"aliases":[],"benchmark_id":"claire-radar-2605-24635","caveat":"HiMed is a Hindi medical dataset and benchmark suite covering Western and Indian medicine. It evaluates medical reasoning in Hindi across multiple tasks, with data and evaluation scripts publicly released on GitHub.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24635"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"HiMed","organization_count":1,"organizations":["claire_radar"],"rank":1441,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24635"},{"aliases":[],"benchmark_id":"claire-radar-2607-18867","caveat":"A black-box audit protocol that measures parametric hindsight leakage in time-indexed LLM decision tasks. It uses date manipulations, memory probes, and six defined metrics, and the paper applies it to 15 models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18867"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"HindsightBench","organization_count":1,"organizations":["claire_radar"],"rank":1442,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.18867"},{"aliases":[],"benchmark_id":"llm-stats-hipho","caveat":"HiPhO is a high-school physics olympiad benchmark evaluating multimodal reasoning over physics problems that include diagrams and figures.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hipho?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"HiPhO","organization_count":1,"organizations":["llm_stats"],"rank":1443,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hipho?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-hle-verified","caveat":"HLE-Verified evaluates multidisciplinary expert reasoning on a verified subset of Humanity's Last Exam.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hle-verified?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"HLE-Verified","organization_count":1,"organizations":["llm_stats"],"rank":1444,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hle-verified?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-02449","caveat":"HLL is a benchmark that evaluates multimodal agents on interactive CAPTCHA verification in a closed-loop GUI environment, covering diverse task types such as text transcription, image selection, sliders, jigsaw puzzles, and logic-based challenges. Scoring is based on task completion and trace-conditioned validation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02449"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"HLL","organization_count":1,"organizations":["claire_radar"],"rank":1445,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02449"},{"aliases":[],"benchmark_id":"llm-stats-hmmt-2025","caveat":"Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hmmt-2025?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"HMMT 2025","organization_count":1,"organizations":["llm_stats"],"rank":1446,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-2025?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-hmmt-feb-26","caveat":"HMMT February 2026 is a math competition benchmark based on problems from the Harvard-MIT Mathematics Tournament, testing advanced mathematical problem-solving and reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hmmt-feb-26?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"HMMT Feb 26","organization_count":1,"organizations":["llm_stats"],"rank":1447,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt-feb-26?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-hmmt25","caveat":"Harvard-MIT Mathematics Tournament 2025 - A prestigious student-organized mathematics competition for high school students featuring two tournaments (November 2025 at MIT and February 2026 at Harvard) with individual tests, team rounds, and guts rounds","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hmmt25?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"HMMT25","organization_count":1,"organizations":["llm_stats"],"rank":1448,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hmmt25?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-01605","caveat":"Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.01605"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Hob-VL","organization_count":1,"organizations":["claire_radar"],"rank":1449,"released":"2026-10-01","source":"claire_radar","url":"https://arxiv.org/abs/2610.01605"},{"aliases":[],"benchmark_id":"claire-radar-2607-27030","caveat":"Evaluates vulnerability discovery in source code using 95 real CVEs across 8 repositories pinned at vulnerable commits, with a detector-blinded judge crediting findings that match code path, root cause, attack condition, and impact.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27030"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"HoF-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1450,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27030"},{"aliases":[],"benchmark_id":"claire-radar-2609-12412","caveat":"A modular benchmarking and deployment toolkit that jointly characterizes accuracy, latency and energy for foundation models across platforms from single-board computers to GPU servers in CPS-IoT applications.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12412"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HoliBench","organization_count":1,"organizations":["claire_radar"],"rank":1451,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12412"},{"aliases":["HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs"],"benchmark_id":"claire-radar-2606-23238","caveat":"HOLMES evaluates higher-order symbolic reasoning in language models using 1,379 natural-language problems paired with formalizations, answers, and verifiable reasoning traces across law and finance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23238"],"document_share":0.0002723311546840959,"domain":"Finance","name":"HOLMES","organization_count":1,"organizations":["claire_radar"],"rank":1452,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23238"},{"aliases":[],"benchmark_id":"opencompass-1581-holobench","caveat":"HoloBench is a benchmark designed to evaluate the ability of long-context language models (LCLMs) to perform holistic reasoning over extended text contexts. HoloBench 是一个用于评估长上下文语言模型（LCLMs）在扩展文本上下文中进行整体推理能力的基准测试。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HoloBench"],"document_share":0.0002723311546840959,"domain":"推理","name":"HoloBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1453,"released":"2024-10-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HoloBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-06420","caveat":"HoloCount evaluates multimodal large language models on visual counting tasks across three levels: semantic counting (atomic and property-based enumeration), analytical counting (logical composition via spatial and set-based reasoning), and robustness testing (adverse scenarios and grounded counter-priors). The benchmark uses a hierarchical taxonomy and provides a dataset for evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06420"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HoloCount","organization_count":1,"organizations":["claire_radar"],"rank":1454,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06420"},{"aliases":["HoloFair: Unified T2I Fairness Evaluation and Fair-GRPO Debiasing"],"benchmark_id":"claire-radar-2605-24687","caveat":"HoloFair evaluates multidimensional demographic bias in text-to-image models using a fairness-oriented dataset, an attribute classifier, and the MGBI metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24687"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HoloFair","organization_count":1,"organizations":["claire_radar"],"rank":1455,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24687"},{"aliases":[],"benchmark_id":"claire-radar-2608-09946","caveat":"HoosierHelp is an interactive benchmark for evaluating LLM agents in social service navigation. Agents interact with simulated users, issue structured resource-search calls, and select final resources from 3,971 Indiana public social service resources. The evaluation focuses on constraint grounding and handling non-ideal user interactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09946"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HoosierHelp","organization_count":1,"organizations":["claire_radar"],"rank":1456,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.09946"},{"aliases":[],"benchmark_id":"claire-radar-2608-01358","caveat":"HopRefusalBench evaluates refusal behavior of search-augmented language model agents on multi-hop questions that are unanswerable, covering three causes of unanswerability and three chain topologies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01358"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HopRefusalBench","organization_count":1,"organizations":["claire_radar"],"rank":1457,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.01358"},{"aliases":[],"benchmark_id":"horizonmath","caveat":"Unsolved research-level mathematics, reported at pass@4, so figures sit far below conventional accuracy scales.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"math","name":"HorizonMath","organization_count":1,"organizations":["Tencent"],"rank":1458,"released":null,"source":"model_reports","url":"https://github.com/ewang26/HorizonMath"},{"aliases":[],"benchmark_id":"llm-stats-horizonmath","caveat":"HorizonMath is an extremely difficult frontier mathematics benchmark designed to test the limits of mathematical reasoning on research-level and competition-beyond problems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/horizonmath?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"HorizonMath","organization_count":1,"organizations":["llm_stats"],"rank":1459,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/horizonmath?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1108-hotpotqa","caveat":"HOTPOTQA is a dataset with 113k Wikipedia-based question-answer pairs. HotpotQA 用于评估大语言模型的推理能力，包含 113,000 个基于维基百科的问题和答案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HotpotQA"],"document_share":0.0002723311546840959,"domain":"强推理","name":"HotpotQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1460,"released":"2018-09-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HotpotQA"},{"aliases":[],"benchmark_id":"claire-radar-2608-12904","caveat":"HounsBench is a CT-centric patient-state benchmark evaluating three task families: readout, reconstruction, and simulation. It provides patient-disjoint splits and per-family metrics for evaluating models on volumetric medical images and clinical language.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12904"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"HounsBench","organization_count":1,"organizations":["claire_radar"],"rank":1461,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.12904"},{"aliases":["Category-Level 3D Correspondence in Camera Space via Morphable Object Priors"],"benchmark_id":"claire-radar-2605-28257","caveat":"A benchmark for monocular category-level 3D correspondence in camera space, with 178k images across 50 household object categories and 3D keypoint annotations on CAD models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28257"],"document_share":0.0002723311546840959,"domain":"CAD","name":"HouseCorr3D","organization_count":1,"organizations":["claire_radar"],"rank":1462,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28257"},{"aliases":[],"benchmark_id":"llm-stats-hr-bench-4k","caveat":"HR-Bench (4k) evaluates image understanding on high-resolution visual inputs with a 4k setting.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hr-bench-4k?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"HR-Bench (4k)","organization_count":1,"organizations":["llm_stats"],"rank":1463,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hr-bench-4k?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-13056","caveat":"HRIBench evaluates intent-aware human-robot collaboration through structured scenario scripts covering roles of Instructor, Collaborator, and Intruder, with 13 tasks and 650 episodes, scoring synchronization, responsiveness, protocol compliance, and safety.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.13056"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"HRIBench","organization_count":1,"organizations":["claire_radar"],"rank":1464,"released":"2026-07-05","source":"claire_radar","url":"https://arxiv.org/abs/2607.13056"},{"aliases":[],"benchmark_id":"claire-radar-2606-19161","caveat":"HT-Bench is a multi-task benchmark for tactile representation learning with egocentric vision and full-hand tactile data, including retrieval, inpainting, synthesis, and prediction tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19161"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"HT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1465,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19161"},{"aliases":[],"benchmark_id":"opencompass-1992-htfllib","caveat":"HtFLlib is a benchmark for heterogeneous federated learning that examines how 40 vision, NLP and sensor models and 10 algorithms collaborate under non-IID data. HtFLlib 是一个面向异构联邦学习算法的综合评测基准，旨在衡量不同模型架构在非 IID 数据环境中的协同学习能力。评测对象覆盖图像、文本与传感信号三类模型，总计 40 个架构及 10 种代表性方法。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HtFLlib"],"document_share":0.0002723311546840959,"domain":"安全","name":"HtFLlib","organization_count":1,"organizations":["opencompass_hub"],"rank":1466,"released":"2025-06-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HtFLlib"},{"aliases":[],"benchmark_id":"claire-radar-2608-26517","caveat":"Visual intelligence seeks to perceive, interpret, and synthesize the visual world and is central to modern computer vision.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26517"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HUG-VIS","organization_count":1,"organizations":["claire_radar"],"rank":1467,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26517"},{"aliases":["HUMAID-NER: A Disaster Tweet Dataset for Joint Named Entity Recognition and Event Classification via Uncertainty-Weighted Multitask Learning"],"benchmark_id":"claire-radar-2609-16964","caveat":"HUMAID-NER is a dataset of 60,000 disaster tweets annotated for named entity recognition across ten entity types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16964"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HUMAID-NER","organization_count":1,"organizations":["claire_radar"],"rank":1468,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.16964"},{"aliases":[],"benchmark_id":"claire-radar-2607-27180","caveat":"HumanCLAW-Bench evaluates how VLMs make embodied decisions. Its 1,218 humanoid find-navigate-interact episodes span 41 indoor scenes, and low-level motor execution is factored out. Models are scored by success rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27180"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"HumanCLAW-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1469,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27180"},{"aliases":[],"benchmark_id":"llm-stats-humaneval","caveat":"A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"HumanEval","organization_count":1,"organizations":["llm_stats"],"rank":1470,"released":"2021-07-08","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-537-humaneval","caveat":"The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. 这是 \"Evaluating Large Language Models Trained on Code\" 论文中描述的 HumanEval 问题解决数据集的评估工具包。它用于测量从文档脚本合成程序的功能正确性。它由 164 个原始编程问题组成，评估语言理解能力、算法和简单数学，其中一些问题与简单的软件面试题类似。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HumanEval"],"document_share":0.0002723311546840959,"domain":"代码","name":"HumanEval","organization_count":1,"organizations":["opencompass_hub"],"rank":1471,"released":"2021-07-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HumanEval"},{"aliases":[],"benchmark_id":"llm-stats-humaneval-plus","caveat":"Enhanced version of HumanEval that extends the original test cases by 80x using EvalPlus framework for rigorous evaluation of LLM-synthesized code functional correctness, detecting previously undetected wrong code","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval-plus?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"HumanEval Plus","organization_count":1,"organizations":["llm_stats"],"rank":1472,"released":"2023-05-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-plus?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-humaneval-2","caveat":"Enhanced version of HumanEval that extends the original test cases by 80x using EvalPlus framework for rigorous evaluation of LLM-synthesized code functional correctness, detecting previously undetected wrong code","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval%2B?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"HumanEval+","organization_count":1,"organizations":["llm_stats"],"rank":1473,"released":"2023-05-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval%2B?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-humaneval-average","caveat":"A variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval-average?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"HumanEval-Average","organization_count":1,"organizations":["llm_stats"],"rank":1474,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-average?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-humaneval-er","caveat":"A variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval-er?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"HumanEval-ER","organization_count":1,"organizations":["llm_stats"],"rank":1475,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-er?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-humaneval-mul","caveat":"A multilingual variant of the HumanEval benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humaneval-mul?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"HumanEval-Mul","organization_count":1,"organizations":["llm_stats"],"rank":1476,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humaneval-mul?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-543-humaneval-x","caveat":"HumanEval-X is a benchmark for evaluating the multilingual ability of code generative models. It consists of 820 high-quality human-crafted data samples (each with test cases) in Python, C++, Java, JavaScript, and Go, and can be used for various tasks, such as code generation and translation. HumanEval-X 是一个用于评估代码生成模型的多语言能力的基准测试。它包含了820个高质量的人工制作的数据样本（每个都有测试案例），包括Python、C++、Java、JavaScript和Go语言，可用于各种任务，如代码生成和翻译。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HumanEval-X"],"document_share":0.0002723311546840959,"domain":"代码","name":"HumanEval-X","organization_count":1,"organizations":["opencompass_hub"],"rank":1477,"released":"2023-03-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HumanEval-X"},{"aliases":["Bootstrapping Niche Multilingual Code Translation via Reinforcement Learning with Execution-Based Verifiable Supervision"],"benchmark_id":"claire-radar-2608-13854","caveat":"HumanEval-X++ is an execution-based benchmark extending HumanEval-X to a broad many-to-many language space for code translation evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13854"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HumanEval-X++","organization_count":1,"organizations":["claire_radar"],"rank":1478,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.13854"},{"aliases":[],"benchmark_id":"llm-stats-humanevalfim-average","caveat":"Average evaluation of HumanEval Fill-in-the-Middle benchmark variants (single-line, multi-line, random-span) for assessing code infilling capabilities of language models","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humanevalfim-average?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"HumanEvalFIM-Average","organization_count":1,"organizations":["llm_stats"],"rank":1479,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humanevalfim-average?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-humanity-s-last-exam","caveat":"Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"Humanity's Last Exam","organization_count":1,"organizations":["llm_stats"],"rank":1480,"released":"2025-01-24","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-humanity-s-last-exam-no-tools-text-only","caveat":"Text-only Humanity's Last Exam variant evaluated without tool use.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28no-tools%2C-text-only%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"Humanity's Last Exam (no tools, text-only)","organization_count":1,"organizations":["llm_stats"],"rank":1481,"released":"2025-01-24","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28no-tools%2C-text-only%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-humanity-s-last-exam-with-tools-text-only","caveat":"Text-only Humanity's Last Exam variant evaluated with tool use enabled.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28with-tools%2C-text-only%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"Humanity's Last Exam (with tools, text-only)","organization_count":1,"organizations":["llm_stats"],"rank":1482,"released":"2025-01-24","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/humanity%27s-last-exam-%28with-tools%2C-text-only%29?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-humanitys-last-exam","caveat":"Reasoning & knowledge","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/humanitys-last-exam"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"Humanity’s Last Exam","organization_count":1,"organizations":["artificial_analysis"],"rank":1483,"released":"2025-01-23","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam"},{"aliases":[],"benchmark_id":"claire-radar-2606-27999","caveat":"HumanMoveVQA evaluates video MLLMs on reasoning about human trajectory and orientation changes in videos, using a first-frame anchored world coordinate system and 10K question-answer pairs across seven reasoning categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27999"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HumanMoveVQA","organization_count":1,"organizations":["claire_radar"],"rank":1484,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.27999"},{"aliases":[],"benchmark_id":"claire-radar-2606-17833","caveat":"HumanoidArena is a simulation benchmark for egocentric hierarchical whole-body learning, evaluating high-level policies that predict whole-body actions for low-level general motion trackers across seven leg-critical human-object and human-scene interaction tasks, with perturbation-conditioned and GMT-conditioned evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17833"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HumanoidArena","organization_count":1,"organizations":["claire_radar"],"rank":1485,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.17833"},{"aliases":[],"benchmark_id":"claire-radar-2610-02089","caveat":"As robotic hardware and learning methods advance, humanoids need tools to perform tasks beyond their inherent physical limits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.02089"],"document_share":0.0002723311546840959,"domain":"Agents","name":"HumanoidToolBench","organization_count":1,"organizations":["claire_radar"],"rank":1486,"released":"2026-10-01","source":"claire_radar","url":"https://arxiv.org/abs/2610.02089"},{"aliases":[],"benchmark_id":"claire-radar-2608-12860","caveat":"Evaluates vision-language navigation for humanoid robots across four embodiments in physics-grounded simulator scenarios. Includes 933 episodes with instructions and multiple stylistic variants, assessing success rate and normalized Dynamic Time Warping.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12860"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"HumanoidVLN","organization_count":1,"organizations":["claire_radar"],"rank":1487,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.12860"},{"aliases":["humanwill-benchmark"],"benchmark_id":"claire-radar-github-humanwill-ai-humanwill-benchmark","caveat":"HumanWill is an assistance-focused evaluation over 424 cybersecurity questions spanning vulnerability validation, source-code security, binary, network and filesystem investigation. It scores false refusal and answer usefulness as separate judged dimensions under configurable per-topic policies, with run evidence and regrading support.","document_count":1,"document_ids":["claire_radar:https://github.com/humanwill-ai/humanwill-benchmark"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"HumanWill Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1488,"released":"2026-09-19","source":"claire_radar","url":"https://github.com/humanwill-ai/humanwill-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2606-06055","caveat":"HUSH-Bench evaluates conversational agents' use of sensitive history under a conservative policy, with 2,400 prompts and matched no-memory references, measuring unsolicited integration and memory access.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06055"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HUSH-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1489,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06055"},{"aliases":[],"benchmark_id":"claire-radar-2607-29196","caveat":"Hy-MultiTurn is a Chinese benchmark for deep multi-turn dialogue understanding, with six controlled evaluation modes covering constraint memory, object localization, and action suppression across 12-76 turn dialogues.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29196"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Hy-MultiTurn","organization_count":1,"organizations":["claire_radar"],"rank":1490,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29196"},{"aliases":[],"benchmark_id":"claire-radar-2606-12620","caveat":"HybridCodeAuthorship is a benchmark of Python files with interleaved human- and AI-authored lines, built from CodeSearchNet, for developing and evaluating AI-generated code detection algorithms at line- and chunk-level.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12620"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HybridCodeAuthorship","organization_count":1,"organizations":["claire_radar"],"rank":1491,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12620"},{"aliases":["Benchmarking Hybrid Deep Research Across Database Querying and Web Search"],"benchmark_id":"claire-radar-2609-09410","caveat":"Evaluates deep-research agents on 380 tasks requiring both SQL database querying and web search to produce a complete end-to-end answer across SQL-to-Search, Search-to-SQL, and Parallel reasoning patterns.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09410"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"HybridDeepResearch","organization_count":1,"organizations":["claire_radar"],"rank":1492,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.09410"},{"aliases":[],"benchmark_id":"claire-radar-2605-21858","caveat":"HyperAlign-Bench is a benchmark introduced in a paper for evaluating hypergraph structural modeling in large language models, covering vertex-level and hyperedge-level tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.21858"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HyperAlign-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1493,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.21858"},{"aliases":[],"benchmark_id":"claire-radar-2610-03574","caveat":"We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13 languages, written by native or highly proficient speakers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03574"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HyperBrowseComp","organization_count":1,"organizations":["claire_radar"],"rank":1494,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03574"},{"aliases":[],"benchmark_id":"claire-radar-2607-21050","caveat":"HyperImageNet is a dataset of 26,084 airborne hyperspectral image patches with 224 spectral bands and 138 fine-grained land-cover categories, providing raw imagery, pixel-level semantic labels, and object-level instance masks for semantic and instance segmentation, along with an open-environment evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21050"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HyperImageNet","organization_count":1,"organizations":["claire_radar"],"rank":1495,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21050"},{"aliases":[],"benchmark_id":"claire-radar-2607-14419","caveat":"HyperShadow evaluates binary classification of 3D point clouds as either native 3D objects or 3D projections of objects in 4-6 spatial dimensions, with static and temporal tracks, four corruption tiers, and fixed train/eval splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14419"],"document_share":0.0002723311546840959,"domain":"General AI","name":"HyperShadow","organization_count":1,"organizations":["claire_radar"],"rank":1496,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.14419"},{"aliases":[],"benchmark_id":"llm-stats-hypersim","caveat":"Hypersim evaluates 3D grounding and depth understanding in synthetic indoor scenes.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/hypersim?top_n=500"],"document_share":0.0002723311546840959,"domain":"spatial_reasoning","name":"Hypersim","organization_count":1,"organizations":["llm_stats"],"rank":1497,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/hypersim?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1782-hypobench","caveat":"HypoBench, a novel benchmark designed to evaluate LLMs and hypothesis generation methods across multiple aspects, including practical utility, generalizability, and hypothesis discovery rate. HypoBench includes 7 real-world tasks and 5 synthetic tasks with 194 distinct datasets. HypoBench，这是一种新颖的基准，旨在从多个方面评估 LLM 和假设生成方法，包括实用性、泛化性和假设发现率。HypoBench 包括 7 个真实任务和 5 个合成任务，具有 194 个不同的数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HypoBench"],"document_share":0.0002723311546840959,"domain":"其他","name":"HypoBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1498,"released":"2025-04-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HypoBench"},{"aliases":[],"benchmark_id":"opencompass-1786-hypoeval","caveat":"HypoEval, Hypothesis-guided Evaluation framework, which first uses a small corpus of human evaluations to generate more detailed rubrics for human judgments and then incorporates a checklist-like approach to combine LLM's assigned scores on each decomposed dimension to acquire overall scores. HypoEval，即假设指导的评估框架，该框架首先使用一小部分人工评估来生成更详细的人类判断量规，然后采用类似清单的方法，将 LLM 在每个分解维度上的分配分数结合起来，以获得总分。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/HypoEval"],"document_share":0.0002723311546840959,"domain":"其他","name":"HypoEval","organization_count":1,"organizations":["opencompass_hub"],"rank":1499,"released":"2025-04-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/HypoEval"},{"aliases":[],"benchmark_id":"claire-radar-2606-00750","caveat":"A benchmark of 19 research papers with expert-built interactive systems for evaluating agents that convert PDFs into executable web applications.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00750"],"document_share":0.0002723311546840959,"domain":"General AI","name":"I-WebGenBench","organization_count":1,"organizations":["claire_radar"],"rank":1500,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00750"},{"aliases":[],"benchmark_id":"opencompass-1320-iac-eval","caveat":"IaC-Eval is meant for quantitatively evaluating the capabilities of LLMs in cloud IaC code generation, containing 458 questions ranging from simple to difficult across various cloud services. IaC-Eval用于定量评估LLM在云IaC代码生成中的功能，其中包含458个从易到难的问题，涵盖了各种云服务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IaC-Eval"],"document_share":0.0002723311546840959,"domain":"代码","name":"IaC-Eval","organization_count":1,"organizations":["opencompass_hub"],"rank":1501,"released":"2024-09-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/IaC-Eval"},{"aliases":[],"benchmark_id":"claire-radar-2608-02171","caveat":"IBA-Bench evaluates LLM agents on implicit behavioral alignment using longitudinal interaction histories, across nine application domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02171"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IBA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1502,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02171"},{"aliases":[],"benchmark_id":"claire-radar-2607-21217","caveat":"ICAE-Bench evaluates coding agents on interactive project-building tasks. Agents receive a fuzzy product requirement and must clarify missing details via an automated user agent, then implement the project in a container. Scoring uses black-box tests and multi-dimensional diagnostics including functional correctness, semantic/API similarity, structural fidelity, design quality, and interaction quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21217"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"ICAE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1503,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21217"},{"aliases":[],"benchmark_id":"claire-radar-2609-32413","caveat":"Interactive chart code generation requires models to reproduce a reference chart's appearance and underlying data and correctly implement the state changes triggered by specified user interactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32413"],"document_share":0.0002723311546840959,"domain":"Code","name":"IChart2Code","organization_count":1,"organizations":["claire_radar"],"rank":1504,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32413"},{"aliases":[],"benchmark_id":"claire-radar-2609-10539","caveat":"A benchmark of 660 evidence-grounded instances evaluating codification-readiness assessment, defect localization, and clarification action generation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10539"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IdeaAMBIG","organization_count":1,"organizations":["claire_radar"],"rank":1505,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10539"},{"aliases":[],"benchmark_id":"claire-radar-2607-08758","caveat":"Evaluates scientific lineage reasoning and lineage-grounded idea generation through two tracks: closed-form IG-Exam and generation IG-Arena with Population-Evolution Score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08758"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IdeaGene-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1506,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08758"},{"aliases":[],"benchmark_id":"claire-radar-2607-03614","caveat":"IDEAL-Bench evaluates Vision-Language Models on holistic 3D layout inference from single images of indoor scenes, scoring predictions across five numerical dimensions and a perceptual render-and-compare protocol. It uses a procedurally generated dataset of 1,000 re-renderable Blender environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03614"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IDEAL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1507,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.03614"},{"aliases":[],"benchmark_id":"claire-radar-2605-22247","caveat":"IdioLink is a retrieval benchmark with 10,700 documents and 2,140 queries across 107 idioms, testing the linking of idiomatic expressions to literal or paraphrased equivalents. Annotations mark core meaning spans, and scoring is based on retrieval quality against these spans.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22247"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IdioLink","organization_count":1,"organizations":["claire_radar"],"rank":1508,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22247"},{"aliases":[],"benchmark_id":"claire-radar-2606-09908","caveat":"IDP-Bench evaluates large language models on interdependent privacy scenarios, covering recognition of co-ownership, identification of contextual integrity parameters, and judgments of sharing appropriateness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09908"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IDP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1509,"released":"2026-06-06","source":"claire_radar","url":"https://arxiv.org/abs/2606.09908"},{"aliases":[],"benchmark_id":"claire-radar-2610-04375","caveat":"Agents built on large language models (LLMs) build and run software through tool calls.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.04375"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IEC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1510,"released":"2026-10-03","source":"claire_radar","url":"https://arxiv.org/abs/2610.04375"},{"aliases":[],"benchmark_id":"llm-stats-if","caveat":"Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/if?top_n=500"],"document_share":0.0002723311546840959,"domain":"structured_output","name":"IF","organization_count":1,"organizations":["llm_stats"],"rank":1511,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/if?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-ifbench","caveat":"Instruction following","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/ifbench"],"document_share":0.0002723311546840959,"domain":"instruction-following","name":"IFBench","organization_count":1,"organizations":["artificial_analysis"],"rank":1512,"released":"2025-07-03","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/ifbench"},{"aliases":[],"benchmark_id":"llm-stats-ifbench","caveat":"Instruction Following Benchmark evaluating model's ability to follow complex instructions","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ifbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"instruction_following","name":"IFBench","organization_count":1,"organizations":["llm_stats"],"rank":1513,"released":"2025-07-03","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ifbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-26072","caveat":"IFCMemoryBench evaluates long-term memory in LLM-based agents for BIM information retrieval, with 143 multi-session tasks across 19 projects and 4,016 prior sessions, requiring integration of remembered context with live IFC queries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26072"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IFCMemoryBench","organization_count":1,"organizations":["claire_radar"],"rank":1514,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.26072"},{"aliases":[],"benchmark_id":"llm-stats-ifeval","caveat":"Instruction-Following Evaluation (IFEval) benchmark for large language models, focusing on verifiable instructions with 25 types of instructions and around 500 prompts containing one or more verifiable constraints","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ifeval?top_n=500"],"document_share":0.0002723311546840959,"domain":"structured_output","name":"IFEval","organization_count":1,"organizations":["llm_stats"],"rank":1515,"released":"2023-11-14","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ifeval?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1136-ifeval","caveat":"IFEval is a straightforward and easy-to reproduce evaluation benchmark. It focuses on a set of “verifiable instructions” such as “write in more than 400 words” and “mention the keyword of AI at least 3 times”. IFEval 是一个简单且易于复现的评估基准。它关注一组“可验证的指令”，例如“写超过 400 个单词”和“至少提到关键词 AI 3 次”。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IFEval"],"document_share":0.0002723311546840959,"domain":"指令跟随","name":"IFEval","organization_count":1,"organizations":["opencompass_hub"],"rank":1516,"released":"2023-11-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/IFEval"},{"aliases":[],"benchmark_id":"claire-radar-2607-27912","caveat":"IFHierBench is a hierarchical instruction-following benchmark with 600 prompts and deterministic checkers, evaluating LLMs on satisfying constraints at different output scopes. It measures prompt-level accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27912"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IFHierBench","organization_count":1,"organizations":["claire_radar"],"rank":1517,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27912"},{"aliases":[],"benchmark_id":"opencompass-1617-ifir","caveat":"IFIR is the first comprehensive benchmark designed to evaluate instruction-following information retrieval (IR) in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature. IFIR，这是第一个旨在评估专家领域指令跟随信息检索（IR）的综合基准。IFIR 包含 2,426 个高质量示例，涵盖四个专业领域（金融、法律、医疗保健和科学文献）的八个子集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IFIR"],"document_share":0.0002723311546840959,"domain":"指令跟随","name":"IFIR","organization_count":1,"organizations":["opencompass_hub"],"rank":1518,"released":"2025-03-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/IFIR"},{"aliases":[],"benchmark_id":"claire-radar-2605-28218","caveat":"Evaluates multilingual translation systems on instruction following across seven languages, six constraint types, and compositional multi-constraint requests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28218"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IFMTBench","organization_count":1,"organizations":["claire_radar"],"rank":1519,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28218"},{"aliases":[],"benchmark_id":"claire-radar-2607-25987","caveat":"IH-Benchmark evaluates instruction-hierarchy robustness in LLMs via conflicting instructions from system, user, and tool outputs, covering 44 constraint families across five domains with a binary pass/fail protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25987"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IH-Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1520,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25987"},{"aliases":[],"benchmark_id":"claire-radar-2606-19595","caveat":"IHBench evaluates post-interruption recovery in voice agents executing state-machine-driven workflows across 10 enterprise domains. It scores task fulfillment and recovery quality for six interruption types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19595"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IHBench","organization_count":1,"organizations":["claire_radar"],"rank":1521,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19595"},{"aliases":[],"benchmark_id":"claire-radar-2608-02018","caveat":"II-Bench evaluates computer-use agents against low-harm adversarial tasks across three platforms, with 444 examples and a testing framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02018"],"document_share":0.0002723311546840959,"domain":"Software & AI Compute","name":"II-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1522,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02018"},{"aliases":[],"benchmark_id":"claire-radar-2608-29210","caveat":"Evaluates text-to-image models on 1,140 prompts with 8,842 combined rules, independently varying instance count and rule constraints to measure compositional instruction following.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29210"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Imag-Eval","organization_count":1,"organizations":["claire_radar"],"rank":1523,"released":"2026-08-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.29210"},{"aliases":[],"benchmark_id":"llm-stats-image2floorplan","caveat":"Image2FloorPlan is an in-house benchmark evaluating multimodal models on generating structured floor plans and interactive frontends directly from images such as design mockups and room photos.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/image2floorplan?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Image2FloorPlan","organization_count":1,"organizations":["llm_stats"],"rank":1524,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/image2floorplan?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-imagemining","caveat":"ImageMining evaluates multimodal models on extracting structured information from images using tool use, measuring ability to combine visual understanding with tool-based retrieval and analysis.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/imagemining?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ImageMining","organization_count":1,"organizations":["llm_stats"],"rank":1525,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/imagemining?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-10789","caveat":"Imaging-101 evaluates coding agents on 57 computational imaging tasks across six scientific domains, with three tracks for planning, function-level unit tests, and end-to-end reconstruction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10789"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Imaging-101","organization_count":1,"organizations":["claire_radar"],"rank":1526,"released":"2026-07-12","source":"claire_radar","url":"https://arxiv.org/abs/2607.10789"},{"aliases":[],"benchmark_id":"claire-radar-2607-07189","caveat":"ImagingBench evaluates agentic AI systems on 20 computational imaging tasks spanning ray and wave optics, image signal processing, inverse reconstruction, computational sensing, and calibration, across three settings: Expert, Planner, and Forward.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.07189"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ImagingBench","organization_count":1,"organizations":["claire_radar"],"rank":1527,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.07189"},{"aliases":[],"benchmark_id":"claire-radar-2607-15641","caveat":"IMBench evaluates intuitive robotic manipulation, combining perception, physical reasoning, action generation, and execution across 35 tasks and 14K trajectories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15641"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"IMBench","organization_count":1,"organizations":["claire_radar"],"rank":1528,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.15641"},{"aliases":[],"benchmark_id":"claire-radar-2606-28556","caveat":"IMCBench evaluates multimodal LLMs in image-grounded, multi-turn medical conversations, scoring safety, accuracy, and uncertainty use on a 1-5 scale.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.28556"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"IMCBench","organization_count":1,"organizations":["claire_radar"],"rank":1529,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.28556"},{"aliases":[],"benchmark_id":"opencompass-1274-imdl-benco","caveat":"IMDL-BenCo offers a comprehensive IMDL benchmark and modular codebase. IMDL-BenCo提供了全面的IMDL基准测试和模块化代码库。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IMDL-BenCo"],"document_share":0.0002723311546840959,"domain":"多模态","name":"IMDL-BenCo","organization_count":1,"organizations":["opencompass_hub"],"rank":1530,"released":"2024-06-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/IMDL-BenCo"},{"aliases":[],"benchmark_id":"claire-radar-2609-18104","caveat":"iMINDBench evaluates models on a shared suite of fifteen decoding tasks across three naturalistic movie-watching iEEG datasets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18104"],"document_share":0.0002723311546840959,"domain":"General AI","name":"iMINDBench","organization_count":1,"organizations":["claire_radar"],"rank":1531,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18104"},{"aliases":[],"benchmark_id":"llm-stats-imo-2025","caveat":"IMO 2025 evaluates models on the six problems from the 2025 International Mathematical Olympiad, requiring rigorous proof-based reasoning. Following the MathArena methodology, proofs are graded against human expert rubrics by dual strong judge models, with the minimum of the two scores taken as final. Maximum score is 42 points.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/imo-2025?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"IMO 2025","organization_count":1,"organizations":["llm_stats"],"rank":1532,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-2025?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-imo-answerbench","caveat":"IMO-AnswerBench is a benchmark for evaluating mathematical reasoning capabilities on International Mathematical Olympiad (IMO) problems, focusing on answer generation and verification.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/imo-answerbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"IMO-AnswerBench","organization_count":1,"organizations":["llm_stats"],"rank":1533,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/imo-answerbench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-imoproof-adv","caveat":"IMOProof-Adv is an advanced benchmark of International Mathematical Olympiad-style proof problems requiring rigorous multi-step mathematical proofs.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/imoproof-adv?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"IMOProof-Adv","organization_count":1,"organizations":["llm_stats"],"rank":1534,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/imoproof-adv?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-24228","caveat":"IMPLICIT-Bench measures implicit bias in text-to-image models under neutral prompts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24228"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IMPLICIT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1535,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24228"},{"aliases":[],"benchmark_id":"claire-radar-2609-16816","caveat":"Benchmark of 169 impossible tasks spanning six impossibility categories, each paired with a verifiable oracle certificate and 48 answerable controls.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16816"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"ImpossibleRubrics","organization_count":1,"organizations":["claire_radar"],"rank":1536,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.16816"},{"aliases":[],"benchmark_id":"claire-radar-2606-09169","caveat":"IMUG-Bench evaluates unified multimodal models on multi-turn interleaved image-text understanding and generation. It includes 3,113 samples and 12,034 interaction turns across Static Spatial, Temporal Causal, and Hybrid classes, with dynamic understanding questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09169"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IMUG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1537,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09169"},{"aliases":[],"benchmark_id":"llm-stats-include","caveat":"Include benchmark - specific documentation not found in official sources","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/include?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"Include","organization_count":1,"organizations":["llm_stats"],"rank":1538,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/include?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-08515","caveat":"Evaluates disability-related bias in text-to-image models using 119K generated images across bias dimensions and contexts, with the Stereotype Content Model Score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08515"],"document_share":0.0002723311546840959,"domain":"General AI","name":"INCLUDE-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":1539,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08515"},{"aliases":[],"benchmark_id":"llm-stats-community-07c9946d-dcf0-4977-a640-a6b1356b4f0b","caveat":null,"document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3A07c9946d-dcf0-4977-a640-a6b1356b4f0b?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"independence-bench","organization_count":1,"organizations":["llm_stats"],"rank":1540,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A07c9946d-dcf0-4977-a640-a6b1356b4f0b?top_n=500"},{"aliases":["indian-agent-benchmark"],"benchmark_id":"claire-radar-github-imaginationai31-sys-indian-agent-benchmark","caveat":"A synthetic multilingual benchmark evaluating computer-use agents on browser-style e-commerce tasks across seven Indian languages, with expected action trajectories, 0-100 task success scoring, and automatic failure-mode classification.","document_count":1,"document_ids":["claire_radar:https://github.com/imaginationai31-sys/indian-agent-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Indian Agent Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1541,"released":"2026-09-10","source":"claire_radar","url":"https://github.com/imaginationai31-sys/indian-agent-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2607-23808","caveat":"Multilingual joint diarization and ASR benchmark for 22 Indian languages, with ~108 hours of human-corrected multi-speaker audio from meetings, far-field, and in-the-wild sources, including code-mixing and overlaps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23808"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Indic DiarBench","organization_count":1,"organizations":["claire_radar"],"rank":1542,"released":"2026-07-26","source":"claire_radar","url":"https://arxiv.org/abs/2607.23808"},{"aliases":[],"benchmark_id":"claire-radar-2609-29167","caveat":"A 799-case benchmark for Indian retail banking spanning five operational domains and twenty primary axes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.29167"],"document_share":0.0002723311546840959,"domain":"Finance","name":"IndicBankBench","organization_count":1,"organizations":["claire_radar"],"rank":1543,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.29167"},{"aliases":[],"benchmark_id":"opencompass-1670-indicmmlu-pro","caveat":"IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models. IndicMMLU-Pro 提供了一个标准化的评估框架，以推动印度语系语言 AI 的研究边界，促进更准确、高效和具有文化敏感性的模型的发展。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IndicMMLU-Pro"],"document_share":0.0002723311546840959,"domain":"语言","name":"IndicMMLU-Pro","organization_count":1,"organizations":["opencompass_hub"],"rank":1544,"released":"2025-01-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/IndicMMLU-Pro"},{"aliases":[],"benchmark_id":"claire-radar-2607-14021","caveat":"The Industrial Dexterity Benchmark (IDB) provides physical boards and tasks for industrial dexterous manipulation, including cable management, cable harness, and gearbox assembly. It evaluates a robot's ability to perform tasks such as cable insertion and grasping via end-to-end imitation learning policies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14021"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"Industrial Dexterity Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1545,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.14021"},{"aliases":[],"benchmark_id":"claire-radar-2609-25562","caveat":"Evaluates six open robot policy models on manipulation capability, robustness, language grounding, and deployability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25562"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"IndustrialVLA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1546,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.25562"},{"aliases":[],"benchmark_id":"claire-radar-2606-14383","caveat":"IndustryBench-MIPU evaluates MLLMs on extracting structured attribute-value pairs from multi-image industrial product data, covering text recognition, visual reasoning, and cross-image integration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14383"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IndustryBench-MIPU","organization_count":1,"organizations":["claire_radar"],"rank":1547,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14383"},{"aliases":["inference-bench"],"benchmark_id":"claire-radar-github-cmcconnell1-inference-bench","caveat":"Evaluates six LLM serving engines (vLLM, SGLang, TGI, Ollama, llama.cpp, NVIDIA NIM) on latency, throughput, and cost per million tokens using fixed workloads and guidellm on Akamai Cloud GPUs.","document_count":1,"document_ids":["claire_radar:https://github.com/cmcconnell1/inference-bench"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"inference-bench: serving engine benchmarks on Akamai Cloud GPUs","organization_count":1,"organizations":["claire_radar"],"rank":1548,"released":"2026-09-27","source":"claire_radar","url":"https://github.com/cmcconnell1/inference-bench"},{"aliases":[],"benchmark_id":"claire-radar-2607-29134","caveat":"InferQ is a database-oriented benchmark for quantum circuit simulation, generating compositional circuits and emitting SQL workloads with feature extraction for workload characterization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29134"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"InferQ","organization_count":1,"organizations":["claire_radar"],"rank":1549,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29134"},{"aliases":[],"benchmark_id":"opencompass-1322-infibench","caveat":"InfiBench is a large-scale freeform question-answering (QA) benchmark for code to our knowledge, comprising 234 carefully selected high-quality Stack Overflow questions that span across 15 programming languages. InfiBench用于评测LLM回答代码相关问题的能力，包括涵盖15种编程语言的234个精心挑选的高质量Stack Overflow问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InfiBench"],"document_share":0.0002723311546840959,"domain":"代码","name":"InfiBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1550,"released":"2024-03-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/InfiBench"},{"aliases":[],"benchmark_id":"llm-stats-infinitebench-en-mc","caveat":"InfiniteBench English Multiple Choice variant - first LLM benchmark featuring average data length surpassing 100K tokens for evaluating long-context capabilities with 12 tasks spanning diverse domains","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.mc?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"InfiniteBench/En.MC","organization_count":1,"organizations":["llm_stats"],"rank":1551,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.mc?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-infinitebench-en-qa","caveat":"InfiniteBench English Question Answering variant - first LLM benchmark featuring average data length surpassing 100K tokens for evaluating long-context capabilities with 12 tasks spanning diverse domains","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.qa?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"InfiniteBench/En.QA","organization_count":1,"organizations":["llm_stats"],"rank":1552,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/infinitebench-en.qa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1085-infobench","caveat":"InfoBench is a benchmark comprising 500 diverse instructions and 2,250 decomposed questions across multiple constraint categories. InfoBench 是一个指令追随评测基准，包含 500 条多样化的指令和 2,250 个分解问题，涵盖多个约束类别。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InfoBench"],"document_share":0.0002723311546840959,"domain":"指令跟随","name":"InfoBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1553,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/InfoBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-33286","caveat":"Multimodal foundation models edit natural photographs at production quality, yet the same models struggle with structured visual content such as infographics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33286"],"document_share":0.0002723311546840959,"domain":"General AI","name":"InfoEdit","organization_count":1,"organizations":["claire_radar"],"rank":1554,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33286"},{"aliases":[],"benchmark_id":"llm-stats-infographicsqa","caveat":"InfographicVQA dataset with 5,485 infographic images and over 30,000 questions requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infographicsqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"InfographicsQA","organization_count":1,"organizations":["llm_stats"],"rank":1555,"released":"2021-04-26","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/infographicsqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-28503","caveat":"InfoOps Bench is an active, constantly updated benchmark measuring the integrity of frontier language models against co-optation for information operations. It uses real examples from a live monitoring pipeline and tests 17 models from 8 providers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28503"],"document_share":0.0002723311546840959,"domain":"General AI","name":"InfoOps Bench","organization_count":1,"organizations":["claire_radar"],"rank":1556,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28503"},{"aliases":[],"benchmark_id":"llm-stats-infovqa","caveat":"InfoVQA dataset with 30,000 questions and 5,000 infographic images requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infovqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"InfoVQA","organization_count":1,"organizations":["llm_stats"],"rank":1557,"released":"2021-04-26","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-infovqatest","caveat":"InfoVQA test set with infographic images requiring joint reasoning over document layout, textual content, graphical elements, and data visualizations with elementary reasoning and arithmetic skills","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/infovqatest?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"InfoVQAtest","organization_count":1,"organizations":["llm_stats"],"rank":1558,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/infovqatest?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-11234","caveat":"InfraBench evaluates AI agents on realistic infrastructure tasks across the system stack and lifecycle, with fine-grained per-check risk assessment and a public leaderboard.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11234"],"document_share":0.0002723311546840959,"domain":"General AI","name":"InfraBench","organization_count":1,"organizations":["claire_radar"],"rank":1559,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.11234"},{"aliases":["InfraOcc: An Infrastructure Occupancy Benchmark with Static-to-Dynamic Reasoning"],"benchmark_id":"claire-radar-2608-30657","caveat":"Evaluates roadside 3D semantic occupancy on 290 multi-modal sequences with static-dynamic annotations, supporting camera, LiDAR, and fusion inputs under a unified scoring protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30657"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"InfraOcc","organization_count":1,"organizations":["claire_radar"],"rank":1560,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30657"},{"aliases":[],"benchmark_id":"claire-radar-2607-24273","caveat":"INS-ActBench evaluates actuarial capability in LLMs across knowledge, case reasoning, and tool-based practice with 12,050 tasks from public exams.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24273"],"document_share":0.0002723311546840959,"domain":"Finance","name":"INS-ActBench","organization_count":1,"organizations":["claire_radar"],"rank":1561,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24273"},{"aliases":[],"benchmark_id":"claire-radar-2610-09671","caveat":"Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decision tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.09671"],"document_share":0.0002723311546840959,"domain":"General AI","name":"InsClaimBench","organization_count":1,"organizations":["claire_radar"],"rank":1562,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.09671"},{"aliases":[],"benchmark_id":"claire-radar-2609-01383","caveat":"Evaluates multimodal agents on claim verification over interactive visualizations, requiring interactions to classify propositions as true, false, or not enough info.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01383"],"document_share":0.0002723311546840959,"domain":"General AI","name":"InSight","organization_count":1,"organizations":["claire_radar"],"rank":1563,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01383"},{"aliases":[],"benchmark_id":"claire-radar-2608-16203","caveat":"INSPIRE evaluates instruction-aware speech retrieval, where natural-language instructions specify relevance criteria including semantic content, speaker identity, speaking style, environmental sounds, and combinations. The benchmark includes a fixed dataset and protocol for evaluating retrieval systems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16203"],"document_share":0.0002723311546840959,"domain":"General AI","name":"INSPIRE","organization_count":1,"organizations":["claire_radar"],"rank":1564,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16203"},{"aliases":[],"benchmark_id":"claire-radar-2609-33233","caveat":"Scientific literature search often begins with an open research problem rather than a known target paper or a fixed candidate set.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33233"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Inspire","organization_count":1,"organizations":["claire_radar"],"rank":1565,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33233"},{"aliases":[],"benchmark_id":"claire-radar-2610-09478","caveat":"Referring Expression Segmentation (RES) links natural-language descriptions to pixel-level object masks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.09478"],"document_share":0.0002723311546840959,"domain":"General AI","name":"InstanceBench","organization_count":1,"organizations":["claire_radar"],"rank":1566,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.09478"},{"aliases":[],"benchmark_id":"llm-stats-instruct-humaneval","caveat":"Instruction-based variant of HumanEval benchmark for evaluating large language models' code generation capabilities with functional correctness using pass@k metric on programming problems","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/instruct-humaneval?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"Instruct HumanEval","organization_count":1,"organizations":["llm_stats"],"rank":1567,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/instruct-humaneval?top_n=500"},{"aliases":["Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation"],"benchmark_id":"claire-radar-2608-02639","caveat":"A repeatable evaluation of instruction following as multiple verifier-checked constraints accumulate in a prompt.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02639"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Instruction Stacking Collapse","organization_count":1,"organizations":["claire_radar"],"rank":1568,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.02639"},{"aliases":[],"benchmark_id":"claire-radar-2608-22990","caveat":"Evaluates instruction-following in robot manipulation through pick-and-place tasks requiring language grounding across category, attribute, spatial, and compositional skills.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22990v1"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"InstructMove","organization_count":1,"organizations":["claire_radar"],"rank":1569,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.22990v1"},{"aliases":[],"benchmark_id":"opencompass-1149-instrusum","caveat":"InstruSum evaluates the task of instruction controllable text summarization, where the model input consists of both a source article and a natural language requirement for desired summary characteristics. InstruSum 是评测指令可控的文本摘要任务，模型输入包括源文章和对所需摘要特征的自然语言要求。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InstruSum"],"document_share":0.0002723311546840959,"domain":"理解","name":"InstruSum","organization_count":1,"organizations":["opencompass_hub"],"rank":1570,"released":"2024-06-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/InstruSum"},{"aliases":[],"benchmark_id":"claire-radar-2608-20220","caveat":"Evaluates whether LLMs recognize legally material missing information in underspecified queries and refrain from premature conclusions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.20220"],"document_share":0.0002723311546840959,"domain":"General AI","name":"InsufficiencyBench","organization_count":1,"organizations":["claire_radar"],"rank":1571,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.20220"},{"aliases":[],"benchmark_id":"claire-radar-2607-21306","caveat":"Int-Bench is a simulation-based evaluation for LLM intervention behavior in tutoring. It simulates a student solving problems across code debugging, mathematics, and brain teasers, with a teacher deciding whether, when, and how to intervene, measuring frequency, timing, and impact on task success and generalization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21306"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Int-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1572,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21306"},{"aliases":[],"benchmark_id":"claire-radar-2608-12345","caveat":"IntegrityBench evaluates language models on research integrity tasks, including misconduct classification, ethical action reasoning, and artifact-grounded decision making, across 36 paired tasks with a 5-level pressure protocol spanning multiple domains and research stages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12345"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IntegrityBench","organization_count":1,"organizations":["claire_radar"],"rank":1573,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.12345"},{"aliases":[],"benchmark_id":"claire-radar-2608-13239","caveat":"Intentbench-Prime is a cleaned subset of the IntentBench audio-visual social QA benchmark with broken and trivially answerable questions removed.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13239"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"Intentbench-Prime","organization_count":1,"organizations":["claire_radar"],"rank":1574,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13239"},{"aliases":[],"benchmark_id":"claire-radar-2605-23187","caveat":"IntentionNav evaluates active object search from implicit human instructions in 176 Isaac Sim scenes. Episodes provide free-text intent, RGB-D observations, and pose, with the target object withheld. The benchmark includes 500 intents over 64 categories and four intent modes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23187"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"IntentionNav","organization_count":1,"organizations":["claire_radar"],"rank":1575,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23187"},{"aliases":["Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis"],"benchmark_id":"claire-radar-2608-20312","caveat":"Large-scale benchmark with 11,388 interaction sequences for multimodal human-human interaction analysis, covering four task categories in generation and perception.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.20312"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Inter-X++","organization_count":1,"organizations":["claire_radar"],"rank":1576,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.20312"},{"aliases":[],"benchmark_id":"claire-radar-2608-29632","caveat":"InteractBench evaluates LLM code generation on interactive competitive programming problems where solutions must query a judge to obtain hidden information under protocol constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29632"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"InteractBench","organization_count":1,"organizations":["claire_radar"],"rank":1577,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29632"},{"aliases":[],"benchmark_id":"claire-radar-2609-36923","caveat":"Existing reactive Graphical User Interface (GUI) agents often fail in long-horizon, dynamic scenarios, where unexpected disturbances trigger attention-diverting and cascading failures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36923"],"document_share":0.0002723311546840959,"domain":"Information retrieval","name":"InterfereBench","organization_count":1,"organizations":["claire_radar"],"rank":1578,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36923"},{"aliases":[],"benchmark_id":"claire-radar-2606-31308","caveat":"InterFLOPBench is a benchmark of 90 C kernels and 1,130 test samples for evaluating LLMs on floating-point error classification across six categories: cancellation, comparison, division by zero, overflow, underflow, and NaN.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31308"],"document_share":0.0002723311546840959,"domain":"General AI","name":"InterFLOPBench","organization_count":1,"organizations":["claire_radar"],"rank":1579,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31308"},{"aliases":[],"benchmark_id":"llm-stats-intergps","caveat":"Interpretable Geometry Problem Solver (Inter-GPS) with Geometry3K dataset of 3,002 geometry problems with dense annotation in formal language using theorem knowledge and symbolic reasoning","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/intergps?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"InterGPS","organization_count":1,"organizations":["llm_stats"],"rank":1580,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/intergps?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-internal-api-instruction-following-hard","caveat":"Internal API instruction following (hard) benchmark - specific documentation not found in official sources","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/internal-api-instruction-following-%28hard%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"structured_output","name":"Internal API instruction following (hard)","organization_count":1,"organizations":["llm_stats"],"rank":1581,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-api-instruction-following-%28hard%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-internal-research-debugging-evaluation","caveat":"The Internal Research Debugging Evaluation measures whether models can debug 41 real bugs from internal OpenAI research experiments (plus alignment-auditing tasks), where the original solutions took experienced researchers hours to days. Passing corresponds to providing assistance that would unblock the user, including partial root-cause explanations or fixes.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/internal-research-debugging-evaluation?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Internal Research Debugging Evaluation","organization_count":1,"organizations":["llm_stats"],"rank":1582,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/internal-research-debugging-evaluation?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2151-interndata-a1","caveat":"InternData-A1: A hybrid synthetic-real manipulation dataset integrating 5 heterogeneous robots, 15 skills, and 200+ scenes, emphasizing multi-robot collaboration under dynamic scenarios. InternData-A1：一个融合了 5 种异构机器人、15 项技能和 200+ 场景的混合合成-真实操作数据集，重点关注动态场景下的多机器人协作。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InternData-A1"],"document_share":0.0002723311546840959,"domain":"多模态","name":"InternData-A1","organization_count":1,"organizations":["opencompass_hub"],"rank":1583,"released":"2025-07-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/InternData-A1"},{"aliases":[],"benchmark_id":"opencompass-2153-interndata-m1","caveat":"InternData-M1: A large-scale synthetic dataset for generalizable pick-and-place over 80K objects, with open-ended instructions covering object recognition, spatial and commonsense reasoning, and long-horizon tasks. InternData-M1：一个大规模合成数据集，用于可泛化的抓取与放置任务，涵盖 8 万+ 对象，配有开放式指令，涉及物体识别、空间与常识推理以及长时任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InternData-M1"],"document_share":0.0002723311546840959,"domain":"多模态","name":"InternData-M1","organization_count":1,"organizations":["opencompass_hub"],"rank":1584,"released":"2025-07-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/InternData-M1"},{"aliases":[],"benchmark_id":"opencompass-2152-interndata-n1","caveat":"InternData-N1: A high-quality navigation dataset with the most diverse scenes and extensive randomization across embodiments/viewpoints, including 3k+ scenes and 830k VLN data. InternData-N1：一个高质量导航数据集，具有最丰富的场景和跨具身/视角的广泛随机化，包含 3000+ 场景和 83 万条 VLN 数据。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/InternData-N1"],"document_share":0.0002723311546840959,"domain":"多模态","name":"InternData-N1","organization_count":1,"organizations":["opencompass_hub"],"rank":1585,"released":"2025-07-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/InternData-N1"},{"aliases":[],"benchmark_id":"claire-radar-2609-15455","caveat":"Benchmark of 210 domestic companion-robot scenarios pairing human and LLM behavior preferences.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15455"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"InterSocialBench","organization_count":1,"organizations":["claire_radar"],"rank":1586,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15455"},{"aliases":["hil-safety-benchmark"],"benchmark_id":"claire-radar-github-yanliang-li-hil-safety-benchmark","caveat":"Evaluates human-in-the-loop safety for tool-using agents over 250 synthetic tasks with four reply types, measuring unsafe action rates, consultation quality, and post-reply behavior.","document_count":1,"document_ids":["claire_radar:https://github.com/yanliang-li/hil-safety-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"InterveneBench","organization_count":1,"organizations":["claire_radar"],"rank":1587,"released":"2026-09-12","source":"claire_radar","url":"https://github.com/yanliang-li/hil-safety-benchmark"},{"aliases":[],"benchmark_id":"opencompass-1976-intphys2","caveat":"IntPhys2 tests intuitive physics—permanence, immutability, continuity and solidity—using synthetic videos. SOTA models perform around chance (~50 %), far below human level. IntPhys 2，一个用于评估深度学习模型直观物理理解能力的视频基准。它围绕永恒性、不可变性、时空连续性和实体性四个核心原则，测试模型区分可能与不可能事件的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IntPhys2"],"document_share":0.0002723311546840959,"domain":"理解","name":"IntPhys2","organization_count":1,"organizations":["opencompass_hub"],"rank":1588,"released":"2025-06-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/IntPhys2"},{"aliases":["Inverse IFEval"],"benchmark_id":"inverse_ifeval","caveat":"Tests ability to override training biases and follow counterintuitive instructions; complementary to IFEval.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"instruction_following","name":"Inverse IFEval","organization_count":1,"organizations":["ByteDance"],"rank":1589,"released":"2025-09-04","source":"model_reports","url":"https://arxiv.org/abs/2509.04292"},{"aliases":[],"benchmark_id":"claire-radar-2606-21844","caveat":"The benchmark evaluates language models on distinguishing human-only vs. human-AI multi-turn dialogues, using paired transcripts and accuracy as the metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21844"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Inverse Turing Bench","organization_count":1,"organizations":["claire_radar"],"rank":1590,"released":"2026-06-20","source":"claire_radar","url":"https://arxiv.org/abs/2606.21844"},{"aliases":[],"benchmark_id":"claire-radar-2608-06108","caveat":"InvestLogicBench evaluates large language models on personalized investment decision-making using 201,247 documented decisions from 151 real-world investors. Each episode traces investor profile, market events, reasoning, decision, and outcome. Tasks include comprehension, profile-conditioned generation, and end-to-end replay.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06108"],"document_share":0.0002723311546840959,"domain":"Finance","name":"InvestLogicBench","organization_count":1,"organizations":["claire_radar"],"rank":1591,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06108"},{"aliases":[],"benchmark_id":"claire-radar-2606-25984","caveat":"InvestPhilBench is a multi-layer benchmark for evaluating procedural reasoning in investment philosophy, spanning eight cognitive tiers from principle identification to novel framework extrapolation. It includes principle cards, decision-framework cards, and QA questions, with an automated scoring pipeline (BASP) and five algorithmic metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25984"],"document_share":0.0002723311546840959,"domain":"Finance","name":"InvestPhilBench","organization_count":1,"organizations":["claire_radar"],"rank":1592,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25984"},{"aliases":[],"benchmark_id":"claire-radar-2606-09764","caveat":"iOSWorld evaluates phone agents on interactive tasks within a native iOS simulator. It includes 26 custom iOS apps with connected personal data, 133 tasks across single-app, multi-app, and memory/personalization categories. Agents are evaluated with rubric-based scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09764"],"document_share":0.0002723311546840959,"domain":"Agents","name":"iOSWorld","organization_count":1,"organizations":["claire_radar"],"rank":1593,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09764"},{"aliases":["little m: An AI Agent for Industrial Process Optimization"],"benchmark_id":"claire-radar-2609-16680","caveat":"IPC-Bench is a multimodal benchmark of 50 industrial process optimization scenarios requiring joint reasoning over text and process diagrams to formulate mathematical optimization models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16680"],"document_share":0.0002723311546840959,"domain":"Agents","name":"IPC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1594,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.16680"},{"aliases":[],"benchmark_id":"llm-stats-ipho-2025","caveat":"International Physics Olympiad 2025 (theory) comprises all 3 theory problems from the official 2025 IPhO competition. Results are based on blinded human evaluation with guidelines based on the official competition scoring, validated by domain experts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ipho-2025?top_n=500"],"document_share":0.0002723311546840959,"domain":"physics","name":"IPhO 2025","organization_count":1,"organizations":["llm_stats"],"rank":1595,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ipho-2025?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-27074","caveat":"Benchmark for interactive proactive intelligence of MLLMs under streaming video, covering proactive monitoring, task management, and interleaved requests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27074"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IPIBench","organization_count":1,"organizations":["claire_radar"],"rank":1596,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27074"},{"aliases":["IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO"],"benchmark_id":"claire-radar-2606-23032","caveat":"IPO Finance Agent evaluates language-model financial analysts on IPO due-diligence questions using expert-reviewed evaluation rubrics and accuracy reporting.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23032"],"document_share":0.0002723311546840959,"domain":"Agents","name":"IPO Finance Agent","organization_count":1,"organizations":["claire_radar"],"rank":1597,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23032"},{"aliases":[],"benchmark_id":"opencompass-1840-iqbench","caveat":"IQBench is a novel benchmark designed to evaluate the fluid intelligence of VisionLanguage Models (VLMs) using standardized visual IQ tests. It consists of 500 manually collected and annotated visual IQ questions covering various domains. IQBench是一个新基准测试，旨在通过标准化视觉智商测试评估视觉语言模型（VLMs）的流体智力。该基准包含500个手动收集和注释的视觉智商问题，涵盖模式识别、类比推理、视觉算术、空间理解等多个领域。与以往仅关注最终答案准确性的基准不同，IQBench强调对模型推理能力的评估，采用双重评估框架：准确性评分和推理评分。实验表明，即使是性能最高的模型（如o4mini、gemini2.5flash和claude3.7sonnet），在3D空间和字母重组任务上也表现出明显不足，凸显了当前VLMs在通用推理能力上的局限性。IQBench为开发更透明、更具认知能力的多模态系统奠定了基础。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IQBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"IQBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1598,"released":"2025-05-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/IQBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-22380","caveat":"A benchmark for infrared multi-frame super-resolution containing 594 video sequences and 275,196 frames with fixed sequence-level splits and a reproducible x4 evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22380"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IR275K","organization_count":1,"organizations":["claire_radar"],"rank":1599,"released":"2026-07-24","source":"claire_radar","url":"https://arxiv.org/abs/2607.22380"},{"aliases":[],"benchmark_id":"claire-radar-2606-15107","caveat":"IRTS-ToolBench is a benchmark of 1,700 questions across 10 task types and 13 domains for evaluating irregular univariate time-series question answering, with standardized inputs and a reproducible evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15107"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IRTS-ToolBench","organization_count":1,"organizations":["claire_radar"],"rank":1600,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2606.15107"},{"aliases":[],"benchmark_id":"opencompass-2091-is-bench","caveat":"IS-Bench is the first evaluation benchmark dedicated to assessing the safety of embodied agents during their interaction with home environments. It encompasses 161 high-risk domestic scenarios, spanning 10 hazard categories, including food poisoning, fire, electric shock, and more. IS-Bench 是首个专注于具身智能体与家用环境交互过程安全性的评测基准。它包含 161 个高风险家居场景及相关日常任务，覆盖食物中毒、火灾、触电等 10 大类常见风险。通过贯穿整个交互过程的动态评测框架，全方位评估具身智能体的安全素养。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/IS-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"IS-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1601,"released":"2025-07-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/IS-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2609-22878","caveat":"A benchmark for computational reasoning using programming games with constrained instruction sets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.22878"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"ISA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1602,"released":"2026-09-19","source":"claire_radar","url":"https://arxiv.org/abs/2609.22878"},{"aliases":[],"benchmark_id":"claire-radar-2605-22079","caveat":"A benchmark for generating Information Delivery Specification (IDS) XML from BIM information requirements, with 166 examples in English and Japanese. It evaluates LLMs on formal validity via IDSAuditTool and content fidelity via facet-level macro-F1 against gold IDS files.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22079"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Ishigaki-IDS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1603,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22079"},{"aliases":[],"benchmark_id":"claire-radar-2608-04703","caveat":"Multi-task dataset of 3,465 QA items from 35 Islamic scholarly works, spanning 12 centuries and seven fields, with three difficulty levels and three task formats.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04703"],"document_share":0.0002723311546840959,"domain":"General AI","name":"IslamicTurathBench","organization_count":1,"organizations":["claire_radar"],"rank":1604,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04703"},{"aliases":[],"benchmark_id":"claire-radar-2607-01431","caveat":"IsoSci evaluates reasoning vs. knowledge retrieval in LLMs using isomorphic cross-domain problem pairs, enabling controlled attribution of reasoning gains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01431"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"IsoSci","organization_count":1,"organizations":["claire_radar"],"rank":1605,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.01431"},{"aliases":["I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning"],"benchmark_id":"claire-radar-2608-07417","caveat":"Evaluates person-centric video reasoning using identity-conditioned queries, with 1,377 real-world videos and QA pairs across six difficulty levels covering identity recognition to causal reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07417"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ISYV-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1606,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07417"},{"aliases":[],"benchmark_id":"opencompass-2077-itbench","caveat":"ITBench is a benchmark designed to evaluate the performance of AI agents in real-world IT automation tasks. ITBench 是一个旨在评估 AI 智能体在真实世界 IT 自动化任务中表现的基准。它涵盖了站点可靠性工程、合规与安全运营以及财务运营等关键维度，并包含 102 个真实场景。该基准提供了一个开源框架和多种基线智能体实现，并集成了CrewAI等工具，以促进AI驱动的IT自动化发展。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ITBench"],"document_share":0.0002723311546840959,"domain":"代码","name":"ITBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1607,"released":"2025-02-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ITBench"},{"aliases":[],"benchmark_id":"artificial-analysis-itbench-aa","caveat":"Kubernetes incident root-cause analysis","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/itbench-aa"],"document_share":0.0002723311546840959,"domain":"agentic","name":"ITBench-AA","organization_count":1,"organizations":["artificial_analysis"],"rank":1608,"released":null,"source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/itbench-aa"},{"aliases":[],"benchmark_id":"claire-radar-2607-19407","caveat":"ITPEval evaluates automated formal proof translation across four interactive theorem provers (Lean 4, Rocq, Isabelle, HOL Light), with 1,560 source files and 6,848 theorems, covering statement and proof translation on 12 directed pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19407"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"ITPEval","organization_count":1,"organizations":["claire_radar"],"rank":1609,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.19407"},{"aliases":[],"benchmark_id":"opencompass-2073-j1-bench","caveat":"J1-Bench is an interactive and comprehensive legal benchmark where LLM agents engage in diverse legal scenarios, completing tasks through interactions with various participants under procedural rules. J1-Bench 是一个交互式的综合法律基准，法律智能体在此参与各种法律情景，根据程序规则通过与不同参与者的互动完成指定的法律任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/J1-Bench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"J1-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1610,"released":"2025-07-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/J1-Bench"},{"aliases":[],"benchmark_id":"opencompass-1337-jailtrickbench","caveat":"JailTrickBench can evaluate the impact of various attack settings on LLM performance, including 8 key factors of implementing jailbreak attacks on LLMs from both target-level and attack-level perspectives. JailTrickBench用于评估LLM应对各种越狱攻击的能力，涵盖从目标级和攻击级2个角度实施越狱攻击的8个关键因素。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/JailTrickBench"],"document_share":0.0002723311546840959,"domain":"安全","name":"JailTrickBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1611,"released":"2024-06-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/JailTrickBench"},{"aliases":[],"benchmark_id":"claire-radar-2608-07508","caveat":"JaleesBench evaluates AI-agent counsel across 140 two-turn faith-related scenarios spanning adversarial pressures and framings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07508"],"document_share":0.0002723311546840959,"domain":"Agents","name":"JaleesBench","organization_count":1,"organizations":["claire_radar"],"rank":1612,"released":"2026-06-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.07508"},{"aliases":["JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines"],"benchmark_id":"claire-radar-2606-19830","caveat":"JamBench evaluates project-level game code framework generation and code completion on 300 manually verified Godot projects using deterministic compilation, structural-completeness, and behavioral-alignment measures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19830"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"JamBench","organization_count":1,"organizations":["claire_radar"],"rank":1613,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.19830"},{"aliases":[],"benchmark_id":"claire-radar-2606-10852","caveat":"JANUS is a benchmark with 160 scenarios across 8 domains, each providing a fixed pool of favorable and adverse facts and paired neutral and goal-directed prompts, to evaluate fact-grounded, goal-conditioned pragmatic distortion in LLM outputs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10852"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Janus","organization_count":1,"organizations":["claire_radar"],"rank":1614,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.10852"},{"aliases":[],"benchmark_id":"claire-radar-2607-16610","caveat":"JarvisBench measures mediation in long-horizon agent workflows with two tracks: agent-collaboration and user-interaction. Built on WildClaw tasks and a reference Jarvis prototype.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.16610"],"document_share":0.0002723311546840959,"domain":"Agents","name":"JarvisBench","organization_count":1,"organizations":["claire_radar"],"rank":1615,"released":"2026-07-18","source":"claire_radar","url":"https://arxiv.org/abs/2607.16610"},{"aliases":[],"benchmark_id":"claire-radar-2609-10451","caveat":"Proposed benchmark for cross-device GUI agent workflows across Android, Windows, and Ubuntu, using input-output transformations and dynamic task composition.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10451"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"JarvisGUI","organization_count":1,"organizations":["claire_radar"],"rank":1616,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10451"},{"aliases":[],"benchmark_id":"claire-radar-2607-02825","caveat":"JavaVulBench is a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset includes about 30,600 Java methods spanning 1,740 CVEs and 700+ projects, with method and line labels, publication dates, and five split strategies. The harness provides a unified schema across multiple backends and includes a contamination audit.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02825"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"JavaVulBench","organization_count":1,"organizations":["claire_radar"],"rank":1617,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02825"},{"aliases":[],"benchmark_id":"claire-radar-2605-23543","caveat":"JEDI is a benchmark suite for the Java Stream API, generated by converting SQL benchmarks into Java benchmarks. It includes both stream-based and imperative query implementations to evaluate performance of parallelization strategies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23543"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"JEDI","organization_count":1,"organizations":["claire_radar"],"rank":1618,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23543"},{"aliases":[],"benchmark_id":"claire-radar-2606-31767","caveat":"JETO-Bench is a benchmark of 660 execution time improvement patches (ETIPs) in Java, with 91 manually verified. It is built using JETO-Mine, a configurable tool that mines ETIPs from GitHub repositories and statistically validates improvements.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31767"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"JETO-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1619,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31767"},{"aliases":["jev-vs-llm-benchmark"],"benchmark_id":"claire-radar-github-akashai7-jev-vs-llm-benchmark","caveat":"Compares Jev, a general-purpose LLM, and a confidence-gated hybrid on fixed-choice decisions using 18 labelled cases across support routing, incident priority, and request intent.","document_count":1,"document_ids":["claire_radar:https://github.com/AkashAi7/jev-vs-llm-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Jev vs LLM Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1620,"released":"2026-09-23","source":"claire_radar","url":"https://github.com/AkashAi7/jev-vs-llm-benchmark"},{"aliases":["jev-deterministic-benchmark"],"benchmark_id":"claire-radar-github-etsabary-jev-deterministic-benchmark","caveat":"1,000 deterministic reasoning decisions across 25 families used to test model Jev.","document_count":1,"document_ids":["claire_radar:https://github.com/etsabary/jev-deterministic-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"JEV-DD-1.0","organization_count":1,"organizations":["claire_radar"],"rank":1621,"released":"2026-09-22","source":"claire_radar","url":"https://github.com/etsabary/jev-deterministic-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-31142","caveat":"Evaluates adversarial robustness of RLCD models with typed questions and single-edit attack variants.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.31142"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"JevAdvBench","organization_count":1,"organizations":["claire_radar"],"rank":1622,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.31142"},{"aliases":[],"benchmark_id":"claire-radar-2607-13245","caveat":"JITOMA-Bench is a suite for long-horizon multi-tasking and multi-step reasoning in robotics, focusing on just-in-time scene graph growth to combat perceptual saturation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.13245"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"JITOMA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1623,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.13245"},{"aliases":[],"benchmark_id":"claire-radar-2606-31745","caveat":"JL1-CC&QA is a multi-task benchmark extending JL1-CD with change captioning and question answering. It includes 17,021 captions and 20,060 QA pairs over 5,000 bi-temporal image pairs from Jilin-1 satellite.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31745"],"document_share":0.0002723311546840959,"domain":"General AI","name":"JL1-CC&QA","organization_count":1,"organizations":["claire_radar"],"rank":1624,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31745"},{"aliases":[],"benchmark_id":"opencompass-1554-jl1-cd","caveat":"JL1-CD is a large-scale, sub-meter, all-inclusive open-source dataset for remote sensing image change detection (CD). It contains 5,000 pairs of 512×512 pixel satellite images with a resolution of 0.5 to 0.75 meters, covering various types of surface changes in multiple regions of China. JL1-CD 是一个大规模、亚米级、全包含的开源遥感影像变化检测（CD）数据集。它包含 5000 对 512×512 像素的卫星影像，分辨率为 0.5 至 0.75 米，覆盖中国多个地区的各种地表变化。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/JL1-CD"],"document_share":0.0002723311546840959,"domain":"其他","name":"JL1-CD","organization_count":1,"organizations":["opencompass_hub"],"rank":1625,"released":"2025-02-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/JL1-CD"},{"aliases":[],"benchmark_id":"claire-radar-2605-22080","caveat":"JMed48k evaluates vision-language models on 48,862 Japanese medical licensing exam questions from 11 national examinations (2005-2025), with images annotated under an 8-type taxonomy. The JMed48k-Eval subset contains 12,484 scored questions, including text-only and with-image items, scored separately.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22080"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"JMed48k","organization_count":1,"organizations":["claire_radar"],"rank":1626,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22080"},{"aliases":[],"benchmark_id":"llm-stats-job-bench","caveat":"Job Bench evaluates AI agents on realistic professional tasks that require multi-step planning, research, and production of work artifacts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/job-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"productivity","name":"Job Bench","organization_count":1,"organizations":["llm_stats"],"rank":1627,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/job-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2130-job-complex","caveat":"JOB-Complex is a challenging benchmark for traditional and learned query optimizers, containing 30 SQL queries and a plan-selection benchmark with nearly 6000 execution plans. JOB-Complex是一个面向传统与学习式查询优化器的挑战性数据库基准，包含30个SQL查询和近6000个执行计划的计划选择基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/JOB-Complex"],"document_share":0.0002723311546840959,"domain":"其他","name":"JOB-Complex","organization_count":1,"organizations":["opencompass_hub"],"rank":1628,"released":"2025-07-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/JOB-Complex"},{"aliases":[],"benchmark_id":"jobbench","caveat":"Professional job tasks aligned with human work; rubric grading moves the number beyond model capability alone.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"professional","name":"JobBench","organization_count":1,"organizations":["Tencent"],"rank":1629,"released":"2026-05-25","source":"model_reports","url":"https://arxiv.org/abs/2605.26329"},{"aliases":["JointAVBench","Joint AV-Bench"],"benchmark_id":"jointavbench","caveat":"An ICLR 2026 poster on joint audio-visual reasoning in Omni-LLMs, built with strict audio-video correlation across five cognitive dimensions and four audio information types. Because the question set requires both modalities, vendors that only report vision-only or audio-only figures are not comparable on this instrument. Automated annotation pipeline; manual verification effort is not quantified.","document_count":1,"document_ids":["model_reports:jointavbench_v2_results_table"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"JointAVBench","organization_count":1,"organizations":["Renmin University of China"],"rank":1630,"released":"2025-12-14","source":"model_reports","url":"https://arxiv.org/abs/2512.12772"},{"aliases":[],"benchmark_id":"claire-radar-2607-16777","caveat":"JOR-Bench is a collection of five Japanese-language benchmarks for LLMs in operations research, covering 1,319 problems from IndustryOR, MAMO, NL4OPT, OptiBench, and OptMATH, with pairs of Japanese problem statements and numerical answers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.16777"],"document_share":0.0002723311546840959,"domain":"General AI","name":"JOR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1631,"released":"2026-07-18","source":"claire_radar","url":"https://arxiv.org/abs/2607.16777"},{"aliases":["Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis"],"benchmark_id":"claire-radar-2606-25369","caveat":"A Japanese speech-synthesis pronunciation benchmark covering the officially designated Joyo kanji and their readings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25369"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Joyo Kanji Yomi Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1632,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25369"},{"aliases":[],"benchmark_id":"claire-radar-2609-35032","caveat":"Evaluates embodied visual reasoning requiring active observation requests and evidence grounding, with questions derived from real-world JRDB robotics data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35032"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"JRDB-AVR","organization_count":1,"organizations":["claire_radar"],"rank":1633,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35032"},{"aliases":[],"benchmark_id":"opencompass-1583-judgebench","caveat":"JudgeBench is a benchmark aimed at evaluating LLM-based judges for objective correctness on challenging response pairs. JudgeBench 是一个旨在评估基于LLM的裁判在具有挑战性的响应对上的客观正确性的基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/JudgeBench"],"document_share":0.0002723311546840959,"domain":"其他","name":"JudgeBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1634,"released":"2024-10-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/JudgeBench"},{"aliases":[],"benchmark_id":"claire-radar-2605-25240","caveat":"A dataset of 30 legal tasks with rubric scores and pairwise preference judgments from practicing attorneys, used to compare rubric-based scoring and comparative judgment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25240"],"document_share":0.0002723311546840959,"domain":"General AI","name":"JudgmentBench","organization_count":1,"organizations":["claire_radar"],"rank":1635,"released":"2026-05-24","source":"claire_radar","url":"https://arxiv.org/abs/2605.25240"},{"aliases":[],"benchmark_id":"claire-radar-2609-09887","caveat":"Evaluates LLM-simulated jurors on controversial U.S. criminal law cases with defendant statements of varying backgrounds and emotional appeal, measuring verdict severity changes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09887"],"document_share":0.0002723311546840959,"domain":"General AI","name":"JuryBench","organization_count":1,"organizations":["claire_radar"],"rank":1636,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09887"},{"aliases":[],"benchmark_id":"claire-radar-2609-12808","caveat":"K-Bench is described as a benchmark that scores LLM unlearning under agentic deployment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12808"],"document_share":0.0002723311546840959,"domain":"General AI","name":"K-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1637,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12808"},{"aliases":[],"benchmark_id":"claire-radar-2609-15855","caveat":"A clinician-calibrated benchmark assessing large language models on 200 multi-turn mental health vignettes spanning suicide, self-harm, domestic violence, substance misuse, and no-risk scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15855"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"K-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1638,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15855"},{"aliases":[],"benchmark_id":"claire-radar-2606-02404","caveat":"K-BrowseComp evaluates web-browsing agents on 400 Korean-context problems, requiring multi-hop or parallel evidence retrieval from public Korean websites and returning a single short answer. Includes a 300-problem manually verified subset and a 100-problem synthetic stress-test split.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02404"],"document_share":0.0002723311546840959,"domain":"Agents","name":"K-BrowseComp","organization_count":1,"organizations":["claire_radar"],"rank":1639,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02404"},{"aliases":[],"benchmark_id":"claire-radar-2605-29523","caveat":"Evaluates hallucination detection in multi-turn Korean financial RAG dialogues, with a taxonomy based on context answerability. Includes training and test splits for fine-tuning and benchmarking detectors.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29523"],"document_share":0.0002723311546840959,"domain":"Finance","name":"K-FinHallu","organization_count":1,"organizations":["claire_radar"],"rank":1640,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29523"},{"aliases":[],"benchmark_id":"opencompass-2076-k-sort-arena","caveat":"K-Sort Arena employs K-wise comparison, allowing K models to participate in a free-for-all, providing richer information than pairwise comparison. It also designs a matching algorithm based on exploration-exploitation and probabilistic modeling to achieve more efficient and reliable model ranking. 本项目提出K-Sort Arena，采用 K-wise 比较，允许 K 个模型参与自由混战，提供比成对比较更丰富的信息，并设计基于探索-利用的匹配算法和概率建模，从而实现更高效和更可靠的模型排名。目前，K-Sort Arena 已收集几千次高质量投票并构建了全面的模型排行榜，已用于评估几十种最先进的视觉生成模型，包括文生图和文生视频模型。K-Sort Arena已经历数月的项目内测，期间收到来自加州大学伯克利分校, 新加坡国立大学, 卡内基梅隆大学, 斯坦福大学, 普林斯顿大学, 北京大学等数十家机构的专业人员的技术反馈，现已公开线上发布。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/K-Sort-Arena"],"document_share":0.0002723311546840959,"domain":"多模态","name":"K-Sort-Arena","organization_count":1,"organizations":["opencompass_hub"],"rank":1641,"released":"2024-09-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/K-Sort-Arena"},{"aliases":[],"benchmark_id":"claire-radar-2607-02680","caveat":"K9-Bench is a benchmark focused on real-world domestic dog videos, with approximately 5,000 question-answer pairs across 907 videos spanning 5 task categories. The tasks test long-form, canine-centric multimodal reasoning, including action and interaction understanding. A VLM/LLM-powered pipeline was used for data generation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02680"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"K9-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1642,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02680"},{"aliases":[],"benchmark_id":"claire-radar-2609-03588","caveat":"A multi-turn benchmark with 238 tasks using a user simulator, stateful tools, and environment assertions to test LLM agents' handling of knowledge conflicts across world, input, and temporal categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03588"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"KC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1643,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03588"},{"aliases":[],"benchmark_id":"llm-stats-kernel-bench-l3","caveat":"Kernel Bench L3 evaluates agentic GPU kernel optimization across 50 problems. Qwen reports two metrics for this benchmark: median per-problem speedup over the PyTorch eager reference and the fraction of problems faster than torch.compile.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kernel-bench-l3?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Kernel Bench L3","organization_count":1,"organizations":["llm_stats"],"rank":1644,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/kernel-bench-l3?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-kernelbench-hard","caveat":"KernelBench Hard evaluates agentic GPU kernel optimization on the hardest problem set. Each question is scored by the agent's submitted operator TFLOPs relative to the theoretical peak of the current hardware, with the benchmark score being the average across all questions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kernelbench-hard?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"KernelBench Hard","organization_count":1,"organizations":["llm_stats"],"rank":1645,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelbench-hard?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-kernelgen-1p","caveat":"KernelGen 1P is an OpenAI AI-self-improvement evaluation that measures whether models can write and optimize compute kernels, part of the suite tracking progress toward accelerating internal research.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kernelgen-1p?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"KernelGen 1P","organization_count":1,"organizations":["llm_stats"],"rank":1646,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/kernelgen-1p?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-27231","caveat":"KernelGenBench evaluates LLM- and agent-generated Triton kernels across 210 operators from three sources (ATen, vLLM, cuBLAS) and six hardware platforms, with automatic accuracy verification and two evaluation tracks (LLM Pass@K and iterative agent generation).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27231"],"document_share":0.0002723311546840959,"domain":"Agents","name":"KernelGenBench","organization_count":1,"organizations":["claire_radar"],"rank":1647,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.27231"},{"aliases":[],"benchmark_id":"claire-radar-2609-25591","caveat":"A benchmark for evaluating coding agents on exploit primitive generation against real operating-system kernels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25591"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"KEX-bench","organization_count":1,"organizations":["claire_radar"],"rank":1648,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.25591"},{"aliases":[],"benchmark_id":"claire-radar-2607-14202","caveat":"KeyFrame-Compass evaluates keyframe-conditioned video generation across 386 curated samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities. It jointly measures keyframe execution (presence, fidelity, temporal ordering, localization, persistence, uniqueness) and overall video quality via evidence-grounded MLLM judgments and specialized perception models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14202"],"document_share":0.0002723311546840959,"domain":"General AI","name":"KeyFrame-Compass","organization_count":1,"organizations":["claire_radar"],"rank":1649,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.14202"},{"aliases":["Evaluation of Pipelines for Data Integration into Knowledge Graphs"],"benchmark_id":"claire-radar-2605-22304","caveat":"KGI-Bench evaluates pipelines that integrate multiple input-data formats into an existing knowledge graph.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22304"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"KGI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1650,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22304"},{"aliases":[],"benchmark_id":"claire-radar-2609-03597","caveat":"A four-task benchmark built from 107 real RS Khatian land records evaluating symbol recognition, base-16 conversion, structured field extraction, and legal QA with 1,634 QA pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03597"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"KhatianDoc","organization_count":1,"organizations":["claire_radar"],"rank":1651,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03597"},{"aliases":[],"benchmark_id":"claire-radar-2605-25510","caveat":"KIDBench evaluates child-facing safety of LLMs for ages 7-11 using realistic queries and multi-turn child-actor simulations, scored by an LLM-as-a-Judge rubric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25510"],"document_share":0.0002723311546840959,"domain":"General AI","name":"KIDBench","organization_count":1,"organizations":["claire_radar"],"rank":1652,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25510"},{"aliases":[],"benchmark_id":"llm-stats-kimi-claw-24-7-bench","caveat":"Kimi Claw 24/7 Bench is Moonshot AI's in-house benchmark for evaluating long-horizon agentic performance in persistent, multi-day coworking tasks. It spans 17 professional scenarios across 610 evaluation points, covering software engineering, ML research, recruiting, trading, and marketing tasks executed through the OpenClaw harness.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kimi-claw-24-7-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Kimi Claw 24/7 Bench","organization_count":1,"organizations":["llm_stats"],"rank":1653,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-claw-24-7-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-kimi-code-bench-v2","caveat":"Kimi Code Bench v2 is Moonshot AI's in-house benchmark for evaluating coding agents on realistic software engineering tasks across 10+ mainstream programming languages and a production tech stack spanning backend services, infrastructure, performance engineering, systems programming, security, frontend development, and ML/data engineering.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kimi-code-bench-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Kimi Code Bench v2","organization_count":1,"organizations":["llm_stats"],"rank":1654,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/kimi-code-bench-v2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-kina","caveat":"KINA is a knowledge-intensive evaluation that measures a model's breadth and depth of factual knowledge across academic and professional domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/kina?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"KINA","organization_count":1,"organizations":["llm_stats"],"rank":1655,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/kina?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-19876","caveat":"KineBench evaluates embodied world models via kinematic grounding, using 20 manipulation tasks in ManiSkill3 and metrics like spectral arc length and manipulability index.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19876"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"KineBench","organization_count":1,"organizations":["claire_radar"],"rank":1656,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.19876"},{"aliases":[],"benchmark_id":"opencompass-1543-kitab-bench","caveat":"KITAB-Bench is a Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding,  and spans 36 sub-domains with over 8,809 samples, carefully curated to rigorously evaluate essential skills required for Arabic OCR and document analysis. KITAB-Bench是一个全面多领域阿拉伯文 OCR 和文档理解基准，包含 36 个子领域，超过 8,809 个样本，经过精心挑选，以严格评估阿拉伯 OCR 和文档分析所需的基本技能。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KITAB-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"KITAB-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1657,"released":"2025-02-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/KITAB-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2609-38480","caveat":"Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38480"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"KlinikeBench","organization_count":1,"organizations":["claire_radar"],"rank":1658,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.38480"},{"aliases":[],"benchmark_id":"opencompass-2126-kmmlu-redux","caveat":"KMMLU-Redux is a reconstructed version of the existing KMMLU, comprising 2,587 problems from Korean National Technical Qualification (KNTQ) exams. KMMLU-Redux是现有 KMMLU 的一个重建版本，包含来自韩国国家技术资格（KNTQ）考试的 2,587 个问题。我们发现了 KMMLU 中的一些关键问题，包括泄露的答案、缺乏清晰度、问题表述不当、符号错误和污染风险。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KMMLU-Redux"],"document_share":0.0002723311546840959,"domain":"知识","name":"KMMLU-Redux","organization_count":1,"organizations":["opencompass_hub"],"rank":1659,"released":"2025-07-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/KMMLU-Redux"},{"aliases":["knowledge_tracing_benchmark"],"benchmark_id":"claire-radar-github-devissaputra-knowledge-tracing-benchmark","caveat":"A learner-disjoint benchmark on ASSISTments 2009 sequences that evaluates knowledge-tracing models on next-response probability with ROC-AUC, average precision, Brier score, log loss, and ECE-10.","document_count":1,"document_ids":["claire_radar:https://github.com/devissaputra/knowledge_tracing_benchmark"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Knowledge Tracing Benchmark on ASSISTments 2009","organization_count":1,"organizations":["claire_radar"],"rank":1660,"released":"2026-09-24","source":"claire_radar","url":"https://github.com/devissaputra/knowledge_tracing_benchmark"},{"aliases":[],"benchmark_id":"opencompass-1635-knowlogic","caveat":"KnowLogic is a knowledge-driven synthetic benchmark designed to evaluate the reasoning abilities of large language models (LLMs). It includes 5400 bilingual (Chinese and English) questions across various domains, covering different aspects of commonsense knowledge and logical reasoning. KnowLogic 是一个以知识驱动的合成基准，旨在评估大型语言模型的推理能力（LLMs）。它包含涵盖各个领域、涵盖常识知识和逻辑推理不同方面的 5400 个中英双语问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KnowLogic"],"document_share":0.0002723311546840959,"domain":"推理","name":"KnowLogic","organization_count":1,"organizations":["opencompass_hub"],"rank":1661,"released":"2025-03-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/KnowLogic"},{"aliases":[],"benchmark_id":"opencompass-1706-koffvqa","caveat":"KOFFVQA is a carefully crafted free-form visual question answering(VQA) benchmark in the Korean language consisting of 275 questions across 10 different tasks. KOFFVQA是一个精心设计的韩语自由形式视觉问答（VQA）基准测试，包含10个不同任务中的275个问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KOFFVQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"KOFFVQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1662,"released":"2025-03-31","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/KOFFVQA"},{"aliases":["Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models"],"benchmark_id":"claire-radar-2609-04720","caveat":"KoNA evaluates selective non-compliance in vision-language models across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. Each task measures query-level and component-level non-compliance using paired single and compound queries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04720"],"document_share":0.0002723311546840959,"domain":"General AI","name":"KoNA","organization_count":1,"organizations":["claire_radar"],"rank":1663,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.04720"},{"aliases":[],"benchmark_id":"claire-radar-2609-19916","caveat":"KoNeoBench evaluates LLM understanding of Korean neologisms through four tasks: context-based identification, component reconstruction, semantic/special-domain classification, and definition generation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19916"],"document_share":0.0002723311546840959,"domain":"General AI","name":"KoNeoBench","organization_count":1,"organizations":["claire_radar"],"rank":1664,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.19916"},{"aliases":["Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe"],"benchmark_id":"claire-radar-2609-05395","caveat":"Evaluates multi-step tool-calling over Korean open public APIs with 145 real-world tasks, designed to measure performance gaps in open-source LLM agents that chain tool calls across live government APIs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05395"],"document_share":0.0002723311546840959,"domain":"General AI","name":"KOPA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1665,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05395"},{"aliases":[],"benchmark_id":"opencompass-1245-kor-bench","caveat":"Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench) encompasses five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. KOR-Bench emphasizes the effectiveness of models in applying new rule descriptions to solve novel rule-driven questions. KOR-Bench用于评估大语言模型的推理能力，包括五个任务类别：操作、逻辑、密码、拼图和反事实。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/KOR-Bench"],"document_share":0.0002723311546840959,"domain":"强推理","name":"KOR-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1666,"released":"2024-10-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/KOR-Bench"},{"aliases":["KORBench"],"benchmark_id":"korbench","caveat":"Knowledge-Orthogonal Reasoning Benchmark; evaluates intrinsic reasoning and planning abilities while minimizing interference from pretrained knowledge.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"KORBench","organization_count":1,"organizations":["ByteDance"],"rank":1667,"released":"2024-10-09","source":"model_reports","url":"https://kor-bench.github.io/"},{"aliases":["Kotlin SWE Benchmark"],"benchmark_id":"claire-radar-jetbrains-kotlin-benchmark-2026","caveat":"Evaluates AI coding agents on real-world Kotlin tasks from nine open-source repositories. The benchmark uses reproducible Docker environments, regression tests, and a scoring protocol that awards pass only when all expected test transitions are met.","document_count":1,"document_ids":["claire_radar:https://kotlinlang.org/benchmark/"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Kotlin Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1668,"released":"2026-07-08","source":"claire_radar","url":"https://kotlinlang.org/benchmark/"},{"aliases":[],"benchmark_id":"claire-radar-2605-28013","caveat":"KSAFE-MM evaluates multimodal LLM safety in Korean contexts, with 12 models tested on general and culture-specific safety risks, including jailbreak-style textual queries paired with local visual cues.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28013"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"KSAFE-MM","organization_count":1,"organizations":["claire_radar"],"rank":1669,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28013"},{"aliases":["KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models"],"benchmark_id":"claire-radar-2608-09412","caveat":"Diagnoses KV-cache compression failures in long-context language models using paired runs and cache, likelihood, attention, and decoding measurements.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09412"],"document_share":0.0002723311546840959,"domain":"General AI","name":"KVDiagnosis","organization_count":1,"organizations":["claire_radar"],"rank":1670,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09412"},{"aliases":[],"benchmark_id":"opencompass-541-l-eval","caveat":"L-Eval is a comprehensive Long Context Language Models (LCLMs) evaluation suite with 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs encompassing diverse question styles, domains, and input length (3k～200k tokens). L-Eval 是一个全面的长上下文语言模型（LCLMs）评估套件，包括 20 个子任务、508 个长文档和超过 2,000 个人工标记的查询-响应对。它涵盖了多种问答风格、领域和输入长度（3,000 至 200,000 个 token）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/L-Eval"],"document_share":0.0002723311546840959,"domain":"长文本","name":"L-Eval","organization_count":1,"organizations":["opencompass_hub"],"rank":1671,"released":"2023-10-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/L-Eval"},{"aliases":[],"benchmark_id":"claire-radar-2608-15535","caveat":"L3Cube-IndicQuest v2 is a multilingual QA benchmark for evaluating India-specific factual knowledge of LLMs. It contains 3,471 curriculum-grounded English QA pairs translated into 19 Indic languages, totaling 69,420 pairs across 20 languages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15535"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"L3Cube-IndicQuest v2","organization_count":1,"organizations":["claire_radar"],"rank":1672,"released":"2026-08-16","source":"claire_radar","url":"https://arxiv.org/abs/2608.15535"},{"aliases":[],"benchmark_id":"llm-stats-labbench2","caveat":"LABBench2 evaluates models on real-world biology research tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/labbench2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LABBench2","organization_count":1,"organizations":["llm_stats"],"rank":1673,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/labbench2?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-16802","caveat":"LabOSBench evaluates multimodal GUI agents on 96 subtasks across eight web-based scientific-instrument simulators, covering workflows from sample loading to result inspection. Agents operate via a browser, with execution-based evaluation on task completion.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16802"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"LabOSBench","organization_count":1,"organizations":["claire_radar"],"rank":1674,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16802"},{"aliases":[],"benchmark_id":"claire-radar-2606-17433","caveat":"LADBench evaluates large vision-language models on detecting logical anomalies in synthetic images across four domains: Residential, Urban, Collaborative, and Nature. It uses a Tiered Prompting Protocol with three progressive disclosure levels and automated scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17433"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LADBench","organization_count":1,"organizations":["claire_radar"],"rank":1675,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.17433"},{"aliases":["AI translation of literary texts is \"fine\", but readers still prefer human translations"],"benchmark_id":"claire-radar-2606-26040","caveat":"LAIT is a reader-centered evaluation resource for comparing human and machine literary translations through excerpt-level and aligned chunk-level judgments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26040"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LAIT (Literary AI Translation)","organization_count":1,"organizations":["claire_radar"],"rank":1676,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.26040"},{"aliases":[],"benchmark_id":"claire-radar-2606-10460","caveat":"LakeQA evaluates search-centric question answering over a 9.5 TB data lake of Wikipedia and government data. Tasks require multi-hop reasoning across heterogeneous structured and unstructured sources, with expert-annotated answers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10460"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LakeQA","organization_count":1,"organizations":["claire_radar"],"rank":1677,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.10460"},{"aliases":[],"benchmark_id":"claire-radar-2607-12310","caveat":"LakeQuest evaluates end-to-end question answering over data lakes with 9,846 QA pairs across three domains (AI/ML metadata, retail banking, biomedical drug info), with exact modality-aware evidence pointers, measuring retrieval and cross-modal synthesis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.12310"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"LakeQuest","organization_count":1,"organizations":["claire_radar"],"rank":1678,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.12310"},{"aliases":[],"benchmark_id":"opencompass-523-lambada","caveat":"The LAMBADA evaluates the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse.\nThe LAMBADA dataset is extracted from BookCorpus and consists of 10'022 passages, divided into 4'869 development and 5'153 test passages, comprising 203 million words. LAMBADA 通过一个单词预测任务来评估计算模型对文本理解的能力。LAMBADA 是有如下特点的一组叙述性文章：如果面对整篇文章，人们可以猜测它们的最后一个单词，但如果他们只看到目标单词前面的最后一句话，就无法猜测。为了在 LAMBADA 上由好的效果，模型不能仅仅依赖于局部上下文，而必须能够跟踪更广泛的话语信息。\nLAMBADA 数据集是从 BookCorpus 中提取的，包括 10,022 段落，分为 4,869 个开发段落和 5,153 个测试段落，共计 2.03 亿个单词。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LAMBADA"],"document_share":0.0002723311546840959,"domain":"理解","name":"LAMBADA","organization_count":1,"organizations":["opencompass_hub"],"rank":1679,"released":"2016-06-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LAMBADA"},{"aliases":["Semi-supervised Source Detection in Astronomical Images: New Benchmark and Strong Baseline"],"benchmark_id":"claire-radar-2606-09219","caveat":"A semi-supervised astronomical source-detection benchmark comprising 18,400 images and 728,898 source instances.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09219"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LAMOST-DET","organization_count":1,"organizations":["claire_radar"],"rank":1680,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09219"},{"aliases":[],"benchmark_id":"opencompass-1924-lamp-qa","caveat":"The benchmark covers questions from three major categories: (1) Arts & Entertainment, (2) Lifestyle & Personal Development, and (3) Society & Culture, encompassing over 45 subcategories in total. 基准旨在评估个性化长篇答案生成。该基准涵盖三大类问题:(1)艺术与娱乐,(2)生活与个人发展,(3)社会与文化,共包含45个以上的子类别。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LaMP-QA"],"document_share":0.0002723311546840959,"domain":"其他","name":"LaMP-QA","organization_count":1,"organizations":["opencompass_hub"],"rank":1681,"released":"2025-05-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LaMP-QA"},{"aliases":["benchmark-llms-landuse-relevance","Land-use relevance: small-LLM benchmark results"],"benchmark_id":"claire-radar-huggingface-noeflandre-benchmark-llms-landuse-relevance","caveat":"This entry is a dataset of predictions and scores for small LLMs on a land-use relevance classification task, not an original standalone benchmark with a stable evaluation contract.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Land-use relevance benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1682,"released":"2026-09-13","source":"claire_radar","url":"https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance"},{"aliases":[],"benchmark_id":"claire-radar-2608-06041","caveat":"LangChoiceBench measures Python preference in project-level code generation across 28 projects and seven software areas, assessing language choice, recommendation-implementation consistency, and language diversity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06041"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"LangChoiceBench","organization_count":1,"organizations":["claire_radar"],"rank":1683,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06041"},{"aliases":[],"benchmark_id":"opencompass-2129-langnavbench","caveat":"LangNavBench is a benchmark for evaluating natural language understanding in semantic navigation, built upon the manually verified LangNav open-set dataset. LangNavBench是一个语义导航中自然语言理解评测基准，基于手工验证的LangNav开放集数据集，评测具身智能体在自然语言指令引导下的目标定位能力，涵盖类别层次理解、对象属性识别和空间关系推理等维度.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LangNavBench"],"document_share":0.0002723311546840959,"domain":"理解","name":"LangNavBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1684,"released":"2025-07-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LangNavBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-06368","caveat":"Closed-loop benchmark for temporally grounded cooperative warnings with six scenario families, 3,272 training sequences, and 120 matched route pairs for evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06368"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LANTERN","organization_count":1,"organizations":["claire_radar"],"rank":1685,"released":"2026-09-06","source":"claire_radar","url":"https://arxiv.org/abs/2609.06368"},{"aliases":[],"benchmark_id":"opencompass-2089-lara","caveat":"LaRA is a focused benchmark for testing Retrieval-Augmented Generation (RAG) and long-context LLMs. LaRA 是专为评估检索增强生成（RAG）和长上下文大型语言模型（LLM）而打造的基准。它围绕信息定位、片段对比、内容推理和幻觉检测四大能力，通过 2 326 条测试用例，对四类问答任务和三种长文本场景（小说、学术论文、财务报表）进行全面测评。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LaRA"],"document_share":0.0002723311546840959,"domain":"长文本","name":"LaRA","organization_count":1,"organizations":["opencompass_hub"],"rank":1686,"released":"2025-02-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LaRA"},{"aliases":[],"benchmark_id":"claire-radar-2609-04173","caveat":"A collection of human-authored and peer-reviewed examples in text, image, audio, and video that break leading machine translation models, with handcrafted verification rules for each example.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04173"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Last Translation Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1687,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.04173"},{"aliases":[],"benchmark_id":"claire-radar-2609-06993","caveat":"Evaluates Markdown boundary failures, specifically CommonMark-level fence-boundary accuracy, across 4,179 prompts using a scoring CLI that separates content correctness from boundary correctness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06993"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"LatentMD","organization_count":1,"organizations":["claire_radar"],"rank":1688,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.06993"},{"aliases":[],"benchmark_id":"claire-radar-2607-27353","caveat":"Evaluates cross-layer reliability of agentic RAG systems on 240 tasks across 8 enterprise domains, with 9 fault scenarios and 2 contract modes; measures success at evidence, tool-contract, authorization, and session-state layers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27353"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LayerRAG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1689,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27353"},{"aliases":[],"benchmark_id":"claire-radar-2609-08657","caveat":"LayerWiseBench evaluates layer-wise chart understanding and editing across 2,800 source charts, with derived questions and editing variants.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08657"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LayerWiseBench","organization_count":1,"organizations":["claire_radar"],"rank":1690,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08657"},{"aliases":[],"benchmark_id":"claire-radar-2607-28880","caveat":"LayoutBench evaluates three cloud storage layout strategies (individual objects, tar archives, Parquet columns) for multimedia data retrieval, measuring retrieval time, data transferred, and cost.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28880"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LayoutBench","organization_count":1,"organizations":["claire_radar"],"rank":1691,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28880"},{"aliases":[],"benchmark_id":"llm-stats-lbpp-v2","caveat":"LBPP (v2) benchmark - specific documentation not found in official sources, possibly related to language-based planning problems","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lbpp-%28v2%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LBPP (v2)","organization_count":1,"organizations":["llm_stats"],"rank":1692,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/lbpp-%28v2%29?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-26525","caveat":"LCS-Bench is a theory-scale benchmark for auto-formalization in logics for computer science. It includes 327 textbook items, over 4,076 Lean declarations, and supports five evaluation tracks with definitional equivalence checkers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26525"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"LCS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1693,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.26525"},{"aliases":[],"benchmark_id":"opencompass-520-lcsts","caveat":"LCSTS is a large corpus of Chinese short text summarization dataset constructed from the Chinese microblogging website Sina Weibo, which is released to the public. This corpus consists of over 2 million real Chinese short texts with short summaries given by the author of each text. LCSTS是一个大规模的中文短文本摘要数据集，从中国微博网站新浪微博中构建而成，并已开源。该数据集包含超过 200 万条真实的中文短文本，每个文本都提供了一个简短的摘要。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LCSTS"],"document_share":0.0002723311546840959,"domain":"理解","name":"LCSTS","organization_count":1,"organizations":["opencompass_hub"],"rank":1694,"released":"2015-06-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LCSTS"},{"aliases":[],"benchmark_id":"claire-radar-2609-23067","caveat":"A benchmark for referring surgical video instrument segmentation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23067"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LD-RSVIS","organization_count":1,"organizations":["claire_radar"],"rank":1695,"released":"2026-09-19","source":"claire_radar","url":"https://arxiv.org/abs/2609.23067"},{"aliases":[],"benchmark_id":"claire-radar-2608-03078","caveat":"LDU-Bench evaluates multimodal LLMs on lithography defect understanding through four tasks: defect triage, morphology recognition, coarse localization, and image-conditioned cause analysis, using task-level metrics and the Lithography Closure Score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03078"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LDU-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1696,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03078"},{"aliases":[],"benchmark_id":"claire-radar-2609-02207","caveat":"Proposes a challenge set of 500 document images annotated with GDPR-aligned PII to evaluate document-level leakage risk in redaction systems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02207"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LeakageBench","organization_count":1,"organizations":["claire_radar"],"rank":1697,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02207"},{"aliases":[],"benchmark_id":"claire-radar-2610-08215","caveat":"Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.08215"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Learn2Play Bench","organization_count":1,"organizations":["claire_radar"],"rank":1698,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.08215"},{"aliases":["Legal Agent Benchmark"],"benchmark_id":"legal_agent_benchmark","caveat":"Open-source agent benchmark from Harvey (github.com/harveyai/harvey- labs): 1,200+ tasks across 24 practice areas, all-pass LLM-judge grading. Absolute scores are very low across all models (0-13%), so ordering at the top is decided by a handful of tasks and small gaps are noise.","document_count":1,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5"],"document_share":0.0002723311546840959,"domain":"legal","name":"Legal Agent Benchmark","organization_count":1,"organizations":["Anthropic"],"rank":1699,"released":"2026-05-06","source":"model_reports","url":"https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark"},{"aliases":[],"benchmark_id":"llm-stats-legal-agent-benchmark","caveat":"The Legal Agent Benchmark (LAB) is Harvey's open-source benchmark for evaluating AI agents on complex, long-horizon legal work. Tasks are scored under an all-pass standard against expert-curated rubrics, where a task passes only if every required rubric criterion (facts, conclusions, citations, structure, and analytical moves) passes.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/legal-agent-benchmark?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"Legal Agent Benchmark","organization_count":1,"organizations":["llm_stats"],"rank":1700,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/legal-agent-benchmark?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-20872","caveat":"LegalCiteTrust evaluates citation trustworthiness in Chinese long-form legal research reports, assessing coverage, support, and citation-level existence, fidelity, and applicability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20872"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LegalCiteTrust","organization_count":1,"organizations":["claire_radar"],"rank":1701,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.20872"},{"aliases":[],"benchmark_id":"claire-radar-2609-14739","caveat":"This entry describes LegalRewardBench as a benchmark for evaluating grounded legal generation, but no dataset, scorer, or public reuse path is evidenced in the provided information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.14739"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LegalRewardBench","organization_count":1,"organizations":["claire_radar"],"rank":1702,"released":"2026-09-13","source":"claire_radar","url":"https://arxiv.org/abs/2609.14739"},{"aliases":[],"benchmark_id":"claire-radar-2609-24172","caveat":"LegendBench is a parametric benchmark for diagnosing legend understanding in visualizations through counterfactual interventions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24172"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LegendBench","organization_count":1,"organizations":["claire_radar"],"rank":1703,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24172"},{"aliases":[],"benchmark_id":"opencompass-2422-lens","caveat":"A multi-level evaluation benchmark of multimodal reasoning with with 3.4K contemporary images and 60K+ human-authored questions covering eight tasks and 12 daily scenarios, forming three progressive task tiers, i.e., perception, understanding, and reasoning. 多层次多模态推理评测基准","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LENS"],"document_share":0.0002723311546840959,"domain":"多模态","name":"LENS","organization_count":1,"organizations":["opencompass_hub"],"rank":1704,"released":"2025-06-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LENS"},{"aliases":[],"benchmark_id":"claire-radar-2608-01094","caveat":"Lethe is a benchmark for federated unlearning in medical imaging. It evaluates twelve methods across eight task families, including classification, segmentation, denoising, cross-modality synthesis, and vision-language question answering. Three forgetting granularities (hospital, class, patient) are assessed against a retrained gold standard on utility, privacy, and cost.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01094"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Lethe","organization_count":1,"organizations":["claire_radar"],"rank":1705,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.01094"},{"aliases":[],"benchmark_id":"claire-radar-2606-05497","caveat":"LEVANTE-bench evaluates vision-language models on six cognitive tasks from the LEVANTE dataset, comparing model performance and error patterns with children aged 5-12 across three countries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05497"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LEVANTE-bench","organization_count":1,"organizations":["claire_radar"],"rank":1706,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05497"},{"aliases":[],"benchmark_id":"claire-radar-2609-09754","caveat":"LexAgentHallu is a hierarchical benchmark for profiling hallucinations in tool-augmented legal agents, containing 3414 instances across 17 legal categories and 6 task types, with dual-layer hallucination taxonomy and fine-grained metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09754"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LexAgentHallu","organization_count":1,"organizations":["claire_radar"],"rank":1707,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09754"},{"aliases":[],"benchmark_id":"claire-radar-2609-02954","caveat":"Designed to evaluate legal issue identification in Chinese civil litigation using 430 real cases and 1,303 expert-annotated disputed issues.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02954"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LexIssue","organization_count":1,"organizations":["claire_radar"],"rank":1708,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02954"},{"aliases":[],"benchmark_id":"claire-radar-2608-09106","caveat":"LexKairos is a benchmark for evaluating the temporal capabilities of LLMs in the Chinese legal context, covering statutory temporal knowledge, case temporal modeling, and statute-case temporal reasoning across nine sub-tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09106"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"LexKairos","organization_count":1,"organizations":["claire_radar"],"rank":1709,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09106"},{"aliases":[],"benchmark_id":"claire-radar-2606-09389","caveat":"LexRubric evaluates open-ended legal tasks in Chinese, with 649 instances from legal consultation and judicial examination. It includes 12,337 expert-written atomic scoring criteria under a six-dimensional framework, enabling fine-grained diagnostic evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09389"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LexRubric","organization_count":1,"organizations":["claire_radar"],"rank":1710,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09389"},{"aliases":[],"benchmark_id":"claire-radar-2607-03920","caveat":"LH-AVLN is a benchmark for long-horizon navigation that combines multi-goal missions, heterogeneous goal specifications, and persistent spatialized audio cues. Agents must execute missions of two to four goals specified by category, language, or reference image, using RGB-D, pose, and binaural audio in indoor 3D environments. It supports ordered and unordered missions with alternating goal-associated sounds.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03920"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"LH-AVLN","organization_count":1,"organizations":["claire_radar"],"rank":1711,"released":"2026-07-04","source":"claire_radar","url":"https://arxiv.org/abs/2607.03920"},{"aliases":[],"benchmark_id":"claire-radar-2608-07958","caveat":"LIBAD is a multimodal anomaly detection dataset from Li-ion battery electrode manufacturing, providing aligned visible-light and X-ray radiography images. It includes benchmarks of methods under inline-compatible settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07958"],"document_share":0.0002723311546840959,"domain":"Manufacturing & Process Control","name":"LIBAD","organization_count":1,"organizations":["claire_radar"],"rank":1712,"released":"2026-08-08","source":"claire_radar","url":"https://arxiv.org/abs/2608.07958"},{"aliases":[],"benchmark_id":"claire-radar-2609-39507","caveat":"General-purpose agents can plan, use tools, and revise their behavior from feedback, but it remains unclear whether these capabilities transfer from digital environments to embodied manipulation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39507"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LIBERO-Agent","organization_count":1,"organizations":["claire_radar"],"rank":1713,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39507"},{"aliases":["Beyond Single-Axis Testing: Paired Evaluation of Compound Robustness in Vision-Language-Action Policies"],"benchmark_id":"claire-radar-2609-15940","caveat":"Not assessable due to insufficient evidence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15940"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LIBERO-CTRL","organization_count":1,"organizations":["claire_radar"],"rank":1714,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15940"},{"aliases":[],"benchmark_id":"claire-radar-2609-36518","caveat":"Robots must often continue a task after a target moves, the viewpoint shifts, or an obstacle appears, even though their earlier observations and committed actions reflect the previous scene.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36518"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"LIBERO-MAX","organization_count":1,"organizations":["claire_radar"],"rank":1715,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36518"},{"aliases":["LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models","LIBERO-Recover"],"benchmark_id":"claire-radar-2609-05178","caveat":"Evaluates failure recovery in robotic manipulation across four recovery levels: action retry, action adaptation, object state recovery, and environmental recovery. Built on LIBERO with 1,000+ scenarios, it assesses spatial understanding, object structure reasoning, interaction understanding, and topological reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05178"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"LIBERO-Recover Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1716,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05178"},{"aliases":[],"benchmark_id":"claire-radar-2608-17600","caveat":"LIBERO-VIFO is a benchmark for visual cue following in vision-language-action models, evaluating capability and safety across eight cue families. However, no public artifacts or official links are provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17600"],"document_share":0.0002723311546840959,"domain":"Embodied manipulation","name":"LIBERO-VIFO","organization_count":1,"organizations":["claire_radar"],"rank":1717,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17600"},{"aliases":[],"benchmark_id":"claire-radar-2609-24350","caveat":"LIBERO-VPro evaluates closed-loop visual robustness of robotic foundation models under visual perturbations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24350"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"LIBERO-VPro","organization_count":1,"organizations":["claire_radar"],"rank":1718,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24350"},{"aliases":[],"benchmark_id":"claire-radar-2606-25402","caveat":"LibEvoBench is a multi-task benchmark for evaluating code generation models on API evolution across versions of Python libraries. It includes tasks spanning multiple library versions and introduces the Software Evolution Understanding Score (SEUS) to measure version-specific knowledge consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25402"],"document_share":0.0002723311546840959,"domain":"Code","name":"LibEvoBench","organization_count":1,"organizations":["claire_radar"],"rank":1719,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25402"},{"aliases":[],"benchmark_id":"claire-radar-2609-36730","caveat":"Agents increasingly build on code written by other agents, and they reimplement rather than reuse, growing the codebases later agents must work in.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36730"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LibraryDesignBench","organization_count":1,"organizations":["claire_radar"],"rank":1720,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36730"},{"aliases":["Do LiDAR Language Models Really Understand Spatio-temporal Relationships?"],"benchmark_id":"claire-radar-2609-24452","caveat":"Evaluates LiDAR language model responses on 10,000 geometry-referenced questions across 150 nuScenes scenes, with strict parsing and diagnostic controls.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24452"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LiDAR-Hallu","organization_count":1,"organizations":["claire_radar"],"rank":1721,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24452"},{"aliases":[],"benchmark_id":"claire-radar-2608-25039","caveat":"Evaluates LLM agents on geo-spatial planning tasks using map data enriched with social media posts, measuring pass rate across four task categories and three difficulty levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25039"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LifePlanner","organization_count":1,"organizations":["claire_radar"],"rank":1722,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.25039"},{"aliases":[],"benchmark_id":"claire-radar-huggingface-mondaycake-lifesci-harbor-bench","caveat":"Evaluates coding agents on 503 ultra-long-horizon, open-ended life science research tasks through an 11-layer verifier combining deterministic scoring and anti-hacking checks.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/mondaycake/lifesci-harbor-bench"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"LifeSci-Harbor-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1723,"released":"2026-09-06","source":"claire_radar","url":"https://huggingface.co/datasets/mondaycake/lifesci-harbor-bench"},{"aliases":[],"benchmark_id":"llm-stats-lifescibench","caveat":"LifeSciBench is an expert-authored, expert-reviewed benchmark of 750 open-ended life-science research tasks spanning seven workflows and seven biological domains. Responses are graded against 19,020 physician- and scientist-written rubric criteria rather than multiple-choice answers, and most tasks require interpreting attached artifacts such as figures, PDFs, and sequence files.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lifescibench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"LifeSciBench","organization_count":1,"organizations":["llm_stats"],"rank":1724,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/lifescibench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-13136","caveat":"LigBench is an automated evaluation benchmark for AI research idea generation, using fine-grained and reliable evaluation across generation distributions. It includes PAIR-IQ, a dataset for training pairwise idea judgment models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13136"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LigBench","organization_count":1,"organizations":["claire_radar"],"rank":1725,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13136"},{"aliases":[],"benchmark_id":"claire-radar-2606-08722","caveat":"LilyBench evaluates symbolic music generation and understanding using LilyPond, with a 200-prompt generation suite and ten understanding tasks, scored via compile rate, descriptor similarity, and FMD.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08722"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LilyBench","organization_count":1,"organizations":["claire_radar"],"rank":1726,"released":"2026-06-07","source":"claire_radar","url":"https://arxiv.org/abs/2606.08722"},{"aliases":[],"benchmark_id":"claire-radar-2608-08045","caveat":"Lingjing is a simulation testbed for heterogeneous multi-agent embodied tasks in urban environments, with a Gym-like interface and support for natural-language missions. It includes engine-based evaluations and attribution-ready replays.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08045"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Lingjing","organization_count":1,"organizations":["claire_radar"],"rank":1727,"released":"2026-08-08","source":"claire_radar","url":"https://arxiv.org/abs/2608.08045"},{"aliases":[],"benchmark_id":"opencompass-1238-lingoly","caveat":"Aiming at evaluating LLM's advanced reasoning abilities, LingOly is composed of  olympiad-level linguistic reasoning puzzles in low-resource and extinct languages. It covers more than 90 mostly low-resource languages, and contains 1,133 problems across 6 formats and 5 levels of human difficulty. LingOly由低资源和已灭绝语言的奥林匹克级别语言推理谜题组成，用于评估大语言模型的高级推理能力。其中涵盖了90多种语言，共有1133个涉及6种格式和5个人工难度级别的问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LINGOLY"],"document_share":0.0002723311546840959,"domain":"推理","name":"LINGOLY","organization_count":1,"organizations":["opencompass_hub"],"rank":1728,"released":"2024-06-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LINGOLY"},{"aliases":[],"benchmark_id":"llm-stats-lingoqa","caveat":"A benchmark for multimodal spatial-language understanding and visual-linguistic question answering.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lingoqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"LingoQA","organization_count":1,"organizations":["llm_stats"],"rank":1729,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/lingoqa?top_n=500"},{"aliases":["On the Limitations of Cross-Lingual Consistency in Multilingual Text-to-image Generation"],"benchmark_id":"claire-radar-2608-11002","caveat":"Evaluates cross-lingual text-to-image generation across content generation and text rendering tasks for 10 languages and 33K prompts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11002"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LingT2I","organization_count":1,"organizations":["claire_radar"],"rank":1730,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.11002"},{"aliases":["What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation"],"benchmark_id":"claire-radar-2608-22948","caveat":"A benchmark for falsifiable research ideation, using a six-field contract centered on a falsifying outcome, with pairwise comparisons of proposals from four models.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22948v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Lit2Test","organization_count":1,"organizations":["claire_radar"],"rank":1731,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.22948v1"},{"aliases":[],"benchmark_id":"claire-radar-2608-07370","caveat":"LitTraceQA evaluates literature-grounded question answering over scientific papers, requiring systems to return paper IDs, evidence locations, and answers in multiple formats. The public split includes 55 examples with gold annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07370"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LitTraceQA","organization_count":1,"organizations":["claire_radar"],"rank":1732,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07370"},{"aliases":["Live Gurbani Tracking: A Benchmark and Reference System for Captioning Sikh Kirtan"],"benchmark_id":"claire-radar-2607-13457","caveat":"A released benchmark for locating exact canonical scripture lines during live or offline Sikh Kirtan captioning under blind and oracle conditions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.13457"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Live Gurbani Tracking","organization_count":1,"organizations":["claire_radar"],"rank":1733,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.13457"},{"aliases":["LiveBench"],"benchmark_id":"livebench","caveat":"Contents change by release date; each date is a different benchmark.","document_count":1,"document_ids":["model_reports:qwen3_technical_report"],"document_share":0.0002723311546840959,"domain":"general","name":"LiveBench","organization_count":1,"organizations":["Qwen"],"rank":1734,"released":"2024-06-06","source":"model_reports","url":"https://livebench.ai/"},{"aliases":[],"benchmark_id":"llm-stats-livebench","caveat":"LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. It comprises tasks across math, coding, reasoning, language, instruction following, and data analysis with verifiable, objective ground-truth answers.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livebench?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"LiveBench","organization_count":1,"organizations":["llm_stats"],"rank":1735,"released":"2024-06-12","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1246-livebench","caveat":"LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-free versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. LiveBench是一个LLM基准测试，涵盖数学、编码、推理、语言、指令遵循和数据分析，包含基于最近发布的数学竞赛、arXiv 论文、新闻文章和数据集的问题，及经典基准测试（如 Big-Bench Hard、AMPS 和 IFEval）的更难、无污染的任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LiveBench"],"document_share":0.0002723311546840959,"domain":"知识","name":"LiveBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1736,"released":"2024-06-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LiveBench"},{"aliases":[],"benchmark_id":"llm-stats-livebench-20241125","caveat":"LiveBench is a challenging, contamination-limited LLM benchmark that addresses test set contamination by releasing new questions monthly based on recently-released datasets, arXiv papers, news articles, and IMDb movie synopses. It comprises tasks across math, coding, reasoning, language, instruction following, and data analysis with verifiable, objective ground-truth answers.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livebench-20241125?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"LiveBench 20241125","organization_count":1,"organizations":["llm_stats"],"rank":1737,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livebench-20241125?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-28721","caveat":"A deep-search evaluation containing 335 human-authored questions about recently published, non-globally-salient facts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28721"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LiveBrowseComp","organization_count":1,"organizations":["claire_radar"],"rank":1738,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28721"},{"aliases":[],"benchmark_id":"artificial-analysis-livecodebench","caveat":"Coding","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/livecodebench"],"document_share":0.0002723311546840959,"domain":"coding","name":"LiveCodeBench","organization_count":1,"organizations":["artificial_analysis"],"rank":1739,"released":"2024-03-12","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/livecodebench"},{"aliases":[],"benchmark_id":"claire-radar-2608-00355","caveat":"CurveShift analyzes progress on LiveCodeBench, releasing a difficulty panel of 66 models and 1,055 problems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00355"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LiveCodeBench","organization_count":1,"organizations":["claire_radar"],"rank":1740,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.00355"},{"aliases":[],"benchmark_id":"llm-stats-livecodebench","caveat":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LiveCodeBench","organization_count":1,"organizations":["llm_stats"],"rank":1741,"released":"2024-03-12","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1413-livecodebench","caveat":"LiveCodeBench evaluates LLMs' coding abilities. It continuously collects new problems over time from contests across LeetCode, AtCoder, and CodeForces. Notably, it also focuses on a broader range of code related capabilities besides code generation. LiveCodeBench用于评估大语言模型的代码能力，包含来自LeetCode、AtCoder和CodeForces的动态更新的问题，并在代码生成能力的基础上将更广泛的相关能力纳入考量。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LiveCodeBench"],"document_share":0.0002723311546840959,"domain":"代码","name":"LiveCodeBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1742,"released":"2024-03-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LiveCodeBench"},{"aliases":["LiveCodeBench Pro","LCB Pro"],"benchmark_id":"livecodebench_pro","caveat":"Reported as an Elo rating against competitive-programming problems, not a pass rate, so it cannot be read on the same axis as LiveCodeBench.","document_count":1,"document_ids":["model_reports:google_gemini_3_1_pro_model_card"],"document_share":0.0002723311546840959,"domain":"coding","name":"LiveCodeBench Pro","organization_count":1,"organizations":["Google"],"rank":1743,"released":"2025-06-13","source":"model_reports","url":"https://livecodebenchpro.com/"},{"aliases":[],"benchmark_id":"llm-stats-livecodebench-pro","caveat":"LiveCodeBench Pro is an advanced evaluation benchmark for large language models for code that uses Elo ratings to rank models based on their performance on coding tasks. It evaluates models on real-world coding problems from programming contests (LeetCode, AtCoder, CodeForces) and provides a relative ranking system where higher Elo scores indicate superior performance.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-pro?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LiveCodeBench Pro","organization_count":1,"organizations":["llm_stats"],"rank":1744,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-pro?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-livecodebench-v5","caveat":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LiveCodeBench v5","organization_count":1,"organizations":["llm_stats"],"rank":1745,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-livecodebench-v5-24-12-25-2","caveat":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5-24.12-25.2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LiveCodeBench v5 24.12-25.2","organization_count":1,"organizations":["llm_stats"],"rank":1746,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v5-24.12-25.2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-livecodebench-v6","caveat":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v6?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LiveCodeBench v6","organization_count":1,"organizations":["llm_stats"],"rank":1747,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench-v6?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-livecodebench-01-09","caveat":"LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livecodebench%2801-09%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LiveCodeBench(01-09)","organization_count":1,"organizations":["llm_stats"],"rank":1748,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livecodebench%2801-09%29?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-17299","caveat":"LiveHouse-TS is a living benchmark for time series foundation models, evaluating models prequentially on real future data across 11 domains and 17 datasets. It shifts benchmarking from snapshot accuracy to continuous temporal validity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17299"],"document_share":0.0002723311546840959,"domain":"Distribution shift","name":"LiveHouse-TS","organization_count":1,"organizations":["claire_radar"],"rank":1749,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17299"},{"aliases":[],"benchmark_id":"claire-radar-2605-26781","caveat":"Dynamic benchmark for multimodal reasoning on high school exam questions in math, physics, chemistry, and biology, with a mock exam evaluation scheme.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26781"],"document_share":0.0002723311546840959,"domain":"Materials & Chemistry","name":"LiveK12Bench","organization_count":1,"organizations":["claire_radar"],"rank":1750,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26781"},{"aliases":[],"benchmark_id":"opencompass-1787-livelongbench","caveat":"LiveLongBench is the first spoken long-text benchmark designed to address the challenges of long-context understanding in real-world dialogues, characterized by speech-specific features, high redundancy, and uneven information density. Existing benchmarks fail to capture these complexities, limiting LiveLongBench 是首个面向口语长文本理解的基准测试，基于直播内容构建，涵盖检索类、推理类及混合类三种任务类型，针对现实对话中存在的语音特性、高冗余性和信息密度不均等挑战。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LiveLongBench"],"document_share":0.0002723311546840959,"domain":"推理","name":"LiveLongBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1751,"released":"2025-04-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LiveLongBench"},{"aliases":[],"benchmark_id":"claire-radar-2610-09872","caveat":"Evaluating agents by outcomes alone can obscure the capabilities that produce them.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.09872"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LiveMACEBench","organization_count":1,"organizations":["claire_radar"],"rank":1752,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.09872"},{"aliases":["Can LLMs Take the Pulse of the Economy? A Real-Time Evaluation of LLM Nowcasts on Macroeconomic Indicators"],"benchmark_id":"claire-radar-2608-30110","caveat":"Evaluates LLM nowcasting of 16 macroeconomic indicators via live pre-release windows, scored by equity returns and simulated trading against institutional baselines.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30110"],"document_share":0.0002723311546840959,"domain":"Finance","name":"LiveMacroEval","organization_count":1,"organizations":["claire_radar"],"rank":1753,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30110"},{"aliases":[],"benchmark_id":"opencompass-1397-livemathbench","caveat":"LiveMathBench can capture LLM capabilities in complex reasoning tasks, including challenging latest question sets from various mathematical competitions. LiveMathBench用于评估大语言模型在复杂推理方面的表现，由极具挑战性的现代数学问题组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LiveMathBench"],"document_share":0.0002723311546840959,"domain":"强推理","name":"LiveMathBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1754,"released":"2024-12-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LiveMathBench"},{"aliases":[],"benchmark_id":"llm-stats-livemathematicianbench","caveat":"LiveMathematicianBench evaluates research-level mathematical reasoning on continuously refreshed, contamination-resistant problems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livemathematicianbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"LiveMathematicianBench","organization_count":1,"organizations":["llm_stats"],"rank":1755,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livemathematicianbench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-livesports-3k","caveat":"LiveSports-3K evaluates fine-grained understanding and commentary of live sports video.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livesports-3k?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"LiveSports-3K","organization_count":1,"organizations":["llm_stats"],"rank":1756,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livesports-3k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-livesqlbench","caveat":"LiveSQLBench evaluates models on generating correct SQL queries against live PostgreSQL databases. The LiveSQLBench-Base-Full v1 dataset contains 600 questions across 22 PostgreSQL databases, testing real-world database reasoning and query generation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/livesqlbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"LiveSQLBench","organization_count":1,"organizations":["llm_stats"],"rank":1757,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/livesqlbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-12069","caveat":"Realistic household simulation must capture not only diverse environments but also the lived-in object arrangements and spatial constraints that shape robot motion and interaction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12069"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"LIVIN","organization_count":1,"organizations":["claire_radar"],"rank":1758,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.12069"},{"aliases":[],"benchmark_id":"claire-radar-2606-02535","caveat":"LL-Bench evaluates large-scale generative models on 16 low-level vision tasks using 2,469 real-world degraded images, with human preference and quality score annotations for model outputs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02535"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1759,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02535"},{"aliases":[],"benchmark_id":"claire-radar-2609-00474","caveat":"Evaluates LLM collaboration with non-language chess engines across six tasks covering imitation, assessment, and explanation, scored on task performance and generalization to out-of-distribution scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00474"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LLAMIA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1760,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2609.00474"},{"aliases":[],"benchmark_id":"opencompass-1360-llava-bench","caveat":"LLaVA-Bench evaluates the model's ability in more challenging tasks, including a diverse set of 24 images with 60 questions in total, including indoor and outdoor scenes, memes, paintings, sketches, etc. LLaVA-Bench用于评估多模态大模型应对复杂任务的能力，内含24 张图像及60个问题，包括室内和室外场景、模因、绘画、素描等。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLaVA-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"LLaVA-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1761,"released":"2023-04-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LLaVA-Bench"},{"aliases":["llm-benchmark-panel"],"benchmark_id":"claire-radar-github-yunusbaysal-llm-benchmark-panel","caveat":"Multi-provider LLM benchmarking harness with task suites including reasoning, coding, and Turkish-language questions; scoring supports exact_match, keywords, regex, and optional llm_judge.","document_count":1,"document_ids":["claire_radar:https://github.com/yunusbaysal/llm-benchmark-panel"],"document_share":0.0002723311546840959,"domain":"Finance","name":"LLM Benchmark Panel","organization_count":1,"organizations":["claire_radar"],"rank":1762,"released":"2026-09-08","source":"claire_radar","url":"https://github.com/yunusbaysal/llm-benchmark-panel"},{"aliases":["llm-math-reasoning-benchmark"],"benchmark_id":"claire-radar-github-buguerito-llm-math-reasoning-benchmark","caveat":"Evaluates final-answer accuracy, reasoning quality, instruction following, and causal error classification across 20 math problems in six categories with deterministic validators and human annotation.","document_count":1,"document_ids":["claire_radar:https://github.com/Buguerito/llm-math-reasoning-benchmark"],"document_share":0.0002723311546840959,"domain":"Finance","name":"LLM Mathematical Reasoning Evaluation Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1763,"released":"2026-09-23","source":"claire_radar","url":"https://github.com/Buguerito/llm-math-reasoning-benchmark"},{"aliases":["llm-quantization-benchmark"],"benchmark_id":"claire-radar-github-sameer-sde-llm-quantization-benchmark","caveat":"Benchmark measuring inference speed, memory usage, and accuracy of MLX and llama.cpp with 4-bit and 8-bit quantization across three 3B-class LLMs on Apple M4 hardware.","document_count":1,"document_ids":["claire_radar:https://github.com/sameer-sde/llm-quantization-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"LLM Quantization Benchmark on Apple Silicon","organization_count":1,"organizations":["claire_radar"],"rank":1764,"released":"2026-09-13","source":"claire_radar","url":"https://github.com/sameer-sde/llm-quantization-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-github-gbbarra-llm-summarization-benchmark-ptbr","caveat":"Evaluates reference-faithful summarization of a technical report in Brazilian Portuguese across 9 local LLMs, using a pre-registered rubric with six runs per model and reported scores, timing, and speed.","document_count":1,"document_ids":["claire_radar:https://github.com/gbbarra/llm-summarization-benchmark-ptbr"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"LLM Summarization Benchmark — Brazilian Portuguese","organization_count":1,"organizations":["claire_radar"],"rank":1765,"released":"2026-08-21","source":"claire_radar","url":"https://github.com/gbbarra/llm-summarization-benchmark-ptbr"},{"aliases":[],"benchmark_id":"opencompass-1838-llm-babybench","caveat":"LLM-BabyBench is a benchmark suite designed to evaluate Large Language Models (LLMs) on grounded planning and reasoning tasks. LLM-BabyBench是一个专门评估大语言模型在交互环境中规划和推理能力的新基准测试套件。基于BabyAI网格世界的文本适配版本，该基准评估LLMs在三个核心方面的表现：预测动作对环境状态的影响（Predict任务）、生成低级动作序列以实现指定目标（Plan任务）、以及将高级指令分解为连贯的子目标序列（Decompose任务）。\n基准包含16个难度级别，提供三种文本格式（Narrative、Structured、JSON），并配备OmniBot专家代理用于生成基准数据。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLM-BabyBench"],"document_share":0.0002723311546840959,"domain":"强推理","name":"LLM-BabyBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1766,"released":"2025-05-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LLM-BabyBench"},{"aliases":[],"benchmark_id":"claire-radar-github-ishant-005-llm-finetuning-benchmark","caveat":"Benchmarks fine-tuning strategies for LLMs using Financial PhraseBank and Qwen 2.5-0.5B.","document_count":1,"document_ids":["claire_radar:https://github.com/Ishant-005/llm-finetuning-benchmark"],"document_share":0.0002723311546840959,"domain":"Finance","name":"llm-finetuning-benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1767,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/Ishant-005/llm-finetuning-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-github-miolonixc-llm-free-benchmark","caveat":"Runs a fixed battery of seven typed tasks against free-tier LLMs, grades each answer programmatically on a 0-5 scale, and reports rankings with per-task scores and latency.","document_count":1,"document_ids":["claire_radar:https://github.com/Miolonixc/llm-free-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"llm-free-benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1768,"released":"2026-08-21","source":"claire_radar","url":"https://github.com/Miolonixc/llm-free-benchmark"},{"aliases":["LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports"],"benchmark_id":"claire-radar-2607-24573","caveat":"A prospective live benchmark in which LLMs forecast unresolved sports events. Forecasts are timestamped and schema-validated, then scored against the real outcomes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24573"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LLM-SoccerArena","organization_count":1,"organizations":["claire_radar"],"rank":1769,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24573"},{"aliases":[],"benchmark_id":"opencompass-1752-llm-srbench","caveat":"LLM-SRBench, a comprehensive benchmark with 239 challenging problems across four scientific domains specifically designed to evaluate LLM-based scientific equation discovery methods while preventing trivial memorization. LLM-SRBench，这是一个包含239个挑战性问题的综合基准测试，涵盖四个科学领域，专门设计用于评估基于LLMs的科学方程式发现方法，同时防止简单记忆。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLM-SRBench"],"document_share":0.0002723311546840959,"domain":"推理","name":"LLM-SRBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1770,"released":"2025-04-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LLM-SRBench"},{"aliases":[],"benchmark_id":"opencompass-1325-llm-uncertainty-bench","caveat":"LLM-Uncertainty-Bench is a new benchmarking approach for LLMs that integrates uncertainty quantification. It spans 5 representative natural language processing tasks, each has a dataset with 10,000 instances. LLM-Uncertainty-Bench将不确定性纳入LLM评估，包含5个具有代表性的自然语言处理任务，每个任务都有包含10000个实例的数据集支撑。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLM-Uncertainty-Bench"],"document_share":0.0002723311546840959,"domain":"其他","name":"LLM-Uncertainty-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1771,"released":"2024-01-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LLM-Uncertainty-Bench"},{"aliases":["LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation"],"benchmark_id":"claire-radar-2609-33521","caveat":"Evaluates LLMs on understanding five trust properties via constructed trust graphs and property understanding tasks, with code and datasets released.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33521"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"LLM4Trust","organization_count":1,"organizations":["claire_radar"],"rank":1772,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33521"},{"aliases":[],"benchmark_id":"claire-radar-2609-32533","caveat":"Inserting advertisements (ads) into consumer-facing LLM output is emerging as a new business model, but there is little shared evidence on how such ad insertion should be evaluated or how it affects user preferences.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32533"],"document_share":0.0002723311546840959,"domain":"Advertising & Pricing","name":"LLMAdBench","organization_count":1,"organizations":["claire_radar"],"rank":1773,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32533"},{"aliases":[],"benchmark_id":"claire-radar-2605-24417","caveat":"LLMTabBench evaluates LLMs on binary tabular classification in zero- and few-shot settings, using real-world and controlled synthetic datasets to study the effect of task descriptions and examples on performance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24417"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LLMTabBench","organization_count":1,"organizations":["claire_radar"],"rank":1774,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24417"},{"aliases":[],"benchmark_id":"opencompass-2047-llmthinkbench","caveat":"LLMThinkBench is a benchmark framework designed to evaluate large language models (LLMs) on basic math reasoning and “overthinking” behaviors, targeting code-executing language models. LLMThinkBench 是一个用于评估大语言模型（LLM）在基础数学推理和“过度思考”行为方面的基准框架，支持对具备代码执行能力的语言模型进行全面评估。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LLMThinkBench"],"document_share":0.0002723311546840959,"domain":"推理","name":"LLMThinkBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1775,"released":"2025-07-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LLMThinkBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-00700","caveat":"LLVM-Bench evaluates LLMs on resolving LLVM compiler issues with 423 real-world tasks, using an automated evaluation platform.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00700"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"LLVM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1776,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.00700"},{"aliases":[],"benchmark_id":"opencompass-2128-lm-evaluation-harness","caveat":"AI Language Proficiency Monitor is a multilingual benchmark platform that systematically assesses LLM performance across up to 200 languages. AI Language Proficiency Monitor是一个多语言大语言模型评测平台，系统性评估模型在多达200种语言上的性能表现，特别关注低资源语言，整合FLORES+、MMLU、GSM8K、TruthfulQA和ARC等数据集评测翻译、问答、数学推理和事实性等能力，提供开源自动更新排行榜和交互式仪表板，覆盖全球80-95%人口使用的语言。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/lm-evaluation-harness"],"document_share":0.0002723311546840959,"domain":"知识","name":"lm-evaluation-harness","organization_count":1,"organizations":["opencompass_hub"],"rank":1777,"released":"2025-07-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/lm-evaluation-harness"},{"aliases":[],"benchmark_id":"opencompass-2088-lmact","caveat":"LMAct is a benchmark for evaluating frontier multimodal models' in-context imitation learning capabilities in long contexts. LMAct是评估前沿多模态模型在长上下文中的上下文模仿学习能力的基准。它评估了诸如井字棋、国际象棋和雅达利等交互式任务的多模式决策。该测试集包含多达100万个令牌上下文和512个专家演示集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LMAct"],"document_share":0.0002723311546840959,"domain":"长文本","name":"LMAct","organization_count":1,"organizations":["opencompass_hub"],"rank":1778,"released":"2024-12-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LMAct"},{"aliases":[],"benchmark_id":"llm-stats-lmarena-text","caveat":"LMArena Text Leaderboard is a blind human preference evaluation benchmark that ranks models based on pairwise comparisons in real-world conversations. The leaderboard uses Elo ratings computed from user preferences in head-to-head model battles, providing a comprehensive measure of overall model capability and style.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lmarena-text?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LMArena Text Leaderboard","organization_count":1,"organizations":["llm_stats"],"rank":1779,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/lmarena-text?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-loca-bench-256k","caveat":"LOCA-Bench is a long-context agentic benchmark. The 256k variant evaluates agents using the official ReAct mode with an environment description length of 256k tokens, measuring how well models reason and act over very long contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/loca-bench-256k?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LOCA-Bench (256k)","organization_count":1,"organizations":["llm_stats"],"rank":1780,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/loca-bench-256k?top_n=500"},{"aliases":["llms-benchmark"],"benchmark_id":"claire-radar-github-diovoib-llms-benchmark","caveat":"Handful of tasks for manual testing of local LLM tool calling and agent loop behavior.","document_count":1,"document_ids":["claire_radar:https://github.com/diovoib/llms-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Local agent benchmark (tools + loop + coding)","organization_count":1,"organizations":["claire_radar"],"rank":1781,"released":"2026-09-10","source":"claire_radar","url":"https://github.com/diovoib/llms-benchmark"},{"aliases":["When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents"],"benchmark_id":"claire-radar-2609-03467","caveat":"Evaluates conversational memory retrieval in long-horizon agents across four query styles using a dataset derived from LoCoMo.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03467"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LOCOMO-CONV","organization_count":1,"organizations":["claire_radar"],"rank":1782,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03467"},{"aliases":[],"benchmark_id":"claire-radar-2610-07338","caveat":"Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.07338"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Logbook","organization_count":1,"organizations":["claire_radar"],"rank":1783,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.07338"},{"aliases":[],"benchmark_id":"claire-radar-2605-28876","caveat":"Benchmark evaluating 11 log reduction tools on 35 GitHub Actions failure cases, scored by 3 LLM debugger families.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28876"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LogDx-CI","organization_count":1,"organizations":["claire_radar"],"rank":1784,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.28876"},{"aliases":["Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment"],"benchmark_id":"claire-radar-2609-11185","caveat":"A benchmark assessing large language models' logical consistency in medical risk-of-bias assessment under a Hierarchical Logical Consistency framework with four dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11185"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"LogiMed-RoB","organization_count":1,"organizations":["claire_radar"],"rank":1785,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11185"},{"aliases":[],"benchmark_id":"opencompass-1572-loki","caveat":"LOKI, a multimodal synthetic data detection benchmark, designed specifically to comprehensively assess the capabilities of LMMs in detecting synthetic data. LOKI是一个多模态合成数据检测基准，专门设计用于全面评估 LMMs 在检测合成数据方面的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LOKI"],"document_share":0.0002723311546840959,"domain":"多模态","name":"LOKI","organization_count":1,"organizations":["opencompass_hub"],"rank":1786,"released":"2024-10-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LOKI"},{"aliases":[],"benchmark_id":"claire-radar-2607-27806","caveat":"LoMeVQA is a benchmark for longitudinal medical visual question answering, with 206K VQA pairs across five tasks: progress classification, progress description, progress report generation, differential region grounding, and differential region description.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27806"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"LoMeVQA","organization_count":1,"organizations":["claire_radar"],"rank":1787,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27806"},{"aliases":[],"benchmark_id":"claire-radar-2609-33356","caveat":"Coding agents now sustain hours-long, tool-driven loops, yet their ability to carry long-horizon analog and mixed-signal circuits to electrical specification remains unmeasured.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33356"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"Long-Horizon Analog Design Bench","organization_count":1,"organizations":["claire_radar"],"rank":1788,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33356"},{"aliases":[],"benchmark_id":"claire-radar-2607-08964","caveat":"Evaluates long-horizon terminal tasks in a containerized environment with hidden verifiers and dense reward grading across 46 tasks and nine categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08964"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"Long-Horizon-Terminal-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1789,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08964"},{"aliases":[],"benchmark_id":"claire-radar-2605-26244","caveat":"LongAV-Compass evaluates minute-scale audio-visual generation across text-to-audio-video, image-to-audio-video, and video-to-audio-video tasks. It includes 284 curated test cases and an evaluation framework combining multimodal metrics with MLLM-assisted assessment across 20+ dimensions covering segment quality, cross-segment consistency, narrative coherence, semantic alignment, and audiovisual synchronization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26244"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongAV-Compass","organization_count":1,"organizations":["claire_radar"],"rank":1790,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26244"},{"aliases":[],"benchmark_id":"opencompass-542-longbench","caveat":"LongBench is a benchmark for bilingual, multitask, and comprehensive assessment of long context understanding capabilities of large language models. LongBench 是一个多任务、中英双语、针对大语言模型长文本理解能力的评测基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LongBench"],"document_share":0.0002723311546840959,"domain":"长文本","name":"LongBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1791,"released":"2023-08-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LongBench"},{"aliases":[],"benchmark_id":"llm-stats-longbench-v2","caveat":"LongBench v2 is a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. It consists of 503 challenging multiple-choice questions with contexts ranging from 8k to 2M words across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longbench-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"LongBench v2","organization_count":1,"organizations":["llm_stats"],"rank":1792,"released":"2024-12-19","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/longbench-v2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-longcodebench","caveat":"LongCodeBench evaluates the code understanding and comprehension abilities of large language models at very long context windows, scaling up to 1M tokens. It tests whether models can reason about extensive codebases provided in a single prompt by answering multiple-choice questions about the code.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longcodebench?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"LongCodeBench","organization_count":1,"organizations":["llm_stats"],"rank":1793,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/longcodebench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-03507","caveat":"Evaluates longitudinal depression tracking from five-session counseling dialogues, with 7,749 trajectories across three datasets and labels for symptom severity and trend.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03507"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongCounsel-8","organization_count":1,"organizations":["claire_radar"],"rank":1794,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03507"},{"aliases":[],"benchmark_id":"claire-radar-2608-15064","caveat":"LongDocBench is a benchmark for Table-of-Contents Hierarchy Recovery and Contextual Relationship Recovery in long documents. It includes 85 real-world documents (financial reports, textbooks, academic papers) spanning 2,582 pages, with human-verified annotations for 3,937 heading nodes and 3,258 contextual relationships.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15064"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongDocBench","organization_count":1,"organizations":["claire_radar"],"rank":1795,"released":"2026-08-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.15064"},{"aliases":[],"benchmark_id":"claire-radar-2605-30434","caveat":"LongDS-Bench evaluates long-horizon, multi-turn data analysis tasks where agents must maintain, update, restore, and compose evolving analytical states. It comprises 68 tasks from real-world Kaggle notebooks spanning 2,225 turns across six domains, with an average dependency span of 11.3 turns.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30434"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongDS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1796,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30434"},{"aliases":[],"benchmark_id":"claire-radar-2608-13344","caveat":"LongEarth-Bench evaluates vision-language models on long-horizon Earth observation reasoning, with ~120k QA samples from 117k images, sequences avg 15.14 frames (up to 30), covering 12 tasks in evolution summarization, spatial reasoning, anomaly identification, and logical prediction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13344"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongEarth-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1797,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13344"},{"aliases":[],"benchmark_id":"claire-radar-2607-02096","caveat":"LongEgoRefer is a benchmark for Video Referring Expression Comprehension in long-form egocentric videos, built from Ego4D. It contains 1,498 referring expressions over videos averaging 45 minutes, requiring temporal grounding of when an occurrence happens and spatial grounding of where the object appears. The benchmark uses evaluation metrics for temporal and spatial grounding.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02096"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongEgoRefer","organization_count":1,"organizations":["claire_radar"],"rank":1798,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02096"},{"aliases":[],"benchmark_id":"llm-stats-longfact","caveat":"LongFact evaluates factual precision over long-form generations containing many individual claims. Each claim is extracted and verified, and the model is scored on claim-level precision, measuring whether extended responses introduce unsupported or false statements.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longfact?top_n=500"],"document_share":0.0002723311546840959,"domain":"factuality","name":"LongFact","organization_count":1,"organizations":["llm_stats"],"rank":1799,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-longfact-concepts","caveat":"LongFact is a benchmark for evaluating long-form factuality in large language models. It comprises 2,280 fact-seeking prompts spanning 38 topics, designed to test a model's ability to generate accurate, long-form responses. The benchmark uses SAFE (Search-Augmented Factuality Evaluator) to evaluate factual accuracy.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longfact-concepts?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LongFact Concepts","organization_count":1,"organizations":["llm_stats"],"rank":1800,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-concepts?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-longfact-objects","caveat":"LongFact is a benchmark for evaluating long-form factuality in large language models. It comprises 2,280 fact-seeking prompts spanning 38 topics, designed to test a model's ability to generate accurate, long-form responses. The benchmark uses SAFE (Search-Augmented Factuality Evaluator) to evaluate factual accuracy.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longfact-objects?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"LongFact Objects","organization_count":1,"organizations":["llm_stats"],"rank":1801,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/longfact-objects?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-38137","caveat":"Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38137"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongHarness Bench","organization_count":1,"organizations":["claire_radar"],"rank":1802,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.38137"},{"aliases":[],"benchmark_id":"claire-radar-2606-01629","caveat":"LongJudgeBench evaluates LLM-as-a-judge performance on long-form outputs across six datasets covering pointwise, pairwise, and listwise protocols, with bilingual tasks and multiple prompt variants. It measures agreement with human judgments using accuracy, Spearman, and Kendall's tau.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01629"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongJudgeBench","organization_count":1,"organizations":["claire_radar"],"rank":1803,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01629"},{"aliases":["From Visual Cues to Spoken Narration: Rethinking Audio Description"],"benchmark_id":"claire-radar-2609-01725","caveat":"Evaluates audio description generation on long untrimmed movie clips, covering multi-segment localization and narration quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01725"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongLSMDC","organization_count":1,"organizations":["claire_radar"],"rank":1804,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01725"},{"aliases":[],"benchmark_id":"claire-radar-2607-09322","caveat":"A benchmark for long-horizon clinical decision-making using EHR data from MIMIC-IV, comprising 335 patients with multi-session interactions and three evaluation suites: fact-based QA, temporal reasoning, and long-horizon decision-making.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09322"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"LongMedBench","organization_count":1,"organizations":["claire_radar"],"rank":1805,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.09322"},{"aliases":[],"benchmark_id":"claire-radar-2608-15242","caveat":"LongRCA Bench is a benchmark for diagnosing responsible roles and root causes in long-horizon agent failures. It comprises 1,140 failed trajectories across five domains, with human labels for the responsible role and the earliest decisive root-cause step. Evaluation focuses on responsible-role accuracy and exact root-step accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15242"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LongRCA Bench","organization_count":1,"organizations":["claire_radar"],"rank":1806,"released":"2026-08-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.15242"},{"aliases":[],"benchmark_id":"llm-stats-longtext-bench","caveat":"LongText-Bench evaluates text-to-image models on their ability to accurately render long text passages within generated images. It includes English (EN) and Chinese (ZH) subsets to assess multilingual text rendering capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longtext-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"image-generation","name":"LongText-Bench","organization_count":1,"organizations":["llm_stats"],"rank":1807,"released":"2025-07-29","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/longtext-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2067-longvale","caveat":"LongVALE is the first long video benchmark integrating fine-grained omni modalities (video, audio, speech) information in videos. It comprises 105K omni-modal events with precise temporal boundaries and detailed omni-modal captions, aiming to advance comprehensive multi-modal video understanding. LongVALE 是首个集成视频中细粒度全模态（视频、音频、语音）信息的长视频理解基准。它包含 10.5 万个具有精确时间边界和详细的全模态描述的事件标注，致力于全面提升多模态视频大语言模型的跨模态推理与细粒度时间感知能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LongVALE"],"document_share":0.0002723311546840959,"domain":"多模态","name":"LongVALE","organization_count":1,"organizations":["opencompass_hub"],"rank":1808,"released":"2025-04-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LongVALE"},{"aliases":[],"benchmark_id":"llm-stats-longvideobench","caveat":"LongVideoBench is a question-answering benchmark featuring video-language interleaved inputs up to an hour long. It includes 3,763 varying-length web-collected videos with subtitles across diverse themes and 6,678 human-annotated multiple-choice questions in 17 fine-grained categories for comprehensive evaluation of long-term multimodal understanding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/longvideobench?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"LongVideoBench","organization_count":1,"organizations":["llm_stats"],"rank":1809,"released":"2024-07-22","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/longvideobench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1510-longvideobench","caveat":"LongVideoBench tests LMMs' understanding of long videos. It's a question-answering benchmark with video-language interleaved inputs up to an hour long and comprises 3,763 web-collected videos with subtitles across diverse themes, LongVideoBench用于评估多模态大模型的长视频理解能力，是基于交错长视频语料构建的问答集，包含3763个不同主题的带字幕的视频。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LongVideoBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"LongVideoBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1810,"released":"2024-07-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LongVideoBench"},{"aliases":[],"benchmark_id":"claire-radar-2608-08630","caveat":"LongVLBench evaluates long-context multimodal understanding over interleaved visual and textual sequences, using tasks derived from video narratives that require holistic, narrative-level reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08630"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongVLBench","organization_count":1,"organizations":["claire_radar"],"rank":1811,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08630"},{"aliases":[],"benchmark_id":"claire-radar-2607-01086","caveat":"LongVQUBench evaluates long-term video quality understanding with 1200+ videos and 1500 questions across three levels of perceptual reasoning complexity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01086"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongVQUBench","organization_count":1,"organizations":["claire_radar"],"rank":1812,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.01086"},{"aliases":[],"benchmark_id":"claire-radar-2606-17727","caveat":"LongWebBench evaluates structural and functional webpage generation of long webpages. It includes 490 webpages for structural fidelity and 129 for functional interactions, using VLM-based metrics and a DOM-augmented agent pipeline.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17727"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LongWebBench","organization_count":1,"organizations":["claire_radar"],"rank":1813,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.17727"},{"aliases":[],"benchmark_id":"claire-radar-2608-23200","caveat":"Evaluates verifiable long-workflow tasks across code generation, agent-environment synthesis, mathematical reasoning, and rule following, with machine-verifiable scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.23200"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LongWoF-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1814,"released":"2026-08-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.23200"},{"aliases":[],"benchmark_id":"claire-radar-2608-28281","caveat":"Evaluates a controller model guiding a fixed coding agent through long-running tasks, scored via contract accuracy, strict success rate, and cost metrics across three complementary settings.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.28281v1"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LoopArena","organization_count":1,"organizations":["claire_radar"],"rank":1815,"released":"2026-08-28","source":"claire_radar","url":"http://arxiv.org/abs/2608.28281v1"},{"aliases":[],"benchmark_id":"opencompass-1944-loopnav","caveat":"A video-action dataset containing many loop-based navigation dataset in Minecraft environment, aiming to boost the spatial consistency and providing insight for the design of memory module 一个视频-动作导航数据集，包括了在Minecraft环境下大量基于回环的导航数据，能够促进世界模型等视频模型空间一致性的训练，启发记忆模块的设计。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LoopNav"],"document_share":0.0002723311546840959,"domain":"多模态","name":"LoopNav","organization_count":1,"organizations":["opencompass_hub"],"rank":1816,"released":"2025-05-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LoopNav"},{"aliases":[],"benchmark_id":"claire-radar-2608-00267","caveat":"LOOPSBENCH evaluates coding agents on long-horizon tasks structured as dependency DAGs with flow-aware test release and regression obligations, spanning 8 languages and 9 domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00267"],"document_share":0.0002723311546840959,"domain":"Agents","name":"LoopsBench","organization_count":1,"organizations":["claire_radar"],"rank":1817,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.00267"},{"aliases":["Benchmarking Recursive-Collapse Warning Claims Under Matched False-Positive Control"],"benchmark_id":"claire-radar-2606-00329","caveat":"Loopzero is a reproducible framework for evaluating recursive-collapse warning claims under a fixed alert-budget condition.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00329"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Loopzero","organization_count":1,"organizations":["claire_radar"],"rank":1818,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.00329"},{"aliases":[],"benchmark_id":"claire-radar-2607-19911","caveat":"LoRFT evaluates long-range vehicle trajectory reconstruction from fixed highway cameras, with 6,601 manually verified trajectories and map-aware evaluation metrics like ADE and FDE.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19911"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"LoRFT","organization_count":1,"organizations":["claire_radar"],"rank":1819,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.19911"},{"aliases":[],"benchmark_id":"claire-radar-2606-14600","caveat":"LoSoNA evaluates LLM agents' ability to infer and adapt to local social norms in group chats, using curated transcripts and an elicitor turn to score norm-conforming responses.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14600"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LoSoNA","organization_count":1,"organizations":["claire_radar"],"rank":1820,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14600"},{"aliases":["Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts"],"benchmark_id":"claire-radar-2608-24707","caveat":"A multilingual spoken hallucination detection benchmark with 12,013 news samples across English, Russian, and Kazakh, including synthetic and real-world fake news items in text and audio.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24707"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Lost in Speech","organization_count":1,"organizations":["claire_radar"],"rank":1821,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24707"},{"aliases":[],"benchmark_id":"claire-radar-2609-17124","caveat":"Maritime simulator providing an integration testbed for prototyping and rehearsing offshore human-robot collaboration workflows.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17124"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LOTUSim-Energy","organization_count":1,"organizations":["claire_radar"],"rank":1822,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17124"},{"aliases":["LPFQA"],"benchmark_id":"lpfqa","caveat":"Long-tail professional forum-based benchmark; evaluates complex reasoning across 20 academic and industrial fields with authentic professional scenarios.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"knowledge","name":"LPFQA","organization_count":1,"organizations":["ByteDance"],"rank":1823,"released":null,"source":"model_reports","url":"https://huggingface.co/datasets/m-a-p/LPFQA"},{"aliases":[],"benchmark_id":"llm-stats-lsat","caveat":"LSAT (Law School Admission Test) benchmark evaluating complex reasoning capabilities across three challenging tasks: analytical reasoning, logical reasoning, and reading comprehension. The LSAT measures skills considered essential for success in law school including critical thinking, reading comprehension of complex texts, and analysis of arguments.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lsat?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"LSAT","organization_count":1,"organizations":["llm_stats"],"rank":1824,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/lsat?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1335-ltmbenchmark","caveat":"LTMbenchmark assess the long-term memory, continual learning, and information integration capabilities of the agents via dynamic conversational tasks. LTMbenchmark通过动态对话任务评估智能体的长期记忆、持续学习和信息集成能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LTMbenchmark"],"document_share":0.0002723311546840959,"domain":"智能体","name":"LTMbenchmark","organization_count":1,"organizations":["opencompass_hub"],"rank":1825,"released":"2024-09-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LTMbenchmark"},{"aliases":[],"benchmark_id":"claire-radar-2607-24101","caveat":"LU-500 evaluates concept unlearning for logos with nearly 10,000 pairs, including explicit and implicit contextual tracks, and a multi-grained protocol measuring local removal and global preservation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24101"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LU-500","organization_count":1,"organizations":["claire_radar"],"rank":1826,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24101"},{"aliases":[],"benchmark_id":"claire-radar-2608-05246","caveat":"LUNAR is a benchmark for evaluating LLM personalization from longitudinal app interaction logs across domains like clothing, food, housing, and mobility. It uses a synthetic data pipeline and evaluates 19 LLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05246"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LUNAR","organization_count":1,"organizations":["claire_radar"],"rank":1827,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.05246"},{"aliases":[],"benchmark_id":"opencompass-564-lv-eval","caveat":"LV-Eval is a challenging long-context benchmark with five length levels (16k, 32k, 64k, 128k, and 256k) reaching up to 256k words. The average number of words is 102,380, and the Min/Max number of words is 11,896/387,406. It features two main tasks, single-hop QA and multi-hop QA, comprising 11 bilingual datasets. LV-Eval是一个具备5个长度等级（16k、32k、64k、128k和256k）、最大文本测试长度达到256k的长文本评测基准。LV-Eval的平均文本长度达到102,380字，最小/最大文本长度为11,896/387,406字。LV-Eval主要有两类评测任务——单跳QA和多跳QA，共包含11个涵盖中英文的评测数据子集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/LV-Eval"],"document_share":0.0002723311546840959,"domain":"长文本","name":"LV-Eval","organization_count":1,"organizations":["opencompass_hub"],"rank":1828,"released":"2024-02-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/LV-Eval"},{"aliases":[],"benchmark_id":"llm-stats-lvbench","caveat":"LVBench is an extreme long video understanding benchmark designed to evaluate multimodal models on videos up to two hours in duration. It contains 6 major categories and 21 subcategories, with videos averaging five times longer than existing datasets. The benchmark addresses applications requiring comprehension of extremely long videos.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/lvbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"LVBench","organization_count":1,"organizations":["llm_stats"],"rank":1829,"released":"2024-06-12","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/lvbench?top_n=500"},{"aliases":["LëtzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents"],"benchmark_id":"claire-radar-2608-21714","caveat":"Cross-lingual page-level retrieval benchmark over Luxembourgish PDF documents with queries in four languages.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.21714v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"LëtzCross","organization_count":1,"organizations":["claire_radar"],"rank":1830,"released":"2026-08-22","source":"claire_radar","url":"http://arxiv.org/abs/2608.21714v1"},{"aliases":["M$^3$ISR: A Multi-Modal Multi-View Benchmark for 3D/4D Gaussian Splatting and Feedforward Compression"],"benchmark_id":"claire-radar-2608-22465","caveat":"Evaluates 3D/4D Gaussian splatting methods on 25 synthetic scenes with five tracks covering synthesis, streaming, and compression.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22465v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"M$^3$ISR","organization_count":1,"organizations":["claire_radar"],"rank":1831,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22465v1"},{"aliases":["M$^3$R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding"],"benchmark_id":"claire-radar-2608-05817","caveat":"Evaluates unified metaphor understanding across 1,000 image-text instances with annotations for occurrence, Target-Source mapping, sentiment, and stage-wise explanations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05817"],"document_share":0.0002723311546840959,"domain":"General AI","name":"M$^3$R-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1832,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05817"},{"aliases":[],"benchmark_id":"claire-radar-2608-03803","caveat":"A multilingual proficiency benchmark covering grammatical error detection, round-trip translation, and tokenizer efficiency across 30 languages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03803"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models","organization_count":1,"organizations":["claire_radar"],"rank":1833,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03803"},{"aliases":[],"benchmark_id":"claire-radar-2608-01189","caveat":"A benchmark of 122 real-world AI models in which agents must deploy each model as a ready-to-call API. Standardized test cases check the result, and agents are compared by deployment success rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01189"],"document_share":0.0002723311546840959,"domain":"Agents","name":"M2ABench","organization_count":1,"organizations":["claire_radar"],"rank":1834,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.01189"},{"aliases":[],"benchmark_id":"claire-radar-2607-29125","caveat":"M3-DuplexBench evaluates full-duplex spoken dialogue models in multi-turn, multilingual (English and Japanese), multidomain settings, with multiple dialogue context settings and turn-taking analysis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29125"],"document_share":0.0002723311546840959,"domain":"General AI","name":"M3-DuplexBench","organization_count":1,"organizations":["claire_radar"],"rank":1835,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29125"},{"aliases":[],"benchmark_id":"claire-radar-2609-23003","caveat":"A benchmark for multi-modal, multi-session ground-to-aerial place recognition in forests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23003"],"document_share":0.0002723311546840959,"domain":"General AI","name":"M3GA-Wild","organization_count":1,"organizations":["claire_radar"],"rank":1836,"released":"2026-09-19","source":"claire_radar","url":"https://arxiv.org/abs/2609.23003"},{"aliases":[],"benchmark_id":"opencompass-1147-m3t","caveat":"M3T is a novel benchmark dataset tailored to evaluate NMT systems on the comprehensive task of translating semi-structured documents. This dataset aims to bridge the evaluation gap in document-level NMT systems, acknowledging the challenges posed by rich text layouts in real-world applications. M3T 是旨在评估神经机器翻译（NMT）系统在翻译半结构化文档的综合任务上的表现。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/M3T"],"document_share":0.0002723311546840959,"domain":"理解","name":"M3T","organization_count":1,"organizations":["opencompass_hub"],"rank":1837,"released":"2024-06-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/M3T"},{"aliases":[],"benchmark_id":"claire-radar-2606-13782","caveat":"MA-ProofBench evaluates LLMs on theorem proving in mathematical analysis using 200 Lean 4 formalized problems split into undergraduate and Ph.D. levels. It covers six core topics and uses Pass@8 with formal verification via Lean server.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13782"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"MA-ProofBench","organization_count":1,"organizations":["claire_radar"],"rank":1838,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13782"},{"aliases":[],"benchmark_id":"claire-radar-2606-07805","caveat":"MAC-Bench evaluates procedural compliance of multi-agent systems under social-engineering pressure, measuring compliance-weighted success rate and Machiavellian gap.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07805"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MAC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1839,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07805"},{"aliases":[],"benchmark_id":"claire-radar-2606-22557","caveat":"MacAgentBench evaluates computer use agents on macOS desktop automation, with 676 tasks across 25 applications, including GUI and CLI interactions. It uses deterministic rule-based evaluation and fine-grained multi-checkpoint scoring to assess sub-goal completion.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22557"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MacAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":1840,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22557"},{"aliases":[],"benchmark_id":"claire-radar-2606-06560","caveat":"MacArena benchmarks computer-use agents on macOS with 421 manually verified tasks spanning 50 applications, running on Apple Virtualization framework on Apple Silicon.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06560"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"MacArena","organization_count":1,"organizations":["claire_radar"],"rank":1841,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06560"},{"aliases":["Translation Tag Team: Formal Rules and LLMs Translate More Macros Together than Apart"],"benchmark_id":"claire-radar-2608-06705","caveat":"MacroBench evaluates macro translation from C to Rust using test cases based on macros sampled from real-world C programs, with both formal-rule and LLM-based translators assessed on translation coverage and correctness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06705"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"MacroBench","organization_count":1,"organizations":["claire_radar"],"rank":1842,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.06705"},{"aliases":[],"benchmark_id":"claire-radar-2607-06929","caveat":"MADB is a large-scale dataset and benchmark for music aesthetic assessment, comprising 9,999 tracks annotated by 30 trained annotators across 10 perceptual dimensions and an overall score, with textual comments. It includes a unified evaluation framework over multiple pretrained models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06929"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MADB","organization_count":1,"organizations":["claire_radar"],"rank":1843,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.06929"},{"aliases":[],"benchmark_id":"claire-radar-2608-09593","caveat":"MADBench is a benchmark for modality-aware audio deepfake detection, treating speech and environmental audio as distinct components. It evaluates detectors across independently manipulated forgery sources.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09593"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"MADBench","organization_count":1,"organizations":["claire_radar"],"rank":1844,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09593"},{"aliases":[],"benchmark_id":"claire-radar-2609-39146","caveat":"Multi-agent debate (MAD) can improve large language model (LLM) reasoning by allowing multiple agents to exchange and critique their answers to the same task.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39146"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MADBench","organization_count":1,"organizations":["claire_radar"],"rank":1845,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39146"},{"aliases":[],"benchmark_id":"claire-radar-2606-30371","caveat":"MaDI-Bench evaluates end-to-end data integration pipelines across schema matching, value normalization, entity matching, and conflict resolution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30371"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MaDI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1846,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30371"},{"aliases":[],"benchmark_id":"claire-radar-2607-10645","caveat":"MafiaScope evaluates LLM agents in the social deduction game Mafia. It probes each agent's private beliefs after every public utterance, scoring them against ground truth without influencing the game. The testbed provides an open-source engine, interactive visualizer, recorded games, and counterfactual replay corpus.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10645"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MafiaScope","organization_count":1,"organizations":["claire_radar"],"rank":1847,"released":"2026-07-12","source":"claire_radar","url":"https://arxiv.org/abs/2607.10645"},{"aliases":[],"benchmark_id":"claire-radar-2607-10079","caveat":"MAG is a benchmark that unifies task execution and guide writing into a single multimodal action and guide task, with grounding over screenshots. It includes a harness for annotation, training, evaluation, and joint metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10079"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MAG","organization_count":1,"organizations":["claire_radar"],"rank":1848,"released":"2026-07-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.10079"},{"aliases":[],"benchmark_id":"claire-radar-2610-13534","caveat":"Long-horizon embodied agents must maintain task-relevant information, coordinate heterogeneous robots, and adapt their decisions as execution unfolds.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13534"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"MAGMA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1849,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13534"},{"aliases":[],"benchmark_id":"claire-radar-2609-35003","caveat":"Vision-language-action (VLA) models have demonstrated strong capabilities in robotic manipulation, but they are typically developed and evaluated with all camera streams available throughout task execution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35003"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"MAIL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1850,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35003"},{"aliases":[],"benchmark_id":"claire-radar-2608-19901","caveat":"Benchmark for detecting malicious agent skills, comprising 9,740 skills (7,505 malicious, 2,235 benign) and evaluating detectors under random, structural-disjoint, and source-disjoint splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.19901"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MaliciousSkillBench","organization_count":1,"organizations":["claire_radar"],"rank":1851,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.19901"},{"aliases":[],"benchmark_id":"claire-radar-2606-07131","caveat":"MalSkillBench is a runtime-verified benchmark of malicious agent skills, with 3,944 malicious skills labeled along a taxonomy, measuring detection tool effectiveness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07131"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MalSkillBench","organization_count":1,"organizations":["claire_radar"],"rank":1852,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07131"},{"aliases":[],"benchmark_id":"claire-radar-2607-14385","caveat":"MamaBench evaluates LLM robustness in maternal and child health diagnosis using counterfactual clinical narratives, with the Bias Trap Rate (BTR) metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14385"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MamaBench","organization_count":1,"organizations":["claire_radar"],"rank":1853,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.14385"},{"aliases":[],"benchmark_id":"llm-stats-management-consulting-tasks","caveat":"Management Consulting Tasks is an internal OpenAI evaluation of long-horizon professional knowledge work drawn from management-consulting workflows, scoring whether models produce correct, decision-ready analyses.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/management-consulting-tasks?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Management Consulting Tasks (Internal)","organization_count":1,"organizations":["llm_stats"],"rank":1854,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/management-consulting-tasks?top_n=500"},{"aliases":["Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalation"],"benchmark_id":"claire-radar-2607-15434","caveat":"A released agentic benchmark measuring coercive escalation and fabricated success when an AI manager encounters a subordinate that refuses a task.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15434"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Manager Coercion Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1855,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.15434"},{"aliases":[],"benchmark_id":"claire-radar-github-msavdert-mandate-bench","caveat":"Evaluates LLM agents' compliance with portfolio-rebalancing mandates through frozen market snapshots and pre-registered metrics.","document_count":1,"document_ids":["claire_radar:https://github.com/msavdert/mandate-bench"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Mandate Bench","organization_count":1,"organizations":["claire_radar"],"rank":1856,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/msavdert/mandate-bench"},{"aliases":["MANIGUARD: A Benchmark and Data Suite for Specification-Grounded Safety Evaluation and Improvement of Robotic Manipulation"],"benchmark_id":"claire-radar-2608-17386","caveat":"ManiGuard-Bench evaluates 1,000 locked scenarios across six contact-rich household task families, with safety specifications checked by LTL$_f$-grounded automaton monitors and metrics separating safe success from engaged-and-safe behavior.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17386"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"ManiGuard-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1857,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17386"},{"aliases":[],"benchmark_id":"claire-radar-2609-17910","caveat":"Human-curated benchmark for compositional language-grounded route planning over pre-built semantic maps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17910"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Map2Route","organization_count":1,"organizations":["claire_radar"],"rank":1858,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17910"},{"aliases":[],"benchmark_id":"claire-radar-2606-22597","caveat":"MapReason-OSM evaluates vision-language models on graph-verifiable mobility decisions from self-rendered OpenStreetMap panels. It covers 12 tasks in routing, facility location, and visual disambiguation, with structured outputs scored against hidden oracles for validity, legality, optimality, and constraint satisfaction, plus cross-zoom consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22597"],"document_share":0.0002723311546840959,"domain":"Logistics & Operations","name":"MapReason-OSM","organization_count":1,"organizations":["claire_radar"],"rank":1859,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22597"},{"aliases":["MariSat: A Maritime Dataset for Instance Segmentation of Objects in Satellite and Aerial Images"],"benchmark_id":"claire-radar-2608-29852","caveat":"MariSat provides 1260 satellite and aerial images with pixel-level instance segmentation annotations for eight maritime object classes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29852"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MariSat","organization_count":1,"organizations":["claire_radar"],"rank":1860,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29852"},{"aliases":[],"benchmark_id":"opencompass-1694-maritimebench","caveat":"MaritimeBench builds a scientific, fair maritime knowledge assessment system. With 1,888 MCQs based on industry standards, we evaluate models' capabilities across shipping domains. MaritimeBench 致力于构建一套科学、公平且严谨的航运知识评估体系。基于行业权威标准，我们持续维护并更新高质量的航运数据集——其中包含1,888道客观选择题（MCQ格式），以全面、多维度地量化模型在航运各领域的能力表现。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MaritimeBench"],"document_share":0.0002723311546840959,"domain":"学科","name":"MaritimeBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1861,"released":"2025-04-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MaritimeBench"},{"aliases":["MARS-Bench"],"benchmark_id":"mars_bench","caveat":"Multi-turn dialogue benchmark built from sports commentary; evaluates ultra multi-turn, cross-turn, and interactive multi-turn capabilities.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MARS-Bench","organization_count":1,"organizations":["ByteDance"],"rank":1862,"released":null,"source":"model_reports","url":"https://syuchin.github.io/MARS-Bench/"},{"aliases":[],"benchmark_id":"claire-radar-2610-04672","caveat":"Large language models (LLMs) have progressively evolved into the core of autonomous agents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.04672"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MASBench","organization_count":1,"organizations":["claire_radar"],"rank":1863,"released":"2026-10-03","source":"claire_radar","url":"https://arxiv.org/abs/2610.04672"},{"aliases":[],"benchmark_id":"claire-radar-2608-07556","caveat":"Evaluates multi-agent systems on 600 productivity tasks measuring task completion and unauthorized action rate across different coordination architectures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07556"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MasDrift","organization_count":1,"organizations":["claire_radar"],"rank":1864,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.07556"},{"aliases":[],"benchmark_id":"claire-radar-2608-22460","caveat":"Evaluates ML models on cross-source mass-shooting risk classification using 6,968 incidents with leave-one-dataset-out evaluation.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22460v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MASH-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1865,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22460v1"},{"aliases":[],"benchmark_id":"llm-stats-mask","caveat":"MASK is a collection of 1000 questions measuring whether models faithfully report their beliefs when pressured to lie. It operationalizes deception as the rate at which the model lies, i.e., knowingly making false statements intended to be received as true. Lower dishonesty rates indicate better honesty.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mask?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MASK","organization_count":1,"organizations":["llm_stats"],"rank":1866,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mask?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1609-mask","caveat":"The MASK evaluation provides a rigorous benchmark for evaluating honesty in large language models by measuring whether models remain truthful when incentivized to lie. The public set contains 1,028 high-quality human-labeled examples across six distinct archetypes. MASK 评估提供了一个严格的基准，用于评估大型语言模型中的诚实度，通过测量模型在受到诱使说谎的激励时是否保持真实性。公共集包含 1,028 个高质量的人标注示例，涵盖六个不同的原型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MASK"],"document_share":0.0002723311546840959,"domain":"其他","name":"MASK","organization_count":1,"organizations":["opencompass_hub"],"rank":1867,"released":"2025-03-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MASK"},{"aliases":[],"benchmark_id":"opencompass-1847-massive-steps","caveat":"Massive-STEPS is a large-scale semantic trajectories dataset designed for understanding and predicting Point-of-Interest (POI) check-ins. Massive-STEPS是一个大规模的语义轨迹数据集，旨在理解和预测兴趣点（POI）签到行为。该数据集基于Semantic Trails数据集构建，覆盖12个全球不同地区的城市，包含2012-2013年和2017-2018年的签到数据，提供了更现代和多样化的POI签到信息。Massive-STEPS不仅丰富了签到数据的语义信息，还通过与Foursquare Open Source Places数据集对齐，增加了POI的地理坐标、名称和地址等元数据。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Massive-STEPS"],"document_share":0.0002723311546840959,"domain":"多模态","name":"Massive-STEPS","organization_count":1,"organizations":["opencompass_hub"],"rank":1868,"released":"2025-05-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Massive-STEPS"},{"aliases":[],"benchmark_id":"opencompass-1645-mastermindeval","caveat":"Evaluating Reasoning Capabilities of LLMs Using the Mastermind Board Game. MastermindEval使用猜谜游戏棋盘评估大型语言模型的推理能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MastermindEval"],"document_share":0.0002723311546840959,"domain":"推理","name":"MastermindEval","organization_count":1,"organizations":["opencompass_hub"],"rank":1869,"released":"2025-03-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MastermindEval"},{"aliases":[],"benchmark_id":"claire-radar-2606-21830","caveat":"Mat-Pref evaluates compositional reasoning in inorganic materials via 10,837 ionic-substitution questions across 11 structure families, with splits for in-distribution performance, held-out families, and cross-property transfer.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21830"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Mat-Pref","organization_count":1,"organizations":["claire_radar"],"rank":1870,"released":"2026-06-20","source":"claire_radar","url":"https://arxiv.org/abs/2606.21830"},{"aliases":[],"benchmark_id":"llm-stats-math","caveat":"MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/math?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MATH","organization_count":1,"organizations":["llm_stats"],"rank":1871,"released":"2021-11-08","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/math?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-534-math","caveat":"MATH is a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution. MATH 是一个包含 12,500 个具有挑战性的竞赛数学问题的新数据集。 MATH 中的每个问题都有完整的分步解决方案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MATH"],"document_share":0.0002723311546840959,"domain":"数学","name":"MATH","organization_count":1,"organizations":["opencompass_hub"],"rank":1872,"released":"2021-11-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MATH"},{"aliases":[],"benchmark_id":"llm-stats-math-cot","caveat":"MATH dataset contains 12,500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels (1-5) across seven mathematical subjects. This variant uses Chain-of-Thought prompting to encourage step-by-step reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/math-%28cot%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MATH (CoT)","organization_count":1,"organizations":["llm_stats"],"rank":1873,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/math-%28cot%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-math-500","caveat":"MATH-500 is a subset of the MATH dataset containing 500 challenging competition mathematics problems from AMC 10, AMC 12, AIME, and other mathematics competitions. Each problem includes full step-by-step solutions and spans multiple difficulty levels across seven mathematical subjects including Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/math-500?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MATH-500","organization_count":1,"organizations":["llm_stats"],"rank":1874,"released":"2023-05-31","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/math-500?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-08964","caveat":"Math-Vision Diagrams evaluates LLMs on mathematical diagram generation from text, covering both text-to-code and text-to-image paradigms, with a curated subset of 2,920 competition problems and multiple metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08964"],"document_share":0.0002723311546840959,"domain":"Mathematics & Formal Science","name":"Math-Vision Diagrams","organization_count":1,"organizations":["claire_radar"],"rank":1875,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08964"},{"aliases":[],"benchmark_id":"claire-radar-2608-25449","caveat":"Evaluates theorem proving and auxiliary tasks including multiple-choice, fill-in-the-blank, and reformulation robustness across 13 mathematical domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25449"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MathAdv","organization_count":1,"organizations":["claire_radar"],"rank":1876,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25449"},{"aliases":[],"benchmark_id":"llm-stats-matharena-apex","caveat":"MathArena Apex is a challenging math contest benchmark featuring the most difficult mathematical problems designed to test advanced reasoning and problem-solving abilities of AI models. It focuses on olympiad-level mathematics and complex multi-step mathematical reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/matharena-apex?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MathArena Apex","organization_count":1,"organizations":["llm_stats"],"rank":1877,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/matharena-apex?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1089-mathbench","caveat":"MathBench, a new benchmark that rigorously assesses the mathematical capabilities of large\nlanguage models. MathBench spans a wide range of mathematical disciplines, offering a\ndetailed evaluation of both theoretical understanding and practical problem-solving skills. MathBench 严格评估大型语言模型的数学能力。MathBench 涉及广泛的数学学科，提供对理论理解和实际问题解决技能的详细评估。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathBench"],"document_share":0.0002723311546840959,"domain":"数学","name":"MathBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1878,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MathBench"},{"aliases":[],"benchmark_id":"opencompass-1115-mathqa","caveat":"MathQA is a new large-scale, diverse dataset of 37k English multiple-choice math word problems covering multiple math domain categories by modeling operation programs\ncorresponding to word problems in the AQuA dataset. MathQA 是一个大规模、多样化的数据集，包含 37,000 道英语多项选择数学文字问题，涵盖多个数学领域类别。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathQA"],"document_share":0.0002723311546840959,"domain":"数学","name":"MathQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1879,"released":"2019-05-31","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MathQA"},{"aliases":[],"benchmark_id":"llm-stats-mathverse","caveat":"MathVerse evaluates multimodal mathematical reasoning, testing whether models genuinely interpret visual math diagrams rather than relying on text.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathverse?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MathVerse","organization_count":1,"organizations":["llm_stats"],"rank":1880,"released":"2024-03-21","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1371-mathverse","caveat":"MathVerse is intended for evaluating MLLMs' visual math problem-solving, containing 2,612 high-quality, multi-subject math problems with diagrams. MathVerse用于评估多模态大模型的视觉数学问题解决能力，包含2612个高质量、多主题的数学问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathVerse"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MathVerse","organization_count":1,"organizations":["opencompass_hub"],"rank":1881,"released":"2024-03-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MathVerse"},{"aliases":[],"benchmark_id":"llm-stats-mathverse-mini","caveat":"MathVerse-Mini is a subset of the MathVerse benchmark for evaluating math reasoning capabilities in vision-language models.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathverse-mini?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MathVerse-Mini","organization_count":1,"organizations":["llm_stats"],"rank":1882,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mathverse-mini?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-12954","caveat":"This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12954"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MathVis","organization_count":1,"organizations":["claire_radar"],"rank":1883,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.12954"},{"aliases":[],"benchmark_id":"llm-stats-mathvision","caveat":"MATH-Vision is a dataset designed to measure multimodal mathematical reasoning capabilities. It focuses on evaluating how well models can solve mathematical problems that require both visual understanding and mathematical reasoning, bridging the gap between visual and mathematical domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathvision?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MathVision","organization_count":1,"organizations":["llm_stats"],"rank":1884,"released":"2024-02-22","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvision?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1370-mathvision","caveat":"MathVision measures multimodal mathematical reasoning capabilities through a meticulously curated collection of 3,040 high-quality mathematical problems spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty. MathVision用于评估多模态大模型的数学推理能力，由涵盖16个数学领域、跨越5个难度级别的3040个高质量数学问题组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathVision"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MathVision","organization_count":1,"organizations":["opencompass_hub"],"rank":1885,"released":"2024-02-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MathVision"},{"aliases":[],"benchmark_id":"llm-stats-mathvista","caveat":"MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathvista?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MathVista","organization_count":1,"organizations":["llm_stats"],"rank":1886,"released":"2023-10-03","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1178-mathvista","caveat":"MathVista is a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets. MathVista用于评估多模态大模型的数学能力，结合了丰富的数学和视觉任务挑战。它由6141个示例组成，来自28个涉及数学的现有多模态数据集和3个新创建的数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MathVista"],"document_share":0.0002723311546840959,"domain":"强推理","name":"MathVista","organization_count":1,"organizations":["opencompass_hub"],"rank":1887,"released":"2023-10-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MathVista"},{"aliases":[],"benchmark_id":"llm-stats-mathvista-mini","caveat":"MathVista-Mini is a smaller version of the MathVista benchmark that evaluates mathematical reasoning in visual contexts. It consists of examples derived from multimodal datasets involving mathematics, combining challenges from diverse mathematical and visual tasks to assess foundation models' ability to solve problems requiring both visual understanding and mathematical reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mathvista-mini?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MathVista-Mini","organization_count":1,"organizations":["llm_stats"],"rank":1888,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mathvista-mini?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-02934","caveat":"MatPhaseBench evaluates vision-language models on understanding materials phase diagrams, using 200 diagram-text pairs from 3681 papers. It targets complex scientific image understanding, with tasks requiring deep comprehension and open-ended responses, covering 189 material systems and 70 elements.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02934"],"document_share":0.0002723311546840959,"domain":"Materials & Chemistry","name":"MatPhaseBench","organization_count":1,"organizations":["claire_radar"],"rank":1889,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.02934"},{"aliases":[],"benchmark_id":"claire-radar-2608-24107","caveat":"A reference-free benchmark for material replacement in interior scenes, evaluating edits on local material correctness, global lighting harmony, outside preservation, and inside structure across three conditioning tracks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24107"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MatReplace","organization_count":1,"organizations":["claire_radar"],"rank":1890,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24107"},{"aliases":[],"benchmark_id":"claire-radar-2609-37053","caveat":"Multimodal GUI agents have achieved impressive results on general software benchmarks, yet their ability to operate professional scientific software remains largely unexplored.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37053"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"MatToolBench","organization_count":1,"organizations":["claire_radar"],"rank":1891,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37053"},{"aliases":[],"benchmark_id":"llm-stats-maverix","caveat":"MAVERIX (Multimodal Audio-Visual Evaluation Reasoning Index) evaluates multimodal models on tasks that demand tight integration of video and audio information. It features challenges like situational awareness and social sentiment analysis where the answer cannot be reliably determined from a single modality, rigorously testing joint audio-visual understanding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/maverix?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MAVERIX","organization_count":1,"organizations":["llm_stats"],"rank":1892,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/maverix?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1848-mavos-dd","caveat":"MAVOS-DD is the first large-scale open-set benchmark for multilingual audio-video deepfake detection. MAVOS-DD是一个大规模的多语言音视频深度伪造检测基准数据集，包含超过250小时的真实和伪造视频，涵盖八种语言。该数据集通过七种不同的深度伪造生成模型生成伪造视频，这些模型基于不同的生成方法，包括说话头像生成、表情转移和换脸。MAVOS-DD设计了多种开放集测试场景，包括开放集模型、开放集语言和全开放集，以评估深度伪造检测模型在未知模型和语言下的泛化能力。实验结果表明，现有的深度伪造检测模型在开放集场景下的性能显著下降，凸显了开发更鲁棒检测技术的必要性。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MAVOS-DD"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MAVOS-DD","organization_count":1,"organizations":["opencompass_hub"],"rank":1893,"released":"2025-05-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MAVOS-DD"},{"aliases":[],"benchmark_id":"llm-stats-maxife","caveat":"MAXIFE is a multilingual benchmark evaluating LLMs on instruction following and execution across multiple languages and cultural contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/maxife?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"MAXIFE","organization_count":1,"organizations":["llm_stats"],"rank":1894,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/maxife?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-11616","caveat":"Evaluates multimodal business ideation agents across six domains using 30K samples, with MLLM-as-a-Judge scoring over six business-oriented criteria.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11616"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MBA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1895,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.11616"},{"aliases":[],"benchmark_id":"claire-radar-2606-00793","caveat":"MBench is a benchmark for memory capability of video world models, decomposing into entity, environment, and causal consistency with 12 sub-dimensions. Includes code, dataset, and leaderboard.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00793"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MBench","organization_count":1,"organizations":["claire_radar"],"rank":1896,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00793"},{"aliases":[],"benchmark_id":"llm-stats-mbpp","caveat":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MBPP","organization_count":1,"organizations":["llm_stats"],"rank":1897,"released":"2021-08-16","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-538-mbpp","caveat":"The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. 该基准测试由大约1000个入门级程序员可以解决的众包Python编程问题组成，涵盖编程基础知识、标准库功能等。每个问题都由任务描述、代码解决方案和3个自动化测试用例组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MBPP"],"document_share":0.0002723311546840959,"domain":"代码","name":"MBPP","organization_count":1,"organizations":["opencompass_hub"],"rank":1898,"released":"2021-08-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MBPP"},{"aliases":[],"benchmark_id":"llm-stats-mbpp-base-version","caveat":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality. This is an enhanced version with additional test cases.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-%2B%2B-base-version?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MBPP ++ base version","organization_count":1,"organizations":["llm_stats"],"rank":1899,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-%2B%2B-base-version?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mbpp-evalplus","caveat":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. EvalPlus extends MBPP with significantly more test cases (35x) for more rigorous evaluation of LLM-synthesized code, providing high-quality and precise evaluation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MBPP EvalPlus","organization_count":1,"organizations":["llm_stats"],"rank":1900,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mbpp-evalplus-base","caveat":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. EvalPlus extends MBPP with significantly more test cases (35x) for more rigorous evaluation of LLM-synthesized code, providing high-quality and precise evaluation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus-%28base%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MBPP EvalPlus (base)","organization_count":1,"organizations":["llm_stats"],"rank":1901,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-evalplus-%28base%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mbpp-pass-1","caveat":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases. This variant uses pass@1 evaluation metric measuring the percentage of problems solved correctly on the first attempt.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-pass%401?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MBPP pass@1","organization_count":1,"organizations":["llm_stats"],"rank":1902,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-pass%401?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mbpp-plus","caveat":"MBPP (Mostly Basic Python Problems) is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem consists of a task description, code solution, and 3 automated test cases covering programming fundamentals and standard library functionality. This is an enhanced version with additional test cases for more rigorous evaluation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp-plus?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MBPP Plus","organization_count":1,"organizations":["llm_stats"],"rank":1903,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp-plus?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mbpp-2","caveat":"MBPP+ is an enhanced version of MBPP (Mostly Basic Python Problems) with significantly more test cases (35x) for more rigorous evaluation. MBPP is a benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers, covering programming fundamentals and standard library functionality.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mbpp%2B?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MBPP+","organization_count":1,"organizations":["llm_stats"],"rank":1904,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mbpp%2B?top_n=500"},{"aliases":["MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models"],"benchmark_id":"claire-radar-2608-24118","caveat":"MC-CXR is a benchmark of 240 chest X-ray cases (2,522 instances) for evaluating context-induced disruption in vision-language models. It pairs reliable and misleading context across text and prior imaging, with visual overlays. Defines three task families and two paired metrics: switch-to-wrong rate and context-aligned error rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24118"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MC-CXR","organization_count":1,"organizations":["claire_radar"],"rank":1905,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24118"},{"aliases":[],"benchmark_id":"claire-radar-2609-14495","caveat":"MCIQA-2K includes 2,000 colorized images generated by five models, annotated across color smearing, semantic color misalignment, and global naturalness for no-reference image quality assessment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.14495"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MCIQA-2K","organization_count":1,"organizations":["claire_radar"],"rank":1906,"released":"2026-09-13","source":"claire_radar","url":"https://arxiv.org/abs/2609.14495"},{"aliases":[],"benchmark_id":"opencompass-1606-mcitebench","caveat":"MCiteBench is a benchmark to evaluate multimodal citation text generation in MLLMs. It consists of 3,000 samples from 1,749 academic papers, featuring 2,000 Explanation tasks and 1,000 Locating tasks, with balanced evidence across text, figures, tables, and mixed modalities. MCiteBench 是一个用于评估多模态大模型中多模态引用文本生成的基准，由 1,749 篇学术论文中的 3,000 个样本组成，包括 2,000 个解释任务和 1,000 个定位任务，在文本、图表、表格和混合模态中平衡证据。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MCiteBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MCiteBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1907,"released":"2025-03-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MCiteBench"},{"aliases":[],"benchmark_id":"llm-stats-mcp-atlas","caveat":"MCP Atlas is a benchmark for evaluating AI models on scaled tool use capabilities, measuring how well models can coordinate and utilize multiple tools across complex multi-step tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mcp-atlas?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MCP Atlas","organization_count":1,"organizations":["llm_stats"],"rank":1908,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-atlas?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-24161","caveat":"MCP-GRANITE is a benchmark framework for evaluating tool-interface granularity in MCP-based LLM agents under edge and IoT scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24161"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MCP-GRANITE Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":1909,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24161"},{"aliases":[],"benchmark_id":"llm-stats-mcp-mark","caveat":"MCP-Mark evaluates LLMs on their ability to use Model Context Protocol (MCP) tools effectively, testing tool discovery, selection, invocation, and result interpretation across diverse MCP server scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mcp-mark?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"MCP-Mark","organization_count":1,"organizations":["llm_stats"],"rank":1910,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-mark?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-02470","caveat":"MCP-Persona evaluates LLM agents on real-world personalized MCP tools across social media, collaboration, email, and content management applications. It includes 173 tool-chain tasks, 139 unique tools, and 18 MCP servers, with a fully automated environment simulation pipeline for reproducible evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02470"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MCP-Persona","organization_count":1,"organizations":["claire_radar"],"rank":1911,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02470"},{"aliases":[],"benchmark_id":"llm-stats-mcp-universe","caveat":"MCP-Universe evaluates LLMs on complex multi-step agentic tasks using Model Context Protocol (MCP) tools across diverse interactive environments, testing planning, tool orchestration, and task completion.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mcp-universe?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"MCP-Universe","organization_count":1,"organizations":["llm_stats"],"rank":1912,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mcp-universe?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-14642","caveat":"The evaluation object is unclear from the provided information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14642"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MCPEvol-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1913,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14642"},{"aliases":[],"benchmark_id":"claire-radar-2606-21023","caveat":"MCR-Bench is a benchmark for evaluating reproducibility in mission-critical LLM tasks, measuring output consistency across heterogeneous hardware.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21023"],"document_share":0.0002723311546840959,"domain":"Finance","name":"MCR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1914,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21023"},{"aliases":[],"benchmark_id":"claire-radar-2608-27442","caveat":"In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27442"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"MCR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1915,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27442"},{"aliases":["Post-Training VLMs for Video Mistake Detection"],"benchmark_id":"claire-radar-2608-28406","caveat":"Evaluates mistake detection in videos via question answering, testing whether methods can discern if a step was executed correctly against its description for seen and unseen actions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.28406"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MD-VQA","organization_count":1,"organizations":["claire_radar"],"rank":1916,"released":"2026-08-28","source":"claire_radar","url":"https://arxiv.org/abs/2608.28406"},{"aliases":[],"benchmark_id":"claire-radar-2608-02642","caveat":"MDArena evaluates coding agents on 50 containerized molecular dynamics tasks from active research projects, covering trajectory analysis, system preparation, free-energy protocols, and enhanced sampling, with strict pass-based scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02642"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MDArena","organization_count":1,"organizations":["claire_radar"],"rank":1917,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.02642"},{"aliases":[],"benchmark_id":"llm-stats-measurebench","caveat":"MeasureBench evaluates multimodal models on visual measurement and quantitative perception tasks across both real and synthetic imagery, reported as the average over the two settings.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/measurebench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MeasureBench","organization_count":1,"organizations":["llm_stats"],"rank":1918,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/measurebench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-35515","caveat":"Scientific discovery requires not only recovering mathematical laws that describe observable behavior, but also identifying the mechanisms that generate them.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35515"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MechBench","organization_count":1,"organizations":["claire_radar"],"rank":1919,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35515"},{"aliases":[],"benchmark_id":"claire-radar-2606-31966","caveat":"MECoBench is a multimodal embodied cooperation benchmark with an evaluation platform. It spans real-world tasks, two cooperation structures, and three collaboration modes, with code and dataset publicly available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31966"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MECoBench","organization_count":1,"organizations":["claire_radar"],"rank":1920,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31966"},{"aliases":[],"benchmark_id":"opencompass-1641-medagents-bench","caveat":"MedAgentsBench features 862 challenging medical questions from seven datasets, focusing on cases where models struggle. It emphasizes multi-step clinical reasoning and addresses limitations in existing benchmarks by eliminating simple questions and standardizing evaluation protocols. MedAgentsBench是一个专注于复杂医学推理的基准测试，从七个医学数据集中精选了862个挑战性问题。这些数据集包括MedQA、PubMedQA、MedMCQA、MedBullets、MedExQA、MedXpertQA和MMLU/MMLU-Pro，涵盖了从医学执照考试到研究文献的多种医学问题。\n该基准选择少于50%模型能正确回答的问题，确保医学知识领域全面覆盖，并优先选择需要多步临床推理的问题。这解决了现有评估中简单问题普遍存在、评估协议不一致，以及缺乏性能-成本-时间分析的局限。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedAgents-Bench"],"document_share":0.0002723311546840959,"domain":"推理","name":"MedAgents-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1921,"released":"2025-03-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedAgents-Bench"},{"aliases":[],"benchmark_id":"opencompass-1893-medal","caveat":"MEDAL is a framework for generating and evaluating multilingual open-domain chatbots and their evaluators. This framework supports various language models and provides a structured approach to creating conversational datasets. MEDAL是一个用于生成和评估多语言开放域聊天机器人及其评估器的框架。该框架支持各种语言模型,并提供了一种结构化的方法来创建对话数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MEDAL"],"document_share":0.0002723311546840959,"domain":"其他","name":"MEDAL","organization_count":1,"organizations":["opencompass_hub"],"rank":1922,"released":"2025-05-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MEDAL"},{"aliases":[],"benchmark_id":"opencompass-1874-medarabiq","caveat":"MedArabiQ introduces a new benchmark dataset consisting of seven Arabic medical tasks, covering multiple specialties and question formats:\n✅ Multiple-choice questions\n✏️ Fill-in-the-blank (with and without choices)\n💬 Patient-doctor question answering 大型语言模型（LLMs）在医疗保健应用中显示出了显著的前景，但由于缺乏高质量的领域特定数据集，它们在阿拉伯语医学领域的表现在很大程度上仍未得到探索。MedArabiQ引入了一个新的基准数据集，包含七个阿拉伯语医学任务，涵盖多个专业和问题格式：\n✅ 多项选择题\n✏️ 填空题（有选项和无选项）\n💬 患者-医生问答\n\n该数据集使用过往医学考试和公开可用资源构建，并进行了修改以评估LLMs在各种能力方面的表现，包括偏见缓解。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedArabiQ"],"document_share":0.0002723311546840959,"domain":"其他","name":"MedArabiQ","organization_count":1,"organizations":["opencompass_hub"],"rank":1923,"released":"2025-05-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedArabiQ"},{"aliases":[],"benchmark_id":"claire-radar-2606-24155","caveat":"MedBench v5 evaluates clinical multimodal models across 63 tasks with process-oriented metrics including stressors and hallucination tracking.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24155"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MedBench","organization_count":1,"organizations":["claire_radar"],"rank":1924,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24155"},{"aliases":[],"benchmark_id":"claire-radar-2610-11312","caveat":"Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible with richly annotated imaging datasets and large language models (LLMs), yet existing automation largely focuses on generating evaluation i…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11312"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MedBench","organization_count":1,"organizations":["claire_radar"],"rank":1925,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11312"},{"aliases":[],"benchmark_id":"opencompass-1287-medbench","caveat":"MedBench is committed to building a scientific, fair and rigorous Chinese medical model evaluation system and open platform. Based on authoritative standards, we constantly update and maintain high-quality datasets, and comprehensively quantify capabilities of models in various medical dimensions. MedBench致力于打造一个科学、公平且严谨的中文医疗大模型评测体系及开放平台。我们基于医学权威标准，不断更新维护高质量的医学数据集，全方位多维度量化模型在各个医学维度的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedBench"],"document_share":0.0002723311546840959,"domain":"知识","name":"MedBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1926,"released":"2023-12-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedBench"},{"aliases":[],"benchmark_id":"opencompass-1920-medbookvqa","caveat":"MedBookVQA is a medical visual question answering (VQA) benchmark constructed from open-access medical textbooks. It includes 5,000 questions across five clinical task types and is hierarchically organized by imaging modality, anatomical structure, and clinical specialty. MedBookVQA 是一个基于开放获取医学教科书构建的医学视觉问答（VQA）基准数据集。它包含 5,000 个问题，涵盖五种临床任务类型，并按照影像模态、解剖结构和临床专科进行分层组织。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedBookVQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MedBookVQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1927,"released":"2025-05-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedBookVQA"},{"aliases":[],"benchmark_id":"opencompass-1832-medbrowsecomp","caveat":"the first benchmark that systematically tests an agent’s ability to reliably retrieve and synthesize multi-hop medical facts from live, domain-specific knowledge bases. MedBrowseComp holds 1,000+ human-curated questions that mirror clinical scenarios\n the first benchmark that systematically tests an agent’s ability to reliably retrieve and synthesize multi-hop medical facts from live, domain-specific knowledge bases. MedBrowseComp holds 1,000+ human-curated questions that mirror clinical scenarios\n","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedBrowseComp"],"document_share":0.0002723311546840959,"domain":"其他","name":"MedBrowseComp","organization_count":1,"organizations":["opencompass_hub"],"rank":1928,"released":"2025-05-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedBrowseComp"},{"aliases":[],"benchmark_id":"opencompass-1241-medcalc-bench","caveat":"MedCalc-Bench focuses on evaluating the medical calculation capability of LLMs. It contains an evaluation set of over 1000 manually reviewed instances from 55 different medical calculation tasks. MedCalc-Bench专注于评估LLM的医学计算能力，包含来自55个不同医学计算任务的1000个经过人工审查的实例。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedCalc-Bench"],"document_share":0.0002723311546840959,"domain":"推理","name":"MedCalc-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":1929,"released":"2024-06-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedCalc-Bench"},{"aliases":[],"benchmark_id":"llm-stats-medchembench","caveat":"MedChemBench is an internal OpenAI evaluation of medicinal-chemistry reasoning, testing whether models can support drug-discovery-relevant chemistry analysis and decision-making.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/medchembench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MedChemBench (Internal)","organization_count":1,"organizations":["llm_stats"],"rank":1930,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/medchembench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-14015","caveat":"MedClawBench evaluates long-horizon temporal reasoning in surgical videos through 1,123 doctor-grounded questions over self-built long neurosurgery recordings and a public lecture-video test split, with fixed evaluation dimensions for comparison.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14015"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MedClawBench","organization_count":1,"organizations":["claire_radar"],"rank":1931,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14015"},{"aliases":[],"benchmark_id":"claire-radar-2606-11702","caveat":"MedCTA evaluates clinical tool agents on 107 clinician-verified, step-implicit tasks with multimodal inputs (radiology images, pathology slides, reports) and 5 deployed tools. Metrics cover tool selection, argument validity, execution stability, trajectory fidelity, and outcome quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11702"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedCTA","organization_count":1,"organizations":["claire_radar"],"rank":1932,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.11702"},{"aliases":[],"benchmark_id":"claire-radar-2606-03203","caveat":"MedCUA-Bench is an interactive benchmark for clinical computer-use agents, covering 18 scenarios in 10 medical domains with deterministic safety evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03203"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"MedCUA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1933,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03203"},{"aliases":[],"benchmark_id":"claire-radar-2607-18999","caveat":"MedDDC-Eval evaluates multi-turn medical consultation agents by decoupling diagnosis from the diagnostic reader, using a frozen shared reader to score policies. It reports diagnostic support, coverage, and efficiency across Record and Dialogue splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18999"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedDDC-Eval","organization_count":1,"organizations":["claire_radar"],"rank":1934,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.18999"},{"aliases":[],"benchmark_id":"claire-radar-2607-15166","caveat":"The evaluation object is unclear from the provided information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15166"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedFailBench","organization_count":1,"organizations":["claire_radar"],"rank":1935,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.15166"},{"aliases":[],"benchmark_id":"claire-radar-2606-01028","caveat":"Provides a continuous-time reinforcement learning environment for dynamic medical treatment recommendation, built from clinical data with physics-informed neural networks. Supports offline and online RL with configurable parameters.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01028"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedGym","organization_count":1,"organizations":["claire_radar"],"rank":1936,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01028"},{"aliases":[],"benchmark_id":"claire-radar-2606-21517","caveat":"MedHal-Loc is a benchmark and metric for localization faithfulness of medical hallucination detectors, comprising a controlled subset of 300 PubMedQA-derived statements with injected span-level errors and a natural subset. It evaluates whether top-ranked error units overlap erroneous spans across four error types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21517"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedHal-Loc","organization_count":1,"organizations":["claire_radar"],"rank":1937,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21517"},{"aliases":[],"benchmark_id":"opencompass-2417-medhalltune","caveat":null,"document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedHallTune"],"document_share":0.0002723311546840959,"domain":"医学","name":"MedHallTune","organization_count":1,"organizations":["opencompass_hub"],"rank":1938,"released":"2025-02-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedHallTune"},{"aliases":[],"benchmark_id":"opencompass-1548-medhallu","caveat":"MedHallu is a comprehensive benchmark dataset designed to evaluate the ability of large language models to detect hallucinations in medical question-answering tasks. MedHallu 旨在评估大型语言模型在医学问题解答任务中检测幻觉的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedHallu"],"document_share":0.0002723311546840959,"domain":"其他","name":"MedHallu","organization_count":1,"organizations":["opencompass_hub"],"rank":1939,"released":"2025-02-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedHallu"},{"aliases":[],"benchmark_id":"claire-radar-2610-15714","caveat":"Evaluates healthcare agents on 342 executable tasks spanning FHIR/EHR operations, healthcare administration interfaces, and multimodal clinical tools, measuring both outcome success and seven process-quality dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.15714"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MedHarnessBench","organization_count":1,"organizations":["claire_radar"],"rank":1940,"released":"2026-10-11","source":"claire_radar","url":"https://arxiv.org/abs/2610.15714"},{"aliases":[],"benchmark_id":"claire-radar-2610-04800","caveat":"Graphical consoles offer a practical interface for medical acquisition assistance, allowing agents to work through the controls and visual feedback used by human operators.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.04800"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedImageOSWorld","organization_count":1,"organizations":["claire_radar"],"rank":1941,"released":"2026-10-03","source":"claire_radar","url":"https://arxiv.org/abs/2610.04800"},{"aliases":[],"benchmark_id":"claire-radar-2607-25300","caveat":"Evaluates message-driven narrative video editing with long-form videos paired with multiple editing messages and multiple professional edits per message, using temporal alignment metrics and additional annotations for message ambiguity and contextfulness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25300"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MEDit-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1942,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25300"},{"aliases":[],"benchmark_id":"opencompass-1326-medjourney","caveat":"MedJourney offers a comprehensive assessment of LLMs' effectiveness in real-world clinical settings. It includes multiple tasks from 4 stages of a typical patient's hospital visit journey and comprises 12 datasets. MedJourney用于评估 LLM 在真实临床环境中的有效性，其中包含多个任务，涵盖来自患者就诊典型流程的4个阶段的12个数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedJourney"],"document_share":0.0002723311546840959,"domain":"推理","name":"MedJourney","organization_count":1,"organizations":["opencompass_hub"],"rank":1943,"released":"2024-09-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedJourney"},{"aliases":[],"benchmark_id":"claire-radar-2606-21194","caveat":"MedLayXPlain is a benchmark for medical lay language generation, pairing medical images with expert and lay captions across 122,789 samples from 8 imaging modalities. It introduces a 3B evaluator model scoring expert-lay alignment on five attributes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21194"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MEDLAYXPLAIN","organization_count":1,"organizations":["claire_radar"],"rank":1944,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21194"},{"aliases":[],"benchmark_id":"claire-radar-2608-03028","caveat":"MedPIC-Bench evaluates patient-specific medication-safety reasoning in LLMs using guideline-following and paired counterfactual questions, with 467 questions annotated across six dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03028"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedPIC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1945,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03028"},{"aliases":[],"benchmark_id":"claire-radar-2608-02520","caveat":"MedPRESS evaluates LLM sycophancy in multi-turn medical dialogues, containing 600 five-turn scenarios across three families with structured judging and safety metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02520"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedPRESS","organization_count":1,"organizations":["claire_radar"],"rank":1946,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02520"},{"aliases":["MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation"],"benchmark_id":"claire-radar-2607-09142","caveat":"A multimodal next-response generation benchmark derived from de-identified Chinese online medical consultations, with case-specific physician-refined rubrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09142"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedRealMM","organization_count":1,"organizations":["claire_radar"],"rank":1947,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.09142"},{"aliases":[],"benchmark_id":"claire-radar-2608-22323","caveat":"Evaluates LLMs on multimodal clinical diagnostic synthesis using 625 expert-validated cases with medical images and ICD-11 diagnoses.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22323v1"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedReaMM","organization_count":1,"organizations":["claire_radar"],"rank":1948,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22323v1"},{"aliases":[],"benchmark_id":"opencompass-1331-medsafetybench","caveat":"MedSafetyBench is designed to measure the medical safety of LLMs. It includes 1,800 medical safety demonstrations, where each safety demonstration consists of a harmful medical request and a corresponding safe response. MedSafetyBench用于评估LLM在医疗安全上的表现，包含1800个由有害请求和安全响应组成的医疗安全场景。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedSafetyBench"],"document_share":0.0002723311546840959,"domain":"安全","name":"MedSafetyBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1949,"released":"2024-03-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedSafetyBench"},{"aliases":["MEDSEGBENCH"],"benchmark_id":"claire-radar-2608-29677","caveat":"The framework supports configuration-driven benchmarking of 2D medical image segmentation, exporting raw predictions and pixel counts for post-hoc analysis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29677"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MEDSEGBENCHMARKER","organization_count":1,"organizations":["claire_radar"],"rank":1950,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29677"},{"aliases":[],"benchmark_id":"claire-radar-2607-01751","caveat":"MedStreamBench is a time-aware benchmark for medical video understanding, integrating 22 medical datasets and 5,419 QA instances across four temporal settings: retrospective, present, future, and proactive. Models are restricted to temporally bounded evidence windows and evaluated on answer correctness, responsiveness, and post-evidence stability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01751"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MedStreamBench","organization_count":1,"organizations":["claire_radar"],"rank":1951,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.01751"},{"aliases":[],"benchmark_id":"llm-stats-medxpertqa","caveat":"A comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning, featuring 4,460 questions spanning 17 specialties and 11 body systems. Includes both text-only and multimodal subsets with expert-level exam questions incorporating diverse medical images and rich clinical information.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MedXpertQA","organization_count":1,"organizations":["llm_stats"],"rank":1952,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1891-medxpertqa","caveat":"We introduce MedXpertQA, a highly challenging and comprehensive medical benchmark to evaluate expert-level medical knowledge and advanced reasoning. \nIt has been accepted by ICML 2025 and selected by Google DeepMind as the benchmark for MedGemma. MedXpertQA是由清华大学和上海人工智能实验室构建的全面且具有高度挑战性的医学基准，用于评估专家级的医学知识和高级推理能力。论文已被ICML 2025接收，并被Google DeepMind使用作为MedGemma的评估基准。MedXpertQA 共包含 4,460 道题目，涵盖 17 个医学专科和 11 个身体系统。该基准包含两个子集：用于文本医学能力评估的 Text 子集，以及用于多模态医学能力评估的 MM 子集。MM 子集首次引入了带有多样化图像和丰富临床信息（如病历和检查结果）的专家级考试题，区别于传统多模态医学基准中基于图像描述生成的简单问答对。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MedXpertQA"],"document_share":0.0002723311546840959,"domain":"学科","name":"MedXpertQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1953,"released":"2025-02-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MedXpertQA"},{"aliases":[],"benchmark_id":"llm-stats-medxpertqa-mm","caveat":"MedXpertQA-MM is the multimodal subset of MedXpertQA, evaluating expert-level medical question answering grounded in medical images.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa-mm?top_n=500"],"document_share":0.0002723311546840959,"domain":"medical","name":"MedXpertQA-MM","organization_count":1,"organizations":["llm_stats"],"rank":1954,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/medxpertqa-mm?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-19235","caveat":"MeetingToM evaluates multimodal LLMs on theory-of-mind reasoning in multi-party meetings, including pseudo-consensus detection, across three levels: subject, dyad, and group. It provides a unified evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19235"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MeetingToM","organization_count":1,"organizations":["claire_radar"],"rank":1955,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.19235"},{"aliases":[],"benchmark_id":"llm-stats-mega-mlqa","caveat":"MLQA as part of the MEGA (Multilingual Evaluation of Generative AI) benchmark suite. A multi-way aligned extractive QA evaluation benchmark for cross-lingual question answering across 7 languages (English, Arabic, German, Spanish, Hindi, Vietnamese, and Simplified Chinese) with over 12K QA instances in English and 5K in each other language.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-mlqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MEGA MLQA","organization_count":1,"organizations":["llm_stats"],"rank":1956,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-mlqa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mega-tydi-qa","caveat":"TyDi QA as part of the MEGA benchmark suite. A question answering dataset covering 11 typologically diverse languages (Arabic, Bengali, English, Finnish, Indonesian, Japanese, Korean, Russian, Swahili, Telugu, and Thai) with 204K question-answer pairs. Features realistic information-seeking questions written by people who want to know the answer but don't know it yet.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-tydi-qa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MEGA TyDi QA","organization_count":1,"organizations":["llm_stats"],"rank":1957,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-tydi-qa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mega-udpos","caveat":"Universal Dependencies POS tagging as part of the MEGA benchmark suite. A multilingual part-of-speech tagging dataset based on Universal Dependencies treebanks, utilizing the universal POS tag set of 17 tags across 38 diverse languages from different language families. Used for evaluating multilingual POS tagging systems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-udpos?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"MEGA UDPOS","organization_count":1,"organizations":["llm_stats"],"rank":1958,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-udpos?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mega-xcopa","caveat":"XCOPA (Cross-lingual Choice of Plausible Alternatives) as part of the MEGA benchmark suite. A typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages, including resource-poor languages like Eastern Apurímac Quechua and Haitian Creole. Requires models to select which choice is the effect or cause of a given premise.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-xcopa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MEGA XCOPA","organization_count":1,"organizations":["llm_stats"],"rank":1959,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xcopa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mega-xstorycloze","caveat":"XStoryCloze as part of the MEGA benchmark suite. A cross-lingual story completion task that consists of professionally translated versions of the English StoryCloze dataset to 10 non-English languages. Requires models to predict the correct ending for a given four-sentence story, evaluating commonsense reasoning and narrative understanding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mega-xstorycloze?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MEGA XStoryCloze","organization_count":1,"organizations":["llm_stats"],"rank":1960,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mega-xstorycloze?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-meld","caveat":"MELD (Multimodal EmotionLines Dataset) is a multimodal multi-party dataset for emotion recognition in conversations. Contains approximately 13,000 utterances from 1,433 dialogues extracted from the TV series Friends. Each utterance is annotated with emotion (Anger, Disgust, Sadness, Joy, Neutral, Surprise, Fear) and sentiment labels across audio, visual, and textual modalities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/meld?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Meld","organization_count":1,"organizations":["llm_stats"],"rank":1961,"released":"2018-10-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/meld?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-24259","caveat":"MemCalib is a benchmark for evaluating memory use in LLM agents in realistic memory-system scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24259"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemCalib","organization_count":1,"organizations":["claire_radar"],"rank":1962,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24259"},{"aliases":[],"benchmark_id":"claire-radar-2607-27798","caveat":"MemeBench is a diagnostic benchmark of 1,253 Chinese and English memes with human-written references and VIKR annotations—Visual clues, Identity links, Knowledge units, and Reasoning mechanisms—for evaluating interpretation in LVLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27798"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemeBench","organization_count":1,"organizations":["claire_radar"],"rank":1963,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27798"},{"aliases":["Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classification"],"benchmark_id":"claire-radar-2609-26907","caveat":"A cue-focused benchmark of 9,895 memes across harm, hate, and sarcasm, with annotations identifying the modality and rationale of the pivotal evidence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.26907"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemeCF","organization_count":1,"organizations":["claire_radar"],"rank":1964,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.26907"},{"aliases":[],"benchmark_id":"claire-radar-2609-01772","caveat":"A multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi paired with cultural context notes and three human-written explanations, plus 54 Bengali dialect memes, for evaluating vision-language models' meme explanation generation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01772"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemeCULT-1K","organization_count":1,"organizations":["claire_radar"],"rank":1965,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01772"},{"aliases":[],"benchmark_id":"claire-radar-2605-26667","caveat":"Diagnostic benchmark isolating failure modes of LLM memory systems by evaluating summarization, storage, and retrieval operations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26667"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemFail","organization_count":1,"organizations":["claire_radar"],"rank":1966,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26667"},{"aliases":["Profile-Graph Memory for LLM Agents: Implicit Cross-Entity Traversal through Narrative Profiles"],"benchmark_id":"claire-radar-2607-19359","caveat":"A multi-hop long-term memory benchmark for LLM agents: 1,000 questions at hop depths 1–5 across 10 social-network scenarios, with per-hop evidence annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19359"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemHop","organization_count":1,"organizations":["claire_radar"],"rank":1967,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.19359"},{"aliases":[],"benchmark_id":"claire-radar-2606-27537","caveat":"MemoBench evaluates world modeling in video generation via a disappear-and-reappear paradigm. It includes 360 ground-truth clips and an evaluation suite with automated metrics and VQA across four diagnostic pillars.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27537"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemoBench","organization_count":1,"organizations":["claire_radar"],"rank":1968,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.27537"},{"aliases":[],"benchmark_id":"claire-radar-2609-07047","caveat":"Evaluates robotic manipulation policies on memory fidelity via 30 history-dependent tasks with annotated checkpoints measuring storage, update, and compression operations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07047"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"MEMOBench","organization_count":1,"organizations":["claire_radar"],"rank":1969,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07047"},{"aliases":[],"benchmark_id":"claire-radar-2607-12893","caveat":"MemOps evaluates conversational memory as a sequence of lifecycle operations (remembering, forgetting, updating, reflecting) with structured traces and six categories of operation-level probes, under adjacent-evidence and long-context settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.12893"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemOps","organization_count":1,"organizations":["claire_radar"],"rank":1970,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.12893"},{"aliases":["MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs"],"benchmark_id":"claire-radar-2609-35312","caveat":"Human-curated benchmark pairing factual reasoning tasks with structurally identical fictitious versions to measure parametric memory effects on contextual reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35312"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"MemoReason","organization_count":1,"organizations":["claire_radar"],"rank":1971,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35312"},{"aliases":[],"benchmark_id":"claire-radar-2606-04442","caveat":"MemoryDocDataSet evaluates joint conversational memory and long-document reasoning through 50 synthetic micro-worlds, each with personas, temporal event graphs, real legal documents, multi-session conversations, and QA pairs. Questions are categorized by reasoning type, with Hybrid questions requiring navigation of conversation history to locate the relevant document and extract answers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04442"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemoryDocDataSet","organization_count":1,"organizations":["claire_radar"],"rank":1972,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04442"},{"aliases":[],"benchmark_id":"claire-radar-2606-24595","caveat":"MEMPROBE evaluates long-term memory in LLM agents by reconstructing hidden user-state from agent memory across 50 simulated users with 31 hidden dimensions each.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24595"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MEMPROBE","organization_count":1,"organizations":["claire_radar"],"rank":1973,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24595"},{"aliases":[],"benchmark_id":"claire-radar-2609-14976","caveat":"Evaluates long-horizon LLM agents on 120 scripted episodes covering five memory-risk categories through deterministic trace-grounded checks, with full trace logging and no LLM-as-judge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.14976"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MemRiskBench","organization_count":1,"organizations":["claire_radar"],"rank":1974,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.14976"},{"aliases":[],"benchmark_id":"claire-radar-2607-27080","caveat":"Evaluates lifecycle security of agent memory systems with 310 cases across 48 contexts, using a Write-Execute-Forget protocol and evidence-based adjudication across seven checkpoints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27080"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MemSecBench","organization_count":1,"organizations":["claire_radar"],"rank":1975,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27080"},{"aliases":[],"benchmark_id":"claire-radar-2607-01071","caveat":"MemSyco-Bench evaluates memory-induced sycophancy in agent systems across five tasks, measuring how memory influences decision-making and personalization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01071"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MemSyco-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1976,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.01071"},{"aliases":[],"benchmark_id":"claire-radar-2606-17328","caveat":"MemTrace evaluates long-term memory in LLM agents at the knowledge point level, probing memory age, question type, and evidence condition. Scoring is based on accuracy across these dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17328"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"MemTrace","organization_count":1,"organizations":["claire_radar"],"rank":1977,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.17328"},{"aliases":[],"benchmark_id":"claire-radar-2609-32313","caveat":"Memory lets an embodied agent reuse past experience, yet retaining useful information does not ensure that the agent can apply it when conditions change.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32313"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"MemTransfer","organization_count":1,"organizations":["claire_radar"],"rank":1978,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32313"},{"aliases":[],"benchmark_id":"claire-radar-2605-28161","caveat":"MeniOmni is a multimodal benchmark for meniscus injury assessment with 746 MRI studies, supporting Stoller severity grading and diagnostic report generation, with risk-aware ordinal evaluation and semantic consistency metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28161"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MeniOmni","organization_count":1,"organizations":["claire_radar"],"rank":1979,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28161"},{"aliases":[],"benchmark_id":"claire-radar-2606-25325","caveat":"MEP-Bench evaluates whether multimodal emotion models use modality-specific evidence sufficiently and faithfully.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25325"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MEP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1980,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25325"},{"aliases":[],"benchmark_id":"opencompass-2079-mer-unibench","caveat":"MER-UniBench is a benchmark designed to evaluate multimodal large language models (MLLMs) for their emotion understanding capabilities across typical multimodal emotion recognition (MER) tasks. MER-UniBench 是一个旨在评估多模态大语言模型（MLLM）在典型多模态情感识别（MER）任务中情感理解能力的评测基准。它涵盖了细粒度情感识别、基本情感识别和情感分析三个主要维度。该基准利用了包括 MER-Caption 在内的多个数据集，其中 MER-Caption 拥有超过 2000 种细粒度情感类别和 11.5 万个样本。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MER-UniBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MER-UniBench","organization_count":1,"organizations":["opencompass_hub"],"rank":1981,"released":"2025-01-27","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MER-UniBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-15661","caveat":"MergeMedBench evaluates model merging methods for medical LVLMs across eight imaging modalities, comprising 16 LoRA fine-tuned models. It provides evaluation datasets and released model checkpoints for benchmarking merging approaches.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15661"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MergeMedBench","organization_count":1,"organizations":["claire_radar"],"rank":1982,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.15661"},{"aliases":[],"benchmark_id":"claire-radar-2609-01306","caveat":"The benchmark evaluates triangle-based neural rendering across optimization and game-engine deployment pipeline.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01306"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MeshSplatBench","organization_count":1,"organizations":["claire_radar"],"rank":1983,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01306"},{"aliases":[],"benchmark_id":"llm-stats-meta-internal-coding-bench","caveat":"Meta's internal evaluation of coding-agent performance.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/meta-internal-coding-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Meta Internal Coding Bench","organization_count":1,"organizations":["llm_stats"],"rank":1984,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/meta-internal-coding-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-18640","caveat":"MetaboNet-Bench evaluates multimodal glucose forecasting in type 1 diabetes, using glucose, insulin, and carbohydrate data. It provides an extensible open-source evaluation framework for comparing forecasting algorithms.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18640"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MetaboNet-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1985,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18640"},{"aliases":[],"benchmark_id":"claire-radar-2605-25461","caveat":"MetaphorVU-Bench evaluates metaphorical video understanding in multimodal LLMs through tasks requiring cross-domain mapping, with a benchmark dataset and evaluation code publicly available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25461"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MetaphorVU-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1986,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25461"},{"aliases":[],"benchmark_id":"claire-radar-2608-00107","caveat":"MetaRoute-Bench is an open framework for comparing meta-decision policies in agentic workflow routing. Contains 180 synthetic task profiles, eight routing policies, and 30 seeds, evaluating success, cost, and latency through 43,200 traces. Metrics include success rate and paired confidence intervals.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00107"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MetaRoute-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1987,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.00107"},{"aliases":[],"benchmark_id":"claire-radar-2609-25841","caveat":"The evaluation object would be metric-spatial reasoning in indoor scenes, but task details, environment, and scoring setup are not provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25841"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"Metric-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1988,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.25841"},{"aliases":["Metric-Bench"],"benchmark_id":"claire-radar-huggingface-yulingxi-metric-bench","caveat":"A reconstructed test release for metric spatial understanding over indoor RGB images with supplied anchor measurements. Each example pairs one image and one English question with a numeric JSON reference answer, covering width, length, height, dimension, center, depth, elevation, distance and box measurements.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/yulingxi/Metric-Bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Metric-Bench Test","organization_count":1,"organizations":["claire_radar"],"rank":1989,"released":"2026-09-11","source":"claire_radar","url":"https://huggingface.co/datasets/yulingxi/Metric-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2609-10016","caveat":"MetroLLM-Bench evaluates language models as transit kiosk policy layers across six real metro systems. It includes 955 cases covering routing, fare calculation, disruptions, accessibility, and adversarial input, with models required to call structured tools and submit machine-renderable terminal states.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10016"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MetroLLM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1990,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10016"},{"aliases":[],"benchmark_id":"claire-radar-2607-15778","caveat":"MEventBench is a multi-event long video understanding benchmark introduced alongside the MoD-VLLM framework, evaluating temporal grounding and semantic understanding.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15778"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MEventBench","organization_count":1,"organizations":["claire_radar"],"rank":1991,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.15778"},{"aliases":[],"benchmark_id":"llm-stats-mewc","caveat":"MEWC is a benchmark that evaluates AI model performance on multi-environment web challenges, testing agents' ability to navigate and complete complex tasks across diverse web environments.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mewc?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MEWC","organization_count":1,"organizations":["llm_stats"],"rank":1992,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mewc?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mgsm","caveat":"MGSM (Multilingual Grade School Math) is a benchmark of grade-school math problems. Contains 250 grade-school math problems manually translated from the GSM8K dataset into ten typologically diverse languages: Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, and Telugu. Evaluates multilingual mathematical reasoning capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mgsm?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MGSM","organization_count":1,"organizations":["llm_stats"],"rank":1993,"released":"2022-10-06","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mgsm?top_n=500"},{"aliases":["When Names Cross Scripts: A Source-Grounded Benchmark for Historical Entity Reconciliation in the Mongol World"],"benchmark_id":"claire-radar-2608-23507","caveat":"A provenance-controlled benchmark for historical entity reconciliation, comprising a Name-only core of 396 pairs and a Source-grounded subset of 160 pairs with entity-disjoint splits.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23507v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MHER","organization_count":1,"organizations":["claire_radar"],"rank":1994,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23507v1"},{"aliases":[],"benchmark_id":"llm-stats-miabench","caveat":"MIABench evaluates multimodal instruction alignment and following capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/miabench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MIABench","organization_count":1,"organizations":["llm_stats"],"rank":1995,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/miabench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2085-mib","caveat":"MIB is a benchmark designed to evaluate mechanistic interpretability methods for neural language models. MIB 是一个旨在评估神经网络语言模型中机械可解释性方法的基准。它主要评估方法在精确地定位因果路径（电路定位）和特定概念（因果变量定位）方面的能力，侧重于忠实度和最小化电路规模。该基准涵盖了IOI和算术等四个任务，并评估了Llama-3.1和Gemma-2等四种模型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MIB"],"document_share":0.0002723311546840959,"domain":"理解","name":"MIB","organization_count":1,"organizations":["opencompass_hub"],"rank":1996,"released":"2025-04-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MIB"},{"aliases":[],"benchmark_id":"claire-radar-2609-33441","caveat":"MIC-Bench evaluates image-claim consistency in multimodal misinformation using 8,812 instances across nine inconsistency types, with verdict, inconsistency type, visual evidence, and explanation scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33441"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MIC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":1997,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33441"},{"aliases":[],"benchmark_id":"opencompass-1667-microvqa","caveat":"MicroVQA is a benchmark that evaluates LLM reasoning on multiple-choice questions about microscopy images, created by expert biologists. MicroVQA，一个评估关于显微镜图像的多选题推理基准，由专家生物学家创建，旨在反映生物研究中能够有意义地协助的任务，每个问题都需要多模态推理。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MicroVQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MicroVQA","organization_count":1,"organizations":["opencompass_hub"],"rank":1998,"released":"2025-03-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MicroVQA"},{"aliases":[],"benchmark_id":"opencompass-1781-mieb","caveat":"Massive Image Embedding Benchmark (MIEB) is to evaluate the performance of image and image-text embedding models across the broadest spectrum to date. MIEB用于评估图像和图像文本嵌入模型在迄今为止最广泛的范围内的性能。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MIEB"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MIEB","organization_count":1,"organizations":["opencompass_hub"],"rank":1999,"released":"2025-04-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MIEB"},{"aliases":[],"benchmark_id":"claire-radar-github-fahmiwol-migancore","caveat":"Evaluates Indonesian LLM hallucination and abstention behavior on a 36-question battery, reporting fabrication, over-refusal, and fact accuracy with confidence intervals.","document_count":1,"document_ids":["claire_radar:https://github.com/fahmiwol/migancore"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"MiganCore","organization_count":1,"organizations":["claire_radar"],"rank":2000,"released":"2026-09-28","source":"claire_radar","url":"https://github.com/fahmiwol/migancore"},{"aliases":[],"benchmark_id":"claire-radar-2607-27654","caveat":"MiGUE-Bench is a benchmark for multi-granularity event analysis, covering event detection, relation reasoning, structure induction, and future prediction across single to cross-document settings. It uses an LLM-driven annotation pipeline.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27654"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MiGUE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2001,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27654"},{"aliases":[],"benchmark_id":"opencompass-1658-milic-eval","caveat":"MiLiC-Eval is an NLP evaluation suite for Minority Languages in China, covering Tibetan (bo), Uyghur (ug), Kazakh (kk, in the Kazakh Arabic script), and Mongolian (mn, in the traditional Mongolian script). MiLiC-Eval 是针对中国少数民族语言的 NLP 评估套件，涵盖藏语（bo）、维吾尔语（ug）、哈萨克语（kk，使用哈萨克阿拉伯文脚本）和蒙古语（mn，使用传统蒙古文脚本）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MiLiC-Eval"],"document_share":0.0002723311546840959,"domain":"其他","name":"MiLiC-Eval","organization_count":1,"organizations":["opencompass_hub"],"rank":2002,"released":"2025-03-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MiLiC-Eval"},{"aliases":[],"benchmark_id":"llm-stats-mimic-cxr","caveat":"MIMIC-CXR is a large publicly available dataset of chest radiographs with free-text radiology reports. Contains 377,110 images corresponding to 227,835 radiographic studies from 65,379 patients at Beth Israel Deaconess Medical Center. The dataset is de-identified and widely used for medical imaging research, automated report generation, and medical AI development.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mimic-cxr?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MIMIC CXR","organization_count":1,"organizations":["llm_stats"],"rank":2003,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mimic-cxr?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mimo-coding-bench","caveat":"MiMo Coding Bench evaluates coding-agent capabilities on software engineering tasks reported with the MiMo model family.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mimo-coding-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"MiMo Coding Bench","organization_count":1,"organizations":["llm_stats"],"rank":2004,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mimo-coding-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-23336","caveat":"A benchmark for grounded minimal-change understanding in image pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23336"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MinCU","organization_count":1,"organizations":["claire_radar"],"rank":2005,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23336"},{"aliases":[],"benchmark_id":"claire-radar-2609-03464","caveat":"A stress test benchmark for PII detection systems across seven distribution shifts, used to compare failure modes of SpaCy, Presidio, and Qwen2.5-3B.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03464"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Mind the Gap: Robustness Risks in PII Detection Systems","organization_count":1,"organizations":["claire_radar"],"rank":2006,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03464"},{"aliases":[],"benchmark_id":"opencompass-1125-mind2web","caveat":"MIND2WEB is the first dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website. MIND2WEB 是首个用于开发和评估通用网页代理的数据集，能够根据语言指令在任何网站上完成复杂任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Mind2Web"],"document_share":0.0002723311546840959,"domain":"其他","name":"Mind2Web","organization_count":1,"organizations":["opencompass_hub"],"rank":2007,"released":"2023-12-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Mind2Web"},{"aliases":[],"benchmark_id":"claire-radar-2607-00491","caveat":"MindEdit-Bench evaluates counterfactual spatial reasoning in VLMs with 1,003 multiple-choice questions from private indoor scenes, covering six task types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00491"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"MindEdit-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2008,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.00491"},{"aliases":[],"benchmark_id":"claire-radar-2609-11900","caveat":"Evaluation of topological intuition across continuity, separation, order, enclosure, and knots grounded in cognitive science.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11900"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MindTopo","organization_count":1,"organizations":["claire_radar"],"rank":2009,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11900"},{"aliases":[],"benchmark_id":"claire-radar-2610-11328","caveat":"Advances in foundation models are driving efforts to introduce agents to assist people in the physical world.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11328"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Mine Odyssey","organization_count":1,"organizations":["claire_radar"],"rank":2010,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11328"},{"aliases":["MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft"],"benchmark_id":"claire-radar-2605-30931","caveat":"A Minecraft benchmark that composes atomic tasks into implicit multi-hop exploration tasks for MLLM agents, scored by rule-based milestone evaluators.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30931"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"MineExplorer","organization_count":1,"organizations":["claire_radar"],"rank":2011,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.30931"},{"aliases":[],"benchmark_id":"llm-stats-minerva","caveat":"Minerva is a benchmark for complex video reasoning, evaluating models on multi-step reasoning over long and information-dense video content.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/minerva?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Minerva","organization_count":1,"organizations":["llm_stats"],"rank":2012,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/minerva?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-04569","caveat":"MineXplore is an open-source MuJoCo-based benchmark for robot navigation in GNSS-denied underground mine environments, derived from the Leung et al. 2017 Chilean copper mine dataset. It reconstructs a 104,423 sq.m tunnel network with octagonal wall cross-sections, LiDAR-sourced jagged wall geometry, three terrain friction zones, a global 5 degree incline, and periodic spot lighting. The benchmark includes an evaluation protocol based on coverage percentage, with a 90% target for single-agent PPO policies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04569"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"MineXplore","organization_count":1,"organizations":["claire_radar"],"rank":2013,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04569"},{"aliases":["Mind the Gaps: A Curated Benchmark for Form Field Detection"],"benchmark_id":"claire-radar-2609-23679","caveat":"Evaluates form field detection models on 47,355 document pages in COCO format with annotations for text, choice, and signature fields.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23679"],"document_share":0.0002723311546840959,"domain":"General AI","name":"mini-CommonForms","organization_count":1,"organizations":["claire_radar"],"rank":2014,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23679"},{"aliases":[],"benchmark_id":"opencompass-1518-minictx","caveat":"A context-rich benchmark for evaluating neural theorem proving in realistic scenarios, providing premises, full context, multi-source benchmark, and temporal splits, enabling evaluation of a model's ability to work with context that evolves over time. 一个用于评估现实场景中神经定理证明的丰富上下文基准。通过提供前提、完整上下文、多源基准和时序分割，突出其评估模型处理随时间演变的上下文能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/miniCTX"],"document_share":0.0002723311546840959,"domain":"强推理","name":"miniCTX","organization_count":1,"organizations":["opencompass_hub"],"rank":2015,"released":"2024-08-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/miniCTX"},{"aliases":["Pythagoras-Prover: Advancing Efficient Formal Proving via Augmented Lean Formalisation"],"benchmark_id":"claire-radar-2606-12594","caveat":"A released, contamination-sensitive benchmark of augmented Lean formal statements for evaluating formal theorem provers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12594"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"MiniF2F-ALF","organization_count":1,"organizations":["claire_radar"],"rank":2016,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12594"},{"aliases":[],"benchmark_id":"opencompass-1853-minilongbench","caveat":"MiniLongBench is a low-cost benchmark for evaluating the Long Context Understanding (LCU) capabilities of LLMs, featuring a compact yet diverse test set of only 237 samples spanning 6 major task categories and 21 distinct tasks. MiniLongBench is a low-cost benchmark for evaluating the Long Context Understanding (LCU) capabilities of LLMs, featuring a compact yet diverse test set of only 237 samples spanning 6 major task categories and 21 distinct tasks.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MiniLongBench"],"document_share":0.0002723311546840959,"domain":"理解","name":"MiniLongBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2017,"released":"2025-05-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MiniLongBench"},{"aliases":[],"benchmark_id":"claire-radar-2610-11233","caveat":"A core challenge in short-video fact-checking is identifying which evidence is sufficient to support a verification conclusion.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11233"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MiniVer-V","organization_count":1,"organizations":["claire_radar"],"rank":2018,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11233"},{"aliases":[],"benchmark_id":"claire-radar-2610-03080","caveat":"Large language models are moving from producing trading signals to writing the code that executes them.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03080"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"MintEval","organization_count":1,"organizations":["claire_radar"],"rank":2019,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03080"},{"aliases":[],"benchmark_id":"claire-radar-2607-19201","caveat":"MIRA-Ev evaluates clinical NLP on evidence detection and relational reasoning in Spanish MIR exam cases, with three tasks: evidence sentence retrieval, argumentative component extraction, and relation classification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19201"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MIRA-Ev","organization_count":1,"organizations":["claire_radar"],"rank":2020,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.19201"},{"aliases":[],"benchmark_id":"claire-radar-2607-07391","caveat":"MIRA-Math evaluates mathematical reasoning where each problem is missing exactly one necessary atomic fact that must be requested in natural language under a strict budget, then integrated into an exact answer. It contains 2,310 instances across 22 typed mathematical families.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.07391"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"MIRA-Math","organization_count":1,"organizations":["claire_radar"],"rank":2021,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.07391"},{"aliases":[],"benchmark_id":"claire-radar-2605-29360","caveat":"MiraBench evaluates action-conditioned reliability in robotic world models through three levels: Physics Adherence, Action-Following Fidelity, and Optimism Bias Detection, using a human-annotated corpus of over 16,000 judgments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29360"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"MiraBench","organization_count":1,"organizations":["claire_radar"],"rank":2022,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29360"},{"aliases":[],"benchmark_id":"opencompass-1844-miracl-vision","caveat":"MIRACL-VISION is a large-scale, multilingual visual document retrieval benchmark built by the NVIDIA team, extending the popular MIRACL multilingual text retrieval benchmark. It covers 18 languages and contains 211 original questions. MIRACL-VISION是一个大规模的多语言视觉文档检索基准测试，由NVIDIA团队构建，扩展了流行的MIRACL多语言文本检索基准。该数据集覆盖18种语言，包含211个原创问题，涵盖边界层分析、WKB方法、非线性偏微分方程的渐近解和振荡积分的渐近性等核心主题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MIRACL-VISION"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MIRACL-VISION","organization_count":1,"organizations":["opencompass_hub"],"rank":2023,"released":"2025-05-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MIRACL-VISION"},{"aliases":[],"benchmark_id":"opencompass-1142-mirage","caveat":"MIRAGE is a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets. MIRAGE 是首个此类基准，包含来自五个医学问答数据集的 7,663 个问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MIRAGE"],"document_share":0.0002723311546840959,"domain":"其他","name":"MIRAGE","organization_count":1,"organizations":["opencompass_hub"],"rank":2024,"released":"2024-08-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MIRAGE"},{"aliases":[],"benchmark_id":"claire-radar-2608-04570","caveat":"Evaluates LLM personalization faithfulness: 150 personas, 6 tasks, four-way faithfulness taxonomy, 12 models across 7 families, 143,616 judged claims.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04570"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MirageBench","organization_count":1,"organizations":["claire_radar"],"rank":2025,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04570"},{"aliases":[],"benchmark_id":"claire-radar-2606-30182","caveat":"MirrorCode evaluates AI agents on reimplementing entire software projects from behavior only, matching outputs on end-to-end tests across 25 programs spanning Unix utilities, data serialization, bioinformatics, interpreters, static analysis, cryptography, and compression.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30182"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"MirrorCode","organization_count":1,"organizations":["claire_radar"],"rank":2026,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30182"},{"aliases":[],"benchmark_id":"claire-radar-2607-29218","caveat":"MirrorCraft is a paired benchmark for evaluating LLM-based agents under hidden rule changes in Minecraft. Each Mirror world copies a Vanilla world with modified server-side rules, while terrain and objectives match. It includes five biomes, six rule suites, three objectives, and uses deterministic milestones and Rule Intervention Effect.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29218"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MirrorCraft","organization_count":1,"organizations":["claire_radar"],"rank":2027,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29218"},{"aliases":[],"benchmark_id":"claire-radar-2609-33893","caveat":"Large audio-language models (LALMs) produce fluent responses about audio but often hallucinate by making plausible yet ungrounded claims.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33893"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MISHAP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2028,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33893"},{"aliases":[],"benchmark_id":"claire-radar-2607-18673","caveat":"MissingBench-Verified evaluates vision-language models on detecting missing object parts in images, covering ten leading models with consistent failure rates and probing mitigation strategies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18673"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MissingBench-Verified","organization_count":1,"organizations":["claire_radar"],"rank":2029,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.18673"},{"aliases":[],"benchmark_id":"claire-radar-2607-22014","caveat":"A benchmark for mission-level evaluation of MLLMs in aerial 3D environments, consisting of 120 missions across five simulated environments and four task families.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22014"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"MissionBench","organization_count":1,"organizations":["claire_radar"],"rank":2030,"released":"2026-07-24","source":"claire_radar","url":"https://arxiv.org/abs/2607.22014"},{"aliases":[],"benchmark_id":"claire-radar-2610-12823","caveat":"Web agents increasingly browse, interpret, and act on web content on behalf of users.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12823"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MisVisAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":2031,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.12823"},{"aliases":[],"benchmark_id":"opencompass-1623-mj-bench","caveat":"MJ-Bench incorporates a comprehensive preference dataset to evaluate multimodal judges in providing feedback for image generation models across four key perspectives: alignment, safety, image quality, and bias. MJ-Bench，它包含了一个综合的偏好数据集，用于从四个关键角度评估多模态评委在为图像生成模型提供反馈方面的能力：对齐、安全性、图像质量和偏见。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MJ-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MJ-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2032,"released":"2024-07-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MJ-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2606-26458","caveat":"MKG-RAG-Bench is a benchmark for retrieval in multimodal knowledge graph-augmented generation, constructed from two knowledge graphs (general and medical) with QA datasets supporting controlled evaluation of retrieval and generation. It uses an LLM-based curation pipeline for structurally grounded queries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26458"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"MKG-RAG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2033,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.26458"},{"aliases":[],"benchmark_id":"opencompass-1099-mkqa","caveat":"MKQA is an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). MKQA 是一个开放域问答评估集，包含 10,000 对问题和答案，涵盖 26 种类型多样的语言（总计 260,000 对问题和答案）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MKQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"MKQA","organization_count":1,"organizations":["opencompass_hub"],"rank":2034,"released":"2021-08-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MKQA"},{"aliases":[],"benchmark_id":"artificial-analysis-mlcr-aa","caveat":"MLCR-AA overall score","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/mlcr-aa"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MLCR-AA","organization_count":1,"organizations":["artificial_analysis"],"rank":2035,"released":null,"source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/mlcr-aa"},{"aliases":[],"benchmark_id":"llm-stats-mle-bench","caveat":"MLE-Bench evaluates AI agents on machine learning engineering tasks by measuring their performance on Kaggle competitions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mle-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"MLE-Bench","organization_count":1,"organizations":["llm_stats"],"rank":2036,"released":"2024-10-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mle-bench-lite","caveat":"MLE-Bench Lite evaluates AI agents on machine learning engineering tasks, testing their ability to build, train, and optimize ML models for Kaggle-style competitions in a lightweight evaluation format.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mle-bench-lite?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"MLE-Bench Lite","organization_count":1,"organizations":["llm_stats"],"rank":2037,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mle-bench-lite?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-04927","caveat":"Agentic machine learning engineering (MLE) is an emerging AI4SE application for complex ML tasks and a step toward recursive self-improvement of AI systems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.04927"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"MLE-InsightBench","organization_count":1,"organizations":["claire_radar"],"rank":2038,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.04927"},{"aliases":[],"benchmark_id":"claire-radar-2606-07706","caveat":"MLingualFC evaluates jailbreak vulnerabilities in multilingual vision-language models using flowchart images encoding harmful instructions in five languages, measuring attack success rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07706"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MLingualFC","organization_count":1,"organizations":["claire_radar"],"rank":2039,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07706"},{"aliases":[],"benchmark_id":"opencompass-1779-mlrc-bench","caveat":"MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions. MLRC-Bench旨在量化大模型代理如何有效地应对具有挑战性的机器学习 （ML） 研究竞赛。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MLRC-Bench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"MLRC-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2040,"released":"2025-04-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MLRC-Bench"},{"aliases":[],"benchmark_id":"llm-stats-mls-bench-lite","caveat":"MLS-Bench Lite is the official 30-task subset of MLS-Bench for evaluating whether AI systems can invent generalizable and scalable machine learning methods across LLM pretraining and post-training, robotics, world models, computer vision, reinforcement learning, optimization, ML systems, and AI for Science.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mls-bench-lite?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MLS-Bench Lite","organization_count":1,"organizations":["llm_stats"],"rank":2041,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mls-bench-lite?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-12809","caveat":"MLUBench evaluates multimodal large language models on lifelong unlearning across 127 entities and 9 classes, providing QA pairs and images. It includes a protocol for sequential unlearning requests and evaluation of forgetting and retention.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12809"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MLUBench","organization_count":1,"organizations":["claire_radar"],"rank":2042,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.12809"},{"aliases":[],"benchmark_id":"llm-stats-mlvu","caveat":"A comprehensive benchmark for multi-task long video understanding that evaluates multimodal large language models on videos ranging from 3 minutes to 2 hours across 9 distinct tasks including reasoning, captioning, recognition, and summarization.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mlvu?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MLVU","organization_count":1,"organizations":["llm_stats"],"rank":2043,"released":"2024-06-06","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1512-mlvu","caveat":"MLVU test MLLMs' understanding of multi-task long videos, encompassing diverse tasks based on various long videos. MLVU用于评估多模态大模型的长视频理解能力，包含面向各种类型长视频的多样化的评估任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MLVU"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MLVU","organization_count":1,"organizations":["opencompass_hub"],"rank":2044,"released":"2024-06-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MLVU"},{"aliases":[],"benchmark_id":"llm-stats-mlvu-m","caveat":"MLVU-M benchmark","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mlvu-m?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"MLVU-M","organization_count":1,"organizations":["llm_stats"],"rank":2045,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mlvu-m?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mm-if-eval","caveat":"A challenging multimodal instruction-following benchmark that includes both compose-level constraints for output responses and perception-level constraints tied to input images, with comprehensive evaluation pipeline.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-if-eval?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MM IF-Eval","organization_count":1,"organizations":["llm_stats"],"rank":2046,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-if-eval?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1558-mm-alignbench","caveat":"A benchmark for evaluating MLLMs' alignment with human preferences. It includes 252 high-quality, human-annotated samples with diverse image types and open-ended questions. Modeled after Arena-style benchmarks, it uses GPT-4o as the judge model and Claude-Sonnet-3 as the reference model. 用于评估 MLLM 与人类偏好的一致性的基准。它包含 252 个高质量、人类标注的样本，具有不同的图像类型和开放式问题。它仿照 Arena 风格的基准，使用 GPT-4o 作为评判模型，Claude-Sonnet-3 作为参考模型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MM-AlignBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MM-AlignBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2047,"released":"2025-02-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MM-AlignBench"},{"aliases":[],"benchmark_id":"llm-stats-mm-browsercomp","caveat":"MM-BrowserComp evaluates multimodal agents on web browsing and information retrieval tasks, testing a model's ability to perceive, navigate, and extract information from real web environments.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-browsercomp?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MM-BrowserComp","organization_count":1,"organizations":["llm_stats"],"rank":2048,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-browsercomp?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mm-clawbench","caveat":"MM-ClawBench evaluates models on MiniMax's Claw-style agent benchmark, measuring practical agentic task completion quality in real-world OpenClaw usage scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-clawbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"MM-ClawBench","organization_count":1,"organizations":["llm_stats"],"rank":2049,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-clawbench?top_n=500"},{"aliases":["MM-ContextASR-Bench"],"benchmark_id":"claire-radar-huggingface-lilonghao-mm-contextasr-bench","caveat":"MM-ContextASR Bench evaluates multimodal contextual ASR using controlled dialogue histories, external corpus tracks, and metrics including CER, SER, and entity Recall.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/lilonghao/MM-ContextASR-Bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MM-ContextASR Bench","organization_count":1,"organizations":["claire_radar"],"rank":2050,"released":"2026-09-15","source":"claire_radar","url":"https://huggingface.co/datasets/lilonghao/MM-ContextASR-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2605-26396","caveat":"MM-CreativityBench evaluates large multimodal models on affordance-grounded creative tool use, requiring scene inspection, entity/part selection, and physically feasible solutions in visually rich environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26396"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MM-CreativityBench","organization_count":1,"organizations":["claire_radar"],"rank":2051,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26396"},{"aliases":[],"benchmark_id":"claire-radar-2609-38523","caveat":"Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures, market expectations, and subtle communication signals.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38523"],"document_share":0.0002723311546840959,"domain":"Finance","name":"MM-FinEval","organization_count":1,"organizations":["claire_radar"],"rank":2052,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.38523"},{"aliases":[],"benchmark_id":"claire-radar-2609-04859","caveat":"MM-IFEval-Pro evaluates vision-language models on instruction-following across Chinese and English, covering four major task categories and 24 subcategories with an average of 3.0 constraints per sample, including adversarial instruction hijacking scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04859"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MM-IFEval-Pro","organization_count":1,"organizations":["claire_radar"],"rank":2053,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.04859"},{"aliases":[],"benchmark_id":"opencompass-1566-mm-iq","caveat":"MM-IQ is a comprehensive evaluation framework comprising 2,710 meticulously curated test items spanning 8 distinct reasoning paradigms. MM-IQ，这是一个包含 2,710 个精心挑选的测试项目的综合评估框架，涵盖了 8 种不同的推理范式。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MM-IQ"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MM-IQ","organization_count":1,"organizations":["opencompass_hub"],"rank":2054,"released":"2025-02-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MM-IQ"},{"aliases":[],"benchmark_id":"llm-stats-mm-mind2web","caveat":"A multimodal web navigation benchmark comprising 2,000 open-ended tasks spanning 137 websites across 31 domains. Each task includes HTML documents paired with webpage screenshots, action sequences, and complex web interactions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-mind2web?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MM-Mind2Web","organization_count":1,"organizations":["llm_stats"],"rank":2055,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mind2web?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mm-mt-bench","caveat":"A multi-turn LLM-as-a-judge evaluation benchmark for testing multimodal instruction-tuned models' ability to follow user instructions in multi-turn dialogues and answer open-ended questions in a zero-shot manner.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mm-mt-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MM-MT-Bench","organization_count":1,"organizations":["llm_stats"],"rank":2056,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mm-mt-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1539-mm-rlhf","caveat":"MM-RLHF, a comprehensive project for aligning Multimodal Large Language Models (MLLMs) with human preferences. The dataset and algorithms enable consistent performance improvements across 10 dimensions and 27 benchmarks for open-source MLLMs. MM-RLHF，这是一个将多模态大型语言模型（MLLMs）与人类偏好对齐的全面项目，使开源多语言机器学习模型在 10 个维度和 27 个基准测试中实现持续的性能提升。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MM-RLHF"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MM-RLHF","organization_count":1,"organizations":["opencompass_hub"],"rank":2057,"released":"2025-02-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MM-RLHF"},{"aliases":[],"benchmark_id":"claire-radar-2606-00622","caveat":"MM-Snowball is a benchmark for diagnosing hallucination snowballing in multimodal multi-turn dialogue, with fine-grained analysis. It includes data and code via a project page.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00622"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"MM-Snowball","organization_count":1,"organizations":["claire_radar"],"rank":2058,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00622"},{"aliases":["MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents"],"benchmark_id":"claire-radar-2607-11818","caveat":"MM-ToolSandBox is a stateful sandbox benchmark for visually grounded tool-calling agents. It has 258 human-verified scenarios across 500+ tools in 16 domains, and models are scored by task success rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11818"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MM-ToolSandBox","organization_count":1,"organizations":["claire_radar"],"rank":2059,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11818"},{"aliases":[],"benchmark_id":"opencompass-1356-mm-vet","caveat":"MM-Vet evaluates the capabilities to deal with complicated multimodal tasks, defining 6 core VL capabilities and examining the 16 integrations of interest derived from the capability combination. MM-Vet用于评估复杂多模态任务能力，涵盖了6个核心视觉语言功能的16种功能组合。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MM-Vet"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MM-Vet","organization_count":1,"organizations":["opencompass_hub"],"rank":2060,"released":"2023-08-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MM-Vet"},{"aliases":[],"benchmark_id":"opencompass-1585-mmad","caveat":"MMAD is the first-ever full-spectrum MLLMs benchmark in industrial Anomaly Detection. \nResearchers defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images. MMAD是第一个工业异常检测领域的全谱 MLLMs 基准，研究人员定义了工业检测中 MLLMs 的七个关键子任务，并设计了一个新颖的流程来生成包含 39,672 个问题以及 8,366 个工业图像的 MMAD 数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMAD"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMAD","organization_count":1,"organizations":["opencompass_hub"],"rank":2061,"released":"2024-10-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMAD"},{"aliases":[],"benchmark_id":"claire-radar-2606-07229","caveat":"MMAE is a benchmark for instruction-based audio editing with 2,000 samples across 7 modalities, 6 complexity levels, and rubric-based evaluation with 17,741 criteria for instruction following and context consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07229"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMAE","organization_count":1,"organizations":["claire_radar"],"rank":2062,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07229"},{"aliases":[],"benchmark_id":"opencompass-1910-mmar","caveat":"We introduce MMAR, a new benchmark comprising 1,000 meticulously curated audio-question-answer triplets, designed to evaluate the deep reasoning capabilities of Audio-Language Models (ALMs) across massive multi-disciplinary tasks. MMAR是一个全新评测基准，旨在评估音频-语言模型（ALMs）的深度推理能力。该基准包含1,000个精心构建的音频与问答，并经过多轮纠错与质量校验以确保高标准。与现有局限于特定声音、音乐或语音领域的评测体系不同，MMAR覆盖现实场景中的混合模态，并采用四级分层分类体系（信号层、感知层、语义层与文化层）。该基准中的每个题目都需要超越表层理解的多层次深度推理，部分问题更要求研究生级别的专业领域知识与感知能力。我们在MMAR上评估了多类模型，测试结果表明该基准具有显著挑战性，分析结果进一步揭示了当前模型在理解与推理能力上的关键局限。我们期待MMAR能推动这个重要但尚未充分探索的研究领域的发展。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMAR"],"document_share":0.0002723311546840959,"domain":"强推理","name":"MMAR","organization_count":1,"organizations":["opencompass_hub"],"rank":2063,"released":"2025-05-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMAR"},{"aliases":[],"benchmark_id":"claire-radar-2608-09281","caveat":"MMArch is a benchmark for multimodal reasoning in architecture and civil engineering, spanning ten subdomains and built from figures in peer-reviewed papers. It contains 1,212 short-answer items requiring perception, principle identification, and application.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09281"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMArch","organization_count":1,"organizations":["claire_radar"],"rank":2064,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09281"},{"aliases":[],"benchmark_id":"llm-stats-mmau","caveat":"A massive multi-task audio understanding and reasoning benchmark comprising 10,000 carefully curated audio clips paired with human-annotated natural language questions spanning speech, environmental sounds, and music. Requires expert-level knowledge and complex reasoning across 27 distinct skills.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmau?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMAU","organization_count":1,"organizations":["llm_stats"],"rank":2065,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmau-music","caveat":"A subset of the MMAU benchmark focused specifically on music understanding and reasoning tasks. Part of a comprehensive multimodal audio understanding benchmark that evaluates models on expert-level knowledge and complex reasoning across music audio clips.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmau-music?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMAU Music","organization_count":1,"organizations":["llm_stats"],"rank":2066,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-music?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmau-sound","caveat":"A subset of the MMAU benchmark focused specifically on environmental sound understanding and reasoning tasks. Part of a comprehensive multimodal audio understanding benchmark that evaluates models on expert-level knowledge and complex reasoning across environmental sound clips.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmau-sound?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMAU Sound","organization_count":1,"organizations":["llm_stats"],"rank":2067,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-sound?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmau-speech","caveat":"A subset of the MMAU benchmark focused specifically on speech understanding and reasoning tasks. Part of a comprehensive multimodal audio understanding benchmark that evaluates models on expert-level knowledge and complex reasoning across speech audio clips.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmau-speech?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMAU Speech","organization_count":1,"organizations":["llm_stats"],"rank":2068,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmau-speech?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmbc","caveat":"MMBC is a multimodal benchmark for vision-language knowledge and reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmbc?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMBC","organization_count":1,"organizations":["llm_stats"],"rank":2069,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbc?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmbench","caveat":"A bilingual benchmark for assessing multi-modal capabilities of vision-language models through multiple-choice questions in both English and Chinese, providing systematic evaluation across diverse vision-language tasks with robust metrics.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMBench","organization_count":1,"organizations":["llm_stats"],"rank":2070,"released":"2023-07-12","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1206-mmbench","caveat":"MMBench is a collection of benchmarks to evaluate the multi-modal understanding capability of large vision language models (LVLMs). This benchmark contains 3,000 multiple-choice questions covering 20 fine-grained assessment dimensions. MMBench是OpenCompass 研究团队自建的视觉语言模型评测数据集，可实现从感知到认知能力逐级细分评估。此评测基准包含3000 道单项选择题 ，覆盖 20个细粒度评估维度。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2071,"released":"2023-07-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-01813","caveat":"MMBench-Live is a continuously evolving multimodal benchmark built by a multi-agent pipeline from MMBench. It contains 5.9K newly generated evaluation instances with executable reasoning, and evaluates vision-language models across question-answer generation and reasoning tasks. Updates cost about USD 30 and take 1-2 hours, with distribution-consistent strategy to maintain comparability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01813"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MMBench-Live","organization_count":1,"organizations":["claire_radar"],"rank":2072,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.01813"},{"aliases":[],"benchmark_id":"llm-stats-mmbench-v1-1","caveat":"Version 1.1 of MMBench, an improved bilingual benchmark for assessing multi-modal capabilities of vision-language models through multiple-choice questions in both English and Chinese, providing systematic evaluation across diverse vision-language tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmbench-v1.1?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMBench-V1.1","organization_count":1,"organizations":["llm_stats"],"rank":2073,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-v1.1?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmbench-video","caveat":"A long-form multi-shot benchmark for holistic video understanding that incorporates approximately 600 web videos from YouTube spanning 16 major categories, with each video ranging from 30 seconds to 6 minutes. Includes roughly 2,000 original question-answer pairs covering 26 fine-grained capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmbench-video?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMBench-Video","organization_count":1,"organizations":["llm_stats"],"rank":2074,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmbench-video?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1207-mmbench-video","caveat":"MMBench-Video is a comprehensive video understanding evaluation benchmark that covers long videos, multiple shots, and evaluates the temporal understanding ability of MLLMs. Contains over 600 videos, 16 categories, and manually annotated Q&A pairs. MMBench-Video是全面视频理解评测基准，覆盖长视频、多镜头，评估MLLMs时序理解能力。包含16类共600+视频以及人工标注问答对。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMBench-Video"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMBench-Video","organization_count":1,"organizations":["opencompass_hub"],"rank":2075,"released":"2024-06-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMBench-Video"},{"aliases":[],"benchmark_id":"claire-radar-2606-00891","caveat":"MMDG-Bench is a benchmark for multi-modal domain generalization, providing two frameworks (D2M and M2D) and unified protocols across action recognition and face anti-spoofing tasks. Includes code repository.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00891"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMDG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2076,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00891"},{"aliases":[],"benchmark_id":"claire-radar-2608-30903","caveat":"Evaluates multimodal dynamic stance classification in social media parent-reply interactions using 3,482 instances with a seven-label taxonomy, including a structured reasoning subset and challenge factor annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30903"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMDS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2077,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30903"},{"aliases":[],"benchmark_id":"opencompass-1334-mmdu","caveat":"MMDU is intended for evaluating the multi-image multi-turn dialogue capabilities. It comprises 110 high-quality multi-image multi-turn dialogues with more than 1600 questions, each accompanied by detailed long-form answers. MMDU用于评估大型视觉语言模型的多图像多轮对话能力，包含110个高质量的多图像多轮对话，由1600多个附有详细的长篇答案的问题组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMDU"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMDU","organization_count":1,"organizations":["opencompass_hub"],"rank":2078,"released":"2024-06-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMDU"},{"aliases":[],"benchmark_id":"llm-stats-mme","caveat":"A comprehensive evaluation benchmark for Multimodal Large Language Models measuring both perception and cognition abilities across 14 subtasks. Features manually designed instruction-answer pairs to avoid data leakage and provides systematic quantitative assessment of MLLM capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mme?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MME","organization_count":1,"organizations":["llm_stats"],"rank":2079,"released":"2023-06-23","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mme?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1357-mme","caveat":"MME is a comprehensive MLLM evaluation benchmark. It measures both perception and cognition abilities on a total of 14 subtasks. MME是一个全面的多模态大模型评估基准，涵盖14 个考察感知和认知能力的子任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MME"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MME","organization_count":1,"organizations":["opencompass_hub"],"rank":2080,"released":"2023-06-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MME"},{"aliases":[],"benchmark_id":"opencompass-1565-mme-cot","caveat":"MME-CoT is a specialized benchmark evaluating the CoT reasoning performance of LMMs, spanning six domains: math, science, OCR, logic, space-time, and general scenes. MME-CoT，一个专门用于评估 LMMs CoT 推理性能的基准，涵盖六个领域：数学、科学、OCR、逻辑、时空和一般场景。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MME-CoT"],"document_share":0.0002723311546840959,"domain":"强推理","name":"MME-CoT","organization_count":1,"organizations":["opencompass_hub"],"rank":2081,"released":"2025-02-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MME-CoT"},{"aliases":[],"benchmark_id":"llm-stats-mme-realworld","caveat":"A comprehensive evaluation benchmark for Multimodal Large Language Models featuring over 13,366 high-resolution images and 29,429 question-answer pairs across 43 subtasks and 5 real-world scenarios. The largest manually annotated multimodal benchmark to date, designed to test MLLMs on challenging high-resolution real-world scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mme-realworld?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MME-RealWorld","organization_count":1,"organizations":["llm_stats"],"rank":2082,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mme-realworld?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1451-mme-realworld","caveat":"MME-RealWorld evaluates MLLMs' real-world recognition, featuring 13,366 high-resolution images averaging 2,000 × 1,500 pixels. MME-RealWorld用于评估多模态大模型对真实场景的理解能力，包含13366个平均2000*1500像素的高分辨率图像。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MME-RealWorld"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MME-RealWorld","organization_count":1,"organizations":["opencompass_hub"],"rank":2083,"released":"2024-08-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MME-RealWorld"},{"aliases":[],"benchmark_id":"claire-radar-2606-24200","caveat":"MMed-Bench-IR evaluates multilingual medical information retrieval across 6 languages with three tasks: cross-lingual QA retrieval, concept discrimination, and evidence retrieval for RAG.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24200"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MMed-Bench-IR","organization_count":1,"organizations":["claire_radar"],"rank":2084,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24200"},{"aliases":[],"benchmark_id":"claire-radar-2609-11601","caveat":"Multi-sensor gait recognition dataset with visible, infrared, depth, LiDAR, and radar modalities in sequence correspondence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11601"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMGait","organization_count":1,"organizations":["claire_radar"],"rank":2085,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11601"},{"aliases":[],"benchmark_id":"claire-radar-2606-22437","caveat":"MMGist is a curated multimodal benchmark covering seven capability dimensions with 7,262 items, derived from 18 existing benchmarks via filtering pipelines. It evaluates vision-language models across dimensions such as Visual Logic and Expert Knowledge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22437"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMGist","organization_count":1,"organizations":["claire_radar"],"rank":2086,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22437"},{"aliases":[],"benchmark_id":"claire-radar-2607-27895","caveat":"MMHBench is a multimodal benchmark for mental health understanding in long-form videos, comprising 268 videos and 2,184 questions across third-person and first-person settings. It evaluates model reasoning about mental states from multimodal evidence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27895"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMHBench","organization_count":1,"organizations":["claire_radar"],"rank":2087,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27895"},{"aliases":[],"benchmark_id":"opencompass-1523-mmie","caveat":"MMIE, a large-scale knowledge-intensive benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs). MMIE，这是一个大规模知识密集型基准，用于评估大型视觉-语言模型（LVLMs）中的交错多模态理解和生成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMIE"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMIE","organization_count":1,"organizations":["opencompass_hub"],"rank":2088,"released":"2024-10-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMIE"},{"aliases":[],"benchmark_id":"opencompass-1547-mmir","caveat":"A benchmark for evaluating Multimodal Large Language Models (MLLMs) on detecting and reasoning about inconsistencies in layout-rich multimodal content. MMIR features 534 challenging samples across five reasoning-heavy inconsistency categories. 用于评估多模态模型（MLLM）检测和推理布局丰富的多模态内容中的不一致性的基准。MMIR 包含 534 个具有挑战性的样本，涉及五个推理能力较强的不一致类别。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMIR"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMIR","organization_count":1,"organizations":["opencompass_hub"],"rank":2089,"released":"2025-02-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMIR"},{"aliases":[],"benchmark_id":"opencompass-1453-mmiu","caveat":"MMIU test MLLMs' multi-image understanding capabilities, encompassesing 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions. MMIU用于评估多模态大模型的多图理解能力，包含7种类型的多图像关系、52个任务、77K图像和11K精心策划的多选题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMIU"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMIU","organization_count":1,"organizations":["opencompass_hub"],"rank":2090,"released":"2024-08-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMIU"},{"aliases":[],"benchmark_id":"claire-radar-2608-25490","caveat":"Evaluates multimodal jailbreak vulnerabilities by factorizing harmful intent, prompt framing, visual semantics, and instruction carrier.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25490"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMJailBench","organization_count":1,"organizations":["claire_radar"],"rank":2091,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25490"},{"aliases":[],"benchmark_id":"opencompass-1578-mmke-bench","caveat":"MMKE-Bench is a benchmark designed to evaluate the ability of LMMs to edit visual knowledge in real-world scenarios. It includes 2,940 pieces of knowledge and 8,363 images across 33 broad categories, with automatically generated, human-verified evaluation questions. MMKE-Bench是一个旨在评估 LMM 在现实场景中编辑视觉知识能力的基准，包括 33 个广泛类别中的 2,940 条知识和 8,363 张图像，以及自动生成并由人工验证的评估问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMKE-Bench"],"document_share":0.0002723311546840959,"domain":"其他","name":"MMKE-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2092,"released":"2025-02-27","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMKE-Bench"},{"aliases":[],"benchmark_id":"opencompass-1851-mmlongbench","caveat":"MMLongBench is a benchmark that evaluates long-context vision-language models across various tasks, image types, and input lengths, revealing that single-task performance is insufficient for gauging overall vision-language long-context capability. MMLongBench 是一个针对长上下文视觉-语言模型的基准，覆盖多种任务、图像类型和输入长度。评测结果表明，单一任务的表现不足以衡量模型的整体长上下文能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMLongBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMLongBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2093,"released":"2025-05-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMLongBench"},{"aliases":[],"benchmark_id":"llm-stats-mmlongbench-128k","caveat":"MMLongBench-128K evaluates multimodal long-context understanding at a 128K token context length, testing how well vision-language models reason over very long mixed text and image inputs.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MMLongBench-128K","organization_count":1,"organizations":["llm_stats"],"rank":2094,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-128k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmlongbench-doc","caveat":"MMLongBench-Doc evaluates long document understanding capabilities in vision-language models.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-doc?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MMLongBench-Doc","organization_count":1,"organizations":["llm_stats"],"rank":2095,"released":"2024-07-01","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlongbench-doc?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1275-mmlongbench-doc","caveat":"MMLongBench-Doc is a long-context multi-modal benchmark comprising 1,062 expert-annotated questions. MMLONGBENCH-DOC是一个长上下文的多模态基准，由1062个专家注释的问题组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMLongBench-Doc"],"document_share":0.0002723311546840959,"domain":"长文本","name":"MMLongBench-Doc","organization_count":1,"organizations":["opencompass_hub"],"rank":2096,"released":"2024-07-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMLongBench-Doc"},{"aliases":[],"benchmark_id":"claire-radar-2606-14747","caveat":"MMLongEmbed is a benchmark for evaluating multimodal embedding models in long-context scenarios, covering four retrieval tasks across text, document, and video modalities with varying context lengths.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14747"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMLongEmbed","organization_count":1,"organizations":["claire_radar"],"rank":2097,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.14747"},{"aliases":[],"benchmark_id":"llm-stats-mmlu","caveat":"Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"MMLU","organization_count":1,"organizations":["llm_stats"],"rank":2098,"released":"2020-09-07","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-498-mmlu","caveat":"MMLU (Massive Multitask Language Understanding) is a new benchmark designed to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This makes the benchmark more challenging and more similar to how we evaluate humans. The benchmark covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from an elementary level to an advanced professional level, and it tests both world knowledge and problem solving ability. Subjects range from traditional areas, such as mathematics and history, to more specialized areas like law and ethics. The granularity and breadth of the subjects makes the benchmark ideal for identifying a model’s blind spots. MMLU (Massive Multitask Language Understanding) 是一个新的基准测试，旨在通过在零次学习和少次学习的环境中评估模型来测量预训练期间获得的知识。这使得基准测试更具挑战性，且更接近我们评估人类的方式。该基准测试涵盖了STEM、人文学科、社会科学等57个主题。其难度范围从小学级别到专业级别，旨在测试世界知识和解决问题的能力。测试主题范围从传统领域，如数学和历史，到更专业的领域，如法律和伦理学。题目的精细度和广度使该基准测试成为识别模型盲点的理想选择。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMLU"],"document_share":0.0002723311546840959,"domain":"学科","name":"MMLU","organization_count":1,"organizations":["opencompass_hub"],"rank":2099,"released":"2020-09-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMLU"},{"aliases":[],"benchmark_id":"llm-stats-mmlu-cot","caveat":"Chain-of-Thought variant of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. This version uses chain-of-thought prompting to elicit step-by-step reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-%28cot%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"MMLU (CoT)","organization_count":1,"organizations":["llm_stats"],"rank":2100,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-%28cot%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmlu-chat","caveat":"Chat-format variant of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. This version uses conversational prompting format for model evaluation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-chat?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"MMLU Chat","organization_count":1,"organizations":["llm_stats"],"rank":2101,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-chat?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmlu-french","caveat":"French language variant of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. This multilingual version tests model performance in French.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-french?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"MMLU French","organization_count":1,"organizations":["llm_stats"],"rank":2102,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-french?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmlu-base","caveat":"Base version of the Massive Multitask Language Understanding benchmark, evaluating language models across 57 tasks including elementary mathematics, US history, computer science, law, and other professional and academic subjects. Designed to comprehensively measure the breadth and depth of a model's academic and professional understanding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-base?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"MMLU-Base","organization_count":1,"organizations":["llm_stats"],"rank":2103,"released":"2020-09-07","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-base?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmlu-pro","caveat":"A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-pro?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"MMLU-Pro","organization_count":1,"organizations":["llm_stats"],"rank":2104,"released":"2024-06-03","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-pro?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1276-mmlu-pro","caveat":"MMLU-Pro is the extension of MMLU, integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. MMLU-Pro是MMLU的扩展版本，涵盖了更具挑战性、以推理为重点的问题，并将选择集从4个选项扩展到10个选项。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMLU-Pro"],"document_share":0.0002723311546840959,"domain":"学科","name":"MMLU-Pro","organization_count":1,"organizations":["opencompass_hub"],"rank":2105,"released":"2024-06-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMLU-Pro"},{"aliases":[],"benchmark_id":"llm-stats-mmlu-prox","caveat":"Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-prox?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"MMLU-ProX","organization_count":1,"organizations":["llm_stats"],"rank":2106,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-prox?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmlu-redux","caveat":"An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MMLU-Redux","organization_count":1,"organizations":["llm_stats"],"rank":2107,"released":"2024-06-06","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmlu-redux-2-0","caveat":"A curated version of the MMLU benchmark featuring manually re-annotated 5,700 questions across 57 subjects to identify and correct errors in the original dataset. Addresses the 6.49% error rate found in MMLU and provides more reliable evaluation metrics for language models.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux-2.0?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MMLU-redux-2.0","organization_count":1,"organizations":["llm_stats"],"rank":2108,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-redux-2.0?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmlu-stem","caveat":"STEM-focused subset of the Massive Multitask Language Understanding benchmark, evaluating language models on science, technology, engineering, and mathematics topics including physics, chemistry, mathematics, and other technical subjects.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmlu-stem?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MMLU-STEM","organization_count":1,"organizations":["llm_stats"],"rank":2109,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmlu-stem?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmmlu","caveat":"Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmlu?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MMMLU","organization_count":1,"organizations":["llm_stats"],"rank":2110,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmlu?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmmu","caveat":"MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMMU","organization_count":1,"organizations":["llm_stats"],"rank":2111,"released":"2023-11-27","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1248-mmmu","caveat":"MMMU is a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks. MMMU用于评估多模态大模型在复杂多学科任务中的表现，包括从大学考试和教科书中精心收集的11.5K多模态问题，涵盖六个核心学科：艺术与设计、商业、科学、健康与医学、人文与社会科学以及技术与工程。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMMU"],"document_share":0.0002723311546840959,"domain":"知识","name":"MMMU","organization_count":1,"organizations":["opencompass_hub"],"rank":2112,"released":"2023-11-27","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMMU"},{"aliases":[],"benchmark_id":"llm-stats-mmmu-val","caveat":"Validation set of the Massive Multi-discipline Multimodal Understanding and Reasoning benchmark. Features college-level multimodal questions across 6 core disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering) spanning 30 subjects and 183 subfields with diverse image types including charts, diagrams, maps, and tables.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28val%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMMU (val)","organization_count":1,"organizations":["llm_stats"],"rank":2113,"released":"2023-11-27","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28val%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmmu-validation","caveat":"Validation set of the Massive Multi-discipline Multimodal Understanding and Reasoning benchmark. Features college-level multimodal questions across 6 core disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering) spanning 30 subjects and 183 subfields with diverse image types including charts, diagrams, maps, and tables.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28validation%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMMU (validation)","organization_count":1,"organizations":["llm_stats"],"rank":2114,"released":"2023-11-27","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-%28validation%29?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-mmmu-pro","caveat":"Visual reasoning","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/mmmu-pro"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMMU-Pro","organization_count":1,"organizations":["artificial_analysis"],"rank":2115,"released":"2024-09-05","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/mmmu-pro"},{"aliases":[],"benchmark_id":"llm-stats-mmmu-pro","caveat":"A more robust multi-discipline multimodal understanding benchmark that enhances MMMU through a three-step process: filtering text-only answerable questions, augmenting candidate options, and introducing vision-only input settings. Achieves significantly lower model performance (16.8-26.9%) compared to original MMMU, providing more rigorous evaluation that closely mimics real-world scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMMU-Pro","organization_count":1,"organizations":["llm_stats"],"rank":2116,"released":"2024-09-04","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmmu-pro-with-tools","caveat":"MMMU-Pro variant evaluated with tool access enabled.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro-with-tools?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMMU-Pro (with tools)","organization_count":1,"organizations":["llm_stats"],"rank":2117,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmu-pro-with-tools?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmmuval","caveat":"Validation set for MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) benchmark, designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning across Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmmuval?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMMUval","organization_count":1,"organizations":["llm_stats"],"rank":2118,"released":"2023-11-27","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmmuval?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-27637","caveat":"MMOOC evaluates multimodal large language models on out-of-context (OOC) and shifted in-context (Shifted IC) visual question answering. It contains over 41K image-question pairs covering three question formats, eight shift types, and six visual scenarios. Responses are scored for accuracy and refusal rate, with an LLM-as-a-judge metric for reasoning correctness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27637"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMOOC","organization_count":1,"organizations":["claire_radar"],"rank":2119,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27637"},{"aliases":["MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs"],"benchmark_id":"claire-radar-2608-29286","caveat":"Evaluates multimodal large language models on proactive critique across 3,146 instances with a taxonomy of 4 error types and 12 subtypes, measuring detection, diagnosis, and resolution using a hierarchical protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29286"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMPCBench","organization_count":1,"organizations":["claire_radar"],"rank":2120,"released":"2026-08-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.29286"},{"aliases":[],"benchmark_id":"claire-radar-2609-09404","caveat":"A benchmark for evaluating multimodal prompt injection attacks on agentic AI frameworks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09404"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMPIBench","organization_count":1,"organizations":["claire_radar"],"rank":2121,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.09404"},{"aliases":[],"benchmark_id":"claire-radar-2606-07069","caveat":"mmPISA-bench consists of 25 multiple-choice questions from PISA in 43 languages with official and machine translations, used to evaluate LLMs' reasoning across languages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07069"],"document_share":0.0002723311546840959,"domain":"General AI","name":"mmPISA-bench","organization_count":1,"organizations":["claire_radar"],"rank":2122,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07069"},{"aliases":[],"benchmark_id":"claire-radar-2610-05398","caveat":"Autonomous research seeks sustained model improvements through iterative experimentation and feedback.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05398"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMPostTrainBench","organization_count":1,"organizations":["claire_radar"],"rank":2123,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05398"},{"aliases":[],"benchmark_id":"opencompass-1829-mms-vpr","caveat":"MMS-VPR is a large-scale multimodal dataset for street-level place recognition in pedestrian areas. It contains 78,575 images and 2,512 videos from 207 locations in Chengdu, China, with rich metadata and spatial graph structure. MMS-VPR 是一个面向复杂城市步行街区的 多模态街景视觉地点识别大规模数据集，采集自成都约 70,800 平方米的开放式商业街区，覆盖 207 个地点，包含 78,575 张图像和 2,512 段视频，每条数据均带有 GPS 坐标、时间戳和文本元信息。相较以车载视角和西方城市为主的传统数据集，MMS-VPR 更贴近真实、密集、多用途的街道空间，具备丰富的视角、时段和模态多样性。此外，该数据集构建了包含 81 个节点、125 条边的空间图结构，支持结构感知的地点识别方法。数据集还定义了两个子集（Edges 和Points），支持精细化和图结构评估任务，助力多模态与地理空间理解的交叉研究。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMS-VPR"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMS-VPR","organization_count":1,"organizations":["opencompass_hub"],"rank":2124,"released":"2025-05-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMS-VPR"},{"aliases":[],"benchmark_id":"llm-stats-mmsearch","caveat":"MMSearch evaluates multimodal models on search-based retrieval and question answering tasks that require processing both visual and textual information from search results.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmsearch?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMSearch","organization_count":1,"organizations":["llm_stats"],"rank":2125,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1584-mmsearch","caveat":"Logo MMSearch is a multimodal search benchmark crafted to evaluate the potential of LMMs to function as a multimodal AI search engine. This benchmark encompasses a meticulously collected dataset of 300 queries spanning 14 subfields. MMSearch 是一个多模态搜索基准，旨在评估大型语言模型（LMMs）作为多模态 AI 搜索引擎的潜力。该基准包含了一个精心收集的包含 300 个查询的数据集，涵盖 14 个子领域。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMSearch"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMSearch","organization_count":1,"organizations":["opencompass_hub"],"rank":2126,"released":"2024-09-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMSearch"},{"aliases":[],"benchmark_id":"llm-stats-mmsearch-plus","caveat":"MMSearch-Plus is an extended variant of MMSearch with harder multimodal search and retrieval tasks requiring deeper reasoning over visual and textual search results.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmsearch-plus?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMSearch-Plus","organization_count":1,"organizations":["llm_stats"],"rank":2127,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsearch-plus?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-29002","caveat":"MMShopBench is a real-log benchmark for multimodal multi-turn shopping agents. It uses cleaned shopping logs with annotations for purchase intent and mandatory product requirements. Agents must infer requirements from images and dialogue, retrieve candidates, and verify product satisfaction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29002"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MMShopBench","organization_count":1,"organizations":["claire_radar"],"rank":2128,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29002"},{"aliases":[],"benchmark_id":"opencompass-1922-mmsi-bench","caveat":"MMSI-Bench is a novel Visual Question Answering (VQA) benchmark specifically designed to evaluate Multi-image Spatial Intelligence in multimodal large language models (MLLMs). Unlike traditional datasets that focus on spatial reasoning within a single image, MMSI-Bench emphasizes real-world inspired MMSI-Bench 是一个全新的多模态空间智能视觉问答（VQA）基准数据集，专为评估多图像空间推理能力而设计。与专注于单图像关系推理的传统数据集不同，MMSI-Bench 更贴近现实世界，聚焦于需在多张图像之间进行逻辑推理的复杂场景。该数据集由六位三维视觉专家耗时超过300小时构建，精心整理出包含1,000个高质量选择题的问题集，题目来自12万余张图像，并附有精心设计的误导选项与逐步推理过程。对34个主流开源和闭源多模态大语言模型的实证评估显示：最先进的模型准确率仅为30%至40%，而人类表现高达97%，揭示该任务的巨大挑战性与模型发展空间。此外，MMSI-Bench 配套提供自动化错误分析","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMSI-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMSI-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2129,"released":"2025-05-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMSI-Bench"},{"aliases":[],"benchmark_id":"llm-stats-mmsibench","caveat":"MMSIBench is a multimodal spatial-intelligence benchmark evaluating spatial reasoning over images.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmsibench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMSIBench","organization_count":1,"organizations":["llm_stats"],"rank":2130,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmsibench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmstar","caveat":"MMStar is an elite vision-indispensable multimodal benchmark comprising 1,500 challenge samples meticulously selected by humans to evaluate 6 core capabilities and 18 detailed axes. The benchmark addresses issues of visual content unnecessity and unintentional data leakage in existing multimodal evaluations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmstar?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMStar","organization_count":1,"organizations":["llm_stats"],"rank":2131,"released":"2024-04-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmstar?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1175-mmstar","caveat":"MMStar is an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans. MMStar benchmarks 6 core capabilities and 18 detailed axes, aiming to evaluate LVLMs’ multi-modal capacities with carefully balanced and purified samples. MMStar 是一个多模态基准，包含 1,500 个经过人工精心挑选的样本。MMStar 评估 6 项核心能力和 18 个详细维度，旨在通过精心平衡和净化的样本，评估 LVLM 的多模态能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMStar"],"document_share":0.0002723311546840959,"domain":"其他","name":"MMStar","organization_count":1,"organizations":["opencompass_hub"],"rank":2132,"released":"2024-04-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMStar"},{"aliases":[],"benchmark_id":"llm-stats-mmt-bench","caveat":"MMT-Bench is a comprehensive multimodal benchmark for evaluating Large Vision-Language Models towards multitask AGI. It comprises 31,325 meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering 32 core meta-tasks and 162 subtasks in multimodal understanding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmt-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMT-Bench","organization_count":1,"organizations":["llm_stats"],"rank":2133,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmt-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1364-mmt-bench","caveat":"MMT-Bench is designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. It comprises 31,325 multi-choice visual questions, covering 32 core meta-tasks and 162 subtasks in multimodal understanding. MMT-Bench考察多模态大模型的视觉识别、定位、推理和规划能力，包括31325个多选视觉问题，涵盖了32个核心元任务和162个多模态理解子任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMT-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MMT-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2134,"released":"2024-04-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMT-Bench"},{"aliases":[],"benchmark_id":"opencompass-1736-mmtb","caveat":"Large language models (LLMs) demonstrate strong potential as agents for tool invocation due to their advanced comprehension and planning capabilities. Users increasingly rely on LLM-based agents to solve complex missions through iterative interactions. However, existing benchmarks predominantly acce 大型语言模型（LLM）凭借其先进的理解能力和规划能力，在作为工具调用的智能体方面展现出了巨大的潜力。用户越来越依赖基于大型语言模型的智能体，通过迭代交互来解决复杂任务。 然而，现有的基准测试主要是在单任务场景中评估智能体，无法体现现实世界的复杂性。为了填补这一空白，我们提出了 Multi-Mission Tool Bench 测试。在这个基准测试中，每个测试用例都包含多个相互关联的任务。这种设计要求智能体能够动态适应不断变化的需求。此外，所提出的基准测试探索了在固定任务数量下所有可能的任务切换模式。具体而言，我们提出了一个多智能体数据生成框架来构建这个基准测试。我们还提出了一种利用动态决策树来","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MMTB"],"document_share":0.0002723311546840959,"domain":"智能体","name":"MMTB","organization_count":1,"organizations":["opencompass_hub"],"rank":2135,"released":"2025-03-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MMTB"},{"aliases":[],"benchmark_id":"claire-radar-2609-04842","caveat":"MMTClinic evaluates large language models on clinical time-series question answering and reasoning across five languages and three tasks: mortality prediction, heart rate forecasting, and SOFA score estimation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04842"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MMTClinic","organization_count":1,"organizations":["claire_radar"],"rank":2136,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.04842"},{"aliases":[],"benchmark_id":"llm-stats-mmvet","caveat":"MM-Vet is an evaluation benchmark that examines large multimodal models on complicated multimodal tasks requiring integrated capabilities. It assesses six core vision-language capabilities: recognition, knowledge, spatial awareness, language generation, OCR, and math through questions that require one or more of these capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmvet?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MMVet","organization_count":1,"organizations":["llm_stats"],"rank":2137,"released":"2023-08-04","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvet?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmvetgpt4turbo","caveat":"MM-Vet evaluation using GPT-4 Turbo for scoring. This variant of MM-Vet examines large multimodal models on complicated multimodal tasks requiring integrated capabilities across six core vision-language abilities: recognition, knowledge, spatial awareness, language generation, OCR, and math.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmvetgpt4turbo?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MMVetGPT4Turbo","organization_count":1,"organizations":["llm_stats"],"rank":2138,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvetgpt4turbo?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mmvu","caveat":"MMVU (Multimodal Multi-disciplinary Video Understanding) is a benchmark for evaluating multimodal models on video understanding tasks across multiple disciplines, testing comprehension and reasoning capabilities on video content.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mmvu?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MMVU","organization_count":1,"organizations":["llm_stats"],"rank":2139,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mmvu?top_n=500"},{"aliases":["MMVU"],"benchmark_id":"mmvu","caveat":"Video understanding benchmark. For non-video-native models, uses 1 fps frame extraction. Scores depend on frame sampling strategy and context length.","document_count":1,"document_ids":["model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0002723311546840959,"domain":"vision","name":"MMVU","organization_count":1,"organizations":["Z.ai"],"rank":2140,"released":"2025-06-01","source":"model_reports","url":"https://github.com/MMVU/MMVU"},{"aliases":["Can Language Models Understand mmWave Data? Benchmarking Large Language Models for mmWave Radar-Based Human Understanding"],"benchmark_id":"claire-radar-2608-14179","caveat":"mmWave-QA is a benchmark for language-conditioned mmWave human perception, aggregating heterogeneous public datasets into six scenarios and five QA tasks for standardized evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14179"],"document_share":0.0002723311546840959,"domain":"General AI","name":"mmWave-QA","organization_count":1,"organizations":["claire_radar"],"rank":2141,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14179"},{"aliases":[],"benchmark_id":"claire-radar-2608-31022","caveat":"Evaluates agents' perceptual state construction in partially observable environments through a sequential glimpse-based MNIST digit search with lookback constraints, scored on recognition accuracy across memory representations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.31022"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MNIST-PRO","organization_count":1,"organizations":["claire_radar"],"rank":2142,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.31022"},{"aliases":[],"benchmark_id":"claire-radar-2609-23980","caveat":"MobileCybench is a benchmark for vulnerability discovery by AI agents in 13 Android applications.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23980"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MobileCybench","organization_count":1,"organizations":["claire_radar"],"rank":2143,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.23980"},{"aliases":["Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation"],"benchmark_id":"claire-radar-2607-28645","caveat":"A benchmark that tests whether multimodal LLMs can turn multi-screen mobile app designs into buildable projects with working navigation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28645"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MobileForge","organization_count":1,"organizations":["claire_radar"],"rank":2144,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.28645"},{"aliases":[],"benchmark_id":"claire-radar-2608-11434","caveat":"MobileJudgeBench evaluates LLM-as-judge methods on mobile agent trajectories. It includes 931 human-annotated trajectories from 6 mobile agent benchmarks, covering 4 agent models and 68 apps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11434"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MobileJudgeBench","organization_count":1,"organizations":["claire_radar"],"rank":2145,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.11434"},{"aliases":[],"benchmark_id":"llm-stats-mobileminiwob-sr","caveat":"MobileMiniWob++ SR (Success Rate) is an adaptation of the MiniWob++ web interaction benchmark for mobile Android environments within AndroidWorld. It comprises 92 web interaction tasks adapted for touch-based mobile interfaces, evaluating agents' ability to navigate and interact with web applications on mobile devices.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mobileminiwob%2B%2B-sr?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MobileMiniWob++_SR","organization_count":1,"organizations":["llm_stats"],"rank":2146,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileminiwob%2B%2B-sr?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-23035","caveat":"Evaluates mobile planning agents on tool-calling and planning across 13 domains and 212 tools in an interactive sandbox, with scoring on three advanced planning dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.23035"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"MobilePA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2147,"released":"2026-08-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.23035"},{"aliases":[],"benchmark_id":"llm-stats-mobileworld","caveat":"MobileWorld is a benchmark for evaluating multimodal agents on real mobile-device tasks, testing GUI grounding, navigation, and multi-step task completion in mobile environments.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mobileworld?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MobileWorld","organization_count":1,"organizations":["llm_stats"],"rank":2148,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mobileworld?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-17659","caveat":"MobileWorldSafety evaluates GUI agents' safety against environmental injection attacks in Android apps. It includes 142 risk tasks on real applications, with programmatically verifiable risk indicators and a two-stage pipeline for verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17659"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MobileWorldSafety","organization_count":1,"organizations":["claire_radar"],"rank":2149,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17659"},{"aliases":[],"benchmark_id":"claire-radar-2610-11064","caveat":"A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11064"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ModeBench","organization_count":1,"organizations":["claire_radar"],"rank":2150,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11064"},{"aliases":[],"benchmark_id":"claire-radar-2609-14857","caveat":"A framework for recursive self-improvement of agent harnesses using benchmark-disjoint evolution tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.14857"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ModularRSI","organization_count":1,"organizations":["claire_radar"],"rank":2151,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.14857"},{"aliases":[],"benchmark_id":"claire-radar-2607-01117","caveat":"MoHallBench evaluates motion hallucination in video LLMs with 11,306 video clips and 40,493 QA pairs, covering three hallucination sources and multiple choice settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01117"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MoHallBench","organization_count":1,"organizations":["claire_radar"],"rank":2152,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.01117"},{"aliases":[],"benchmark_id":"claire-radar-2609-27349","caveat":"A scenario-grounded benchmark for evaluating tool-augmented LLM agents in molecular design tasks, intended to reflect real-world molecular design contexts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27349"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MolDesignBench","organization_count":1,"organizations":["claire_radar"],"rank":2153,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27349"},{"aliases":[],"benchmark_id":"claire-radar-2609-06966","caveat":"MOLE evaluates threat-detection monitors over 150 AI-operated accounts sharing 9 stateful services across 30 workdays, covering 12 insider threat types and 8 model-generated corpora.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06966"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MOLE","organization_count":1,"organizations":["claire_radar"],"rank":2154,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.06966"},{"aliases":[],"benchmark_id":"opencompass-1273-molpuzzle","caveat":"MolPuzzle is a benchmark for evaluating MLM's reasoning ability, comprising 234 instances of structure elucidation, which feature over 18,000 QA samples. MolPuzzle用于考察MLM的推理能力，包含234个结构解析实例以及超过18000个QA样本。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MolPuzzle"],"document_share":0.0002723311546840959,"domain":"推理","name":"MolPuzzle","organization_count":1,"organizations":["opencompass_hub"],"rank":2155,"released":"2024-09-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MolPuzzle"},{"aliases":[],"benchmark_id":"claire-radar-2607-00464","caveat":"MolSafeEval evaluates safety risks in AI-generated molecules across four task types, using structured knowledge graphs and LLM-based reasoning to detect unsafe features.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00464"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MolSafeEval","organization_count":1,"organizations":["claire_radar"],"rank":2156,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.00464"},{"aliases":[],"benchmark_id":"claire-radar-2606-02522","caveat":"Moment-Video evaluates video MLLMs on momentary visual event understanding through 1,000 human-verified video-QA pairs across four task types: temporal occurrence, temporal counting, action description, and temporal reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02522"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Moment-Video","organization_count":1,"organizations":["claire_radar"],"rank":2157,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02522"},{"aliases":[],"benchmark_id":"claire-radar-2606-00832","caveat":"Momento benchmarks persistent agentic task completion in multi-session service environments, requiring agents to resolve temporal dependencies and evolving user goals across sessions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00832"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Momento","organization_count":1,"organizations":["claire_radar"],"rank":2158,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00832"},{"aliases":["Racing in Volume with Flow Ensembles"],"benchmark_id":"claire-radar-2609-16310","caveat":"Photorealistic 4D reconstruction benchmark with Formula 1 sequences captured from sparse external cameras under varied illumination, including trackside, onboard, and drone viewpoints with dense ground truth.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16310"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Monaco4D","organization_count":1,"organizations":["claire_radar"],"rank":2159,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.16310"},{"aliases":[],"benchmark_id":"opencompass-1704-mono2stereo","caveat":"For evaluating stereo image conversion, it provides test data for five different scenarios: animation, indoor, outdoor, complex, and simple, with a total of approximately 2,500 test sample pairs. It also offers evaluation metrics for assessing the stereo effect. 用于测评立体影像转换，提供了动画，室内，室外，复杂，简单共五种场景的测试数据，总共约2500对测试样本。并提供了用于测评立体效果的评价指标-立体交并比。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Mono2Stereo"],"document_share":0.0002723311546840959,"domain":"其他","name":"Mono2Stereo","organization_count":1,"organizations":["opencompass_hub"],"rank":2160,"released":"2025-03-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Mono2Stereo"},{"aliases":[],"benchmark_id":"claire-radar-2607-02956","caveat":"MORE evaluates multilingual document parsing across 149 languages, covering text, formulas, tables, code blocks, catalogs, and reading order, using 1,288 real-world document images with human-refined annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02956"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MORE","organization_count":1,"organizations":["claire_radar"],"rank":2161,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.02956"},{"aliases":[],"benchmark_id":"opencompass-1999-morse-500","caveat":"MORSE-500 introduces 500 programmatically generated videos testing 6 reasoning types: abstract, physical, planning, spatial, temporal, mathematical. Its controllable generation (via Manim, Matplotlib, generative models etc.) enables scalable difficulty, designed to evolve as SOTA models improve. 当前多模态推理基准存在三大不足：依赖静态图像、偏重数学解题、易饱和。为此，我们提出MORSE-500视频基准：包含500个脚本化视频片段，涵盖抽象、物理、规划、空间、时间、数学六类推理问题。其核心在于程序化生成（使用Manim、Matplotlib等），可精确控制视觉复杂度、干扰物密度和时序动态，从而系统化提升难度。与易过时的静态基准不同，MORSE-500具备可持续演进能力，其可控生成流程能无限创建新挑战实例。在顶尖模型（Gemini 2.5 Pro、OpenAI o3等）上的测试揭示了显著性能差距，尤其在抽象和规划任务上。我们开源数据集、生成脚本及评估工具，以促进透明、可复现的前沿研究。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MORSE-500"],"document_share":0.0002723311546840959,"domain":"强推理","name":"MORSE-500","organization_count":1,"organizations":["opencompass_hub"],"rank":2162,"released":"2025-06-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MORSE-500"},{"aliases":[],"benchmark_id":"claire-radar-2606-19416","caveat":"MortarBench evaluates LLM-based agents on mortgage loan origination tasks, covering application, underwriting, approval, and funding. It uses synthetic data with edge-case coverage and exact match accuracy scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19416"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MortarBench","organization_count":1,"organizations":["claire_radar"],"rank":2163,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19416"},{"aliases":["Beyond Global Scalars: Synergizing Token-Level Statistics and Deep Semantics for Adversarial AIGC Text Detection"],"benchmark_id":"claire-radar-2608-28009","caveat":"Evaluates adversarial robustness of machine-generated text detection across 16,000 samples covering token-level to document-level attacks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.28009"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MOSAIC","organization_count":1,"organizations":["claire_radar"],"rank":2164,"released":"2026-08-28","source":"claire_radar","url":"https://arxiv.org/abs/2608.28009"},{"aliases":[],"benchmark_id":"claire-radar-2605-30727","caveat":"MosaicLeaks is a benchmark of 1,001 multi-hop research tasks to evaluate privacy leakage in deep research agents, but no artifacts are provided in this article.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30727"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MosaicLeaks","organization_count":1,"organizations":["claire_radar"],"rank":2165,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.30727"},{"aliases":[],"benchmark_id":"llm-stats-motionbench","caveat":"MotionBench is a benchmark for evaluating multimodal models on motion understanding in videos, testing the ability to comprehend temporal dynamics, movement patterns, and action sequences.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/motionbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MotionBench","organization_count":1,"organizations":["llm_stats"],"rank":2166,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/motionbench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1682-motionbench","caveat":"MotionBench, is a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models. MotionBench是一个综合性的评估基准，旨在评估视频理解模型的细粒度运动理解能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MotionBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MotionBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2167,"released":"2025-01-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MotionBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-09528","caveat":"A contrastive benchmark of self-recorded video pairs testing physically grounded motion understanding in video-LLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09528"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MotionBlind","organization_count":1,"organizations":["claire_radar"],"rank":2168,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.09528"},{"aliases":[],"benchmark_id":"claire-radar-2609-25689","caveat":"MotionForge is described as a simulation benchmark and data-generation pipeline for long-horizon manipulation of dynamic objects under domain shifts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25689"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"MotionForge","organization_count":1,"organizations":["claire_radar"],"rank":2169,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.25689"},{"aliases":[],"benchmark_id":"claire-radar-2606-23061","caveat":"MotionHalluc is a benchmark evaluating kinematic hallucinations in cross-video motion comparison. It includes 1540 questions over 553 video pairs, assessing directional, attributional, and temporal hallucinations in generated instructions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23061"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MotionHalluc","organization_count":1,"organizations":["claire_radar"],"rank":2170,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23061"},{"aliases":[],"benchmark_id":"opencompass-2154-motionmillion","caveat":"MotionMillion: The largest open-sourced 3D human motion dataset with text annotation, including 2.5k hours 1.9M episodes. MotionMillion：目前最大的开源带文本标注的 3D 人体动作数据集，包含 2500 小时、190 万个片段。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MotionMillion"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MotionMillion","organization_count":1,"organizations":["opencompass_hub"],"rank":2171,"released":"2025-07-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MotionMillion"},{"aliases":[],"benchmark_id":"claire-radar-2605-28192","caveat":"MOV-Bench contains 519 questions requiring multi-hop reasoning over temporally dispersed audio-visual evidence, evaluating omni-modal LLMs on cross-modal reasoning tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28192"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MOV-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2172,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28192"},{"aliases":[],"benchmark_id":"claire-radar-2609-15780","caveat":"Probabilistic wildlife movement forecasting benchmark with 2.6M GPS locations and environmental covariates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15780"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MoveBench","organization_count":1,"organizations":["claire_radar"],"rank":2173,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15780"},{"aliases":[],"benchmark_id":"claire-radar-2607-18789","caveat":"Moving Alphabet is a procedural testbed for controlled experiments on text-to-video training data, generating synthetic videos with ground-truth metadata to study data distribution and caption quality effects.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18789"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Moving Alphabet","organization_count":1,"organizations":["claire_radar"],"rank":2174,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.18789"},{"aliases":[],"benchmark_id":"claire-radar-2608-10444","caveat":"MPAR-Bench evaluates multi-point associative reasoning in LLMs across English and Chinese using 1,000 items with diverse clues. Scoring includes exact-match accuracy, ANLS, embedding similarity, and reasoning-trace verification, with four perturbation types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10444"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MPAR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2175,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10444"},{"aliases":[],"benchmark_id":"claire-radar-2606-11416","caveat":"MPC-Patch-Bench evaluates LLM-based code repair on repository-level Secure Multi-Party Computation (MPC) software. It provides 205 verified instances with Fail-to-Pass/Pass-to-Pass tests and a verifier that checks cryptographic safety and numerical fidelity via differential testing and static analysis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11416"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"MPC-Patch-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2176,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11416"},{"aliases":[],"benchmark_id":"claire-radar-2605-22100","caveat":"MPDocBench-Parse is a benchmark for multi-page document parsing, covering 433 manually annotated documents with 3,246 pages across 15 document types in English and Chinese. It evaluates content fidelity and logical structure, including text, table, formula recognition, reading order, and heading hierarchy recovery.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22100"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MPDocBench-Parse","organization_count":1,"organizations":["claire_radar"],"rank":2177,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22100"},{"aliases":[],"benchmark_id":"claire-radar-2607-27616","caveat":"MPIE-Bench evaluates multi-person image editing models on tasks involving multiple named people in contact interactions such as embrace, carry, or grapple. The benchmark provides a 2,500-sample test set with 14 interaction categories and four contact densities, and scores outputs on six axes including anatomy and interaction via mesh reconstruction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27616"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MPIE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2178,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27616"},{"aliases":[],"benchmark_id":"opencompass-930-mr-ben-meta-reasoning-benchmark","caveat":"Welcome to the dataset page for the Meta-Reasoning Benchmark associated with our recent publication `Mr-Ben: A Comprehensive Meta-Reasoning Benchmark for Large Language Models`. We have provided a demo evaluate script for you to try out benchmark in mere two steps. We encourage everyone to try out our benchmark in the SOTA models and return its results to us. We would be happy to include it in the eval_results and update the evaluation tables below for you. 本工作联合MIT,清华,剑桥等知名院校, 提出了一个评测大语言模型对复杂问题的推理过程的“阅卷”批改能力的评测数据集，有别于以前的以结果匹配为评测模式的数据集MR-Ben，我们的数据集基于GSM8K[1], MMLU[2], LogiQA[3], MHPP[4]等数据集经由细致的高水平人工标注构建而成，显著地增加了难度及区分度。我们细致地分析了包括claude3.5, GPT4-Turbo, Kimi, Zhipu, Yi-Large, Qwen2, DeepseekCoderv2 等国内外一线的大语言模型，发现开源的模型在复杂推理的场景下有望追上顶尖的闭源模型。该评测数据集的所有数据均已开源，并且支持一键评测。欢迎所有做大模型训练的小伙伴向我们分享你的评测结果，我们会及时更新榜单。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MR-Ben-Meta-Reasoning-Benchmark"],"document_share":0.0002723311546840959,"domain":"推理","name":"MR-Ben-Meta-Reasoning-Benchmark","organization_count":1,"organizations":["opencompass_hub"],"rank":2179,"released":"2024-06-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MR-Ben-Meta-Reasoning-Benchmark"},{"aliases":[],"benchmark_id":"opencompass-1587-mr-gsm8k","caveat":"MR-GSM8K is a challenging benchmark designed to evaluate the meta-reasoning capabilities of state-of-the-art Large Language Models (LLMs). It goes beyond traditional evaluation metrics by focusing on the reasoning process rather than just the final answer. MR-GSM8K 是一个旨在评估最先进大型语言模型（LLMs）元推理能力的挑战性基准。它超越了传统的评估指标，专注于推理过程而非仅仅关注最终答案，从而对模型的认知能力进行更细致的评估。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MR-GSM8K"],"document_share":0.0002723311546840959,"domain":"推理","name":"MR-GSM8K","organization_count":1,"organizations":["opencompass_hub"],"rank":2180,"released":"2023-12-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MR-GSM8K"},{"aliases":[],"benchmark_id":"claire-radar-2605-24777","caveat":"MR-LiDAR is a multi-resolution LiDAR benchmark for roadside perception. It includes point cloud data and annotations from 16-, 32-, 80-, and 128-beam LiDARs in identical scenarios, covering vehicles and vulnerable road users at various distances.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24777"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MR-LiDAR","organization_count":1,"organizations":["claire_radar"],"rank":2181,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24777"},{"aliases":["Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents"],"benchmark_id":"claire-radar-2609-11318","caveat":"Benchmark of 102 research questions with explicit dependency graphs and step checklists for long-horizon multimodal deep research agents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11318"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Mr.LHDR","organization_count":1,"organizations":["claire_radar"],"rank":2182,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11318"},{"aliases":[],"benchmark_id":"opencompass-1586-mrag-bench","caveat":"MRAG-Bench consists of 16,130 images and 1,353 human-annotated multiple-choice questions across 9 distinct scenarios, providing a robust and systematic evaluation of Large Vision Language Model (LVLM)’s vision-centric multimodal retrieval-augmented generation (RAG) abilities. MRAG-Bench 包含 16,130 张图片和 1,353 个跨越 9 个不同场景的人标注多选题，为大型视觉语言模型（LVLM）的视觉中心多模态检索增强生成（RAG）能力提供了稳健和系统的评估。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MRAG-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MRAG-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2183,"released":"2024-10-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MRAG-Bench"},{"aliases":[],"benchmark_id":"llm-stats-mrcr","caveat":"MRCR (Multi-Round Coreference Resolution) is a synthetic long-context reasoning task where models must navigate long conversations to reproduce specific model outputs. It tests the ability to distinguish between similar requests and reason about ordering while maintaining attention across extended contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR","organization_count":1,"organizations":["llm_stats"],"rank":2184,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-128k-2-needle","caveat":"MRCR (Multi-Round Coreference Resolution) at 128K context length with 2 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 128K-token contexts with 2 items to retrieve.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%282-needle%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR 128K (2-needle)","organization_count":1,"organizations":["llm_stats"],"rank":2185,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%282-needle%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-128k-4-needle","caveat":"MRCR (Multi-Round Coreference Resolution) at 128K context length with 4 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 128K-token contexts with 4 items to retrieve.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%284-needle%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR 128K (4-needle)","organization_count":1,"organizations":["llm_stats"],"rank":2186,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%284-needle%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-128k-8-needle","caveat":"MRCR (Multi-Round Coreference Resolution) at 128K context length with 8 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 128K-token contexts with 8 items to retrieve.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%288-needle%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR 128K (8-needle)","organization_count":1,"organizations":["llm_stats"],"rank":2187,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-128k-%288-needle%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-1m","caveat":"MRCR 1M is a variant of the Multi-Round Coreference Resolution benchmark designed for testing extremely long context capabilities with approximately 1 million tokens. It evaluates models' ability to maintain reasoning and attention across ultra-long conversations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR 1M","organization_count":1,"organizations":["llm_stats"],"rank":2188,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-1m-pointwise","caveat":"MRCR 1M (pointwise) is a variant of the Multi-Round Coreference Resolution benchmark that uses pointwise evaluation for ultra-long contexts (~1M tokens). This version evaluates each response independently rather than comparatively, testing models' absolute performance on long-context reasoning tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m-%28pointwise%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR 1M (pointwise)","organization_count":1,"organizations":["llm_stats"],"rank":2189,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-1m-%28pointwise%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-64k-2-needle","caveat":"MRCR (Multi-Round Coreference Resolution) at 64K context length with 2 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 64K-token contexts with 2 items to retrieve.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%282-needle%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR 64K (2-needle)","organization_count":1,"organizations":["llm_stats"],"rank":2190,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%282-needle%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-64k-4-needle","caveat":"MRCR (Multi-Round Coreference Resolution) at 64K context length with 4 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 64K-token contexts with 4 items to retrieve.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%284-needle%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR 64K (4-needle)","organization_count":1,"organizations":["llm_stats"],"rank":2191,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%284-needle%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-64k-8-needle","caveat":"MRCR (Multi-Round Coreference Resolution) at 64K context length with 8 needles. Models must navigate long conversations to reproduce specific model outputs, testing attention and reasoning across 64K-token contexts with 8 items to retrieve.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%288-needle%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR 64K (8-needle)","organization_count":1,"organizations":["llm_stats"],"rank":2192,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-64k-%288-needle%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-v2","caveat":"MRCR v2 (Multi-Round Coreference Resolution version 2) is an enhanced version of the synthetic long-context reasoning task. It extends the original MRCR framework with improved evaluation criteria and additional complexity for testing models' ability to maintain attention and reasoning across extended contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR v2","organization_count":1,"organizations":["llm_stats"],"rank":2193,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-v2-8-needle","caveat":"MRCR v2 (8-needle) is a variant of the Multi-Round Coreference Resolution benchmark that includes 8 needle items to retrieve from long contexts. This tests models' ability to simultaneously track and reason about multiple pieces of information across extended conversations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-%288-needle%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR v2 (8-needle)","organization_count":1,"organizations":["llm_stats"],"rank":2194,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-%288-needle%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mrcr-v2-8-needle-512k-1m","caveat":"MRCR v2 8-needle variant evaluated on contexts from 512K to 1M tokens.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-8-needle-512k-1m?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MRCR v2 (8-needle, 512K-1M)","organization_count":1,"organizations":["llm_stats"],"rank":2195,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mrcr-v2-8-needle-512k-1m?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-22236","caveat":"Evaluates LALMs on audio degradation perception through multi-turn dialogues over multiple audio inputs, covering speech, music, and sound.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22236v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MRMAD","organization_count":1,"organizations":["claire_radar"],"rank":2196,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22236v1"},{"aliases":[],"benchmark_id":"claire-radar-2608-05196","caveat":"MS-MLB evaluates machine learning models for classifying multiple sclerosis versus healthy controls from whole blood RNA expression data (GSE17048). It uses a shared pipeline with nested cross-validation and a holdout set, and reports the MS Research Score composite metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05196"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"MS-MLB","organization_count":1,"organizations":["claire_radar"],"rank":2197,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.05196"},{"aliases":[],"benchmark_id":"opencompass-1102-ms-marco","caveat":"MS MARCO comprises of 1,010,916 anonymized questions—sampled from Bing’s search query logs—each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages. MS MARCO 数据集包含 1,010,916 个来自 Bing 的搜索查询日志的匿名问题，每个问题都有一个人工生成的答案和 182,669 个完全由人重写的生成答案。此外，该数据集还包含从 3,563,535 个由 Bing 检索的网页文档中提取的 8,841,823 个段落，这些段落提供了策划自然语言答案所需的信息。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MS_MARCO"],"document_share":0.0002723311546840959,"domain":"知识","name":"MS_MARCO","organization_count":1,"organizations":["opencompass_hub"],"rank":2198,"released":"2018-10-31","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MS_MARCO"},{"aliases":["Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution"],"benchmark_id":"claire-radar-2607-13034","caveat":"MSE-Bench measures agent performance and execution redundancy across deterministic editing tasks in a controlled simulator.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.13034"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MSE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2199,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.13034"},{"aliases":["An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding"],"benchmark_id":"claire-radar-2607-27877","caveat":"MSEval evaluates multi-agent software development on authentic full-stack projects using deterministic, requirement-based grading.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27877"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MSEval","organization_count":1,"organizations":["claire_radar"],"rank":2200,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.27877"},{"aliases":[],"benchmark_id":"claire-radar-2609-24812","caveat":"MSI-Bench evaluates multi-speaker voice interaction for collaborative AI agents using 1,152 audio cases with six patterns, scored via LLM judge and deterministic tool validation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24812"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MSI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2201,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24812"},{"aliases":[],"benchmark_id":"claire-radar-2608-22232","caveat":"Evaluates multimodal large language models on discrimination, understanding, and reasoning under situational illusions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.22232"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MSIBench","organization_count":1,"organizations":["claire_radar"],"rank":2202,"released":"2026-08-23","source":"claire_radar","url":"https://arxiv.org/abs/2608.22232"},{"aliases":[],"benchmark_id":"claire-radar-2609-26872","caveat":"A benchmark suite of 22 full-body musculoskeletal motor-control tasks across postural stabilization, locomotion, and contact-rich interaction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.26872"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"MSK-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2203,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.26872"},{"aliases":[],"benchmark_id":"claire-radar-2608-03708","caveat":"MSP-Bench evaluates multi-subject personalized text-to-image generation. It scores identity fidelity, attribute binding accuracy, and attribute misalignment through a dual-pathway protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03708"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MSP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2204,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03708"},{"aliases":[],"benchmark_id":"claire-radar-2607-00724","caveat":"MSQA evaluates 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers, testing cultural knowledge in a multilingual context.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00724"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MSQA","organization_count":1,"organizations":["claire_radar"],"rank":2205,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.00724"},{"aliases":[],"benchmark_id":"llm-stats-msqa","caveat":"MSQA is a multilingual question-answering benchmark that measures knowledge and reasoning across a diverse set of languages.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/msqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MSQA","organization_count":1,"organizations":["llm_stats"],"rank":2206,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/msqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-26525","caveat":"A benchmark for generating three- to five-minute, multi-shot video continuations from an observed frame or clip while maintaining state and narrative progression.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26525"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MSVE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2207,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26525"},{"aliases":[],"benchmark_id":"llm-stats-mt-aime-2025","caveat":"MT-AIME 2025 is Cohere's internal multilingual translation of AIME 2025, evaluated for Arabic, Japanese, and Korean in the Command A+ release.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mt-aime-2025?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"MT-AIME 2025","organization_count":1,"organizations":["llm_stats"],"rank":2208,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-aime-2025?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-mt-bench","caveat":"MT-Bench is a challenging multi-turn benchmark that measures the ability of large language models to engage in coherent, informative, and engaging conversations. It uses strong LLMs as judges for scalable and explainable evaluation of multi-turn dialogue capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mt-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MT-Bench","organization_count":1,"organizations":["llm_stats"],"rank":2209,"released":"2023-06-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mt-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1172-mt-bench-101","caveat":"MT-Bench-101 is specifically designed to evaluate the finegrained abilities of LLMs in multi-turn dialogues. MT-Bench-101 专门设计用于评估 LLMs 在多轮对话中的细粒度能力。通过对真实多轮对话数据的详细分析，构建了一个三层级的能力分类法，涵盖 1388 个多轮对话中的 4208 个轮次，涉及 13 种不同的任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MT-Bench-101"],"document_share":0.0002723311546840959,"domain":"理解","name":"MT-Bench-101","organization_count":1,"organizations":["opencompass_hub"],"rank":2210,"released":"2024-06-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MT-Bench-101"},{"aliases":[],"benchmark_id":"claire-radar-2608-03474","caveat":"MT-Web2Code evaluates coding agents on multi-turn web UI reconstruction and modification tasks across 102 tasks in 16 domains, with a dual-axis protocol measuring target-region fidelity and preservation of unaffected content.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03474"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MT-Web2Code","organization_count":1,"organizations":["claire_radar"],"rank":2211,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03474"},{"aliases":[],"benchmark_id":"claire-radar-2609-14992","caveat":"Benchmark for instruction-following in multi-turn software development with constraint checklists and verification scripts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.14992"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"MTAC-IFBench","organization_count":1,"organizations":["claire_radar"],"rank":2212,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.14992"},{"aliases":[],"benchmark_id":"claire-radar-2605-28035","caveat":"MTAVG-Bench 2.0 evaluates omni large language models on diagnosing high-level cinematic failures in multi-talker audio-video generation, with over 10,000 QA instances covering acting, narrative, atmosphere, and audio-visual language.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28035"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MTAVG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2213,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28035"},{"aliases":[],"benchmark_id":"opencompass-1929-mtcmb","caveat":"We introduce MTCMB-a Multi-Task Benchmark for Evaluating LLMs on TCM Knowledge, Reasoning, and Safety. Developed in collaboration with certified TCM experts, MTCMB comprises 12 sub-datasets spanning five major categories: knowledge QA, language understanding, diagnostic reasoning, formula generation 大语言模型在中医领域的应用日益增多，到底大语言模型在这一古老的学科表现如何？虽然有一些零碎的评测数据集，但大都以单选题、病案分析、粗糙的医患对话为主，很难全面评估大语言模型在中医领域的实际能力。近日，中山大学联合湖南中医药大学等团队推出全球首个中医多任务评测基准（Benchmark） —— MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MTCMB"],"document_share":0.0002723311546840959,"domain":"推理","name":"MTCMB","organization_count":1,"organizations":["opencompass_hub"],"rank":2214,"released":"2025-05-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MTCMB"},{"aliases":[],"benchmark_id":"opencompass-2034-mteb","caveat":"MTEB is a benchmark for evaluating text embedding models, aiming to enhance long-term usability and reproducibility, focusing on task extensibility, data integrity validation, and result generalizability. MTEB 是一个评估文本嵌入模型的基准，旨在提升其长期可用性和可复现性，涵盖任务扩展性、数据完整性验证和结果通用性等维度。该基准包含包括分类、检索和聚类在内的多种任务，覆盖多个语言和领域。MTEB 引入了自动化测试管道、持续集成框架和社区贡献机制，以支持基准的可扩展性和质量控制。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MTEB"],"document_share":0.0002723311546840959,"domain":"其他","name":"MTEB","organization_count":1,"organizations":["opencompass_hub"],"rank":2215,"released":"2025-06-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MTEB"},{"aliases":[],"benchmark_id":"claire-radar-2607-04581","caveat":"MTEB-BR is a text embedding benchmark for Brazilian Portuguese with 22 native tasks across seven categories, including classification, clustering, retrieval, and reranking. It evaluates models on native Portuguese data and provides a public leaderboard.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04581"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MTEB-BR","organization_count":1,"organizations":["claire_radar"],"rank":2216,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.04581"},{"aliases":[],"benchmark_id":"claire-radar-2605-27649","caveat":"MTM-Bench is a controlled benchmark for language-conditioned task execution in multilingual settings, enumerating all 27 instruction-content-response language triplets across English, Spanish, and Chinese. It contains 2,430 instances per model across semantic reversal, final-state extraction, and language purity tasks, with decomposed metrics for semantic correctness, language adherence, constraint satisfaction, and joint success.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27649"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MTM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2217,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27649"},{"aliases":["MTOB","Machine Translation from One Book"],"benchmark_id":"mtob","caveat":"Translation from a single grammar book, reported as half-book and full-book variants that are not interchangeable.","document_count":1,"document_ids":["model_reports:meta_llama_4"],"document_share":0.0002723311546840959,"domain":"long_context","name":"MTOB","organization_count":1,"organizations":["Meta"],"rank":2218,"released":"2023-09-28","source":"model_reports","url":"https://arxiv.org/abs/2309.16575"},{"aliases":[],"benchmark_id":"claire-radar-2608-30241","caveat":"Evaluates multi-turn scientific diagram refinement via 292 images annotated with 3,518 user requirements, using a user simulator that provides iterative feedback and measures requirement satisfaction and diagram quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30241"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MTPaperBananaBench","organization_count":1,"organizations":["claire_radar"],"rank":2219,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30241"},{"aliases":[],"benchmark_id":"claire-radar-2609-20152","caveat":"A benchmark for evaluating language models within cascaded voice agents across multi-turn interactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.20152"],"document_share":0.0002723311546840959,"domain":"Agents","name":"MTVA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2220,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.20152"},{"aliases":[],"benchmark_id":"llm-stats-mtvqa","caveat":"MTVQA (Multilingual Text-Centric Visual Question Answering) is the first benchmark featuring high-quality human expert annotations across 9 diverse languages, consisting of 6,778 question-answer pairs across 2,116 images. It addresses visual-textual misalignment problems in multilingual text-centric VQA.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mtvqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MTVQA","organization_count":1,"organizations":["llm_stats"],"rank":2221,"released":"2024-05-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mtvqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1502-mtvqa","caveat":"MTVQA is designed to evaluate LMMs' multilingual text understanding, featuring high-quality human expert annotations across 9 diverse languages. MTVQA用于评估多模态大模型理解多语言文本的能力，包含来自9种语言的由人类专家注释的高质量数据。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MTVQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MTVQA","organization_count":1,"organizations":["opencompass_hub"],"rank":2222,"released":"2024-06-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MTVQA"},{"aliases":[],"benchmark_id":"claire-radar-2608-29477","caveat":"MUDDLE separates the effects of distractor topical similarity and added context length on document question answering, using 270 questions across five controlled conditions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29477"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MUDDLE","organization_count":1,"organizations":["claire_radar"],"rank":2223,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29477"},{"aliases":[],"benchmark_id":"llm-stats-muirbench","caveat":"A comprehensive benchmark for robust multi-image understanding capabilities of multimodal LLMs. Consists of 12 diverse multi-image tasks involving 10 categories of multi-image relations (e.g., multiview, temporal relations, narrative, complementary). Comprises 11,264 images and 2,600 multiple-choice questions created in a pairwise manner, where each standard instance is paired with an unanswerable variant for reliable assessment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/muirbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MuirBench","organization_count":1,"organizations":["llm_stats"],"rank":2224,"released":"2024-06-13","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/muirbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-23416","caveat":"The proposed benchmark would include 5,038 open-ended questions over in-the-wild audio recordings across 16 languages and eight domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23416"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MuLA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2225,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23416"},{"aliases":[],"benchmark_id":"claire-radar-2607-23870","caveat":"An offline protocol-conditioned benchmark for Vision-Language-Action UAV agents, evaluating operational context understanding, multimodal evidence arbitration, degradation-aware reasoning, and risk-aware action planning across 3,024 samples with semantic scoring and structural diagnostics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23870"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MulRobBench","organization_count":1,"organizations":["claire_radar"],"rank":2226,"released":"2026-07-26","source":"claire_radar","url":"https://arxiv.org/abs/2607.23870"},{"aliases":[],"benchmark_id":"llm-stats-multichallenge","caveat":"MultiChallenge is a realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key categories: instruction retention (maintaining instructions throughout conversations), inference memory (recalling and connecting details from previous turns), reliable versioned editing (adapting to evolving instructions during collaborative editing), and self-coherence (avoiding contradictions in responses). The benchmark evaluates models on sustained, contextually complex dialogues across diverse topics including travel planning, technical documentation, and professional communication.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multichallenge?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Multi-Challenge","organization_count":1,"organizations":["llm_stats"],"rank":2227,"released":"2025-01-29","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/multichallenge?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-multi-if","caveat":"Multi-IF benchmarks LLMs on multi-turn and multilingual instruction following. It expands upon IFEval by incorporating multi-turn sequences and translating English prompts into 7 other languages, resulting in 4,501 multilingual conversations with three turns each. The benchmark reveals that current leading LLMs struggle with maintaining accuracy in multi-turn instructions and shows higher error rates for non-Latin script languages.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multi-if?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Multi-IF","organization_count":1,"organizations":["llm_stats"],"rank":2228,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-if?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-20517","caveat":"Multi-LCB evaluates code generation across twelve programming languages (C++, C#, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, Kotlin, Scala, PHP) by transforming Python tasks from LiveCodeBench while preserving its contamination controls and evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20517"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Multi-LCB","organization_count":1,"organizations":["claire_radar"],"rank":2229,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.20517"},{"aliases":[],"benchmark_id":"claire-radar-2605-29738","caveat":"Multi-Legal-Bench evaluates LLMs on legal reasoning across six countries, four language families, and five tasks, using structured metadata from court registries for classification and extraction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29738"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Multi-Legal-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2230,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29738"},{"aliases":[],"benchmark_id":"claire-radar-github-caitlin-leonard-ct-segmentation-benchmark","caveat":"Compares MOOSE, TotalSegmentator, and VoxTell on public CT datasets using Dice and IoU scores per organ, with a staged design from a 4-organ pilot to 16 organs across up to 50 subjects.","document_count":1,"document_ids":["claire_radar:https://github.com/caitlin-leonard/ct-segmentation-benchmark"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Multi-Organ CT Segmentation Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2231,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/caitlin-leonard/ct-segmentation-benchmark"},{"aliases":[],"benchmark_id":"llm-stats-multi-swe-bench","caveat":"A multilingual benchmark for issue resolving that evaluates Large Language Models' ability to resolve software issues across diverse programming ecosystems. Covers 7 programming languages (Java, TypeScript, JavaScript, Go, Rust, C, and C++) with 1,632 high-quality instances carefully annotated by 68 expert annotators. Addresses limitations of existing benchmarks that focus almost exclusively on Python.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multi-swe-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Multi-SWE-Bench","organization_count":1,"organizations":["llm_stats"],"rank":2232,"released":"2025-04-03","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/multi-swe-bench?top_n=500"},{"aliases":["Multi-SWE-Bench"],"benchmark_id":"multi_swe_bench","caveat":"Multilingual SWE-bench variant spanning 7 programming languages; cross-repository context transfer is the measured capability.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"coding_agent","name":"Multi-SWE-Bench","organization_count":1,"organizations":["ByteDance"],"rank":2233,"released":"2025-04-03","source":"model_reports","url":"https://github.com/multi-swe-bench/multi-swe-bench/"},{"aliases":[],"benchmark_id":"claire-radar-2609-34752","caveat":"Reproduction test generation translates a natural-language issue description into executable tests that fail on the original code and pass after the issue is resolved, providing executable evidence for verifying candidate patches.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34752"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Multi-SWT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2234,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34752"},{"aliases":[],"benchmark_id":"claire-radar-2608-26535","caveat":"Audio-video generation is rapidly moving from prompt-driven synthesis toward multimodal conditioning, where text, images, audio, and video can jointly shape the generated output.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26535"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Multi2AV-Safety","organization_count":1,"organizations":["claire_radar"],"rank":2235,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26535"},{"aliases":["FaceSnap: Real-Time Personalized Lightstage Facial Performance Capture"],"benchmark_id":"claire-radar-2608-31033","caveat":"Evaluates 4D facial reconstruction in lightstage settings using a public dataset that enables topology-invariant geometric comparison across methods.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.31033"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Multi4D","organization_count":1,"organizations":["claire_radar"],"rank":2236,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.31033"},{"aliases":["MultiChallenge"],"benchmark_id":"multi_challenge","caveat":"Multi-turn instruction retention, graded by an LLM judge.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MultiChallenge","organization_count":1,"organizations":["ByteDance"],"rank":2237,"released":"2025-01-29","source":"model_reports","url":"https://arxiv.org/abs/2501.17399"},{"aliases":["MultiChallenge"],"benchmark_id":"multichallenge","caveat":"Multi-turn instruction retention, graded by an LLM judge.","document_count":1,"document_ids":["model_reports:qwen3_5_model_card"],"document_share":0.0002723311546840959,"domain":"instruction_following","name":"MultiChallenge","organization_count":1,"organizations":["Qwen"],"rank":2238,"released":"2025-01-29","source":"model_reports","url":"https://arxiv.org/abs/2501.17399"},{"aliases":[],"benchmark_id":"claire-radar-2609-02379","caveat":"Contains 928 machine-generated books across six languages and three scripts for evaluating long-form LLM-generated text attribution under distribution shifts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02379"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MultiGhostBench","organization_count":1,"organizations":["claire_radar"],"rank":2239,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02379"},{"aliases":[],"benchmark_id":"claire-radar-2608-03882","caveat":"MultiGlobeQA evaluates geospatial reasoning in large language models across 46,060 question-answer pairs in 17 languages, covering 14 spatial-function families and 15 answer formats, with ground truth from three knowledge graphs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03882"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"MultiGlobeQA","organization_count":1,"organizations":["claire_radar"],"rank":2240,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03882"},{"aliases":[],"benchmark_id":"llm-stats-multilf","caveat":"MultiLF benchmark","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multilf?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"MultiLF","organization_count":1,"organizations":["llm_stats"],"rank":2241,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/multilf?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-multilingual-mgsm-cot","caveat":"Multilingual Grade School Math (MGSM) benchmark evaluates language models' chain-of-thought reasoning abilities across ten typologically diverse languages. Contains 250 grade-school math problems manually translated from GSM8K dataset into languages including Bengali and Swahili.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mgsm-%28cot%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"Multilingual MGSM (CoT)","organization_count":1,"organizations":["llm_stats"],"rank":2242,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mgsm-%28cot%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-multilingual-mmlu","caveat":"MMLU-ProX is a comprehensive multilingual benchmark covering 29 typologically diverse languages, building upon MMLU-Pro. Each language version consists of 11,829 identical questions enabling direct cross-linguistic comparisons. The benchmark evaluates large language models' reasoning capabilities across linguistic and cultural boundaries through challenging, reasoning-focused questions with 10 answer choices.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mmlu?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Multilingual MMLU","organization_count":1,"organizations":["llm_stats"],"rank":2243,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/multilingual-mmlu?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1751-multiloko","caveat":"MultiLoKo is a multilingual knowledge benchmark, covering 30 languages plus English. MultiLoKo是一个多语言知识基准，涵盖30种语言及英语。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MultiLoKo"],"document_share":0.0002723311546840959,"domain":"语言","name":"MultiLoKo","organization_count":1,"organizations":["opencompass_hub"],"rank":2244,"released":"2025-04-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MultiLoKo"},{"aliases":[],"benchmark_id":"claire-radar-2609-13076","caveat":"MP-Bench evaluates conversational speech systems acting as active participants within multi-party conversation contexts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.13076"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Multiparty Bench (MP-Bench)","organization_count":1,"organizations":["claire_radar"],"rank":2245,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.13076"},{"aliases":[],"benchmark_id":"llm-stats-multipl-e","caveat":"MultiPL-E is a scalable and extensible system for translating unit test-driven code generation benchmarks to multiple programming languages. It extends HumanEval and MBPP Python benchmarks to 18 additional programming languages, enabling evaluation of neural code generation models across diverse programming paradigms and language features.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multipl-e?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"MultiPL-E","organization_count":1,"organizations":["llm_stats"],"rank":2246,"released":"2022-08-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-multipl-e-humaneval","caveat":"MultiPL-E is a scalable and extensible approach to benchmarking neural code generation that translates unit test-driven code generation benchmarks across multiple programming languages. It extends the HumanEval benchmark to 18 additional programming languages, enabling evaluation of code generation models across diverse programming paradigms and providing insights into how models generalize programming knowledge across language boundaries.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-humaneval?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"Multipl-E HumanEval","organization_count":1,"organizations":["llm_stats"],"rank":2247,"released":"2022-08-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-humaneval?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-multipl-e-mbpp","caveat":"MultiPL-E extends the Mostly Basic Python Problems (MBPP) benchmark to 18+ programming languages for evaluating multilingual code generation capabilities. MBPP contains 974 crowd-sourced programming problems designed to be solvable by entry-level programmers, covering programming fundamentals and standard library functionality. Each problem includes a task description, code solution, and automated test cases.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-mbpp?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Multipl-E MBPP","organization_count":1,"organizations":["llm_stats"],"rank":2248,"released":"2022-08-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/multipl-e-mbpp?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-14189","caveat":"MultiRef-Compass evaluates multi-reference-to-audio-video generation systems on 350 curated samples. It assesses Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following using 14 sub-metrics, combining automatic metrics with an MLLM-as-a-Judge framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14189"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MultiRef-Compass","organization_count":1,"organizations":["claire_radar"],"rank":2249,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.14189"},{"aliases":[],"benchmark_id":"claire-radar-2605-26320","caveat":"MultiSeismo is a multimodal seismic dataset with over 16K events integrating waveforms, intensity maps, population exposure, and text, plus MISCE instruction set for seismic reasoning tasks like retrieval and cross-modal analysis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26320"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MULTISEISMO","organization_count":1,"organizations":["claire_radar"],"rank":2250,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26320"},{"aliases":[],"benchmark_id":"claire-radar-2607-26465","caveat":"Evaluates multimodal motivation reasoning in story-driven visual narratives, using 1,000 narratives with 16,092 multi-label questions grounded in Maslow and Reiss frameworks across definition and practical reasoning tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26465"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MultivationBench","organization_count":1,"organizations":["claire_radar"],"rank":2251,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.26465"},{"aliases":[],"benchmark_id":"claire-radar-2607-08970","caveat":"Evaluates multi-view integration in vision-language models using diagnostic tasks for 3D scene comprehension, with a fixed-view baseline and proposed ViewNavigator method.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08970"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MultiView-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2252,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08970"},{"aliases":[],"benchmark_id":"claire-radar-2610-11723","caveat":"Multiplayer world models must ensure that independently controlled views remain consistent with one shared and persistent world.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11723"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MultiWorldBench","organization_count":1,"organizations":["claire_radar"],"rank":2253,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11723"},{"aliases":[],"benchmark_id":"claire-radar-2605-28579","caveat":"MUSE evaluates text-to-CAD generation of complex B-Rep assemblies via design specifications. It scores models on code validity, geometric correctness, and design-intent alignment using rubrics covering functionality, manufacturability, and assemblability. A VLM judge with human validation is used for scalable scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28579"],"document_share":0.0002723311546840959,"domain":"CAD","name":"MUSE","organization_count":1,"organizations":["claire_radar"],"rank":2254,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28579"},{"aliases":[],"benchmark_id":"claire-radar-2609-19088","caveat":"A benchmark for evaluating large vision-language models on artistic image understanding in situated educational applications.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19088"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MUSE","organization_count":1,"organizations":["claire_radar"],"rank":2255,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.19088"},{"aliases":[],"benchmark_id":"claire-radar-2609-15087","caveat":"A multimodal time series forecasting benchmark with heterogeneous context, covering 14 datasets across 8 domains and 6 context types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15087"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MUSE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2256,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15087"},{"aliases":[],"benchmark_id":"claire-radar-2606-30026","caveat":"MuseBench evaluates multimodal large language models on intent-level understanding of audiovisual arts, covering cinematic arts, static visual arts, stage performing arts, and game arts with 4,016 questions in single- and multi-select formats.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30026"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MuseBench","organization_count":1,"organizations":["claire_radar"],"rank":2257,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30026"},{"aliases":[],"benchmark_id":"llm-stats-musiccaps","caveat":"MusicCaps is a dataset composed of 5,521 music examples, each labeled with an English aspect list and a free text caption written by musicians. The dataset contains 10-second music clips from AudioSet paired with rich textual descriptions that capture sonic qualities and musical elements like genre, mood, tempo, instrumentation, and rhythm. Created to support research in music-text understanding and generation tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/musiccaps?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MusicCaps","organization_count":1,"organizations":["llm_stats"],"rank":2258,"released":"2023-01-26","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/musiccaps?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-huggingface-bryel-labs-musp-bench","caveat":"490-question benchmark for musical score understanding, performance listening, and combined score-performance reasoning, with multiple input modalities and accepted ground-truth answers.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/bryel-labs/MuSP-Bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MuSP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2259,"released":"2026-08-20","source":"claire_radar","url":"https://huggingface.co/datasets/bryel-labs/MuSP-Bench"},{"aliases":[],"benchmark_id":"llm-stats-musr","caveat":"MuSR (Multistep Soft Reasoning) is a benchmark for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Created through a neurosymbolic synthetic-to-natural generation algorithm, it generates complex reasoning scenarios like murder mysteries roughly 1000 words in length that challenge current LLMs including GPT-4. The benchmark tests chain-of-thought reasoning capabilities across domains involving commonsense reasoning about physical and social situations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/musr?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"MuSR","organization_count":1,"organizations":["llm_stats"],"rank":2260,"released":"2023-10-24","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/musr?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-29300","caveat":"MusTBENCH evaluates temporal grounding in Large Audio-Language Models through five temporally grounded question-answering tasks, validated by music experts, to test alignment with audio regions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29300"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MusTBENCH","organization_count":1,"organizations":["claire_radar"],"rank":2261,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29300"},{"aliases":[],"benchmark_id":"claire-radar-2609-16090","caveat":"A music retrieval benchmark with 3,440 tracks and seven relation-dependent tasks using natural-language instructions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16090"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MUUNRiver-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2262,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.16090"},{"aliases":[],"benchmark_id":"claire-radar-2607-19910","caveat":"MV-Bench evaluates multimodal language models on coordinating multi-view interface construction using Tableau workbooks, converting specifications into executable web interfaces.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19910"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MV-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2263,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.19910"},{"aliases":["MV2: Multi-View Multi-Vehicle Driving Dataset for Novel View Synthesis"],"benchmark_id":"claire-radar-2608-12442","caveat":"Evaluates novel view synthesis under large viewpoint changes using synchronized captures from car, scooter, and drone across 50 scenes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12442"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MV2","organization_count":1,"organizations":["claire_radar"],"rank":2264,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.12442"},{"aliases":[],"benchmark_id":"llm-stats-mvbench","caveat":"A comprehensive multi-modal video understanding benchmark covering 20 challenging video tasks that require temporal understanding beyond single-frame analysis. Tasks span from perception to cognition, including action recognition, temporal reasoning, spatial reasoning, object interaction, scene transition, and counterfactual inference. Uses a novel static-to-dynamic method to systematically generate video tasks from existing annotations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/mvbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"MVBench","organization_count":1,"organizations":["llm_stats"],"rank":2265,"released":"2023-11-28","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/mvbench?top_n=500"},{"aliases":["MVbench","MVBench"],"benchmark_id":"mvbench","caveat":"Video understanding benchmark. For non-video-native models, uses 1 fps frame extraction. Scores depend on frame sampling strategy and context length.","document_count":1,"document_ids":["model_reports:zai_glm_5_3_flash_model_card"],"document_share":0.0002723311546840959,"domain":"vision","name":"MVbench","organization_count":1,"organizations":["Z.ai"],"rank":2266,"released":"2025-06-01","source":"model_reports","url":"https://github.com/MVBench/MVBench"},{"aliases":[],"benchmark_id":"opencompass-1509-mvbench","caveat":"MVBench can test MLLMs' temporal understanding in the dynamic video tasks. It covers 20 challenging video tasks that cannot be effectively solved with a single frame. MVBench用于评估多模态大模型在动态视频任务中的时间理解能力，由20个单帧内容无法有效解决的挑战性的视频任务组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MVBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MVBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2267,"released":"2023-11-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MVBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-27461","caveat":"A browser-based benchmark with 600 live CAPTCHA instances and a matched foreground-only control benchmark, designed to evaluate GUI agents against motion-based CAPTCHAs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27461"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MVCAP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2268,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27461"},{"aliases":[],"benchmark_id":"claire-radar-2610-12961","caveat":"We introduce MVGraph, a benchmark for evaluating large vision-language models (LVLMs) on multi-graph reasoning where the evidence required by a query is distributed across multiple interrelated visual graph views.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12961"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MVGraph","organization_count":1,"organizations":["claire_radar"],"rank":2269,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.12961"},{"aliases":[],"benchmark_id":"opencompass-1537-mvl-sib","caveat":"MVL-SIB is a multilingual dataset that provides image-sentence pairs spanning 205 languages and 7 topical categories (entertainment, geography, health, politics, science, sports, travel). It was constructed by extending the SIB-200 benchmark. MVL-SIB 是一个多语言数据集，提供了涵盖 205 种语言和 7 个主题类别的图像-句子对（ entertainment ， geography ， health ， politics ， science ， sports ， travel ）。它通过扩展 SIB-200 基准构建而成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MVL-SIB"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MVL-SIB","organization_count":1,"organizations":["opencompass_hub"],"rank":2270,"released":"2025-02-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MVL-SIB"},{"aliases":[],"benchmark_id":"opencompass-1905-mvpbench","caveat":"MVPBench, a curated benchmark designed to rigorously evaluate visual physical reasoning through the lens of visual CoT. Each example features interleaved multi-image inputs and demands not only the correct final answer but also a coherent, step-by-step reasoning path grounded in evolving visual cues MVPBench专注于视觉物理推理中的视觉链式思维（CoT）能力评估。该基准涵盖真实图像、多步逻辑与多条可行思维路径，每个样例均配有图像证据，要求模型在剥离文本提示依赖的前提下，不仅得出正确答案，还需正确完成每一个中间推理步骤，模拟人类的逐步图像推理过程。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/MVPBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"MVPBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2271,"released":"2025-06-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/MVPBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-30952","caveat":"Evaluates multi-view video reasoning in vision-language models using real-world multi-camera data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30952"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"MVVBench","organization_count":1,"organizations":["claire_radar"],"rank":2272,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.30952"},{"aliases":[],"benchmark_id":"claire-radar-2606-16748","caveat":"MyPCBench tests computer-use agents as personal assistants on a Linux desktop seeded with 17 simulated web apps for a single persona. It has 184 tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16748"],"document_share":0.0002723311546840959,"domain":"General AI","name":"MyPCBench","organization_count":1,"organizations":["claire_radar"],"rank":2273,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16748"},{"aliases":[],"benchmark_id":"claire-radar-2609-27086","caveat":"Shared task for multidialectal Arabic speech processing with evaluation and rankings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27086"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NADI 2026","organization_count":1,"organizations":["claire_radar"],"rank":2274,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.27086"},{"aliases":[],"benchmark_id":"claire-radar-2609-27517","caveat":"A procedural generator of solver-certified instances spanning four semantic viewpoints: SLDNF, well-founded semantics, and credulous and skeptical reasoning under stable-model semantics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27517"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NAFBench","organization_count":1,"organizations":["claire_radar"],"rank":2275,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27517"},{"aliases":[],"benchmark_id":"llm-stats-nanogpt","caveat":"NanoGPT is an OpenAI AI-self-improvement evaluation that measures whether models can optimize training recipes for small GPT-style models, part of the suite tracking progress toward accelerating internal research.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nanogpt?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"NanoGPT","organization_count":1,"organizations":["llm_stats"],"rank":2276,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nanogpt?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-17391","caveat":"NarrativeWorldBench evaluates long-horizon narrative generation in audio drama across nine structural metrics and four Indic languages, across horizons from 10 to 200 episodes. Scoring is based on plot-beat F1 and other metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17391"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NarrativeWorldBench","organization_count":1,"organizations":["claire_radar"],"rank":2277,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.17391"},{"aliases":[],"benchmark_id":"claire-radar-2608-13210","caveat":"NARU evaluates multimodal models on Japanese long-form video understanding across narrative evolution (character evolution, sequential flow, plot progression, thematic development) and cultural nuance (aizuchi, reading the air, subtext, cultural context, sentiment) via multiple-choice QA.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13210"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NARU","organization_count":1,"organizations":["claire_radar"],"rank":2278,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13210"},{"aliases":[],"benchmark_id":"llm-stats-natural-questions","caveat":"Natural Questions is a question answering dataset featuring real anonymized queries issued to Google search engine. It contains 307,373 training examples where annotators provide long answers (passages) and short answers (entities) from Wikipedia pages, or mark them as unanswerable.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/natural-questions?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Natural Questions","organization_count":1,"organizations":["llm_stats"],"rank":2279,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/natural-questions?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-natural2code","caveat":"NaturalCodeBench (NCB) is a challenging code benchmark designed to mirror the complexity and variety of real-world coding tasks. It comprises 402 high-quality problems in Python and Java, selected from natural user queries from online coding services, covering 6 different domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/natural2code?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Natural2Code","organization_count":1,"organizations":["llm_stats"],"rank":2280,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/natural2code?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1081-naturalcodebench","caveat":"NaturalCodeBench is challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. NaturalCodeBench 是一个具有挑战性的代码基准，旨在反映真实编码任务中的复杂性和多样性。NaturalCodeBench 包含 402 个高质量的 Python 和 Java 问题，这些问题是从在线编码服务的自然用户查询中精心挑选的，涵盖了 6 个不同的领域。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NaturalCodeBench"],"document_share":0.0002723311546840959,"domain":"代码","name":"NaturalCodeBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2281,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/NaturalCodeBench"},{"aliases":[],"benchmark_id":"opencompass-1117-naturalproofs","caveat":"NATURALPROOFS is a multi-domain corpus of mathematical statements and their proofs, written in natural mathematical language. NATURALPROOFS unifies broad coverage, deep coverage, and low-resource mathematical sources, allowing for evaluating both in-distribution and zero-shot generalization. NATURALPROOFS 是一个多领域的数学语句及其证明的语料库，采用自然数学语言编写。整合了广泛覆盖、深入覆盖和低资源数学来源，便于评估分布内和零样本泛化的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NaturalProofs"],"document_share":0.0002723311546840959,"domain":"数学","name":"NaturalProofs","organization_count":1,"organizations":["opencompass_hub"],"rank":2282,"released":"2021-06-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/NaturalProofs"},{"aliases":[],"benchmark_id":"claire-radar-2606-24530","caveat":"NatureBench evaluates AI coding agents on 90 tasks distilled from Nature-family publications across 6 scientific domains, scoring against each paper's reported state of the art.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24530"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NatureBench","organization_count":1,"organizations":["claire_radar"],"rank":2283,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24530"},{"aliases":[],"benchmark_id":"claire-radar-2607-19695","caveat":"NavVerse evaluates indoor-to-outdoor embodied navigation in continuous robot simulation, with 10,000 episodes across object, vision-language, and place navigation tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19695"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"NavVerse","organization_count":1,"organizations":["claire_radar"],"rank":2284,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.19695"},{"aliases":[],"benchmark_id":"claire-radar-2608-09200","caveat":"NBA_Streaming is a benchmark for online fine-grained basketball commentary generation, containing 307.5 hours of broadcasts and approximately 35K temporally aligned events with annotations of event boundaries, player identities, fine-grained actions, event chains, and natural-language commentary.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09200"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NBA_Streaming","organization_count":1,"organizations":["claire_radar"],"rank":2285,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09200"},{"aliases":[],"benchmark_id":"claire-radar-2608-08160","caveat":"NCP-Bench evaluates narrative commitment preservation in interactive narratives. It provides 100 movie-synopsis-based environments with structured narrative specifications and an automatic evaluator that checks fact, commitment, and trajectory consistency across narrator responses to player actions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08160"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"NCP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2286,"released":"2026-08-08","source":"claire_radar","url":"https://arxiv.org/abs/2608.08160"},{"aliases":[],"benchmark_id":"claire-radar-2607-12076","caveat":"NEEDL-Bench is a microscopy detection benchmark for Swiss Needle Cast and stomata detection, with 3250 annotated images from 1082 Douglas-fir needles, annotated for keypoint and bounding-box detectors, and two evaluation splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.12076"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NEEDL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2287,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.12076"},{"aliases":[],"benchmark_id":"claire-radar-2609-06774","caveat":"Combines five astronomical streak datasets in YOLO format with predefined within-source and leave-one-source-out splits to evaluate cross-source generalization of detectors.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06774"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NEO-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2288,"released":"2026-09-06","source":"claire_radar","url":"https://arxiv.org/abs/2609.06774"},{"aliases":[],"benchmark_id":"claire-radar-2610-15732","caveat":"NepPlural is a dataset and evaluation suite for modeling societal personas in Nepali migration discourse, with human gold annotations, LLM protocol audit, encoder baselines, and human-judged steerability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.15732"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NepPlural","organization_count":1,"organizations":["claire_radar"],"rank":2289,"released":"2026-10-11","source":"claire_radar","url":"https://arxiv.org/abs/2610.15732"},{"aliases":["Net-Ev$^2$: A Generative Simulator for Network Event Evolution"],"benchmark_id":"claire-radar-2606-12494","caveat":"Net-Ev$^2$-6.5M evaluates generated network dynamics using aligned event and traffic data from four large-scale road networks and the topology-aware JL-MMD metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12494"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Net-Ev$^2$-6.5M","organization_count":1,"organizations":["claire_radar"],"rank":2290,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12494"},{"aliases":[],"benchmark_id":"claire-radar-2609-09849","caveat":"NetArtifactBench evaluates the ability of AI agents to repair inconsistent network experiment records while preserving claims that remain supported. It contains 52 instances with injected inconsistencies ranging from direct contradictions to unstated relations across multiple artifacts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09849"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NetArtifactBench","organization_count":1,"organizations":["claire_radar"],"rank":2291,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09849"},{"aliases":[],"benchmark_id":"claire-radar-2608-23179","caveat":"Evaluates LLM agents in closed-loop network configuration across 480 task instances from 96 protocol-focused templates, scoring based on hidden executable test cases.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.23179"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NetConfArena","organization_count":1,"organizations":["claire_radar"],"rank":2292,"released":"2026-08-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.23179"},{"aliases":[],"benchmark_id":"claire-radar-2607-10490","caveat":"NetInjectBench evaluates LLM agents for network operations under indirect prompt injection. It comprises 130 scenarios: 40 benign, 40 weak-attack, 40 strong-attack, and 10 approved high-impact changes, with separated untrusted artifact text, trusted policy metadata, and evaluation labels. Scoring measures unsafe tool-action rate, usefulness, and overblocking across models and defenses.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10490"],"document_share":0.0002723311546840959,"domain":"Agents","name":"NetInjectBench","organization_count":1,"organizations":["claire_radar"],"rank":2293,"released":"2026-07-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.10490"},{"aliases":[],"benchmark_id":"claire-radar-2608-12197","caveat":"NetlistBench evaluates LLM reliability in recognizing and manipulating SPICE netlists, covering parameter and connectivity recognition, edits, hierarchical operations, equivalence judgment, and compound editing, with deterministic structure-aware scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12197"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"NetlistBench","organization_count":1,"organizations":["claire_radar"],"rank":2294,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.12197"},{"aliases":[],"benchmark_id":"claire-radar-2607-25232","caveat":"Evaluates machine learning models on the Neurai-VN dataset for mental health classification. Four binary tasks (healthy control vs. depression, anxiety, clinical, and depression vs. anxiety) are defined using subject-wise cross-validation and standardized feature groups. Baseline models include linear, tree-based, and neural networks, with mean F1 scores reported.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25232"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Neurai-VN Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2295,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25232"},{"aliases":[],"benchmark_id":"claire-radar-2609-16919","caveat":"Dataset-centric effort that benchmarks neuro-symbolic and ontological caption generation using pre-trained grounding and learned auto-regressive captioning networks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16919"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"NeuroSymbEAD","organization_count":1,"organizations":["claire_radar"],"rank":2296,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.16919"},{"aliases":[],"benchmark_id":"opencompass-1074-newsbench","caveat":"NewsBench is a novel evaluation framework to systematically assess the capabilities of Large Language Models (LLMs) for editorial capabilities in Chinese journalism. NewsBench 是一个新颖的评估框架，旨在系统性地评估大型语言模型在中文新闻编辑能力上的表现。构建的基准数据集聚焦于写作能力的四个方面和安全遵循的六个方面，包含 1,267 个手动精心设计的测试样本，类型包括选择题和简答题，涵盖 24 个新闻领域的五项编辑任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NewsBench"],"document_share":0.0002723311546840959,"domain":"创作","name":"NewsBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2297,"released":"2024-06-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/NewsBench"},{"aliases":[],"benchmark_id":"claire-radar-2606-04773","caveat":"NextMotionQA is a benchmark for human motion understanding with VLMs, featuring three tasks: multiple-choice QA, video captioning, and fine-grained error correction. It includes expert-verified annotations across three semantic axes and three complexity levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04773"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"NextMotionQA","organization_count":1,"organizations":["claire_radar"],"rank":2298,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04773"},{"aliases":[],"benchmark_id":"llm-stats-nexus","caveat":"NexusRaven benchmark for evaluating function calling capabilities of large language models in zero-shot scenarios across cybersecurity tools and API interactions","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nexus?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"Nexus","organization_count":1,"organizations":["llm_stats"],"rank":2299,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nexus?top_n=500"},{"aliases":["Accelerometry-Derived Digital Biomarkers for Cardiometabolic Risk: A Population-Representative Tabular Benchmark with Uncertainty Quantification"],"benchmark_id":"claire-radar-2606-30702","caveat":"A tabular benchmark for predicting cardiometabolic biomarkers from accelerometry-derived activity phenotypes and lifestyle covariates in a population-representative NHANES cohort.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30702"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"NHANES Accelerometry Cardiometabolic Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2300,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30702"},{"aliases":[],"benchmark_id":"llm-stats-nih-multi-needle","caveat":"Multi-needle in a haystack benchmark for evaluating long-context comprehension capabilities of language models by testing retrieval of multiple target pieces of information from extended documents","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nih-multi-needle?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"NIH/Multi-needle","organization_count":1,"organizations":["llm_stats"],"rank":2301,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nih-multi-needle?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-29700","caveat":"NL-PDDL-Bench evaluates natural-language-to-PDDL specification generation. It consists of multi-domain instances from IPC domains with planner-verified executability, difficulty scaled by object count, and a suite for parseability, solvability, and plan-level consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29700"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NL-PDDL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2302,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.29700"},{"aliases":[],"benchmark_id":"llm-stats-nl2repo","caveat":"NL2Repo evaluates long-horizon coding capabilities including repository-level understanding, where models must generate or modify code across entire repositories from natural language specifications.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nl2repo?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"NL2Repo","organization_count":1,"organizations":["llm_stats"],"rank":2303,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nl2repo?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-22061","caveat":"NL2Scratch is an executable benchmark for natural-language-to-Scratch generation, consisting of 311,648 parser-valid NL-program pairs extracted from real Scratch projects. It includes a semantically validated pool of 23,594 examples and an 800-example diagnostic set. Evaluation uses Semantic Alignment Consistency (SAC), an interpretable slot-level metric for measuring semantic agreement.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22061"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"NL2Scratch","organization_count":1,"organizations":["claire_radar"],"rank":2304,"released":"2026-06-20","source":"claire_radar","url":"https://arxiv.org/abs/2606.22061"},{"aliases":[],"benchmark_id":"claire-radar-2608-07530","caveat":"A benchmark suite for translating natural language requirements into SHACL shapes, evaluating semantic equivalence beyond string comparison.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07530"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NL2SHACL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2305,"released":"2026-07-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.07530"},{"aliases":["nlp-sentiment-analysis-benchmark"],"benchmark_id":"claire-radar-github-anisha-das-kts-nlp-sentiment-analysis-benchmark","caveat":"A reproducible NLP benchmarking project comparing three sentiment-analysis approaches on the NLTK Movie Reviews dataset, reporting predictive performance alongside training cost, latency, and error analysis.","document_count":1,"document_ids":["claire_radar:https://github.com/anisha-das-kts/nlp-sentiment-analysis-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NLP Sentiment Analysis Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2306,"released":"2026-09-06","source":"claire_radar","url":"https://github.com/anisha-das-kts/nlp-sentiment-analysis-benchmark"},{"aliases":[],"benchmark_id":"llm-stats-nmos","caveat":"NMOS evaluation benchmark for assessing model performance on specialized tasks","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nmos?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"NMOS","organization_count":1,"organizations":["llm_stats"],"rank":2307,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nmos?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-39673","caveat":"Can a pretrained graph model replace training and tuning a separate predictor for each dataset?","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39673"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NodeGround","organization_count":1,"organizations":["claire_radar"],"rank":2308,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39673"},{"aliases":[],"benchmark_id":"llm-stats-community-2256e9c9-b256-4444-b639-7cc3b1855d96","caveat":null,"document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3A2256e9c9-b256-4444-b639-7cc3b1855d96?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"nolima","organization_count":1,"organizations":["llm_stats"],"rank":2309,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A2256e9c9-b256-4444-b639-7cc3b1855d96?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-nolima-128k","caveat":"NoLiMa evaluated at a 131072-token context length. Tests latent associative reasoning in long contexts with minimal lexical overlap between questions and needles.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nolima-128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"NoLiMa 128K","organization_count":1,"organizations":["llm_stats"],"rank":2310,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-128k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-nolima-32k","caveat":"NoLiMa evaluated at a 32768-token context length. Tests latent associative reasoning in long contexts with minimal lexical overlap between questions and needles.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nolima-32k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"NoLiMa 32K","organization_count":1,"organizations":["llm_stats"],"rank":2311,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-32k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-nolima-64k","caveat":"NoLiMa evaluated at a 65536-token context length. Tests latent associative reasoning in long contexts with minimal lexical overlap between questions and needles.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nolima-64k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"NoLiMa 64K","organization_count":1,"organizations":["llm_stats"],"rank":2312,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nolima-64k?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-04397","caveat":"NOLLI is a procedurally generated English-Korean puzzle benchmark with 15 puzzle types (25 tasks, 7,500 items). Each instance is seed-regenerable, verified to have a unique solution, and scored deterministically. Difficulty is calibrated behaviorally to target accuracy bands.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04397"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NOLLI","organization_count":1,"organizations":["claire_radar"],"rank":2313,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04397"},{"aliases":[],"benchmark_id":"claire-radar-2606-04806","caveat":"NoRA is a visual first-person video benchmark requiring models to generate candidate next actions and justify them via fact-reason-action support graphs. It includes 1,420 annotated video clips and uses a grounded reasonableness score combining action alignment, factual grounding, and support binding.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04806"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NoRA","organization_count":1,"organizations":["claire_radar"],"rank":2314,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04806"},{"aliases":[],"benchmark_id":"claire-radar-2606-27826","caveat":"NormAct evaluates embodied agents on 550 TongSim scenarios where the same goal permits norm-compliant or norm-violating action sequences, testing whether agents infer and apply scene-relevant social norms during ordinary tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27826"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"NormAct","organization_count":1,"organizations":["claire_radar"],"rank":2315,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.27826"},{"aliases":[],"benchmark_id":"claire-radar-2606-08932","caveat":"NormBench evaluates defeasible scope parsing in legal texts using Span-Grounded Deontic Trees, with 2,290 provisions across multiple languages. It focuses on identifying clause overrides and includes whole-tree fidelity metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08932"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"NormBench","organization_count":1,"organizations":["claire_radar"],"rank":2316,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.08932"},{"aliases":[],"benchmark_id":"llm-stats-nova-63","caveat":"NOVA-63 is a multilingual evaluation benchmark covering 63 languages, designed to assess LLM performance across diverse linguistic contexts and tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nova-63?top_n=500"],"document_share":0.0002723311546840959,"domain":"general","name":"NOVA-63","organization_count":1,"organizations":["llm_stats"],"rank":2317,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nova-63?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-03657","caveat":"NovelAPIBench is a dynamic benchmark for evaluating LLM tool use with novel APIs, covering knowledge components and diagnostic failure categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03657"],"document_share":0.0002723311546840959,"domain":"Agents","name":"NovelAPIBench","organization_count":1,"organizations":["claire_radar"],"rank":2318,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03657"},{"aliases":[],"benchmark_id":"claire-radar-2609-11234","caveat":"Human-anchored assessment of LLM capabilities in diagnosing paper novelty.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11234"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NovGauge","organization_count":1,"organizations":["claire_radar"],"rank":2319,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11234"},{"aliases":[],"benchmark_id":"opencompass-1783-nppc","caveat":"Nondeterministic Polynomial-time Problem Challenge (NPPC), an ever-scaling reasoning benchmark for LLMs. 非确定性多项式时间问题挑战 （NPPC），这是一个不断扩展的 LLM 推理基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NPPC"],"document_share":0.0002723311546840959,"domain":"推理","name":"NPPC","organization_count":1,"organizations":["opencompass_hub"],"rank":2320,"released":"2025-04-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/NPPC"},{"aliases":[],"benchmark_id":"llm-stats-nq","caveat":"Natural Questions (NQ) benchmark containing real user questions issued to Google search with answers found from Wikipedia, designed for training and evaluation of automatic question answering systems","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nq?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"NQ","organization_count":1,"organizations":["llm_stats"],"rank":2321,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nq?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-513-nq","caveat":"NQ (NaturalQuestion) corpus contains questions from real users, and it requires QA systems to read and comprehend an entire Wikipedia article that may or may not contain the answer to the question. The inclusion of real user questions, and the requirement that solutions should read an entire page to find the answer, cause NQ to be a more realistic and challenging task than prior QA datasets. NQ 数据集来自于真实用户的问题，它要求 QA 系统阅读和理解整个维基百科文章，这些文章可能包含也可能不包含问题的答案。由真实用户问题构成，以及需要阅读整个页面才能找到答案的要求，比以往的 QA 数据集更现实和更具挑战性的任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NQ"],"document_share":0.0002723311546840959,"domain":"知识","name":"NQ","organization_count":1,"organizations":["opencompass_hub"],"rank":2322,"released":"2019-06-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/NQ"},{"aliases":[],"benchmark_id":"claire-radar-2605-23668","caveat":"NQP-Bench is a dataset within the OnePred paper for evaluating next-query prediction in multi-turn conversations. It spans three diverse subsets and is used to compare OnePred against baselines.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23668"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NQP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2323,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23668"},{"aliases":[],"benchmark_id":"claire-radar-2606-19727","caveat":"A dataset of 9,260 question-answer pairs across 12 languages evaluating cultural knowledge in global dance traditions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19727"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NRITYAM","organization_count":1,"organizations":["claire_radar"],"rank":2324,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.19727"},{"aliases":[],"benchmark_id":"claire-radar-2606-20408","caveat":"Evaluates multi-turn red-teaming of LLM agents acting as operators of a simulated nuclear power plant control room. Safety is measured by objective loss of critical safety functions (CSFs) under adaptive attacks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20408"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NRT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2325,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.20408"},{"aliases":[],"benchmark_id":"claire-radar-2606-27047","caveat":"NuclearQAv2 evaluates LLMs on nuclear engineering knowledge with approximately 1,240 QA pairs covering boolean, numeric, and verbal questions. It uses structured prompting for automated question generation and response evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27047"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"NuclearQAv2","organization_count":1,"organizations":["claire_radar"],"rank":2326,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.27047"},{"aliases":[],"benchmark_id":"claire-radar-2608-28206","caveat":"Evaluates counting accuracy in text-to-image models using 640,000 prompts across 1,600 categories and counts from 1 to 100, with factorial variation of composition, layout, and appearance.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.28206v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NumBench","organization_count":1,"organizations":["claire_radar"],"rank":2327,"released":"2026-08-28","source":"claire_radar","url":"http://arxiv.org/abs/2608.28206v1"},{"aliases":["NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models"],"benchmark_id":"claire-radar-2608-15425","caveat":"Evaluates zero-shot numerosity perception in vision-language models on 10,800 synthetic images across six controlled conditions manipulating object size, spatial arrangement, and numerosity while ablating texture, shape, and color.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15425"],"document_share":0.0002723311546840959,"domain":"General AI","name":"NumerosityVLM","organization_count":1,"organizations":["claire_radar"],"rank":2328,"released":"2026-08-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.15425"},{"aliases":[],"benchmark_id":"llm-stats-nuscene","caveat":"A multimodal benchmark for scene understanding and reasoning over the nuScenes autonomous driving domain.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/nuscene?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Nuscene","organization_count":1,"organizations":["llm_stats"],"rank":2329,"released":"2019-03-26","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/nuscene?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-07135","caveat":"Benchmarks embodied agents on nutrition management via QA pairs from cooking videos across five task families.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07135"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"NutriBench-Kitchen","organization_count":1,"organizations":["claire_radar"],"rank":2330,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07135"},{"aliases":[],"benchmark_id":"opencompass-1533-nutritionqa","caveat":"CoSyn-400K dataset contains 9 categories of synthetc text-rich images with 2.7M instruction-tuning data 通过代码引导的合成多模态数据生成扩展文本丰富图像理解，CoSyn-400K 数据集包含 9 类合成文本丰富图像，以及 270 万条指令微调数据。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/NutritionQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"NutritionQA","organization_count":1,"organizations":["opencompass_hub"],"rank":2331,"released":"2025-02-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/NutritionQA"},{"aliases":["newt-bench"],"benchmark_id":"claire-radar-github-vsbdev-little-hermione","caveat":"Deliberately contaminated LoRA experiment with a 75-question Harry Potter evaluation set and training code that reproduces the leakage demonstration.","document_count":1,"document_ids":["claire_radar:https://github.com/VSBDev/little-hermione"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"O.W.L. & N.E.W.T. Bench","organization_count":1,"organizations":["claire_radar"],"rank":2332,"released":"2026-09-08","source":"claire_radar","url":"https://github.com/VSBDev/little-hermione"},{"aliases":[],"benchmark_id":"claire-radar-2606-30262","caveat":"OAO-AttackBench is a set of counterfactual prompts for one-and-only objects, used to evaluate text-to-image alignment in a specific study.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30262"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OAO-AttackBench","organization_count":1,"organizations":["claire_radar"],"rank":2333,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30262"},{"aliases":[],"benchmark_id":"claire-radar-2609-07601","caveat":"Evaluates long-context EHR decision-making for obstetrics and gynecology using 1,500 rule-grounded cases across Evidence-only, Visit-level, and History-level settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07601"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ObGynLongBench","organization_count":1,"organizations":["claire_radar"],"rank":2334,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07601"},{"aliases":[],"benchmark_id":"claire-radar-2609-21822","caveat":"Introduces two benchmark concepts for uncertainty-aware object detection and label error detection.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21822"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty","organization_count":1,"organizations":["claire_radar"],"rank":2335,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21822"},{"aliases":[],"benchmark_id":"llm-stats-objectron","caveat":"Objectron evaluates 3D object detection and pose estimation capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/objectron?top_n=500"],"document_share":0.0002723311546840959,"domain":"3d","name":"Objectron","organization_count":1,"organizations":["llm_stats"],"rank":2336,"released":"2020-12-18","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/objectron?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-08264","caveat":"Evaluates operational skill unlearning in deployed agents across 88 attack episodes, measuring attack success rate and impact-weighted exposure after workflow-level defenses.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08264"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OBLIVION","organization_count":1,"organizations":["claire_radar"],"rank":2337,"released":"2026-08-08","source":"claire_radar","url":"https://arxiv.org/abs/2608.08264"},{"aliases":[],"benchmark_id":"claire-radar-2607-23537","caveat":"ObsDriveBench evaluates multimodal understanding in autonomous driving under adverse weather, covering observability awareness, spatial reliability, and risk-aware decision-making with multiple-choice tasks over camera, LiDAR, and radar inputs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23537"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"ObsDriveBench","organization_count":1,"organizations":["claire_radar"],"rank":2338,"released":"2026-07-26","source":"claire_radar","url":"https://arxiv.org/abs/2607.23537"},{"aliases":["ObserverBench: Testing Mechanistic Estimates for Intervention and Control"],"benchmark_id":"claire-radar-2609-03026","caveat":"Benchmark framework that tests whether mechanistic interpretability observers are adequate for intervention, control, or safety tasks by reporting estimation accuracy and action loss separately.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03026"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ObserverBench","organization_count":1,"organizations":["claire_radar"],"rank":2339,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.03026"},{"aliases":[],"benchmark_id":"claire-radar-2608-00012","caveat":"Obshazard-bench is a real-time benchmark for disaster intelligence, integrating raw satellite and ground-station data. It covers 8 disaster categories and 28 sub-categories across 60+ countries, with VQA samples and a three-stage evaluation taxonomy: Predictive Crisis Anticipation, Active Evolution Reasoning, and Multi-faceted Impact Quantification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00012"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Obshazard-bench","organization_count":1,"organizations":["claire_radar"],"rank":2340,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.00012"},{"aliases":[],"benchmark_id":"claire-radar-2607-07051","caveat":"OCCUR-Bench evaluates temporal preservation in conversational image editing, providing occlusion-and-revelation scenarios with historical restoration references to assess faithful restoration of occluded content.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.07051"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OCCUR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2341,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.07051"},{"aliases":[],"benchmark_id":"opencompass-525-ocnli","caveat":"OCNLI is a Chinese natural language inference task, which requires to determine the logical relation between two sentences, with three relations: entailment, contradiction and neutral. OCNLI是一个中文自然语言推理任务，要求根据两个句子判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OCNLI"],"document_share":0.0002723311546840959,"domain":"推理","name":"OCNLI","organization_count":1,"organizations":["opencompass_hub"],"rank":2342,"released":"2020-10-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OCNLI"},{"aliases":["OCR-MetaReasoning Benchmark"],"benchmark_id":"claire-radar-2608-30678","caveat":"Evaluates MLLMs on text-rich image understanding by separating deduction, induction, and abduction across 1,500 samples with automatic answer scoring and reasoning-process compliance metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30678"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OCR-MetaReasoning","organization_count":1,"organizations":["claire_radar"],"rank":2343,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30678"},{"aliases":[],"benchmark_id":"llm-stats-ocrbench","caveat":"OCRBench: Comprehensive evaluation benchmark for assessing Optical Character Recognition (OCR) capabilities in Large Multimodal Models across text recognition, scene text VQA, and document understanding tasks","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ocrbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"image_to_text","name":"OCRBench","organization_count":1,"organizations":["llm_stats"],"rank":2344,"released":"2024-01-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-557-ocrbench","caveat":"OCRBench provides a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks including Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER). OCRBench对 GPT4V 和 Gemini 等大型多模态模型在各种文本相关的视觉任务中的表现进行了全面的评估，包括文本识别、场景文本为中心的视觉问答 (VQA)、面向文档的 VQA、关键信息提取 (KIE) 和手写数学表达式识别 (HMER)。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OCRBench"],"document_share":0.0002723311546840959,"domain":"语言","name":"OCRBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2345,"released":"2024-01-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OCRBench"},{"aliases":[],"benchmark_id":"llm-stats-ocrbench-v2-en","caveat":"OCRBench v2 English subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with English text content","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28en%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"image_to_text","name":"OCRBench-V2 (en)","organization_count":1,"organizations":["llm_stats"],"rank":2346,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28en%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ocrbench-v2-zh","caveat":"OCRBench v2 Chinese subset: Enhanced benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with Chinese text content","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28zh%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"image_to_text","name":"OCRBench-V2 (zh)","organization_count":1,"organizations":["llm_stats"],"rank":2347,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2-%28zh%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ocrbench-v2","caveat":"OCRBench v2: Enhanced large-scale bilingual benchmark for evaluating Large Multimodal Models on visual text localization and reasoning with 10,000 human-verified question-answering pairs across 8 core OCR capabilities","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"image_to_text","name":"OCRBench_V2","organization_count":1,"organizations":["llm_stats"],"rank":2348,"released":"2024-12-31","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ocrbench-v2?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-16609","caveat":"OCT-Bench evaluates multimodal large language models on OCT image understanding with 10,076 multiple-choice questions from 4,137 images across seven public datasets, covering 20 tasks in Perception, Cognition, and Reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.16609"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"OCT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2349,"released":"2026-07-18","source":"claire_radar","url":"https://arxiv.org/abs/2607.16609"},{"aliases":[],"benchmark_id":"llm-stats-octocodingbench","caveat":"Octopus coding benchmark for evaluating multi-language programming capabilities","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/octocodingbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"OctoCodingBench","organization_count":1,"organizations":["llm_stats"],"rank":2350,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/octocodingbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-10703","caveat":"Eyeglass reflection removal is important across smartphone imaging, video conferencing, and other face-centric visual applications.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10703"],"document_share":0.0002723311546840959,"domain":"Mobile & Personal Computing","name":"OcuBench","organization_count":1,"organizations":["claire_radar"],"rank":2351,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10703"},{"aliases":[],"benchmark_id":"claire-radar-2608-18595","caveat":"OdinEval is a benchmark for program repair in the Odin programming language, built from documented defects, with issue-to-commit bindings, regression tests, and execution records.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.18595"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"OdinEval","organization_count":1,"organizations":["claire_radar"],"rank":2352,"released":"2026-08-19","source":"claire_radar","url":"https://arxiv.org/abs/2608.18595"},{"aliases":[],"benchmark_id":"llm-stats-odinw","caveat":"Object Detection in the Wild (ODinW) benchmark for evaluating object detection models' task-level transfer ability across diverse real-world datasets in terms of prediction accuracy and adaptation efficiency","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/odinw?top_n=500"],"document_share":0.0002723311546840959,"domain":"vision","name":"ODinW","organization_count":1,"organizations":["llm_stats"],"rank":2353,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/odinw?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-11941","caveat":"Evaluates language models on formalized open mathematical conjectures from OEIS, scored by theorem resolution under a compute budget.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11941"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OEIS Open","organization_count":1,"organizations":["claire_radar"],"rank":2354,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.11941"},{"aliases":[],"benchmark_id":"claire-radar-2608-20106","caveat":"Evaluates LLM knowledge in the wine domain via 3,266 multiple-choice questions across six pillars and four difficulty tiers, sourced from verified facts and scored against a calibrated audit.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.20106"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"OenoBench","organization_count":1,"organizations":["claire_radar"],"rank":2355,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.20106"},{"aliases":["office-agent-benchmark"],"benchmark_id":"claire-radar-github-basilos-ai-office-agent-benchmark","caveat":"Evaluates office agents on documents, spreadsheets, presentations, daily workflows, instruction following, and safety constraints through executable scoring contracts.","document_count":1,"document_ids":["claire_radar:https://github.com/basilos-ai/office-agent-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Office Agent Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2356,"released":"2026-09-20","source":"claire_radar","url":"https://github.com/basilos-ai/office-agent-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2607-01245","caveat":"A two-track benchmark evaluating structural, visual, and domain-grounded comprehension of native Word, Excel, and PowerPoint files.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01245"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Office Comprehension Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2357,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.01245"},{"aliases":[],"benchmark_id":"claire-radar-2609-34469","caveat":"A small Office edit creates two obligations: propagate every required update and leave protected state untouched.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34469"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"OfficeEditBench","organization_count":1,"organizations":["claire_radar"],"rank":2358,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34469"},{"aliases":[],"benchmark_id":"llm-stats-officeqa-pro","caveat":"OfficeQA Pro evaluates AI models on professional knowledge-work questions and tasks drawn from real office workflows, including document analysis, spreadsheet reasoning, and information synthesis across business domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/officeqa-pro?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"OfficeQA Pro","organization_count":1,"organizations":["llm_stats"],"rank":2359,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/officeqa-pro?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-19951","caveat":"OffNadirLoc evaluates UAV-to-satellite geo-localization under large off-nadir views, with structure-aware contextual weighting and view-coherent learning for viewpoint-invariant features.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19951"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OffNadirLoc","organization_count":1,"organizations":["claire_radar"],"rank":2360,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.19951"},{"aliases":[],"benchmark_id":"llm-stats-ojbench","caveat":"OJBench is a competition-level code benchmark designed to assess the competitive-level code reasoning abilities of large language models. It comprises 232 programming competition problems from NOI and ICPC, categorized into Easy, Medium, and Hard difficulty levels. The benchmark evaluates models' ability to solve complex competitive programming challenges using Python and C++.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ojbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"OJBench","organization_count":1,"organizations":["llm_stats"],"rank":2361,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ojbench-cpp","caveat":"OJBench (C++) is the C++ subset of OJBench, a competition-level code benchmark that evaluates large language models on programming competition problems from NOI and ICPC using C++ as the implementation language.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ojbench-cpp?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"OJBench (C++)","organization_count":1,"organizations":["llm_stats"],"rank":2362,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ojbench-cpp?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-02986","caveat":"Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.02986"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OLMo-Detect","organization_count":1,"organizations":["claire_radar"],"rank":2363,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.02986"},{"aliases":[],"benchmark_id":"opencompass-1705-olymmath","caveat":"A benchmark of 200 Olympiad math problems across algebra, geometry, number theory, and combinatorics. Available in English and Chinese, it features two difficulty levels: EASY (AIME-level) to test standard reasoning, and HARD to challenge advanced models. 一个包含 200 道奥林匹克数学题的基准测试，涵盖代数、几何、数论和组合。我们提供英文和中文版本，并提供两个难度等级：EASY（AIME水平）用于测试标准推理能力，以及 HARD 用于挑战高级模型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OlymMATH"],"document_share":0.0002723311546840959,"domain":"强推理","name":"OlymMATH","organization_count":1,"organizations":["opencompass_hub"],"rank":2364,"released":"2025-03-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OlymMATH"},{"aliases":[],"benchmark_id":"llm-stats-olympiadbench","caveat":"A challenging benchmark for promoting AGI with Olympiad-level bilingual multimodal scientific problems. Comprises 8,476 math and physics problems from international and Chinese Olympiads and the Chinese college entrance exam, featuring expert-level annotations for step-by-step reasoning. Includes both text-only and multimodal problems in English and Chinese.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/olympiadbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"OlympiadBench","organization_count":1,"organizations":["llm_stats"],"rank":2365,"released":"2024-02-21","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/olympiadbench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1070-olympiadbench","caveat":"OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring\n8,476 problems from Olympiad-level mathematics and physics competitions, including the\nChinese college entrance exam. Each problem is detailed with expert-level annotations\nfor step-by-step reasoning. OlympiadBench 是一个奥林匹克级别的双语多模态科学基准，包含来自奥林匹克级数学和物理竞赛的8,476道题目，包括中国高考。每道题目都配有专家级别的注释，提供逐步推理的详细说明。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OlympiadBench"],"document_share":0.0002723311546840959,"domain":"学科","name":"OlympiadBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2366,"released":"2024-06-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OlympiadBench"},{"aliases":[],"benchmark_id":"opencompass-1329-olympicarena","caveat":"OlympicArena evaluates cognitive reasoning abilities. It includes 11,163 bilingual problems spanning seven fields and 62 international Olympic competitions. OlympicArena用于评估大模型的认知推理能力，包含来自7个领域、62项国际奥林匹克比赛的11163个双语问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OlympicArena"],"document_share":0.0002723311546840959,"domain":"推理","name":"OlympicArena","organization_count":1,"organizations":["opencompass_hub"],"rank":2367,"released":"2024-06-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OlympicArena"},{"aliases":[],"benchmark_id":"claire-radar-2607-27155","caveat":"Evaluates LLM agents on long-horizon office-suite tasks from 100 practitioner-derived scenarios, scoring deliverable quality through code-based verifiers and comparing performance against economic signals of human labor time and price.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27155"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmegaUse-OfficeVal","organization_count":1,"organizations":["claire_radar"],"rank":2368,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27155"},{"aliases":[],"benchmark_id":"claire-radar-2609-23435","caveat":"The proposed benchmark would consist of 1,160 expert-validated questions covering DNA regulation, RNA processing, and protein function.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23435"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"OmicsBench","organization_count":1,"organizations":["claire_radar"],"rank":2369,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23435"},{"aliases":["LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It"],"benchmark_id":"claire-radar-2608-31016","caveat":"Assesses whether automated judges detect omissions in AI-generated clinical notes using 500 pairs of clean and single-error notes scored on paired discrimination and false-alarm rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.31016"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"OmissionBench","organization_count":1,"organizations":["claire_radar"],"rank":2370,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.31016"},{"aliases":[],"benchmark_id":"opencompass-1244-omni-math","caveat":"Omni-MATH focuses exclusively on mathematics and comprises a vast collection of 4428 competition-level problems with rigorous human annotation. These problems are meticulously categorized into over 33 sub-domains and span more than 10 distinct difficulty levels Omni-MATH用于评估LLM在奥林匹克水平上的数学推理能力，包括4428道竞赛级问题，并带有严格的人工注释。这些问题被精心分类为超过33个子领域，涵盖10多个不同的难度级别。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Omni-MATH"],"document_share":0.0002723311546840959,"domain":"强推理","name":"Omni-MATH","organization_count":1,"organizations":["opencompass_hub"],"rank":2371,"released":"2024-10-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Omni-MATH"},{"aliases":[],"benchmark_id":"claire-radar-2609-37317","caveat":"Omnimodal evaluation should go beyond independent text, image, and speech production: individually plausible outputs may not express a coherent shared event.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37317"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Omni-StoryBench","organization_count":1,"organizations":["claire_radar"],"rank":2372,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37317"},{"aliases":[],"benchmark_id":"claire-radar-2609-23417","caveat":"The proposed benchmark would cover audiovisual website development with 918 bilingual instances involving edit steps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23417"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Omni2Web","organization_count":1,"organizations":["claire_radar"],"rank":2373,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23417"},{"aliases":[],"benchmark_id":"claire-radar-2607-14989","caveat":"The evaluation object is unclear from the provided information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14989"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniaBench","organization_count":1,"organizations":["claire_radar"],"rank":2374,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14989"},{"aliases":[],"benchmark_id":"opencompass-1553-omnialign-v","caveat":"OmniAlign-V datasets mainly focus on improving the alignment of Multi-modal Large Language Models(MLLMs) with human preference. It contains 205k high-quality Image-Quetion-Answer pairs with open-ended, creative quetions and long, knowledge-rich, comprehensive answers. OmniAlign-V 数据集主要关注提高多模态大型语言模型（MLLMs）与人类偏好的对齐。它包含 205k 个高质量的图像-问答对，包含开放式、创意性问题以及长篇、知识丰富、内容全面的答案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniAlign-V"],"document_share":0.0002723311546840959,"domain":"多模态","name":"OmniAlign-V","organization_count":1,"organizations":["opencompass_hub"],"rank":2375,"released":"2025-02-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OmniAlign-V"},{"aliases":[],"benchmark_id":"llm-stats-omnibench","caveat":"A novel multimodal benchmark designed to evaluate large language models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. Comprises 1,142 question-answer pairs covering 8 task categories from basic perception to complex inference, with a unique constraint that accurate responses require integrated understanding of all three modalities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnibench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OmniBench","organization_count":1,"organizations":["llm_stats"],"rank":2376,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1981-omnibench","caveat":"OmniBench is a multi-dimensional benchmark for virtual agents, designed to systematically evaluate ten core capabilities such as planning, decision-making, and instruction comprehension through automatically generated task graphs with controllable complexity. OmniBench 是一个面向虚拟智能体的多维度评测基准，旨在通过自动化流程生成具有可控复杂度的任务图，系统评估智能体在计划、决策、指令理解等十个核心能力上的表现。该基准包含 36,000 个图结构任务，覆盖 20 个真实场景，并引入 OmniEval 框架，实现子任务级别的细粒度评估，显著提升了评测的效率和可扩展性。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniBench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"OmniBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2377,"released":"2025-06-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OmniBench"},{"aliases":[],"benchmark_id":"llm-stats-omnibench-music","caveat":"Music component of OmniBench, a comprehensive benchmark for evaluating omni-language models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. The music category includes various compositions and performances that require integrated understanding across text, image, and audio modalities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnibench-music?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OmniBench Music","organization_count":1,"organizations":["llm_stats"],"rank":2378,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/omnibench-music?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-22637","caveat":"Evaluates VLMs on assembly-aware 3D spatial reasoning across 25k mechanical assemblies with component-level, relational, and tool-augmented tasks.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22637v1"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"OmniCAD","organization_count":1,"organizations":["claire_radar"],"rank":2379,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22637v1"},{"aliases":[],"benchmark_id":"claire-radar-2606-08572","caveat":"OmniCap-IF evaluates instruction following in omni-modal video captioning with 50 constraint types, 1,920 samples, and checklist-based scoring for format and content correctness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08572"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniCap-IF","organization_count":1,"organizations":["claire_radar"],"rank":2380,"released":"2026-06-07","source":"claire_radar","url":"https://arxiv.org/abs/2606.08572"},{"aliases":[],"benchmark_id":"claire-radar-2610-12458","caveat":"Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12458"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniCapBench","organization_count":1,"organizations":["claire_radar"],"rank":2381,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.12458"},{"aliases":[],"benchmark_id":"llm-stats-omnidocbench","caveat":"OmniDocBench evaluates multimodal models on document understanding tasks such as OCR, layout parsing, and structured document comprehension.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OmniDocBench","organization_count":1,"organizations":["llm_stats"],"rank":2382,"released":"2024-12-10","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1942-omnidocbench","caveat":"OmniDocBench is a comprehensive benchmark for evaluating document parsing in real-world scenarios. It includes 981 PDF pages across 9 document types, annotated with dense paragraph-level bboxes with text and attributes. Along with its designed evaluation methods, it provides Fine-grained results. OmniDocBench是一个用于评估真实场景下多样性文档解析效果的评测集，它包含了981个页面，覆盖9种文档类型（包括研报、教材、报纸、手写笔记、杂志等），具有段落级别的位置标注和内容标注，还有阅读顺序标注和属性标注，并开发了配套的评测方法，使其既具备单模块的评测能力（包括布局检测，公式识别，表格识别，文本识别），又具备端到端的评测能力，针对不同元素提供了分页面以及分属性的精细化评测结果，精准定位模型文档解析的痛点问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniDocBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"OmniDocBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2383,"released":"2024-12-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OmniDocBench"},{"aliases":[],"benchmark_id":"llm-stats-omnidocbench-1-5","caveat":"OmniDocBench 1.5 is a comprehensive benchmark for evaluating multimodal large language models on document understanding tasks, including OCR, document parsing, information extraction, and visual question answering across diverse document types. Lower Overall Edit Distance scores are better.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench-1.5?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OmniDocBench 1.5","organization_count":1,"organizations":["llm_stats"],"rank":2384,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/omnidocbench-1.5?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-23407","caveat":"The proposed benchmark would unify spatial audio-visual perception and audio-vision-language navigation tasks for embodied agents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23407"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"OmniEchoBench","organization_count":1,"organizations":["claire_radar"],"rank":2385,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23407"},{"aliases":[],"benchmark_id":"claire-radar-2608-05049","caveat":"OmniEdit-Bench evaluates instruction-based video editing with dimensions including spatial, temporal, audio, and reference-based editing, and assesses accuracy, preservation, realism, and consistency with an accuracy-aware penalty.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05049"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniEdit-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2386,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.05049"},{"aliases":[],"benchmark_id":"claire-radar-2606-00815","caveat":"OmniEEG-Bench is a unified benchmark for EEG foundation models, organizing evaluation into six task families and unifying 54 EEG datasets. It provides a leaderboard and code repository.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00815"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniEEG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2387,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00815"},{"aliases":[],"benchmark_id":"claire-radar-2607-08423","caveat":"A benchmark for evaluating Vision-Language Models on nutrient reasoning and personalized health advice, with progressive capabilities: basic perception, quantitative reasoning, and safety-critical advisory.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08423"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"OmniFood-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2388,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08423"},{"aliases":[],"benchmark_id":"llm-stats-omnigaia","caveat":"OmniGAIA evaluates multimodal perception and reasoning in agentic contexts, testing a model's ability to process diverse inputs and perform complex multi-step reasoning tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnigaia?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OmniGAIA","organization_count":1,"organizations":["llm_stats"],"rank":2389,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/omnigaia?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-09826","caveat":"OmniGameArena evaluates VLM game agents across twelve Unreal Engine 5 games spanning Solo, PvP, and Coop play. It provides unified action interfaces and two evaluation clocks (PDQ for decision quality, LCRT for real-time latency). The Improvement Dynamics Curve (IDC) measures agent improvement through reflection rounds.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09826"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniGameArena","organization_count":1,"organizations":["claire_radar"],"rank":2390,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09826"},{"aliases":[],"benchmark_id":"opencompass-1801-omnigirl","caveat":"A multi-modal, multi-language benchmark dataset for the GitHub Issue Resolution task. 一个面向 GitHub Issue ResoLution任务的多语言、多模态基准数据集，包含以下特点: 1.支持 Python、Java、JS、TS 四种主流编程语言，2. 输入信息涵盖文本、图像、网页等多种模态，3. 提供可复现的 Docker 评估环境。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniGIRL"],"document_share":0.0002723311546840959,"domain":"多模态","name":"OmniGIRL","organization_count":1,"organizations":["opencompass_hub"],"rank":2391,"released":"2025-05-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OmniGIRL"},{"aliases":[],"benchmark_id":"claire-radar-2608-18586","caveat":"Evaluates multimodal LLMs and OCR systems on handwritten text and mathematical expression recognition across six subtasks and twelve subsets with 77.57K images.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.18586"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniHandwritingOCR","organization_count":1,"organizations":["claire_radar"],"rank":2392,"released":"2026-08-19","source":"claire_radar","url":"https://arxiv.org/abs/2608.18586"},{"aliases":[],"benchmark_id":"claire-radar-2605-26485","caveat":"OmniInteract is a streaming benchmark for real-time omnimodal LLMs evaluated through native online inference over audio-visual streams. It contains 250 videos with 1,430 temporally grounded response slots (1Q1A and 1QnA), with each slot including trigger, response window, and target answer. Metrics include IA-QTF1, Interruption Diagnostic Suite, and Nested Chain Completion Score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26485"],"document_share":0.0002723311546840959,"domain":"Mathematics & Formal Science","name":"OmniInteract","organization_count":1,"organizations":["claire_radar"],"rank":2393,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26485"},{"aliases":[],"benchmark_id":"claire-radar-2607-03261","caveat":"OmniLayout evaluates language models on printed-circuit-board (PCB) layout placement reasoning under geometric, routing, and connectivity constraints, with four tasks including geometric placement, routability-aware placement, electrical functionality, and tool-augmented agentic reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03261"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"OmniLayout","organization_count":1,"organizations":["claire_radar"],"rank":2394,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.03261"},{"aliases":[],"benchmark_id":"claire-radar-2607-09068","caveat":"A visual question-answering benchmark of 2,096 manually annotated questions over 1,603 map documents in nine categories. It is scored by accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09068"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniMapBench","organization_count":1,"organizations":["claire_radar"],"rank":2395,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.09068"},{"aliases":[],"benchmark_id":"claire-radar-2605-29833","caveat":"OmniMatBench assesses multimodal reasoning in materials science across 19 subfields with 3,171 expert-curated QA and calculation problems, spanning four domains from fundamental knowledge to applied materials.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29833"],"document_share":0.0002723311546840959,"domain":"Materials & Chemistry","name":"OmniMatBench","organization_count":1,"organizations":["claire_radar"],"rank":2396,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29833"},{"aliases":[],"benchmark_id":"llm-stats-omnimath","caveat":"A Universal Olympiad Level Mathematic Benchmark for Large Language Models containing 4,428 competition-level problems with rigorous human annotation, categorized into over 33 sub-domains and spanning more than 10 distinct difficulty levels","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omnimath?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"OmniMath","organization_count":1,"organizations":["llm_stats"],"rank":2397,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/omnimath?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-05539","caveat":"OmniMech evaluates vision-language models on four tasks: CAD program synthesis from engineering drawings, diagram-to-3D reasoning, annotation-grounded reasoning, and tool-augmented agentic reasoning using industrial mechanical data with 251k drawings and associated CAD models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05539"],"document_share":0.0002723311546840959,"domain":"CAD","name":"OmniMech","organization_count":1,"organizations":["claire_radar"],"rank":2398,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05539"},{"aliases":[],"benchmark_id":"opencompass-2142-omnimmi","caveat":"OmniMMI is a cutting-edge benchmark for multi-modal interaction, specifically designed for OmniLLMs in streaming video environments. It includes 1,121 videos and 2,290 questions, tackling the challenges of streaming video understanding and proactive reasoning across six unique subtasks. OmniMMI is a cutting-edge benchmark for multi-modal interaction, specifically designed for OmniLLMs in streaming video environments. It includes 1,121 videos and 2,290 questions, tackling the challenges of streaming video understanding and proactive reasoning across six unique subtasks.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OmniMMI"],"document_share":0.0002723311546840959,"domain":"多模态","name":"OmniMMI","organization_count":1,"organizations":["opencompass_hub"],"rank":2399,"released":"2025-04-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OmniMMI"},{"aliases":[],"benchmark_id":"claire-radar-2607-04033","caveat":"A cross-domain benchmark for comparing optimizers in large-scale model training. It covers 24+ optimizers across two stages: Stage 1 sweeps on C4 with LLaMA-3 architectures (60M to 1B), and Stage 2 transfers to FineWeb-Edu with four architectures (Transformer++, GLA, DeltaNet, Gated DeltaNet) at 340M and 1B scales. Controlled-variable protocol with fixed architecture, data, and schedule settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04033"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniOpt","organization_count":1,"organizations":["claire_radar"],"rank":2400,"released":"2026-07-04","source":"claire_radar","url":"https://arxiv.org/abs/2607.04033"},{"aliases":[],"benchmark_id":"claire-radar-2607-25641","caveat":"OmniPhys is a benchmark of 1,551 text-to-image generation samples grounded in a Physical Knowledge Graph, aligned with PhET simulations and curricula. It evaluates physical commonsense in generated images using a dual-path verification protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25641"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"OmniPhys","organization_count":1,"organizations":["claire_radar"],"rank":2401,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25641"},{"aliases":[],"benchmark_id":"claire-radar-2608-25398","caveat":"Evaluates multimodal physics understanding, reasoning, and generation on 15,246 questions with 19,850 images from middle-school to university levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25398"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniPhys","organization_count":1,"organizations":["claire_radar"],"rank":2402,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25398"},{"aliases":[],"benchmark_id":"claire-radar-2609-39490","caveat":"Recent advances have enabled unified omni-modal models in understanding audio, vision, and language.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39490"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniReasoningBench","organization_count":1,"organizations":["claire_radar"],"rank":2403,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39490"},{"aliases":[],"benchmark_id":"claire-radar-2606-26947","caveat":"OmniRef-Bench is introduced in the paper but primarily serves to evaluate the proposed DyRef method. It is not presented as a standalone public benchmark.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26947"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniRef-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2404,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.26947"},{"aliases":[],"benchmark_id":"claire-radar-2605-26641","caveat":"A released 12-direction audio-video-text retrieval benchmark containing 3,782 triples and reporting aggregate retrieval performance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26641"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniRetriever-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2405,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26641"},{"aliases":[],"benchmark_id":"claire-radar-2608-04434","caveat":"OmniRouting evaluates LLMs on PCB routing reasoning under real-world constraints with 1,681 industrial designs across four tasks: geometric routing, design-rule-aware routing, electrical functionality, and tool-augmented agentic routing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04434"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"OmniRouting","organization_count":1,"organizations":["claire_radar"],"rank":2406,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04434"},{"aliases":[],"benchmark_id":"llm-stats-omniscience","caveat":"OmniScience is a broad scientific knowledge and reasoning benchmark that measures both answer accuracy and non-hallucination (calibrated abstention) across science domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omniscience?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"OmniScience","organization_count":1,"organizations":["llm_stats"],"rank":2407,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-omniscience-non-hallucination-rate","caveat":"OmniScience variant that reports the non-hallucination rate, defined as one minus the hallucination rate.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/omniscience-non-hallucination-rate?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"OmniScience (non-hallucination rate)","organization_count":1,"organizations":["llm_stats"],"rank":2408,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/omniscience-non-hallucination-rate?top_n=500"},{"aliases":["Re-engineering SORT-based algorithms for low-cost small object tracking from omnidirectional footage"],"benchmark_id":"claire-radar-2609-07547","caveat":"OmniSmall evaluates multi-object tracking of flying animals in omnidirectional video, focusing on seam-aware motion and association for small, fast-moving targets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07547"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"OmniSmall","organization_count":1,"organizations":["claire_radar"],"rank":2409,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07547"},{"aliases":[],"benchmark_id":"claire-radar-2609-32569","caveat":"Smart-home assistants are expected to handle diverse, realistic requests that arise in daily life.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32569"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniSmartHome","organization_count":1,"organizations":["claire_radar"],"rank":2410,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32569"},{"aliases":[],"benchmark_id":"claire-radar-2605-26322","caveat":"OmniToM evaluates theory of mind in LLMs by requiring explicit belief modeling, extracting belief propositions and labeling them with seven-dimensional schema labels across 895 stories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26322"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniToM","organization_count":1,"organizations":["claire_radar"],"rank":2411,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26322"},{"aliases":[],"benchmark_id":"claire-radar-2606-15749","caveat":"OmniTraffic is a controllable generation pipeline and benchmark for spatio-temporal traffic reasoning. It provides 8M VQA samples and a 3K human-verified test set across 12 reconstructed 3D intersections, with a three-level task hierarchy covering scene perception, multi-view and temporal reasoning, and decision support.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15749"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniTraffic","organization_count":1,"organizations":["claire_radar"],"rank":2412,"released":"2026-06-14","source":"claire_radar","url":"https://arxiv.org/abs/2606.15749"},{"aliases":[],"benchmark_id":"claire-radar-2609-22069","caveat":"Evaluates omni reference-to-video generation using a large-scale dataset.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.22069"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniVBench","organization_count":1,"organizations":["claire_radar"],"rank":2413,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.22069"},{"aliases":[],"benchmark_id":"claire-radar-2609-37773","caveat":"Artificial Intelligence Virtual Cells (AIVCs) are envisioned as scientific agents that simulate cellular responses, explain underlying mechanisms, and support hypothesis-driven discovery.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37773"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OmniVCBench","organization_count":1,"organizations":["claire_radar"],"rank":2414,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37773"},{"aliases":["OmniVideoBench","Omni Video-Bench"],"benchmark_id":"omnivideobench","caveat":"1,000 audio-visual reasoning questions over 628 videos (seconds to 30 minutes long), targeting synergistic audio-visual understanding in Omni MLLMs, by the NJU-LINK team; an ICLR 2026 poster. Video length and audio-video correlation vary widely, so per-model results depend heavily on which subset is sampled and on frame/audio chunking strategy.","document_count":1,"document_ids":["model_reports:omnivideobench_v3_results_table"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OmniVideoBench","organization_count":1,"organizations":["NJU-LINK (Nanjing University)"],"rank":2415,"released":"2025-10-12","source":"model_reports","url":"https://arxiv.org/abs/2510.10689"},{"aliases":["Towards One-to-Many Temporal Grounding"],"benchmark_id":"claire-radar-2606-06294","caveat":"OMTG Bench evaluates one-to-many temporal grounding, in which a textual query can correspond to multiple disjoint video segments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06294"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OMTG Bench","organization_count":1,"organizations":["claire_radar"],"rank":2416,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06294"},{"aliases":[],"benchmark_id":"claire-radar-2606-11144","caveat":"OncoTraj is a public benchmark for predicting acquired resistance to first-line osimertinib in EGFR-mutant non-small-cell lung cancer. It provides a harmonized dataset of 813 patients from three real-world sources, with locked splits, an evaluation harness, and six baselines, defining three tasks: 12-month progression classification, time-to-progression regression, and resistance mechanism classification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11144"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"OncoTraj","organization_count":1,"organizations":["claire_radar"],"rank":2417,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11144"},{"aliases":[],"benchmark_id":"claire-radar-2608-02615","caveat":"OncoTriad-QA is a patient-level benchmark for pan-cancer question answering, integrating radiology, pathology, genomics, and clinical metadata from TCGA. It includes 86.1k questions across 9,281 cases and 32 cancer cohorts, with annotations derived from curated labels and diagnostic reports.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02615"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"OncoTriad-QA","organization_count":1,"organizations":["claire_radar"],"rank":2418,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2608.02615"},{"aliases":["One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning"],"benchmark_id":"claire-radar-2609-05260","caveat":"The paper describes a real-robot benchmark evaluating language-conditioned embodied reasoning with instruction pairs, but the benchmark name is not formally declared in the abstract; it uses the phrase 'introduce One Word, Different Action' without explicit benchmark declaration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05260"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"One Word, Different Action","organization_count":1,"organizations":["claire_radar"],"rank":2419,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05260"},{"aliases":["ONE CYLinder: A Benchmark for Graph-Based Surrogate Modeling of Unsteady Bluff-Body Flows"],"benchmark_id":"claire-radar-2609-08947","caveat":"ONECYL evaluates graph-based surrogate models for unsteady flow past a circular cylinder across laminar, transitional, and high-Reynolds-number regimes using 450 high-fidelity simulations with randomized geometries, time-resolved fields, and mesh connectivity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08947"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ONECYL","organization_count":1,"organizations":["claire_radar"],"rank":2420,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08947"},{"aliases":[],"benchmark_id":"llm-stats-onemillion-bench","caveat":"OneMillion Bench evaluates AI agents on high-economic-value tasks that require sustained, reliable execution across long-horizon real-world workflows.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/onemillion-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"OneMillion Bench","organization_count":1,"organizations":["llm_stats"],"rank":2421,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/onemillion-bench?top_n=500"},{"aliases":["OneMillionBench","One Million Bench"],"benchmark_id":"onemillionbench","caveat":"Long-context recall and use; a \"with tools\" figure is not comparable to a no-tools run.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"long_context","name":"OneMillionBench","organization_count":1,"organizations":["Tencent"],"rank":2422,"released":null,"source":"model_reports","url":"https://github.com/humanlaya/OneMillion-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2608-02595","caveat":"onepot-Bench 0 is a proprietary benchmark suite evaluating language models on synthetic chemistry capabilities, including cheminformatics literacy, safety behavior, and reaction outcome prediction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02595"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"onepot-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2423,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02595"},{"aliases":[],"benchmark_id":"claire-radar-2609-08174","caveat":"A tiered biomedical retrieval benchmark with query-document pairs for concept grounding, relational retrieval, and compositional phenotype-based retrieval.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08174"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"OntologyBench","organization_count":1,"organizations":["claire_radar"],"rank":2424,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08174"},{"aliases":["OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning"],"benchmark_id":"claire-radar-2607-12523","caveat":"OOD-RL-Bench is an extensible framework for testing out-of-distribution detectors on anomalies injected into RL trajectories. It scores detectors with matched-time AUROC, AUPRC, false-positive rate, and detection delay.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.12523"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OOD-RL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2425,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.12523"},{"aliases":[],"benchmark_id":"claire-radar-2606-06481","caveat":"OpAI-Bench evaluates AI-text detection across document, sentence, token, and span granularities using operation-guided progressive human-to-AI revision trajectories with nine versions per sample.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06481"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OpAI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2426,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06481"},{"aliases":[],"benchmark_id":"llm-stats-open-rewrite","caveat":"OpenRewriteEval is a benchmark for evaluating open-ended rewriting of long-form texts, covering a wide variety of rewriting types expressed through natural language instructions including formality, expansion, conciseness, paraphrasing, and tone and style transfer.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/open-rewrite?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"Open-rewrite","organization_count":1,"organizations":["llm_stats"],"rank":2427,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/open-rewrite?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-openai-mmlu","caveat":"MMLU (Massive Multitask Language Understanding) is a comprehensive benchmark that measures a text model's multitask accuracy across 57 diverse academic and professional subjects. The test covers elementary mathematics, US history, computer science, law, morality, business ethics, clinical knowledge, and many other domains spanning STEM, humanities, social sciences, and professional fields. To attain high accuracy, models must possess extensive world knowledge and problem-solving ability.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-mmlu?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"OpenAI MMLU","organization_count":1,"organizations":["llm_stats"],"rank":2428,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mmlu?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-openai-mrcr-2-needle-128k","caveat":"Multi-round Co-reference Resolution (MRCR) benchmark for evaluating an LLM's ability to distinguish between multiple needles hidden in long context. Models are given a long, multi-turn synthetic conversation and must retrieve a specific instance of a repeated request, requiring reasoning and disambiguation skills beyond simple retrieval.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"OpenAI-MRCR: 2 needle 128k","organization_count":1,"organizations":["llm_stats"],"rank":2429,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-128k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-openai-mrcr-2-needle-1m","caveat":"Multi-Round Co-reference Resolution benchmark that tests an LLM's ability to distinguish between multiple similar needles hidden in long conversations. Models must reproduce specific instances of content (e.g., 'Return the 2nd poem about tapirs') from multi-turn synthetic conversations, requiring reasoning about context, ordering, and subtle differences between similar outputs.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-1m?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"OpenAI-MRCR: 2 needle 1M","organization_count":1,"organizations":["llm_stats"],"rank":2430,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-1m?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-openai-mrcr-2-needle-256k","caveat":"Multi-Round Co-reference Resolution (MRCR) benchmark that tests long-context reasoning by evaluating a model's ability to distinguish between similar outputs, reason about ordering, and reproduce specific content from multi-turn conversations containing multiple writing requests on overlapping topics at 256k tokens.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-256k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"OpenAI-MRCR: 2 needle 256k","organization_count":1,"organizations":["llm_stats"],"rank":2431,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-mrcr%3A-2-needle-256k?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-github-crispyberry-openbenchmark","caveat":"Evaluates AI agents and models on user-defined tasks generated from imported agent trajectories, scoring correctness, cost, and speed.","document_count":1,"document_ids":["claire_radar:https://github.com/crispyberry/OpenBenchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"OpenBenchmark","organization_count":1,"organizations":["claire_radar"],"rank":2432,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/crispyberry/OpenBenchmark"},{"aliases":[],"benchmark_id":"llm-stats-openbookqa","caveat":"OpenBookQA is a question-answering dataset modeled after open book exams for assessing human understanding. It contains 5,957 multiple-choice elementary-level science questions that probe understanding of 1,326 core science facts and their application to novel situations, requiring combination of open book facts with broad common knowledge through multi-hop reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openbookqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"OpenBookQA","organization_count":1,"organizations":["llm_stats"],"rank":2433,"released":"2018-09-08","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/openbookqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-518-openbookqa","caveat":"OpenBookQA contains questions that require multi-step reasoning, application of common-sense knowledge, and in-depth comprehension of text. It is a new type of question-answering dataset, modeled after open-book exams, designed to assess human understanding of a specific topic. OpenBookQA包含需要多步推理、运用常识知识、深入理解文本等能力的问题，是一种新型的问答数据集，其模式借鉴了开放式书本考试，用于评估人类对某一主题理解的程度。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OpenbookQA"],"document_share":0.0002723311546840959,"domain":"理解","name":"OpenbookQA","organization_count":1,"organizations":["opencompass_hub"],"rank":2434,"released":"2018-09-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OpenbookQA"},{"aliases":[],"benchmark_id":"opencompass-631-openfindata","caveat":"OpenFinData is an open-source financial evaluation dataset jointly released by EastMoney and Shanghai Artificial Intelligence Laboratory. This dataset represents the most realistic industrial scenario requirements and is currently the most comprehensive and professional financial evaluation dataset. OpenFinData是由东方财富与上海人工智能实验室联合发布的开源金融评测数据集。该数据集代表了最真实的产业场景需求，是目前场景最全、专业性最深的金融评测数据集。它基于东方财富实际金融业务的多样化丰富场景，旨在为金融科技领域的研究者和开发者提供一个高质量的数据资源。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OpenFinData"],"document_share":0.0002723311546840959,"domain":"知识","name":"OpenFinData","organization_count":1,"organizations":["opencompass_hub"],"rank":2435,"released":"2023-12-29","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OpenFinData"},{"aliases":[],"benchmark_id":"claire-radar-2606-06959","caveat":"OpenHalDet is a unified benchmark for hallucination detection across 17 datasets, supporting black-box, gray-box, and white-box detectors with standardized pipelines, scoring via AUROC and Cost@N.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06959"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"OpenHalDet","organization_count":1,"organizations":["claire_radar"],"rank":2436,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.06959"},{"aliases":[],"benchmark_id":"claire-radar-2608-16022","caveat":"OpenHarmony Bench evaluates coding agents on 153 app-level ArkTS tasks across three input sources, with 242 feature points and device-based verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16022"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"OpenHarmony Bench","organization_count":1,"organizations":["claire_radar"],"rank":2437,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16022"},{"aliases":[],"benchmark_id":"claire-radar-2610-01497","caveat":"Molecular tumor boards integrate genomic findings, clinical context, and therapeutic evidence to support precision oncology.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.01497"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"OpenMTB-Audit","organization_count":1,"organizations":["claire_radar"],"rank":2438,"released":"2026-10-01","source":"claire_radar","url":"https://arxiv.org/abs/2610.01497"},{"aliases":[],"benchmark_id":"llm-stats-openrca","caveat":"OpenRCA is a benchmark for evaluating AI models on root cause analysis tasks. For each failure case, the model receives 1 point if all generated root-cause elements match the ground-truth ones, and 0 points if any mismatch is identified. The overall accuracy is the average score across all failure cases.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openrca?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"OpenRCA","organization_count":1,"organizations":["llm_stats"],"rank":2439,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/openrca?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-19108","caveat":"OpenRTAG evaluates text-attributed graph learning under nine degradation scenarios (sparsity, noise, imbalance) across nine TAG datasets and three downstream tasks. It provides a standardized testbed for robustness comparison.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19108"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OpenRTAG","organization_count":1,"organizations":["claire_radar"],"rank":2440,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.19108"},{"aliases":[],"benchmark_id":"claire-radar-2607-02047","caveat":"OpenSafeIntent is a benchmark of controlled prompt-sets that vary user intent while holding the underlying task fixed. Each datapoint contains benign, dual-use, and malicious variants of the same task, and models are evaluated on whether they calibrate assistance across intent shifts rather than only appearing safe on average.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02047"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OpenSafeIntent","organization_count":1,"organizations":["claire_radar"],"rank":2441,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02047"},{"aliases":[],"benchmark_id":"claire-radar-2607-28692","caveat":"OpenSciToolBench is a benchmark with 900 tasks across four difficulty levels for evaluating LLM agents in open-world scientific tool acquisition.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28692"],"document_share":0.0002723311546840959,"domain":"Agents","name":"OpenSciToolBench","organization_count":1,"organizations":["claire_radar"],"rank":2442,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28692"},{"aliases":[],"benchmark_id":"claire-radar-2607-20121","caveat":"OpenSkillRisk evaluates LLM-based agents on their ability to recognize and avoid safety risks when using third-party skills, with 263 risky skills in seven threat categories and sandboxed execution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20121"],"document_share":0.0002723311546840959,"domain":"Agents","name":"OpenSkillRisk","organization_count":1,"organizations":["claire_radar"],"rank":2443,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.20121"},{"aliases":[],"benchmark_id":"claire-radar-2609-32810","caveat":"Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions, yet benchmarks rarely capture these real-world trajectories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32810"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"OpenTumorBoard","organization_count":1,"organizations":["claire_radar"],"rank":2444,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32810"},{"aliases":[],"benchmark_id":"opencompass-1754-openturingbench","caveat":"OpenTuringBench, a new benchmark based on OLLMs, designed to train and evaluate machine-generated text detectors on the Turing Test and Authorship Attribution problems. OpenTuringBench，这是一个基于OLLMs的新基准测试，旨在训练和评估机器生成文本检测器在图灵测试和作者归属问题上的性能。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OpenTuringBench"],"document_share":0.0002723311546840959,"domain":"其他","name":"OpenTuringBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2445,"released":"2025-04-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OpenTuringBench"},{"aliases":[],"benchmark_id":"opencompass-1979-openunlearning","caveat":"OpenUnlearning is an efficient and modular benchmark platform designed to support and drive research on \"unlearning\" in large language models (LLMs). OpenUnlearning 是一个高效且模块化的基准平台，旨在支持和推动大型语言模型（LLM）中的“遗忘”（unlearning）研究。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OpenUnlearning"],"document_share":0.0002723311546840959,"domain":"其他","name":"OpenUnlearning","organization_count":1,"organizations":["opencompass_hub"],"rank":2446,"released":"2025-06-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OpenUnlearning"},{"aliases":[],"benchmark_id":"claire-radar-2610-07922","caveat":"World-action models (WAMs) couple future prediction with robot control, yet existing systems often vary the video backbone, interaction structure, supervision, and inference procedure simultaneously, making their design choices difficult t…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.07922"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"OpenWAM","organization_count":1,"organizations":["claire_radar"],"rank":2447,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.07922"},{"aliases":[],"benchmark_id":"claire-radar-2609-35052","caveat":"Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories, video reference, or selected revisit viewpoints that can confound the assessment of a model's tr…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35052"],"document_share":0.0002723311546840959,"domain":"Logistics & Operations","name":"OPIS","organization_count":1,"organizations":["claire_radar"],"rank":2448,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35052"},{"aliases":[],"benchmark_id":"opencompass-1990-opt-bench","caveat":"OPT-BENCH is a comprehensive benchmark designed to evaluate large language model (LLM) agents on large-scale search space optimization problems, focusing on their iterative reasoning and problem-solving capabilities. OPT-BENCH 是一个面向大型语言模型（LLM）智能体的大规模搜索空间优化评测基准，旨在系统评估模型在迭代推理和解决复杂优化问题中的能力。 该基准包含 30 个任务，包括 20 个来自 Kaggle 的真实机器学习任务和 10 个经典 NP 问题，涵盖预测建模、图论和组合优化等领域。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OPT-BENCH"],"document_share":0.0002723311546840959,"domain":"智能体","name":"OPT-BENCH","organization_count":1,"organizations":["opencompass_hub"],"rank":2449,"released":"2025-06-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OPT-BENCH"},{"aliases":[],"benchmark_id":"claire-radar-2607-10768","caveat":"Opti-Agent-Bench evaluates LLM agents on the end-to-end optimization R&D pipeline, from business-language description to mathematical modeling, algorithm selection, code implementation, and report generation, with modular evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10768"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Opti-Agent-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2450,"released":"2026-07-12","source":"claire_radar","url":"https://arxiv.org/abs/2607.10768"},{"aliases":[],"benchmark_id":"claire-radar-2609-32227","caveat":"Static QA and code-generation benchmarks only partially capture the role that large language models (LLMs) now play as coding agents and research tools.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32227"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OptiArena","organization_count":1,"organizations":["claire_radar"],"rank":2451,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32227"},{"aliases":[],"benchmark_id":"llm-stats-community-64d67847-06bd-423a-923c-c2acfab82281","caveat":null,"document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3A64d67847-06bd-423a-923c-c2acfab82281?top_n=500"],"document_share":0.0002723311546840959,"domain":"other","name":"OptimBench","organization_count":1,"organizations":["llm_stats"],"rank":2452,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3A64d67847-06bd-423a-923c-c2acfab82281?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-23055","caveat":"A controlled comparison of seven optimizers across four diffusion formulations, including masked, uniform, and Gaussian diffusion tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23055"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Optimizers for Diffusion Models: A Controlled Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2453,"released":"2026-09-19","source":"claire_radar","url":"https://arxiv.org/abs/2609.23055"},{"aliases":[],"benchmark_id":"claire-radar-2610-05715","caveat":"Vision-language models (VLMs) can detect that an object has rotated across views, but cannot reliably tell by how much.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05715"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2454,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.05715"},{"aliases":[],"benchmark_id":"opencompass-2086-or-bench","caveat":"OR Bench is a large-scale benchmark designed to evaluate the excessive rejection behavior of large language models. OR-Bench 是一个旨在评估大型语言模型过度拒绝行为的大规模基准。它衡量LLM在过度拒绝和有害提示拒绝方面的表现，涵盖暴力、隐私等10个类别。基准包含8万个、1千个困难及6百个有害提示，通过Mixtral等工具自动化生成。它为未来安全对齐研究提供强大测试平台。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OR-Bench"],"document_share":0.0002723311546840959,"domain":"理解","name":"OR-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2455,"released":"2024-05-31","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OR-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2605-28158","caveat":"A full-lifecycle workspace benchmark for industrial optimization agents, evaluating model construction, revision, and grounded explanation across three task modes (Build, Revise, Explain) using executable multi-file workspaces with task-specific evaluators.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28158"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OR-Space","organization_count":1,"organizations":["claire_radar"],"rank":2456,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28158"},{"aliases":[],"benchmark_id":"claire-radar-github-cl1110goo-afk-oraclephys","caveat":"Evaluates LLM ranking of stories by inter-story drift and identification of the governing story for multi-story 2-D steel frames, with scoring by top-1 accuracy and Spearman correlation against oracle truth.","document_count":1,"document_ids":["claire_radar:https://github.com/cl1110goo-afk/oraclephys"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"OraclePhys","organization_count":1,"organizations":["claire_radar"],"rank":2457,"released":"2026-08-23","source":"claire_radar","url":"https://github.com/cl1110goo-afk/oraclephys"},{"aliases":[],"benchmark_id":"claire-radar-2606-19787","caveat":"ORAgentBench evaluates autonomous agents on end-to-end operations research tasks. Each task involves a natural-language brief, multi-file data, configuration artifacts, and a required submission schema. Agents write and run solution code, validated for schema, hard constraints, and objective quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19787"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"ORAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":2458,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.19787"},{"aliases":[],"benchmark_id":"opencompass-1917-orak","caveat":"Orak (오락) is a foundational benchmark for evaluating Large Language Model (LLM) agents in diverse popular video games. Orak是一个基础性的基准,用于评估在各种流行视频游戏中的大型语言模型(LLM)代理。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Orak"],"document_share":0.0002723311546840959,"domain":"智能体","name":"Orak","organization_count":1,"organizations":["opencompass_hub"],"rank":2459,"released":"2025-06-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Orak"},{"aliases":[],"benchmark_id":"claire-radar-2609-30749","caveat":"Evaluates LLMs on data science code translation across curated tasks and project-level translations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30749"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ORCA","organization_count":1,"organizations":["claire_radar"],"rank":2460,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.30749"},{"aliases":[],"benchmark_id":"claire-radar-2607-28545","caveat":"ORCA-bench evaluates language model agents on root cause analysis in a production-fidelity oncall setting, using a live OpenTelemetry-instrumented microservice system with 1,079 tasks varying in report specificity, time-to-detection, and fault co-occurrence. Agents access metrics, logs, traces, and source code via real telemetry interfaces.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28545"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"ORCA-bench","organization_count":1,"organizations":["claire_radar"],"rank":2461,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28545"},{"aliases":[],"benchmark_id":"claire-radar-2607-25656","caveat":"OrchBench evaluates multi-agent orchestration plans in isolation using deterministic simulation. It constructs DAGs from real-world tasks and scores plans on result quality, makespan, and token cost without executing worker agents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25656"],"document_share":0.0002723311546840959,"domain":"Agents","name":"OrchBench","organization_count":1,"organizations":["claire_radar"],"rank":2462,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25656"},{"aliases":[],"benchmark_id":"claire-radar-2606-15034","caveat":"OSGuard is a dual-granularity benchmark suite for evaluating safety in computer-use agents, with an action-level benchmark for local guardrail decisions and a risk-augmented execution suite for end-to-end evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15034"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OSGuard","organization_count":1,"organizations":["claire_radar"],"rank":2463,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2606.15034"},{"aliases":[],"benchmark_id":"opencompass-2005-oss-bench","caveat":"OSS-Bench replaces functions with LLM-generated code and evaluates them using three natural metrics: compilability, functional correctness, and memory safety, leveraging robust signals like compilation failures, test-suite violations, and sanitizer alerts as ground truth. OSS-Bench，这是一个基准生成器，它可以从真实的开源软件中自动构建大规模的实时评估任务。OSS-Bench 将函数替换为 LLM 生成的代码，并使用三个自然指标（可编译性、功能正确性和内存安全性）对其进行评估，并利用编译失败、测试套件违规和Sanitizer警报等稳健信号作为基准事实。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/OSS-Bench"],"document_share":0.0002723311546840959,"domain":"代码","name":"OSS-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2464,"released":"2025-06-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/OSS-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2609-15128","caveat":"A diagnostic benchmark for omni-streaming thinking that fixes video while editing audio across conditions including agreement, absence, contradiction, coexistence, and subtitle-speech conflict.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15128"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OST-DiagBench","organization_count":1,"organizations":["claire_radar"],"rank":2465,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15128"},{"aliases":[],"benchmark_id":"llm-stats-osworld","caveat":"OSWorld: The first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS with 369 computer tasks involving real web and desktop applications, OS file I/O, and multi-application workflows","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OSWorld","organization_count":1,"organizations":["llm_stats"],"rank":2466,"released":"2024-04-11","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-29537","caveat":"Evaluates computer-use agents on 108 long-horizon real-world workflows across everyday and professional tasks, scored by binary completion at 500 steps and partial scores.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29537"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"OSWorld 2.0","organization_count":1,"organizations":["claire_radar"],"rank":2467,"released":"2026-06-28","source":"claire_radar","url":"https://arxiv.org/abs/2606.29537"},{"aliases":[],"benchmark_id":"llm-stats-osworld-2-0","caveat":"OSWorld 2.0 is a benchmark of 108 long-horizon, real-world computer-use workflows spanning everyday and professional tasks. Each task is an end-to-end workflow that takes human users a median of about 1.6 hours, scored with a binary-completion metric, and targets challenges such as dynamic environments, cross-source reasoning, and implicit-state inference.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-2.0?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OSWorld 2.0","organization_count":1,"organizations":["llm_stats"],"rank":2468,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-2.0?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-osworld-extended","caveat":"OSWorld is a scalable, real computer environment benchmark for evaluating multimodal agents on open-ended tasks across Ubuntu, Windows, and macOS. It comprises 369 computer tasks involving real web and desktop applications, OS file I/O, and multi-application workflows. The benchmark evaluates agents' ability to interact with computer interfaces using screenshots and actions in realistic computing environments.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-extended?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OSWorld Extended","organization_count":1,"organizations":["llm_stats"],"rank":2469,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-extended?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-osworld-screenshot-only","caveat":"OSWorld Screenshot-only: A variant of the OSWorld benchmark that evaluates multimodal AI agents using only screenshot observations to complete open-ended computer tasks across real operating systems (Ubuntu, Windows, macOS). Tests agents' ability to perform complex workflows involving web apps, desktop applications, file I/O, and multi-application tasks through visual interface understanding and GUI grounding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-screenshot-only?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OSWorld Screenshot-only","organization_count":1,"organizations":["llm_stats"],"rank":2470,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-screenshot-only?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-osworld-g","caveat":"OSWorld-G (Grounding) evaluates screenshot grounding accuracy for OS automation tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-g?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OSWorld-G","organization_count":1,"organizations":["llm_stats"],"rank":2471,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-g?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-39903","caveat":"Scientific software presents a demanding test for computer-using agents based on visual language models (VLMs): completing a research workflow requires interpreting specialized interfaces, manipulating scientific objects, and producing ver…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39903"],"document_share":0.0002723311546840959,"domain":"Agents","name":"OSWorld-Science","organization_count":1,"organizations":["claire_radar"],"rank":2472,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39903"},{"aliases":[],"benchmark_id":"llm-stats-osworld-verified","caveat":"OSWorld-Verified is a verified subset of OSWorld, a scalable real computer environment for multimodal agents supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/osworld-verified?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OSWorld-Verified","organization_count":1,"organizations":["llm_stats"],"rank":2473,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/osworld-verified?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-12552","caveat":"OV-SGG-Bench measures open-vocabulary relation prediction across six axes scored over datasets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12552"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OV-SGG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2474,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12552"},{"aliases":[],"benchmark_id":"claire-radar-2608-17110","caveat":"OV3D-Bench is a diagnostic benchmark for open-vocabulary monocular 3D detection, evaluating localization, semantic robustness, and cross-domain transfer across seven datasets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17110"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OV3D-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2475,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.17110"},{"aliases":[],"benchmark_id":"llm-stats-ovbench","caveat":"OVBench is an online video understanding benchmark that evaluates a model's ability to perceive, memorize, and reason about real-time video streams as they unfold.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ovbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OVBench","organization_count":1,"organizations":["llm_stats"],"rank":2476,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ovbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-27278","caveat":"Evaluates open-vocabulary Earth observation models on category breadth and query diversity, covering mask and box localization across vocabulary, referring, and reasoning queries under a unified zero-shot protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27278"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OVEarth-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2477,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27278"},{"aliases":[],"benchmark_id":"claire-radar-2609-20812","caveat":"OverclaimBench evaluates frontier LLM agents across five file-review scenarios using transcript-based coverage measurements to detect overclaiming against registered planted defects.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.20812"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"OverclaimBench","organization_count":1,"organizations":["claire_radar"],"rank":2478,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.20812"},{"aliases":[],"benchmark_id":"claire-radar-github-kcelestinomaria-overlooked-bench","caveat":"Evaluates AI models on 91 items across five tracks: ethics and philosophy, niche academic domains, small-organisation tasks, education, and matched institutional-criticism prompts, using rubric-based scoring and cross-judge validation.","document_count":1,"document_ids":["claire_radar:https://github.com/kcelestinomaria/overlooked-bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"overlooked-bench","organization_count":1,"organizations":["claire_radar"],"rank":2479,"released":"2026-09-07","source":"claire_radar","url":"https://github.com/kcelestinomaria/overlooked-bench"},{"aliases":[],"benchmark_id":"claire-radar-2608-22279","caveat":"Evaluates VLMs on online video question answering under user interruptions, with offline simulation and multi-dimensional metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.22279"],"document_share":0.0002723311546840959,"domain":"General AI","name":"OVIBench","organization_count":1,"organizations":["claire_radar"],"rank":2480,"released":"2026-08-23","source":"claire_radar","url":"https://arxiv.org/abs/2608.22279"},{"aliases":[],"benchmark_id":"claire-radar-2606-03890","caveat":"OVO-S-Bench evaluates streaming spatial intelligence in MLLMs with 1,680 human-annotated questions across four abstraction levels, using streaming prefixes and evidence intervals.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03890"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"OVO-S-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2481,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03890"},{"aliases":[],"benchmark_id":"llm-stats-ovobench","caveat":"OVOBench (Online Video Online Benchmark) evaluates streaming video understanding, testing a model's ability to perceive and respond to video content in real time as it unfolds.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ovobench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"OVOBench","organization_count":1,"organizations":["llm_stats"],"rank":2482,"released":"2025-01-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ovobench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1283-p-mmeval","caveat":"P-MMEval is a comprehensive multilingual multitask benchmark, covering effective fundamental and capability-specialized datasets. P-MMEval是一个全面的多语言多任务基准，涵盖了高效的基础和专项能力数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/P-MMEval"],"document_share":0.0002723311546840959,"domain":"语言","name":"P-MMEval","organization_count":1,"organizations":["opencompass_hub"],"rank":2483,"released":"2024-11-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/P-MMEval"},{"aliases":[],"benchmark_id":"claire-radar-2606-16753","caveat":"P3B3 is an expert-curated benchmark of conversational prompts for evaluating variety bias and controllability in LLMs across European and Brazilian Portuguese.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16753"],"document_share":0.0002723311546840959,"domain":"General AI","name":"P3B3","organization_count":1,"organizations":["claire_radar"],"rank":2484,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16753"},{"aliases":[],"benchmark_id":"claire-radar-2606-11152","caveat":"P3D-Bench evaluates multimodal large language models on parametric 3D generation from text, image, and assembly specifications, scoring executability, geometric fidelity, topology, text-grounded constraints, multiview semantic alignment, and part-level structure.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11152"],"document_share":0.0002723311546840959,"domain":"General AI","name":"P3D-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2485,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11152"},{"aliases":[],"benchmark_id":"claire-radar-2608-14441","caveat":"PACE-Bench is a simulator-grounded benchmark with 144 source-to-target adaptation pairs across six physics domains. Each pair presents a code-driven design that succeeds in a source environment but fails in a mutated target environment, and agents must iteratively adapt the design using diagnostic sandbox feedback within a limited attempt budget.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14441"],"document_share":0.0002723311546840959,"domain":"Agents","name":"PACE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2486,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14441"},{"aliases":["PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?"],"benchmark_id":"claire-radar-2609-18605","caveat":"PACT evaluates LLM assistants' compliance with standing rules under nine realistic pressures across 48 scenarios and 12 regulated domains, scoring multi-turn decisions with PACTScore and six axes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18605"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PACT","organization_count":1,"organizations":["claire_radar"],"rank":2487,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18605"},{"aliases":[],"benchmark_id":"claire-radar-2606-15144","caveat":"PACUTE is a diagnostic benchmark of 4,600 tasks evaluating morphological understanding in Filipino, covering six compositional levels from morpheme decomposition to syllabification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15144"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PACUTE","organization_count":1,"organizations":["claire_radar"],"rank":2488,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2606.15144"},{"aliases":[],"benchmark_id":"claire-radar-2606-00188","caveat":"PaintBench is a procedurally generated benchmark for precise visual editing, covering 20 tasks across geometric, structural, color, and symbolic categories. Stable scoring uses pixel-level mIoU with fixed seeds.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00188"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"PaintBench","organization_count":1,"organizations":["claire_radar"],"rank":2489,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.00188"},{"aliases":[],"benchmark_id":"claire-radar-2609-34195","caveat":"Figural divergent thinking is the ability to develop a given shape fragment into an original drawing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34195"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PainterBench","organization_count":1,"organizations":["claire_radar"],"rank":2490,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34195"},{"aliases":[],"benchmark_id":"claire-radar-2607-01360","caveat":"PAIR-Bench evaluates code improvement by transforming incorrect programs into more correct ones through feedback-guided refinement. It uses progressive hinting with failure-region and hint-depth controls to measure repair of targeted failures, generalization beyond the hint, preservation of correct behavior, and required assistance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01360"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"PAIR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2491,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.01360"},{"aliases":["Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark"],"benchmark_id":"claire-radar-2608-30065","caveat":"Pak3H evaluates Helpfulness, Harmlessness, and Honesty alignment in Urdu through three suites: PakAlpaca, PakBeaverTails, and PakTruthfulQA.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30065"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Pak3H","organization_count":1,"organizations":["claire_radar"],"rank":2492,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.30065"},{"aliases":[],"benchmark_id":"claire-radar-2606-16175","caveat":"PAL-Bench evaluates evidence-grounded profile reconstruction from longitudinal personal albums, scoring agents on owner facts, identities, and relations with a seven-metric protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16175"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"PAL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2493,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16175"},{"aliases":["Pancasila-Dilemmas: Evaluating Large Language Models on Indonesian Human Value Dilemmas Grounded in Pancasila"],"benchmark_id":"claire-radar-2607-18066","caveat":"An evaluation dataset of 1,834 dilemma questions from Indonesian news that measures LLM alignment with the five Pancasila values, scored by PMS and MVAS against answers from Indonesian annotators.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18066"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Pancasila-Dilemmas","organization_count":1,"organizations":["claire_radar"],"rank":2494,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.18066"},{"aliases":["onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction"],"benchmark_id":"claire-radar-2609-24983","caveat":"A dataset and benchmark for token-level correction in LLM and agent alignment data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24983"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Panda-CVL","organization_count":1,"organizations":["claire_radar"],"rank":2495,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24983"},{"aliases":[],"benchmark_id":"llm-stats-paperbench","caveat":"PaperBench is a benchmark for evaluating AI agents on their ability to replicate research papers. It tests models on complex, multi-step workflows involving code implementation, experimentation, and reproducing scientific results from academic publications.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/paperbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"PaperBench","organization_count":1,"organizations":["llm_stats"],"rank":2496,"released":"2025-04-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/paperbench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1730-paperbench","caveat":"PaperBench is a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research. PaperBench，这是一个评估AI代理复制最新AI研究能力的基准测试。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PaperBench"],"document_share":0.0002723311546840959,"domain":"理解","name":"PaperBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2497,"released":"2025-04-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PaperBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-22689","caveat":"ParaGUIBench aims to benchmark parallel execution and coordination of multiple GUI agents across separate desktop instances, with a dataset of 233 tasks and efficiency metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22689"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"ParaGUIBench","organization_count":1,"organizations":["claire_radar"],"rank":2498,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.22689"},{"aliases":[],"benchmark_id":"claire-radar-2608-03071","caveat":"ParamBench is a benchmark for evaluating LLM tool call parameter generation, built from real cloud-network APIs with difficulty tiers and exact match metrics. It is used to evaluate the proposed probe-guided training framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03071"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ParamBench","organization_count":1,"organizations":["claire_radar"],"rank":2499,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03071"},{"aliases":[],"benchmark_id":"claire-radar-2610-08662","caveat":"As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.08662"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"ParanoiaEval","organization_count":1,"organizations":["claire_radar"],"rank":2500,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.08662"},{"aliases":[],"benchmark_id":"claire-radar-2606-24648","caveat":"ParaPairAudioBench evaluates LALMs as judges for paralinguistic speech across five dimensions with 5,175 audio pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24648"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ParaPairAudioBench","organization_count":1,"organizations":["claire_radar"],"rank":2501,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24648"},{"aliases":[],"benchmark_id":"claire-radar-2609-12345","caveat":"ParaRecover evaluates LLM-based parallel tool-use agents on error localization and recovery, testing whether agents detect execution errors, diagnose root causes, and repair plans when tool calls fail. It scores replanner output across structural, diagnostic, and evolutionary dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12345"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ParaRecover","organization_count":1,"organizations":["claire_radar"],"rank":2502,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12345"},{"aliases":[],"benchmark_id":"claire-radar-2607-23225","caveat":"ParasGB evaluates graph neural network predictions of parasitic capacitance and resistance on circuit graphs from analog/mixed-signal designs, with node-level ground capacitance, edge-level resistance, and edge-level coupling capacitance tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23225"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"ParasGB","organization_count":1,"organizations":["claire_radar"],"rank":2503,"released":"2026-07-25","source":"claire_radar","url":"https://arxiv.org/abs/2607.23225"},{"aliases":["ParsHate: A Benchmark Dataset for Hate and Target Detection in Persian"],"benchmark_id":"claire-radar-2609-16393","caveat":"Evaluates hate speech and target detection on 10,000 manually annotated Persian tweets spanning 2013-2022.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16393"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ParsHate","organization_count":1,"organizations":["claire_radar"],"rank":2504,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.16393"},{"aliases":["particle-event-quantum-benchmark"],"benchmark_id":"claire-radar-github-starswithboutaina-particle-event-quantum-benchmark","caveat":"Evaluates classical and future hybrid quantum classifiers on a synthetic particle-physics-inspired binary classification task with controlled overlap, noise, and imbalance; Phase 1 includes majority, logistic regression, and random forest baselines with held-out test set.","document_count":1,"document_ids":["claire_radar:https://github.com/starswithboutaina/particle-event-quantum-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Particle Event Classification with Classical and Hybrid Quantum Models: A Reproducible Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2505,"released":"2026-09-08","source":"claire_radar","url":"https://github.com/starswithboutaina/particle-event-quantum-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2606-02443","caveat":"PaSBench-Video is a 740-video benchmark for proactive safety warning with 481 risk and 259 no-risk videos across driving, healthcare, daily life, and industrial production. Annotations include frame-level risk onset and accident boundaries. Models must process video causally and output a warning that is temporally calibrated and content-correct.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02443"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PaSBench-Video","organization_count":1,"organizations":["claire_radar"],"rank":2506,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02443"},{"aliases":[],"benchmark_id":"opencompass-1836-pashtoocr","caveat":"PsOCR is a large-scale synthetic dataset for Optical Character Recognition in low-resource Pashto language. PsOCR is a large-scale synthetic dataset for Optical Character Recognition in low-resource Pashto language.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PashtoOCR"],"document_share":0.0002723311546840959,"domain":"多模态","name":"PashtoOCR","organization_count":1,"organizations":["opencompass_hub"],"rank":2507,"released":"2025-05-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PashtoOCR"},{"aliases":[],"benchmark_id":"claire-radar-2605-26978","caveat":"A dated automated screening benchmark for Pashto text-to-speech systems covering completion, transcription error, script fidelity, and audio-language identification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26978"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PashtoTTS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2508,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26978"},{"aliases":[],"benchmark_id":"claire-radar-2608-04003","caveat":"PAST-Bench evaluates recursive self-improvement in personal AI agents by testing whether retained experience improves performance on future tasks. It spans 26 scenarios and 204 episodes across memory, procedural reuse, information gathering, and update capabilities, with matched persistence on/off controls.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04003"],"document_share":0.0002723311546840959,"domain":"Agents","name":"PAST-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2509,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.04003"},{"aliases":[],"benchmark_id":"claire-radar-2609-04075","caveat":"Evaluates AI agents on realistic C/C++ vulnerability patching tasks where ground-truth fixes lie outside the crash stack, using vulnerability transplant and code mutations to reduce memorization and surface-level fixes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04075"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"PatchBench","organization_count":1,"organizations":["claire_radar"],"rank":2510,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.04075"},{"aliases":[],"benchmark_id":"claire-radar-2610-10276","caveat":"An LLM safety patch can pass a benchmark while still being a poor repair.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10276"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PatchBench","organization_count":1,"organizations":["claire_radar"],"rank":2511,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10276"},{"aliases":[],"benchmark_id":"claire-radar-2607-01597","caveat":"PatchFuseBench is a fixed-pool benchmark for evaluating repair candidate fusion, built from existing SWE-bench Verified, SWE-bench Multilingual, and Defects4J candidate patches. The benchmark pools candidate patches for 500 bugs on SWE-bench Verified, 300 on Multilingual, and 371 on Defects4J, and evaluates methods that fuse or select a final patch.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01597"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"PatchFuseBench","organization_count":1,"organizations":["claire_radar"],"rank":2512,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.01597"},{"aliases":[],"benchmark_id":"claire-radar-2607-19261","caveat":"PathAgentBench evaluates vision-language models on whole-slide pathology images across four capabilities: image-to-text matching, text-to-image retrieval, diagnostic-region localization, and multi-scale reasoning. It includes 1,822 TCGA WSIs and 17,135 diagnostic paths.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19261"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PathAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":2513,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.19261"},{"aliases":[],"benchmark_id":"claire-radar-2610-11329","caveat":"Pathology vision-language models (VLMs) have shown strong visual perception ability, but their robustness in the language domain remains poorly characterized.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11329"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PathLang","organization_count":1,"organizations":["claire_radar"],"rank":2514,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11329"},{"aliases":[],"benchmark_id":"llm-stats-pathmcqa","caveat":"PathMMU is a massive multimodal expert-level benchmark for understanding and reasoning in pathology, containing 33,428 multimodal multi-choice questions and 24,067 images validated by seven pathologists. It evaluates Large Multimodal Models (LMMs) performance on pathology tasks, with the top-performing model GPT-4V achieving only 49.8% zero-shot performance compared to 71.8% for human pathologists.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pathmcqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"PathMCQA","organization_count":1,"organizations":["llm_stats"],"rank":2515,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/pathmcqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-17607","caveat":"PathoArgus-Bench evaluates evidence-grounded visual reasoning in whole-slide pathology. It comprises 22,078 multiple-choice questions from 4,913 patients across 15 TCGA projects, testing availability, accessibility, use, and responsiveness of evidence under a fixed reader budget.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17607"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PathoArgus-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2516,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17607"},{"aliases":["PathReportEval: A Systematic Benchmark for Pathology Report Generation"],"benchmark_id":"claire-radar-2607-18448","caveat":"A standardized benchmark for generating pathology reports from whole-slide images. It covers TCGA, HistAI and REG 2025 with three foundation encoders and scores reports with the Clinical Report Quality Score (CRQS).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18448"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PathReportEval","organization_count":1,"organizations":["claire_radar"],"rank":2517,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.18448"},{"aliases":[],"benchmark_id":"claire-radar-2607-28318","caveat":"PathVU is a benchmark for fine-grained multiscale visual understanding in pathology, built from 23 public datasets with human-supervised labels and spatial annotations, covering region and slide fields of view, with 14 VQA-style tasks and 308k samples.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28318"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PathView-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2518,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28318"},{"aliases":[],"benchmark_id":"claire-radar-2607-25485","caveat":"A benchmark framework that evaluates patient-facing healthcare agents in simulated, tool-using conversations with patients, scored by an LLM jury against clinician-grounded criteria.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25485"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PatientAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":2519,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25485"},{"aliases":["Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs"],"benchmark_id":"claire-radar-2608-12781","caveat":"PatternEval is a diagnostic benchmark with 2,415 multimodal prompts testing four response-pattern failures: chain-of-thought leakage, repetition, contradiction, and performative reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12781"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"PatternEval","organization_count":1,"organizations":["claire_radar"],"rank":2520,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.12781"},{"aliases":[],"benchmark_id":"claire-radar-2607-27354","caveat":"A user-centric benchmark for evaluating personal AI assistants in stateful, service-integrated environments, with multi-regime evaluation and user simulation for long-horizon tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27354"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PAUSE","organization_count":1,"organizations":["claire_radar"],"rank":2521,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27354"},{"aliases":[],"benchmark_id":"claire-radar-2608-27345","caveat":"Evaluates video generators as stochastic samplers of world dynamics across 50 scenarios, comparing the distribution of possible behaviors under identical initial observations and actions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27345"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PAWBench","organization_count":1,"organizations":["claire_radar"],"rank":2522,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27345"},{"aliases":[],"benchmark_id":"opencompass-1076-pca-bench","caveat":"PCA-Bench is a multimodal decisionmaking benchmark for evaluating the integrated capabilities of Multimodal Large Language Models (MLLMs). Departing from previous\nbenchmarks focusing on simplistic tasks and individual model capability. PCA-Bench 是一个多模态决策基准，用于评估多模态大型语言模型（MLLMs）的综合能力。与之前专注于简单任务和单个模型能力的基准不同，PCA-Bench 引入了三个复杂场景：自动驾驶、家庭机器人和开放世界游戏。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PCA-Bench"],"document_share":0.0002723311546840959,"domain":"其他","name":"PCA-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2523,"released":"2024-02-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PCA-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2608-27923","caveat":"Consists of over 300 real-world PCB schematic images with annotated pins and paired SPICE netlists, intended for schematic-to-netlist conversion evaluation.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.27923v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PCBnet","organization_count":1,"organizations":["claire_radar"],"rank":2524,"released":"2026-08-28","source":"claire_radar","url":"http://arxiv.org/abs/2608.27923v1"},{"aliases":[],"benchmark_id":"claire-radar-2609-34930","caveat":"Large language model (LLM) agents are evolving from tool-calling systems that execute isolated instructions into task-oriented agents that pursue user goals through sustained, multi-step interactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34930"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PDEU-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2525,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34930"},{"aliases":[],"benchmark_id":"claire-radar-2608-24509","caveat":"PeakBench is a benchmark of executable multi-tool workflows for evaluating resource-aware tool invocation in LLM agents. It includes execution-grounded dependency annotations and measured resource profiles, with a two-part evaluation framework distinguishing logical planning from physical scheduling. Code is available on GitHub.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24509"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PeakBench","organization_count":1,"organizations":["claire_radar"],"rank":2526,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24509"},{"aliases":[],"benchmark_id":"claire-radar-2605-28819","caveat":"PEFT-Arena benchmark jointly evaluates target-domain performance and retention of pretrained capabilities for parameter-efficient finetuning methods. It covers mathematical and medical reasoning as target domains and measures general capability retention on benchmarks including BBH, IFEval, and NQ with SFT and RLVR training settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28819"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PEFT-Arena","organization_count":1,"organizations":["claire_radar"],"rank":2527,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28819"},{"aliases":[],"benchmark_id":"claire-radar-2609-00906","caveat":"Evaluates robotic insertion performance on a reconfigurable peg-in-hole setup with modular components, emphasizing task generalization across varied high-precision insertion scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00906"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"Peg-in-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2528,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.00906"},{"aliases":["pelican-bike-benchmark"],"benchmark_id":"claire-radar-github-qrzzzz-pelican-bike-benchmark","caveat":"Evaluates AI-generated single-file HTML implementations of a pelican-riding-a-bicycle prompt using a fixed prompt and static site, with optional scoring across runnable, visual, animation, accessibility, and quality dimensions.","document_count":1,"document_ids":["claire_radar:https://github.com/Qrzzzz/pelican-bike-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Pelican Bike Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2529,"released":"2026-09-12","source":"claire_radar","url":"https://github.com/Qrzzzz/pelican-bike-benchmark"},{"aliases":["Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?"],"benchmark_id":"claire-radar-2608-21827","caveat":"Evaluates poetic logic in modern Chinese poetry through four tasks across stanza, line, and imagery levels using six mainstream LLMs.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.21827v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Peony","organization_count":1,"organizations":["claire_radar"],"rank":2530,"released":"2026-08-22","source":"claire_radar","url":"http://arxiv.org/abs/2608.21827v1"},{"aliases":[],"benchmark_id":"claire-radar-2607-24957","caveat":"PerceptionBench evaluates atomic visual perception in MLLMs with 3,000 verified questions isolating ten perceptual capabilities, based on an error taxonomy from 42 benchmarks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24957"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PerceptionBench","organization_count":1,"organizations":["claire_radar"],"rank":2531,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24957"},{"aliases":[],"benchmark_id":"llm-stats-perceptionbench","caveat":"PerceptionBench is Moonshot AI's internal benchmark for evaluating atomic visual perception capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/perceptionbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"PerceptionBench","organization_count":1,"organizations":["llm_stats"],"rank":2532,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptionbench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-perceptiontest","caveat":"A novel multimodal video benchmark designed to evaluate perception and reasoning skills of pre-trained models across video, audio, and text modalities. Contains 11.6k real-world videos (average 23 seconds) filmed by participants worldwide, densely annotated with six types of labels. Focuses on skills (Memory, Abstraction, Physics, Semantics) and reasoning types (descriptive, explanatory, predictive, counterfactual). Shows significant performance gap between human baseline (91.4%) and state-of-the-art video QA models (46.2%).","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/perceptiontest?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"PerceptionTest","organization_count":1,"organizations":["llm_stats"],"rank":2533,"released":"2023-05-23","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/perceptiontest?top_n=500"},{"aliases":["PereStruct: Multimodal Semantic Assembly for Robust Historical Document Parsing"],"benchmark_id":"claire-radar-2606-07661","caveat":"A 93-page benchmark for mapping detected blocks from complex historical newspaper layouts to their corresponding articles.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07661"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PereStruct benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2534,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.07661"},{"aliases":[],"benchmark_id":"claire-radar-2607-07744","caveat":"PERFOPT-Bench evaluates coding agents on software performance optimization tasks, requiring profiling, diagnosing bottlenecks, editing code while preserving correctness, and verifying reproducible speedups. Scoring includes hidden correctness tests, verified-speedup measurement, and trajectory-level audit.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.07744"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"PERFOPT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2535,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.07744"},{"aliases":[],"benchmark_id":"claire-radar-2605-25535","caveat":"PerMemBench evaluates personalized memory systems for LLM agents using multi-year, multi-domain interaction histories across 20 user personas, measuring memory retention accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25535"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PerMemBench","organization_count":1,"organizations":["claire_radar"],"rank":2536,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25535"},{"aliases":[],"benchmark_id":"claire-radar-2605-27015","caveat":"PersLitEval is a benchmark of 4,514 Persian literature multiple-choice questions across eight categories including spelling, literary devices, grammar, vocabulary, word formation, and conceptual understanding, sourced from Konkur university entrance exam materials.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27015"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PersLitEval","organization_count":1,"organizations":["claire_radar"],"rank":2537,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27015"},{"aliases":[],"benchmark_id":"claire-radar-2608-19746","caveat":"Evaluates personalized text generation through LUAR, LLM-as-judge, and stylometrics across 50 authors and 1,000 generations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.19746"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PersonalBench","organization_count":1,"organizations":["claire_radar"],"rank":2538,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.19746"},{"aliases":[],"benchmark_id":"opencompass-1959-personalens","caveat":"PersonaLens  a large-scale benchmark specifically designed to evaluate personalization in task-oriented dialogues. The benchmark features 1,500 in-depth user profiles, each integrating real demographic data, detailed cross-domain preferences, and rich interaction histories. PersonaLens是一个专为任务导向型对话设计的、大规模的个性化能力评测基准。它包含1,500个深度用户画像，每个画像都集成了真实的人口统计信息、详尽的个人偏好及历史互动记录。这些画像与覆盖20个领域的111项真实世界任务相结合，并辅以动态的“情景上下文”来模拟现实世界的复杂性。为了实现自动化、可扩展的评估，该基准引入了两个LLM驱动的智能体：一个“用户智能体”负责模拟真人与AI进行对话，另一个“评判智能体”则对个性化水平、任务成功率和对话质量进行系统性打分。PersonaLens旨在为研究社区提供一个强大可靠的工具，共同推动下一代更懂你、更智能的AI助手的研发。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PersonaLens"],"document_share":0.0002723311546840959,"domain":"语言","name":"PersonaLens","organization_count":1,"organizations":["opencompass_hub"],"rank":2539,"released":"2025-06-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PersonaLens"},{"aliases":[],"benchmark_id":"claire-radar-2608-16717","caveat":"PersonaShot benchmarks person-centric narrative continuity in multi-shot video generation with ~1,000 segments and 16 metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16717"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PersonaShot","organization_count":1,"organizations":["claire_radar"],"rank":2540,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16717"},{"aliases":[],"benchmark_id":"claire-radar-2607-20482","caveat":"PersonaTrail evaluates personalized web agents using realistic browsing trajectories as user history, assessing preference inference and information recall. Operates in a managed open web environment with two tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20482"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"PersonaTrail","organization_count":1,"organizations":["claire_radar"],"rank":2541,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.20482"},{"aliases":[],"benchmark_id":"claire-radar-2606-08878","caveat":"PerspectiveGap is a benchmark with 110 scenarios for evaluating multi-agent orchestration prompting, using distractor-mixed tasks and topologies from the authors' practice.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08878"],"document_share":0.0002723311546840959,"domain":"Agents","name":"PerspectiveGap","organization_count":1,"organizations":["claire_radar"],"rank":2542,"released":"2026-06-07","source":"claire_radar","url":"https://arxiv.org/abs/2606.08878"},{"aliases":[],"benchmark_id":"opencompass-2087-perteval-scfm","caveat":"PertEval-scFM is a standardized framework to evaluate single-cell foundation models (scFMs) for predicting cellular perturbation effects. PertEval-scFM 是一个评估单细胞基础模型（scFM）扰动效应预测的标准化框架。它从分布偏移泛化性、扰动强度及上下文对齐三方面评测。测试集包含Norman、Replogle等数据集，涵盖数千扰动样本和基因。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PertEval-scFM"],"document_share":0.0002723311546840959,"domain":"其他","name":"PertEval-scFM","organization_count":1,"organizations":["opencompass_hub"],"rank":2543,"released":"2024-10-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PertEval-scFM"},{"aliases":[],"benchmark_id":"claire-radar-2609-04598","caveat":"Evaluates large language and vision-language models on Korean veterinary question answering, covering text and multimodal questions with expert answers and multiple evaluation metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04598"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning","organization_count":1,"organizations":["claire_radar"],"rank":2544,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.04598"},{"aliases":[],"benchmark_id":"claire-radar-2609-19883","caveat":"Compact and scalable benchmark for evaluating LLM reasoning over dynamic state spaces using Petri nets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19883"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PetriBench","organization_count":1,"organizations":["claire_radar"],"rank":2545,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.19883"},{"aliases":[],"benchmark_id":"claire-radar-2609-28921","caveat":"Benchmark with four task interfaces for evaluating language models on protein modification, including single-mutant generation and multi-mutant ranking.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.28921"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PFArena","organization_count":1,"organizations":["claire_radar"],"rank":2546,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.28921"},{"aliases":["PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology"],"benchmark_id":"claire-radar-2605-29710","caveat":"A real-robot leaderboard for vision-language-action policies on a Franka FR3. It scores policies with Human-Relative Throughput computed from time-to-success distributions and uses Kolmogorov-Smirnov significance tests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29710"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"PhAIL (Physical AI Leaderboard)","organization_count":1,"organizations":["claire_radar"],"rank":2547,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29710"},{"aliases":[],"benchmark_id":"claire-radar-2606-11105","caveat":"PhantomBench evaluates language models' ability to abstain from answering about non-existent entities. It comprises over 60,000 non-existent terms derived from real concepts across domains, and provides a pipeline for generating further instances.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11105"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PhantomBench","organization_count":1,"organizations":["claire_radar"],"rank":2548,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11105"},{"aliases":[],"benchmark_id":"claire-radar-2607-20492","caveat":"PhantomFill measures schema-coerced fabrication in language models by asking questions on unanswerable inputs under three output formats: free text, JSON with an escape option, and JSON with required fields. It reports Coerced Fabrication Rate and Escape Utilization Rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20492"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PhantomFill","organization_count":1,"organizations":["claire_radar"],"rank":2549,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.20492"},{"aliases":[],"benchmark_id":"claire-radar-2609-10055","caveat":"Provides 13,390 samples across seven HPO normalization datasets for mapping free-text phenotype mentions to standardized concepts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10055"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PhenoNormBench","organization_count":1,"organizations":["claire_radar"],"rank":2550,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10055"},{"aliases":[],"benchmark_id":"llm-stats-phibench","caveat":"PhiBench is an internal benchmark designed to evaluate diverse skills and reasoning abilities of language models, covering a wide range of tasks including coding (debugging, extending incomplete code, explaining code snippets) and mathematics (identifying proof errors, generating related problems). Created by Microsoft's research team to address limitations of standard academic benchmarks and guide the development of the Phi-4 model.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/phibench?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"PhiBench","organization_count":1,"organizations":["llm_stats"],"rank":2551,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/phibench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-09789","caveat":"PHITSBench evaluates AI-assisted generation of PHITS radiation-transport input via natural language across 282 tasks in three workflows: Edit, Repair, and Reproduce. Scoring uses a Composite Metric Score combining execution success and agreement with reference transport observables.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09789"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PHITSBench","organization_count":1,"organizations":["claire_radar"],"rank":2552,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.09789"},{"aliases":[],"benchmark_id":"claire-radar-2606-14832","caveat":"PhoneHarness Bench evaluates phone-use agents on verifiable mobile workflows with mixed GUI, CLI, and tool actions, scored by observable side effects from auditable execution traces.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14832"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"PhoneHarness","organization_count":1,"organizations":["claire_radar"],"rank":2553,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14832"},{"aliases":[],"benchmark_id":"claire-radar-2610-10455","caveat":"Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10455"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PHRBench","organization_count":1,"organizations":["claire_radar"],"rank":2554,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10455"},{"aliases":[],"benchmark_id":"claire-radar-2607-00436","caveat":"PHREEQC-MCQ-200 evaluates tool-augmented agents on 200 multiple-choice questions derived from 21 validated PHREEQC scenarios. Agents must construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers. Scoring is based on exact match of selected answer.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00436"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"PHREEQC-MCQ-200","organization_count":1,"organizations":["claire_radar"],"rank":2555,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.00436"},{"aliases":[],"benchmark_id":"claire-radar-2606-07300","caveat":"Phun-Bench evaluates LLMs' phonological understanding in Chinese across three dimensions: Homophony, Rhyme, and Phonetic Similarity, with diverse tasks designed to isolate genuine phonological ability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07300"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Phun-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2556,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07300"},{"aliases":[],"benchmark_id":"llm-stats-phybench","caveat":"PHYBench is a benchmark of real-world physics problems spanning mechanics, electromagnetism, thermodynamics, optics, and modern physics, designed to evaluate physical perception and multi-step quantitative reasoning in large language models.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/phybench?top_n=500"],"document_share":0.0002723311546840959,"domain":"physics","name":"PHYBench","organization_count":1,"organizations":["llm_stats"],"rank":2557,"released":"2025-04-22","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/phybench?top_n=500"},{"aliases":["PhyBench"],"benchmark_id":"phybench","caveat":"Physics problem-solving benchmark; multi-step reasoning quality affects the final answer.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"science","name":"PhyBench","organization_count":1,"organizations":["ByteDance"],"rank":2558,"released":"2025-04-22","source":"model_reports","url":"https://github.com/phybench-official/phybench"},{"aliases":[],"benchmark_id":"claire-radar-2606-26551","caveat":"PhyEditBench evaluates physics-aware image editing. It includes 238 real-world instances and 35 synthetic anti-physics instances across 12 physical subclasses, with VLM-based scoring on consistency, instruction following, physical plausibility, and image quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26551"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PhyEditBench","organization_count":1,"organizations":["claire_radar"],"rank":2559,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.26551"},{"aliases":[],"benchmark_id":"opencompass-2080-phygenbench","caveat":"PhyGenBench is a benchmark designed to evaluate physical commonsense correctness in text-to-video generation models. I PhyGenBench 是一个评估T2V模型物理常识的基准。它涵盖力学、光学、热学和材料特性四大领域27种物理定律，包含160个提示。结合PhyGenEval框架，利用视听和大语言模型自动化评估模型的物理理解能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PhyGenBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"PhyGenBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2560,"released":"2024-10-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PhyGenBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-23695","caveat":"The benchmark evaluates action selection in UAV mission traces through multiple-choice decision instances derived from conversational data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23695"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"PhysAI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2561,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23695"},{"aliases":[],"benchmark_id":"claire-radar-2606-18613","caveat":"PhysAssistBench evaluates interactive doctor-patient-EHR assistance. It contains 1,296 physician-validated turns from MIMIC-IV cases, testing coordination of clinical knowledge, patient communication, and EHR tool use.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18613"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PhysAssistBench","organization_count":1,"organizations":["claire_radar"],"rank":2562,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18613"},{"aliases":[],"benchmark_id":"claire-radar-2608-25097","caveat":"Evaluates multimodal LLMs on 11,586 bilingual Olympiad-level physics problems with visual diagrams, step-by-step solutions, and answer accuracy metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25097"],"document_share":0.0002723311546840959,"domain":"Mathematics & Formal Science","name":"PhysElite","organization_count":1,"organizations":["claire_radar"],"rank":2563,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.25097"},{"aliases":[],"benchmark_id":"claire-radar-2609-34072","caveat":"Multimodal large language models (MLLMs) are increasingly envisioned as core components of scientific and engineering agents, yet their ability to interpret physical fields remains poorly understood.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34072"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PhysFieldBench","organization_count":1,"organizations":["claire_radar"],"rank":2564,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34072"},{"aliases":[],"benchmark_id":"claire-radar-2609-15562","caveat":"PhysForensics-Bench evaluates AI-generated audio-video detection using paired real and generated clips across nine event-centric scenes from two generators.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15562"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PhysForensics-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2565,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15562"},{"aliases":[],"benchmark_id":"claire-radar-2606-18943","caveat":"A revised benchmark for measuring whether video generative models reproduce the physical behavior observed in real-world experiment videos.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18943"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Physics-IQ Verified","organization_count":1,"organizations":["claire_radar"],"rank":2566,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18943"},{"aliases":[],"benchmark_id":"claire-radar-2608-24056","caveat":"PhysicsBench is a unified benchmark and leaderboard for generative and predictive AI models in engineering design and simulation. It spans seven tasks across 1D, 2D, and 3D domains, ranks 66 models on nine datasets, and uses a common metric suite including BenchRank for debiased ranking. Evaluation covers data scales from S to XL.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24056"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PhysicsBench","organization_count":1,"organizations":["claire_radar"],"rank":2567,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24056"},{"aliases":[],"benchmark_id":"llm-stats-physicsfinals","caveat":"PHYSICS is a comprehensive benchmark for university-level physics problem solving, containing 1,297 expert-annotated problems covering six core areas: classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics. Each problem requires advanced physics knowledge and mathematical reasoning. Even advanced models like o3-mini achieve only 59.9% accuracy.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/physicsfinals?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"PhysicsFinals","organization_count":1,"organizations":["llm_stats"],"rank":2568,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/physicsfinals?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-15722","caveat":"PhySimCode evaluates video-to-code generation for physics simulations, requiring models to infer physical laws and parameters from videos and produce runnable Python code rendered and scored against the input video.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.15722"],"document_share":0.0002723311546840959,"domain":"Code","name":"PhySimCode","organization_count":1,"organizations":["claire_radar"],"rank":2569,"released":"2026-10-11","source":"claire_radar","url":"https://arxiv.org/abs/2610.15722"},{"aliases":[],"benchmark_id":"claire-radar-2607-01938","caveat":"A robot manipulation benchmark with 16 tasks involving fast, dynamically moving targets. It covers simulation and real-world robot experiments and scores policies by task success rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01938"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"PhysMani-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2570,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.01938"},{"aliases":[],"benchmark_id":"opencompass-1576-physreason","caveat":"PhysReason is a comprehensive physics-based reasoning benchmark consisting of 1,200 physics problems spanning multiple domains, with a focus on both knowledge-based (25%) and reasoning-based (75%) questions. PhysReason 是一个包含 1,200 个物理问题的综合物理推理基准，涵盖多个领域，重点关注基于知识（25%）和推理（75%）的问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PhysReason"],"document_share":0.0002723311546840959,"domain":"学科","name":"PhysReason","organization_count":1,"organizations":["opencompass_hub"],"rank":2571,"released":"2025-02-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PhysReason"},{"aliases":[],"benchmark_id":"claire-radar-2606-10803","caveat":"PhysTool-Bench evaluates multimodal LLMs on physical tool use through two tasks: recognizing all tools in a scene and selecting and sequencing tools for a given task, using 2,510 queries over 2,678 real-world tools.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10803"],"document_share":0.0002723311546840959,"domain":"Agents","name":"PhysTool-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2572,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.10803"},{"aliases":[],"benchmark_id":"opencompass-2350-picabench","caveat":null,"document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PICABench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"PICABench","organization_count":1,"organizations":["opencompass_hub"],"rank":2573,"released":"2025-10-27","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PICABench"},{"aliases":[],"benchmark_id":"claire-radar-2609-15017","caveat":"Evaluates prompt-injection detectors across multiple axes including over-defense, obfuscation, and distribution shift, with fixed thresholds and reproducible metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15017"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PIDS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2574,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15017"},{"aliases":[],"benchmark_id":"llm-stats-pinchbench","caveat":"PinchBench evaluates coding agents on real-world agentic coding tasks, measuring both best-case and average performance across complex software engineering scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pinchbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"PinchBench","organization_count":1,"organizations":["llm_stats"],"rank":2575,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/pinchbench?top_n=500"},{"aliases":["Decision-Aware Evaluation of Physics-Informed Surrogates"],"benchmark_id":"claire-radar-2606-07146","caveat":"An open benchmark for evaluating physics-informed surrogates in material-conditioned lattice design across prediction fidelity, physical validity, retrieval, and decision regret.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07146"],"document_share":0.0002723311546840959,"domain":"General AI","name":"pinn-gym","organization_count":1,"organizations":["claire_radar"],"rank":2576,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07146"},{"aliases":[],"benchmark_id":"claire-radar-2605-30639","caveat":"PInVerify is an offline embodied benchmark for Active Instance Verification, where agents select viewpoints around a candidate object to decide if it matches a fine-grained description.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30639"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"PInVerify","organization_count":1,"organizations":["claire_radar"],"rank":2577,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30639"},{"aliases":[],"benchmark_id":"claire-radar-2607-06440","caveat":"PIPBench evaluates personalized image generation, where models must align outputs with a user's implicit visual preferences based on a few historically preferred images and a short prompt. It includes real-user and agent-based data across psychological and demographic profiles.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06440"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PIPBench","organization_count":1,"organizations":["claire_radar"],"rank":2578,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06440"},{"aliases":[],"benchmark_id":"claire-radar-2606-08481","caveat":"PIPE-Cypher is a pipeline that generates NL-to-Cypher benchmarks from live property graphs, producing executable query pairs with validation, diversity controls, and local LLM judges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08481"],"document_share":0.0002723311546840959,"domain":"Agents","name":"PIPE-Cypher","organization_count":1,"organizations":["claire_radar"],"rank":2579,"released":"2026-06-07","source":"claire_radar","url":"https://arxiv.org/abs/2606.08481"},{"aliases":["Pipette: An Embodied Simulation Platform, Benchmark, and Data-Efficient Augmentation Framework for Wet-Lab Robotics"],"benchmark_id":"claire-radar-2606-12936","caveat":"Pipette evaluates wet-lab robotic agents across 12 embodied tasks using task success rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12936"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"Pipette","organization_count":1,"organizations":["claire_radar"],"rank":2580,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.12936"},{"aliases":[],"benchmark_id":"llm-stats-piqa","caveat":"PIQA (Physical Interaction: Question Answering) is a benchmark dataset for physical commonsense reasoning in natural language. It tests AI systems' ability to answer questions requiring physical world knowledge through multiple choice questions with everyday situations, focusing on atypical solutions inspired by instructables.com. The dataset contains 21,000 multiple choice questions where models must choose the most appropriate solution for physical interactions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/piqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"physics","name":"PIQA","organization_count":1,"organizations":["llm_stats"],"rank":2581,"released":"2019-11-26","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/piqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-532-piqa","caveat":"PIQA is a physical interaction question answering task, which requires to select the most reasonable solution based on the given scenario and two possible solutions. This task is designed to test the model's knowledge in physical commonsense. This dataset consists of 16k training samples, 800 development samples and 2k test samples, all on English text. PIQA是一个物理交互问答任务，要求根据给定的场景和两个可能的解决方案，选择最合理的方案。这个任务是为了测试模型在物理常识方面的知识。这个数据集包含了16000个训练样本，800个开发样本和2000个测试样本，所有的文本都是英文文本。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PIQA"],"document_share":0.0002723311546840959,"domain":"推理","name":"PIQA","organization_count":1,"organizations":["opencompass_hub"],"rank":2582,"released":"2019-11-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PIQA"},{"aliases":[],"benchmark_id":"claire-radar-2605-26176","caveat":"PitchBench evaluates pitch hearing in audio-language models across 28 experiments spanning absolute and relative pitch perception in sequences and chords, varying acoustic conditions and response formats.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26176"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PitchBench","organization_count":1,"organizations":["claire_radar"],"rank":2583,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26176"},{"aliases":["SIC-Agents: Benchmarking and Building an Adaptive Simulator for Pediatric Serious Illness Communication Training","SIC-Agents"],"benchmark_id":"claire-radar-2608-29481","caveat":"Evaluates parent-simulator agents in pediatric serious illness communication training using turn-level PitfallBench and dialogue-level DialogueBench.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29481"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"PitfallBench","organization_count":1,"organizations":["claire_radar"],"rank":2584,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29481"},{"aliases":["PIVOT: A Multi-Trajectory Dataset and Testbed for Pose, Intrinsics, and Novel Viewpoint Evaluation in Real-World 3D Reconstruction"],"benchmark_id":"claire-radar-2608-25401","caveat":"Evaluates novel-view synthesis under diverse camera trajectories, measured vs optimized poses, and calibrated vs optimized intrinsics using five real-world scenes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25401"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PIVOT","organization_count":1,"organizations":["claire_radar"],"rank":2585,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25401"},{"aliases":[],"benchmark_id":"claire-radar-2606-23092","caveat":"PIVOTSBench evaluates multimodal large language models on fine-grained interpersonal relationship reasoning. It includes tasks predicting bidirectional relationship dimensions from videos and auxiliary tasks on visual cue identification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23092"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PIVOTSBench","organization_count":1,"organizations":["claire_radar"],"rank":2586,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23092"},{"aliases":[],"benchmark_id":"claire-radar-2608-19646","caveat":"Dataset of 11,000 NBA possession clips with 31,567 annotated events for basketball video understanding tasks like event recognition, captioning, localization, and anticipation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.19646"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PL-NBA","organization_count":1,"organizations":["claire_radar"],"rank":2587,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.19646"},{"aliases":[],"benchmark_id":"claire-radar-2609-21263","caveat":"Evaluates reasoning-driven macro placement in chip design using open RTL designs, EDA tools, and technology libraries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21263"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"PlaceReasoner-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2588,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21263"},{"aliases":[],"benchmark_id":"claire-radar-2606-02747","caveat":"Plan2Map evaluates document-grounded geospatial boundary reconstruction from UK planning records. Systems input a planning document and output a GeoJSON boundary, scored against held-out reference boundaries via IoU.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02747"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Plan2Map","organization_count":1,"organizations":["claire_radar"],"rank":2589,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02747"},{"aliases":[],"benchmark_id":"opencompass-1251-planbench","caveat":"PlanBench is an extensible benchmark suite based on the kinds of domains used in the automated planning community, especially in the International Planning Competition, to test the capabilities of LLMs in planning or reasoning about actions and change. PlanBench用于评估LLM的规划能力，基于自动化规划社区（尤其是在国际规划竞赛）中涉及的各种领域来测试大模型在规划或推理行动和变更方面的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PlanBench"],"document_share":0.0002723311546840959,"domain":"强推理","name":"PlanBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2590,"released":"2022-06-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PlanBench"},{"aliases":[],"benchmark_id":"claire-radar-2606-05744","caveat":"PlanBench-V evaluates vision-language models on spatial planning map interpretation through an expert-annotated dataset of 223 maps and 1629 question-answer pairs, assessing perception, reasoning, association, and implementation capabilities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05744"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PlanBench-V","organization_count":1,"organizations":["claire_radar"],"rank":2591,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05744"},{"aliases":[],"benchmark_id":"claire-radar-2606-22388","caveat":"PlanBench-XL evaluates LLM tool-use agents on long-horizon planning in large-scale tool ecosystems. It includes 327 retail tasks across 1,665 tools, requiring iterative tool retrieval and use. Optional blocker mechanisms inject missing, failing, or distracting tools to test adaptive planning. Scoring is based on final answer accuracy and auxiliary metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22388"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"PlanBench-XL","organization_count":1,"organizations":["claire_radar"],"rank":2592,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22388"},{"aliases":[],"benchmark_id":"llm-stats-plawbench","caveat":"PLawBench evaluates language models on professional legal knowledge and reasoning tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/plawbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"PLawBench","organization_count":1,"organizations":["llm_stats"],"rank":2593,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/plawbench?top_n=500"},{"aliases":["PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?"],"benchmark_id":"claire-radar-2609-34314","caveat":"PlaylistEval evaluates video-language judges on 630 question-answer pairs derived from 100-hour playlists, testing retrieval and judgment across seven domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34314"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PlaylistEval","organization_count":1,"organizations":["claire_radar"],"rank":2594,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34314"},{"aliases":[],"benchmark_id":"claire-radar-2608-13552","caveat":"PlayWorld evaluates interactive video world models across 171 scenarios with long-horizon objectives. Agent players adaptively control each model, and performance is scored on geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution, plus basic video quality and controllability metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13552"],"document_share":0.0002723311546840959,"domain":"Agents","name":"PlayWorld","organization_count":1,"organizations":["claire_radar"],"rank":2595,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13552"},{"aliases":[],"benchmark_id":"claire-radar-2608-26882","caveat":"Evaluates autonomous LLM agents on real programmable logic controllers connected to closed-loop process simulations, measuring their ability to achieve sustained physical impact through fixed diagnostic flags applied to runner, communication, PLC-object, and process records.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26882"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"PLCBENCH","organization_count":1,"organizations":["claire_radar"],"rank":2596,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26882"},{"aliases":[],"benchmark_id":"claire-radar-2610-02982","caveat":"Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.02982"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PLCWorld","organization_count":1,"organizations":["claire_radar"],"rank":2597,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.02982"},{"aliases":[],"benchmark_id":"claire-radar-2608-15931","caveat":"PLSQLBench evaluates LLMs' ability to write executable PL/SQL programs through execution-based tests. It contains 2,865 instances including single-turn and multi-turn tasks, covering schema-grounded and procedural problems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15931"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"PLSQLBench","organization_count":1,"organizations":["claire_radar"],"rank":2598,"released":"2026-08-16","source":"claire_radar","url":"https://arxiv.org/abs/2608.15931"},{"aliases":[],"benchmark_id":"claire-radar-2608-08700","caveat":"PluginEval evaluates tool routing in LLMs via three decision types (missed, spurious, parameter errors) across difficulty levels, using deterministic validation and real API execution for reliable signals.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08700"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PluginEval","organization_count":1,"organizations":["claire_radar"],"rank":2599,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08700"},{"aliases":[],"benchmark_id":"claire-radar-2607-12385","caveat":"PM-Bench evaluates prospective memory in LLM agents through a text-based simulated seven-day week. Agents must maintain user intentions, execute delayed intentions, and monitor latent environment changes while performing ongoing activities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.12385"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2600,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.12385"},{"aliases":[],"benchmark_id":"llm-stats-pmc-vqa","caveat":"A medical visual question answering benchmark built on biomedical literature and medical figures.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pmc-vqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"PMC-VQA","organization_count":1,"organizations":["llm_stats"],"rank":2601,"released":"2023-05-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/pmc-vqa?top_n=500"},{"aliases":["pni-3d-benchmark"],"benchmark_id":"claire-radar-huggingface-tryforge-pni-3d-benchmark","caveat":"Evaluates fine-grained 3D CAD matching, multimodal embedding alignment, and geometric retrieval under sensor distortions using 100 parametric CAD families across 12 representations.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/tryforge/pni-3d-benchmark"],"document_share":0.0002723311546840959,"domain":"CAD","name":"PNI-3D Multimodal CAD & Geometric Robustness Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2602,"released":"2026-09-10","source":"claire_radar","url":"https://huggingface.co/datasets/tryforge/pni-3d-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2605-29299","caveat":"Pocket-Dentist is an efficiency-aware benchmark for dental multimodal question answering that combines three datasets, five task types, and seven metrics to evaluate VLMs on accuracy and computational cost.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29299"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Pocket-Dentist","organization_count":1,"organizations":["claire_radar"],"rank":2603,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29299"},{"aliases":[],"benchmark_id":"claire-radar-2605-28237","caveat":"POINav-Bench evaluates vision-language navigation agents in real-world POI-goal navigation across 11 reconstructed commercial areas covering 126,398 m² with 163 POIs, using traversability-aware annotations and reference trajectories for closed-loop evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28237"],"document_share":0.0002723311546840959,"domain":"General AI","name":"POINav-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2604,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28237"},{"aliases":[],"benchmark_id":"llm-stats-pointgrounding","caveat":"PointArena is a comprehensive platform for evaluating multimodal pointing across diverse reasoning scenarios. It includes Point-Bench, a curated dataset of ~1,000 pointing tasks across five categories: Spatial (positional references), Affordance (functional part identification), Counting (attribute-based grouping), Steerable (relative pointing), and Reasoning (open-ended visual inference). The benchmark evaluates language-guided pointing capabilities in vision-language models.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pointgrounding?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"PointGrounding","organization_count":1,"organizations":["llm_stats"],"rank":2605,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/pointgrounding?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-28241","caveat":"PointQ-Bench is a benchmark for point cloud quality assessment, extending from scalar scoring to comprehensive quality understanding, with 3,083 point clouds and tasks like anomaly sensing, defect diagnosis, usability grading, and open-ended quality reporting.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28241"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PointQ-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2606,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28241"},{"aliases":[],"benchmark_id":"claire-radar-2605-23168","caveat":"PoisonForge benchmarks task-level targeted poisoning of instruction-tuned LLMs, parameterizing bias type, poisoning mode, appearance count, and target output length. It evaluates 12 open-weight models across five families with primarily 1% poison budget, using attack success rate as the main metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23168"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"PoisonForge","organization_count":1,"organizations":["claire_radar"],"rank":2607,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23168"},{"aliases":[],"benchmark_id":"opencompass-1675-pokerbench","caveat":"PokerBench contains natural language game scenarios and optimal decisions computed by solvers in No Limit Texas Hold’em. It is divided into pre-flop and post-flop datasets, each with training and test splits. PokerBench包含自然语言游戏场景和由求解器在无限制德州扑克中计算出的最优决策。它分为前注和后注数据集，每个数据集都包含训练集和测试集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PokerBench"],"document_share":0.0002723311546840959,"domain":"其他","name":"PokerBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2608,"released":"2025-01-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PokerBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-05910","caveat":"PolicyShiftBench evaluates policy-adaptive image guardrailing: given an image and a current policy, a model must output a pass/block decision plus optional violated category IDs. The benchmark comprises 2,000 policy-discriminative instances over 265 images, each paired with multiple policy-conditioned prompts. Scoring uses binary pass/block accuracy and category attribution metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05910"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PolicyShiftBench","organization_count":1,"organizations":["claire_radar"],"rank":2609,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.05910"},{"aliases":[],"benchmark_id":"claire-radar-2607-25953","caveat":"Polistemics evaluates LLMs as mediators of political information across controlled settings varying evidence clarity, noise, and consistency, using a diagnostic benchmark grounded in Epistemic Modesty.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25953"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Polistemics","organization_count":1,"organizations":["claire_radar"],"rank":2610,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25953"},{"aliases":["An open benchmark for machine learning-based polymer property prediction"],"benchmark_id":"claire-radar-2609-27036","caveat":"Benchmark dataset for machine learning-based polymer property prediction across eight physical properties.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27036"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PolyBench26","organization_count":1,"organizations":["claire_radar"],"rank":2611,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.27036"},{"aliases":[],"benchmark_id":"claire-radar-2610-11618","caveat":"As large language models increasingly move toward repository-level software engineering, existing code-generation benchmarks remain fragmented across language coverage, task granularity, and evaluation protocols, impeding systematic compar…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11618"],"document_share":0.0002723311546840959,"domain":"Code","name":"PolyCodeEval","organization_count":1,"organizations":["claire_radar"],"rank":2612,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11618"},{"aliases":[],"benchmark_id":"claire-radar-2608-14741","caveat":"PolyComp is a benchmark for compositional 3D spatial reasoning using polycube problems with 120 problems across four geometry families.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14741"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PolyComp","organization_count":1,"organizations":["claire_radar"],"rank":2613,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.14741"},{"aliases":[],"benchmark_id":"llm-stats-polymath","caveat":"Polymath is a challenging multi-modal mathematical reasoning benchmark designed to evaluate the general cognitive reasoning abilities of Multi-modal Large Language Models (MLLMs). The benchmark comprises 5,000 manually collected high-quality images of cognitive textual and visual challenges across 10 distinct categories, including pattern recognition, spatial reasoning, and relative reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/polymath?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"PolyMATH","organization_count":1,"organizations":["llm_stats"],"rank":2614,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-polymath-en","caveat":"PolyMath is a multilingual mathematical reasoning benchmark covering 18 languages and 4 difficulty levels from easy to hard, ensuring difficulty comprehensiveness, language diversity, and high-quality translation. The benchmark evaluates mathematical reasoning capabilities of large language models across diverse linguistic contexts, making it a highly discriminative multilingual mathematical benchmark.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/polymath-en?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"PolyMath-en","organization_count":1,"organizations":["llm_stats"],"rank":2615,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/polymath-en?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-01016","caveat":"PolySpeech-100 evaluates speech understanding in speech-large language models across 110 linguistic variants, including 19 Chinese dialects and over 80 low-resource languages, using tasks that assess semantic reasoning beyond transcription.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01016"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PolySpeech-100","organization_count":1,"organizations":["claire_radar"],"rank":2616,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01016"},{"aliases":[],"benchmark_id":"claire-radar-2607-06008","caveat":"PolyWorkBench evaluates LLM agents on multilingual, long-horizon workplace workflows across five domains: commerce, knowledge work, legal analysis, localization, and manufacturing. It includes 67 tasks, structured scoring via Grade, executable state verification with Pytest, and LLM-as-Judge diagnostics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06008"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PolyWorkBench","organization_count":1,"organizations":["claire_radar"],"rank":2617,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06008"},{"aliases":[],"benchmark_id":"claire-radar-2608-05162","caveat":"PoolBench isolates pooling strategies as the experimental variable in concept representation evaluation for decoder-only LLMs. It covers 17 concepts, 19 pooling strategies, and 3 open-weight models (Llama-3.1-8B, Gemma-2-9B, Mistral-7B) on a corpus of 37,693 real-text passages, with primary axis linear separability (AUROC) and diagnostic axes for steering and disentanglement.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05162"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PoolBench","organization_count":1,"organizations":["claire_radar"],"rank":2618,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.05162"},{"aliases":[],"benchmark_id":"llm-stats-pope","caveat":"Polling-based Object Probing Evaluation (POPE) is a benchmark for evaluating object hallucination in Large Vision-Language Models (LVLMs). POPE addresses the problem where LVLMs generate objects inconsistent with target images by using a polling-based query method that asks yes/no questions about object presence in images, providing more stable and flexible evaluation of object hallucination.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/pope?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"POPE","organization_count":1,"organizations":["llm_stats"],"rank":2619,"released":"2023-05-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/pope?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1362-pope","caveat":"POPE is an improved evaluation method for LVLMs' object hallucination by proposing a polling-based query method. It offers a more stable and flexible solution. POPE用于评估视觉语言模型的物体幻觉，基于轮询的查询方法设计，提供了一种更稳定、更灵活的评估方案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/POPE"],"document_share":0.0002723311546840959,"domain":"多模态","name":"POPE","organization_count":1,"organizations":["opencompass_hub"],"rank":2620,"released":"2023-05-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/POPE"},{"aliases":["Large Language Models Systematically Favor Popular Options: Evidence and Mitigation Across MCQs"],"benchmark_id":"claire-radar-2608-29257","caveat":"Evaluates multiple-choice question answering across 73,260 MCQ items with six strategies that systematically vary distractor popularity, using accuracy, HPSR, and other bias metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29257"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"PopMCQ","organization_count":1,"organizations":["claire_radar"],"rank":2621,"released":"2026-08-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.29257"},{"aliases":[],"benchmark_id":"llm-stats-popqa","caveat":"PopQA is an entity-centric open-domain question-answering dataset consisting of 14,000 QA pairs designed to evaluate language models' ability to memorize and recall factual knowledge across entities with varying popularity levels. The dataset probes both parametric memory (stored in model parameters) and non-parametric memory effectiveness, with questions covering 16 diverse relationship types from Wikidata converted to natural language using templates. Created by sampling knowledge triples from Wikidata and converting them to natural language questions, focusing on long-tail entities to understand LMs' strengths and limitations in memorizing factual knowledge.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/popqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"PopQA","organization_count":1,"organizations":["llm_stats"],"rank":2622,"released":"2022-12-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/popqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-10255","caveat":"POPSICLE benchmarks cryoET segmentation and macromolecular localization using data from the CryoET Data Portal. It spans eukaryotic and prokaryotic systems, purified and in situ samples, and covers dense voxel-wise segmentation and sparse localization tasks. Built on the CryoET Data Portal, it can expand with new data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10255"],"document_share":0.0002723311546840959,"domain":"Software & AI Compute","name":"POPSICLE","organization_count":1,"organizations":["claire_radar"],"rank":2623,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.10255"},{"aliases":[],"benchmark_id":"claire-radar-2605-27887","caveat":"PortBench evaluates LLM-driven portfolio management via a static QA dataset (6,269 questions across seven task templates) and a dynamic five-stage allocation pipeline, spanning six asset classes over ten years. Scoring includes a dual-layer correlation score and CEPS, with evaluation under three stress regimes and investor profiles.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27887"],"document_share":0.0002723311546840959,"domain":"Finance","name":"PortBench","organization_count":1,"organizations":["claire_radar"],"rank":2624,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27887"},{"aliases":[],"benchmark_id":"claire-radar-2606-19096","caveat":"PorTEXTO evaluates visual text extraction from contemporary and culturally relevant European Portuguese (pt-PT) images. The benchmark includes synthetic and real-world samples with native-speaker-reviewed transcriptions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19096"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PorTEXTO","organization_count":1,"organizations":["claire_radar"],"rank":2625,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19096"},{"aliases":["Benchmarking Automated Security Patch Backporting: How Far Are We?"],"benchmark_id":"claire-radar-2608-17671","caveat":"Porting Benchmark evaluates automated security patch backporting using 1,234 curated cases across cross-version, cross-branch, and cross-repository scenarios, with a common evaluation framework and dynamic validation on a subset.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17671"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Porting Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2626,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17671"},{"aliases":[],"benchmark_id":"claire-radar-2610-05010","caveat":"Portrait aesthetic assessment assigns comparable scores according to how effectively human-centered images fulfill their photographic intent.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05010"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PortraitAes-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2627,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05010"},{"aliases":["KAYROS: An Anytime and Exact Open-Source Solver for Duration-Minimization Time-Dependent Vehicle Routing. A Technical Report and a Case Study in Human-AI Engineering"],"benchmark_id":"claire-radar-2607-23116","caveat":"Poryos2026 evaluates duration-minimization routing on 1,080 paired static and time-dependent instances derived from road networks and controlled traffic.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23116"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Poryos2026","organization_count":1,"organizations":["claire_radar"],"rank":2628,"released":"2026-07-25","source":"claire_radar","url":"https://arxiv.org/abs/2607.23116"},{"aliases":[],"benchmark_id":"claire-radar-2608-13560","caveat":"PosterBench evaluates academic paper-to-poster generation. It includes a 100-paper Main Track across five disciplines and a 10-paper mini subset, with automated scoring and human evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13560"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PosterBench","organization_count":1,"organizations":["claire_radar"],"rank":2629,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13560"},{"aliases":["PosterHarness: Turning Scientific Poster Generation into an Auditable Instruction-Following Benchmark"],"benchmark_id":"claire-radar-2607-03006","caveat":"An auditable evaluation harness for testing whether scientific-poster generators follow layout, placeholder, abstention, text, and provenance requirements.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03006"],"document_share":0.0002723311546840959,"domain":"General AI","name":"POSTERHARNESS","organization_count":1,"organizations":["claire_radar"],"rank":2630,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.03006"},{"aliases":[],"benchmark_id":"claire-radar-2609-20794","caveat":"Benchmark evaluating whether generative scientific inverse solvers recover full posterior distributions across four physics-based inverse tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.20794"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PosteriorBench","organization_count":1,"organizations":["claire_radar"],"rank":2631,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.20794"},{"aliases":[],"benchmark_id":"opencompass-1579-postersum","caveat":"The PosterSum dataset is a multimodal benchmark designed for the summarization of scientific posters into research paper abstracts. The dataset consists of 16,305 research posters collected from major machine learning conferences, including ICLR, ICML, and NeurIPS, spanning the years 2022-2024. PosterSum 数据集是一个多模态基准数据集，旨在将科学海报总结成研究论文摘要。该数据集包含从2022-2024年的主要机器学习会议（包括 ICLR、ICML 和 NeurIPS）收集的 16,305 篇研究海报。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PosterSum"],"document_share":0.0002723311546840959,"domain":"多模态","name":"PosterSum","organization_count":1,"organizations":["opencompass_hub"],"rank":2632,"released":"2025-02-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PosterSum"},{"aliases":[],"benchmark_id":"llm-stats-posttrainbench","caveat":"PostTrainBench evaluates a model's ability to autonomously post-train base models. Given pretrain-only base models, the agent must complete the full pipeline of data synthesis, training, evaluation, and iteration within a time budget, scored across downstream benchmarks such as AIME2025, BFCL, GPQA Main, GSM8K, and HumanEval.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"PostTrainBench","organization_count":1,"organizations":["llm_stats"],"rank":2633,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-posttrainbench-lite","caveat":"PostTrainBench Lite measures whether an agent can design and execute a full post-training strategy (data, prompts, RL recipe, and eval loop) for a pretrained base model under a constrained time budget, scored as normalized mean reward over the improvement window.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench-lite?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"PostTrainBench Lite","organization_count":1,"organizations":["llm_stats"],"rank":2634,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/posttrainbench-lite?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-27288","caveat":"A benchmark extending ARC to assess understanding of underlying abstract rules across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27288"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PotARCin","organization_count":1,"organizations":["claire_radar"],"rank":2635,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27288"},{"aliases":[],"benchmark_id":"claire-radar-2606-20950","caveat":"Power Systems Agent Benchmark is an executable benchmark for power-engineering agents. It includes 41 task families with deterministic evaluators that recompute engineering quantities and check constraints, plus held-out generation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20950"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Power Systems Agent Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2636,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.20950"},{"aliases":[],"benchmark_id":"claire-radar-2609-34492","caveat":"PowerBench evaluates LLM agents on 300 questions across three task families requiring autonomous evidence retrieval and reasoning over interconnected power system data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34492"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PowerBench","organization_count":1,"organizations":["claire_radar"],"rank":2637,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34492"},{"aliases":[],"benchmark_id":"claire-radar-2605-31478","caveat":"PowerCodeBench is a benchmark generator for power system code generation, paired with an intervention method, but no artifacts are provided in this article.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31478"],"document_share":0.0002723311546840959,"domain":"Code","name":"PowerCodeBench","organization_count":1,"organizations":["claire_radar"],"rank":2638,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31478"},{"aliases":[],"benchmark_id":"claire-radar-2607-02897","caveat":"PPE-Bench evaluates machine unlearning in multimodal large language models under private-public entanglement, where images contain a target individual to forget and public elements to preserve.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02897"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PPE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2639,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.02897"},{"aliases":[],"benchmark_id":"claire-radar-2610-13435","caveat":"Photoplethysmography (PPG) is increasingly recorded by wearable and clinical devices at enormous scale, and a growing number of PPG foundation models (FMs) have been developed to compute reusable representations for various downstream task…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13435"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PPG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2640,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13435"},{"aliases":[],"benchmark_id":"claire-radar-2606-31154","caveat":"PPT-Eval is a benchmark of 120 PowerPoint tasks across 12 files for computer-use agents. It covers content creation and editing, with rubric-based evaluation that awards partial credit and provides natural language feedback.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31154"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PPT-Eval","organization_count":1,"organizations":["claire_radar"],"rank":2641,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31154"},{"aliases":["PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides","Einsia PPTBench"],"benchmark_id":"pptbench","caveat":"Einsia / Navers Lab’s 500 scientific flow-diagram reconstruction tasks, distinct from the earlier benchmark of the same name. Mean score is on a 0–100 scale with invalid and majority-gated outputs scored zero, not a success percentage. Results include the model, native agent harness and reasoning effort. Protocol-development and judge-validation tasks remain in the full evaluation. The source figures retain their individual paper licenses.","document_count":1,"document_ids":["model_reports:pptbench_v2_results_table"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"PPTBench (Einsia)","organization_count":1,"organizations":["Navers Lab / Einsia.AI"],"rank":2642,"released":"2026-08-31","source":"model_reports","url":"https://lab.einsia.ai/pptbench/"},{"aliases":[],"benchmark_id":"claire-radar-2609-09664","caveat":"Evaluates personalized guidance through long-term conversation histories and evidence annotations, testing retrieval accuracy and memory-grounded reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09664"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PRAGMA","organization_count":1,"organizations":["claire_radar"],"rank":2643,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09664"},{"aliases":[],"benchmark_id":"claire-radar-2608-09772","caveat":"PragMatch is a controlled set of 3,000 image-text pairs derived from MMSD2.0 for studying pragmatic incongruity in multimodal sarcasm detection, with original sarcastic examples and constructed literal and hard-negative pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09772"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PragMatch","organization_count":1,"organizations":["claire_radar"],"rank":2644,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09772"},{"aliases":[],"benchmark_id":"llm-stats-prbench-finance","caveat":"PRBench-Finance evaluates professional reasoning on finance tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/prbench-finance?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"PRBench-Finance","organization_count":1,"organizations":["llm_stats"],"rank":2645,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-finance?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-prbench-legal","caveat":"PRBench-Legal evaluates professional reasoning on legal tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/prbench-legal?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"PRBench-Legal","organization_count":1,"organizations":["llm_stats"],"rank":2646,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/prbench-legal?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-01829","caveat":"Pre-Flight evaluates large language models on aviation operational knowledge via 300 multiple-choice questions drawn from international standards and airport ground operations material, covering ground operations, ICAO and FAA regulations, general aviation knowledge, and operational scenarios. Scoring is by accuracy under a standard multiple-choice protocol using the Inspect framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01829"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Pre-Flight","organization_count":1,"organizations":["claire_radar"],"rank":2647,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.01829"},{"aliases":[],"benchmark_id":"claire-radar-2606-09890","caveat":"PreActBench is a benchmark for predictive monitoring in LLMs, consisting of 1,000 paired ethical and unethical action trajectories across five domains. It evaluates whether models can infer if a partial trajectory will culminate in unethical action, using the Prefix Foresight F1 metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09890"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PreAct-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2648,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.09890"},{"aliases":[],"benchmark_id":"claire-radar-2606-09376","caveat":"A multilingual grounded-generation benchmark that evaluates both support precision and coverage of complete-oracle facts using Formula 1 telemetry, with a second complete-oracle weather domain.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09376"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle","organization_count":1,"organizations":["claire_radar"],"rank":2649,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09376"},{"aliases":[],"benchmark_id":"claire-radar-2608-02372","caveat":"PredAct-Bench evaluates dialogue agents paired with imperfect tools using educational datasets, measuring AI-assisted decision-making and trust metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02372"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PredAct-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2650,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02372"},{"aliases":[],"benchmark_id":"llm-stats-presentbench","caveat":"PresentBench evaluates AI agents on producing presentation-style deliverables, such as generating lesson-plan slides and structured documents from source materials.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/presentbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"PresentBench","organization_count":1,"organizations":["llm_stats"],"rank":2651,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/presentbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-31468","caveat":"Evaluates LLM price, quality, and brand preferences from hotel-booking choices using a logit choice model.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.31468"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PriceBench","organization_count":1,"organizations":["claire_radar"],"rank":2652,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.31468"},{"aliases":[],"benchmark_id":"claire-radar-2609-04200","caveat":"Evaluates Newtonian physics reasoning in video models through relational consistency between paired objects across eight phenomena, using a calibration-independent consistency score derived from real-world scenes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04200"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Principia","organization_count":1,"organizations":["claire_radar"],"rank":2653,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.04200"},{"aliases":[],"benchmark_id":"claire-radar-2608-14329","caveat":"Principle-Bench contains 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles, with paraphrase, adversarial keyword-stuffing, and boundary perturbations for evaluating LLM-as-judge on accuracy, paraphrase robustness, adversarial robustness, and calibration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14329"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Principle-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2654,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14329"},{"aliases":[],"benchmark_id":"claire-radar-2605-28375","caveat":"Named entity recognition dataset from PubMed abstracts on prion disease, with 317 abstracts annotated for 15 coarse and 31 fine-grained entity types, plus evaluation scripts and train/test splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28375"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"PrionNER","organization_count":1,"organizations":["claire_radar"],"rank":2655,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28375"},{"aliases":[],"benchmark_id":"claire-radar-2608-17962","caveat":"PRISM is a dataset of over 5,000 trajectories across 25 industrial manipulation tasks with multimodal sensing. It provides teleoperated demonstrations for contact-rich operations but lacks a standardized evaluation protocol or scoring contract.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17962"],"document_share":0.0002723311546840959,"domain":"Contact-rich control","name":"PRISM","organization_count":1,"organizations":["claire_radar"],"rank":2656,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17962"},{"aliases":[],"benchmark_id":"claire-radar-2609-04867","caveat":"Evaluates text-to-audio-video generation with an audio-centric diagnostic benchmark comprising 900 human-verified samples, factorizing audio evaluation by audio type and sound-source visibility, and assessing four perceptual dimensions via 35 fine-grained criteria using an MLLM-as-a-Judge protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04867"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PRISM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2657,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.04867"},{"aliases":[],"benchmark_id":"claire-radar-2609-27395","caveat":"A multi-axis discriminative benchmark for compact vision-language models that scores items along seven axes and combines them into a single PScore, recycling items from fifteen public benchmarks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27395"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PRISM-VLM","organization_count":1,"organizations":["claire_radar"],"rank":2658,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27395"},{"aliases":["PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say"],"benchmark_id":"claire-radar-2606-00152","caveat":"A 1,182-case benchmark across 7 acquisition behaviours and 16 domains that audits whether tool-using LLM agents acquire sensitive data beyond task scope.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00152"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PrivacyPeek","organization_count":1,"organizations":["claire_radar"],"rank":2659,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.00152"},{"aliases":[],"benchmark_id":"claire-radar-2608-29624","caveat":"Evaluates text-to-text privatization methods across defined desiderata through modular metrics and a live leaderboard.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29624"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PrivBench","organization_count":1,"organizations":["claire_radar"],"rank":2660,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29624"},{"aliases":["PrivDrift: Auditing User-Secret Leakage Under Topic Drift in Active LLM Conversations"],"benchmark_id":"claire-radar-2609-30094","caveat":"The benchmark evaluates whether user-disclosed secrets remain recoverable in active LLM conversations after topic drift and persuasion-based probing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30094"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PrivDrift","organization_count":1,"organizations":["claire_radar"],"rank":2661,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.30094"},{"aliases":[],"benchmark_id":"claire-radar-2607-16311","caveat":"PriVE-Bench evaluates vision-language models' visual grounding using paired original and counterfactual images, with PriVE-Tools extending to tool-derived evidence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.16311"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PriVE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2662,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.16311"},{"aliases":["PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment"],"benchmark_id":"claire-radar-2608-14284","caveat":"A toolkit that converts robotic rollout videos into progress curves and computes metrics for failure progress, recovery, and execution quality, with a benchmark and evaluation suite for process reward models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14284"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"PRM-as-a-Judge 1.5","organization_count":1,"organizations":["claire_radar"],"rank":2663,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14284"},{"aliases":[],"benchmark_id":"opencompass-1681-prmbench-preview","caveat":"PRMBench is a benchmark dataset for evaluating process-level reward models (PRMs). It consists of 6,216 data instances, each containing a question, a solution process, and a modified process with errors. PRMBench 是一个用于评估过程级奖励模型（PRM）的基准数据集。它包含 6,216 个数据实例，每个实例包含一个问题、一个解决方案过程以及一个包含错误的修改过程。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/PRMBench_Preview"],"document_share":0.0002723311546840959,"domain":"学科","name":"PRMBench_Preview","organization_count":1,"organizations":["opencompass_hub"],"rank":2664,"released":"2025-01-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/PRMBench_Preview"},{"aliases":[],"benchmark_id":"claire-radar-2608-11149","caveat":"PRMU evaluates corpus-free multimodal unlearning of person-related knowledge in MLLMs, using textual and visual probes including adversarial evaluation and locality analysis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11149"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"PRMU","organization_count":1,"organizations":["claire_radar"],"rank":2665,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.11149"},{"aliases":[],"benchmark_id":"claire-radar-2609-27076","caveat":"A prompt-conditioned benchmark for open-vocabulary visual grounding in heterogeneous, real-world environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27076"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"Pro-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2666,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.27076"},{"aliases":[],"benchmark_id":"claire-radar-2609-12658","caveat":"ProactiveBench evaluates models at one-second stream intervals without an explicit response cue in streaming video settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12658"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ProactiveBench","organization_count":1,"organizations":["claire_radar"],"rank":2667,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12658"},{"aliases":[],"benchmark_id":"claire-radar-2609-37470","caveat":"A probability used for a decision should refer to the same event across equivalent requests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37470"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Probability Contracts","organization_count":1,"organizations":["claire_radar"],"rank":2668,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.37470"},{"aliases":[],"benchmark_id":"opencompass-1632-probench","caveat":"ProBench is a benchmark that contains open-ended multimodal queries that require intensive expert-level knowledge to solve. ProBench是一个包含需要大量专家级知识来解决的开放式多模态查询的基准。ProBench 包含 10 个任务领域和 56 个子领域，支持 17 种语言，并支持最多 13 轮对话。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ProBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ProBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2669,"released":"2025-03-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ProBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-06527","caveat":"Potential benchmark for LLM-driven PL/SQL development from natural language in direct and interactive scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06527"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ProcArena","organization_count":1,"organizations":["claire_radar"],"rank":2670,"released":"2026-09-06","source":"claire_radar","url":"https://arxiv.org/abs/2609.06527"},{"aliases":[],"benchmark_id":"opencompass-1621-processbench","caveat":"ProcessBench can measure the ability to identify erroneous steps in mathematical reasoning. It consists of 3,400 test cases, primarily focused on competition- and Olympiad-level math problems. ProcessBench，用于衡量识别数学推理中错误步骤的能力。它包含 3,400 个测试案例，主要关注竞赛和奥林匹克级别的数学问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ProcessBench"],"document_share":0.0002723311546840959,"domain":"学科","name":"ProcessBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2671,"released":"2024-12-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ProcessBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-17701","caveat":"ProEvent is an event-centric benchmark for proactive agents, evaluating their ability to maintain a user's timetable from instant messaging chats. It assesses response timing, single-step correctness, and multi-step correctness using synthesized realistic chat scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.17701"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ProEvent","organization_count":1,"organizations":["claire_radar"],"rank":2672,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.17701"},{"aliases":[],"benchmark_id":"llm-stats-profbench","caveat":"ProfBench evaluates models on professional-domain reasoning and knowledge-work tasks, including search-augmented question answering across expert fields.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/profbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ProfBench","organization_count":1,"organizations":["llm_stats"],"rank":2673,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/profbench?top_n=500"},{"aliases":["Long-Horizon Forecasting of Complete Financial Statements with Forma"],"benchmark_id":"claire-radar-2608-11327","caveat":"Evaluates joint probabilistic forecasting of complete quarterly financial statements across 78 line items and 1-20 quarter horizons.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11327"],"document_share":0.0002723311546840959,"domain":"Finance","name":"ProForma-20Q","organization_count":1,"organizations":["claire_radar"],"rank":2674,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.11327"},{"aliases":[],"benchmark_id":"llm-stats-program-bench","caveat":"Program Bench evaluates code-generation agents by asking them to recreate a program's behavior from only a compiled binary and documentation. It spans 200 tasks from small CLI tools to large systems such as FFmpeg and SQLite, with submissions judged against more than 248,000 fuzz-generated behavioral tests.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/program-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Program Bench","organization_count":1,"organizations":["llm_stats"],"rank":2675,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/program-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-18805","caveat":"Evaluates coding agents on tasks derived from interactive web applications.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18805"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"ProgramDistill","organization_count":1,"organizations":["claire_radar"],"rank":2676,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18805"},{"aliases":[],"benchmark_id":"claire-radar-2605-30284","caveat":"A framework for evaluating scientific hypothesis generation in LLMs under progressive information disclosure, comparing model hypotheses to original paper conclusions via semantic similarity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30284"],"document_share":0.0002723311546840959,"domain":"AI Scientist","name":"ProjectionBench","organization_count":1,"organizations":["claire_radar"],"rank":2677,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30284"},{"aliases":[],"benchmark_id":"opencompass-1633-projudge","caveat":"ProJudge is a comprehensive, multi-modal, multi-discipline, and multi-difficulty benchmark specifically designed for evaluating abilities of MLLM-based process judges.It comprises 2,400 test cases and 50,118 step-level labels, spanning four scientific disciplines with diverse difficulty levels and ProJudge 是一个针对基于 MLLM 的过程裁判能力的全面、多模态、多学科和多难度的基准。它包含 2,400 个测试案例和 50,118 个步骤级标签，涵盖四个科学学科，难度级别和内容多样化。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ProJudge"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ProJudge","organization_count":1,"organizations":["opencompass_hub"],"rank":2678,"released":"2025-03-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ProJudge"},{"aliases":[],"benchmark_id":"claire-radar-2607-09876","caveat":"Prompting-MammAlps is a camera-trap text-to-video retrieval benchmark, evaluating video-language models on fine-grained retrieval of ecological events. It includes a test set of 135 queries and 775 candidate videos, with a proposed method that combines action localization with LLM-based parsing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09876"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Prompting-MammAlps","organization_count":1,"organizations":["claire_radar"],"rank":2679,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.09876"},{"aliases":[],"benchmark_id":"opencompass-1120-proofnet","caveat":"ProofNet is a benchmark for autoformalization and formal proving of undergraduate-level mathematics. The ProofNet benchmarks consists of 371 examples, each consisting of a formal theorem statement in Lean 3, a natural language theorem statement, and a natural language proof. ProofNet 是一个用于本科数学的自动形式化和形式证明的基准。包含 371 个示例，每个示例包括一个 Lean 3 中的形式定理陈述、一个自然语言定理陈述和一个自然语言证明。这些问题主要来自流行的本科纯数学教材，涵盖实分析、复分析、线性代数、抽象代数和拓扑等主题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ProofNet"],"document_share":0.0002723311546840959,"domain":"数学","name":"ProofNet","organization_count":1,"organizations":["opencompass_hub"],"rank":2680,"released":"2023-02-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ProofNet"},{"aliases":[],"benchmark_id":"llm-stats-protocolqa","caveat":"ProtocolQA is a multiple-choice benchmark on troubleshooting failed experimental outcomes from common biological laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/protocolqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"ProtocolQA","organization_count":1,"organizations":["llm_stats"],"rank":2681,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/protocolqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-00451","caveat":"ProtStructQA is an executable benchmark for protein structural question answering, with questions generated from DSL programs and answers obtained by executing on AlphaFold-predicted structures. Released 382.2K questions covering confidence, distances, PAE, solvent exposure, secondary structure, topology, and contacts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00451"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ProtStructQA","organization_count":1,"organizations":["claire_radar"],"rank":2682,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00451"},{"aliases":[],"benchmark_id":"claire-radar-2608-02693","caveat":"PRWeaver evaluates LLM-based code auditors against malicious pull requests across repository history, using 208 attacks and multiple renderings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02693"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"PRWeaver","organization_count":1,"organizations":["claire_radar"],"rank":2683,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02693"},{"aliases":[],"benchmark_id":"claire-radar-2606-18060","caveat":"PseudoBench evaluates agentic auto-research systems on their ability to identify and resist pseudoscientific narratives. It contains 200 curated pseudoscientific claim-evidence pairs across five domains and assesses performance through an end-to-end research pipeline from experiment design to report writing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18060"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PseudoBench","organization_count":1,"organizations":["claire_radar"],"rank":2684,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18060"},{"aliases":[],"benchmark_id":"claire-radar-2605-29653","caveat":"PTCG-Bench evaluates LLM agents on the Pokémon Trading Card Game at two levels: single-environment decision-making and self-evolution through accumulated experience, with modular harness ablation to separate agent performance from harness design.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29653"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"PTCG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2685,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29653"},{"aliases":[],"benchmark_id":"claire-radar-2608-17379","caveat":"PTXBench evaluates LLMs in generating architecture-specific PTX for GPU kernel optimization. It measures functional correctness, execution of target instructions, and speedup over frontier libraries across GEMM and attention workloads on H100 and B200 GPUs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17379"],"document_share":0.0002723311546840959,"domain":"Architecture-specific programming","name":"PTXBench","organization_count":1,"organizations":["claire_radar"],"rank":2686,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17379"},{"aliases":[],"benchmark_id":"claire-radar-2610-12423","caveat":"We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12423"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Pumpire","organization_count":1,"organizations":["claire_radar"],"rank":2687,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.12423"},{"aliases":["Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation"],"benchmark_id":"claire-radar-2606-17188","caveat":"A multilingual visual-reasoning benchmark comprising parallel Punjabi image-text instances in Gurmukhi, Shahmukhi, and Roman scripts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17188"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PuMVR","organization_count":1,"organizations":["claire_radar"],"rank":2688,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.17188"},{"aliases":["Internalizing Geometric Law: Learning from Solver Residuals for Precision-Critical Generation"],"benchmark_id":"claire-radar-2606-09278","caveat":"A 300-problem stratified benchmark for generating precise geometric constructions from natural-language descriptions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09278"],"document_share":0.0002723311546840959,"domain":"General AI","name":"PyGeoX-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2689,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09278"},{"aliases":[],"benchmark_id":"claire-radar-2606-03858","caveat":"PyraMathBench is a hierarchical benchmark with 32,505 questions derived from math word problems, evaluating numerical reasoning and mathematical capabilities across cognitive aspects and modalities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03858"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"PyraMathBench","organization_count":1,"organizations":["claire_radar"],"rank":2690,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03858"},{"aliases":[],"benchmark_id":"opencompass-1000-q-bench","caveat":"Q-Bench/Q-Bench+ is a benchmark for general-purpose foundation models on low-level vision. Q-Bench/Q-Bench+是一个面向多模态大模型底层视觉理解的数据集。此数据集从底层视觉的感知、描述、评价能力出发来对多模态大模型进行完整的测试，测试的对象既包括单张图像也包括图像对。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Q-Bench"],"document_share":0.0002723311546840959,"domain":"理解","name":"Q-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2691,"released":"2024-08-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Q-Bench"},{"aliases":["Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems"],"benchmark_id":"claire-radar-2609-08887","caveat":"A large-scale retrieval evaluation with a 190M-document web corpus and 70k agentic search queries in ten languages, judged by agent citations, production rankings, and a combined relevance set.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08887"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Q2D-Web","organization_count":1,"organizations":["claire_radar"],"rank":2692,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08887"},{"aliases":[],"benchmark_id":"opencompass-1103-qasc","caveat":"QASC is a question-answering dataset with a focus on sentence composition. It consists of 9,980 8-way multiple-choice questions about grade school science (8,134 train, 926 dev, 920 test), and comes with a corpus of 17M sentences. QASC 是一个专注于句子组合的问答数据集。它包含 9,980 道小学科学的多项选择题（8,134 道用于训练，926 道用于开发，920 道用于测试），并配有一个包含 1,700 万个句子的语料库。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/QASC"],"document_share":0.0002723311546840959,"domain":"知识","name":"QASC","organization_count":1,"organizations":["opencompass_hub"],"rank":2693,"released":"2020-02-04","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/QASC"},{"aliases":[],"benchmark_id":"llm-stats-qasper","caveat":"QASPER is a dataset of 5,049 information-seeking questions and answers anchored in 1,585 NLP research papers. Questions are written by NLP practitioners who read only titles and abstracts, while answers require understanding the full paper text and provide supporting evidence. The dataset challenges models with complex reasoning across document sections for academic document question answering. Each question seeks information present in the full text and is answered by a separate set of NLP practitioners who also provide supporting evidence to answers.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qasper?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"Qasper","organization_count":1,"organizations":["llm_stats"],"rank":2694,"released":"2021-05-07","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qasper?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-35581","caveat":"We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35581"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"QC-Stark","organization_count":1,"organizations":["claire_radar"],"rank":2695,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35581"},{"aliases":[],"benchmark_id":"claire-radar-2609-17439","caveat":"A set of 49 expert-authored tasks for verified autonomy in quantum engineering.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17439"],"document_share":0.0002723311546840959,"domain":"General AI","name":"QIQCBench","organization_count":1,"organizations":["claire_radar"],"rank":2696,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17439"},{"aliases":["Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation"],"benchmark_id":"claire-radar-2605-27210","caveat":"Qiskit QuantumKatas is a 350-task evaluation of LLM quantum-computing capabilities across 26 categories, using deterministic circuit-simulation tests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27210"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Qiskit QuantumKatas","organization_count":1,"organizations":["claire_radar"],"rank":2697,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27210"},{"aliases":[],"benchmark_id":"llm-stats-qmsum","caveat":"QMSum is a benchmark for query-based multi-domain meeting summarization consisting of 1,808 query-summary pairs over 232 meetings across academic, product, and committee domains. The dataset enables models to select and summarize relevant spans of meetings in response to specific queries. Published at NAACL 2021, QMSum presents significant challenges in long meeting summarization where models must identify and summarize relevant content based on user queries.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qmsum?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"QMSum","organization_count":1,"organizations":["llm_stats"],"rank":2698,"released":"2021-04-13","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qmsum?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-04646","caveat":"QO-Bench is a diagnostic benchmark for query-operator question answering over typed event tuples. It covers 22,984 news articles, 614 corporate events, and 18 query templates, with 785 questions. Gold answers are deterministically computed and scored by recall via exact match.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04646"],"document_share":0.0002723311546840959,"domain":"General AI","name":"QO-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2699,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04646"},{"aliases":["Quant Arena"],"benchmark_id":"quant_arena","caveat":"Agents research stock-scoring strategies on 2022–2023 data, freeze one program, and receive one sealed 2024 cash-account assessment. The 0–100 total combines 30% research-window and 70% sealed-window scores, not investment return. Reasoning effort and iterative versus one-shot settings remain separate; the proprietary Kepler-1 reference has undisclosed construction. These historical results do not establish future trading performance.","document_count":1,"document_ids":["model_reports:einsia_quant_arena_v1_results"],"document_share":0.0002723311546840959,"domain":"finance","name":"Quant-Arena","organization_count":1,"organizations":["Navers Lab / Einsia AI"],"rank":2700,"released":"2026-10-21","source":"model_reports","url":"https://lab.einsia.ai/quant-arena/"},{"aliases":["variant-pathogenicity-benchmark"],"benchmark_id":"claire-radar-github-rathanpai-variant-pathogenicity-benchmark","caveat":"A comparative framework for genetic variant pathogenicity prediction using ClinVar and gnomAD-derived biological features, benchmarking classical ML, deep learning, and quantum machine learning models.","document_count":1,"document_ids":["claire_radar:https://github.com/RathanPai/variant-pathogenicity-benchmark"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"Quantum-Enhanced Genetic Variant Pathogenicity Prediction","organization_count":1,"organizations":["claire_radar"],"rank":2701,"released":"2026-09-16","source":"claire_radar","url":"https://github.com/RathanPai/variant-pathogenicity-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2606-23943","caveat":"A comparison of tokenization strategies (BPE, Unigram LM, WordPiece, PRPE) for Southern Quechua using a 200k-sentence corpus and a finite-state morphological analyzer as reference, with metrics including fertility rate, OOV rate, and morphological boundary accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23943"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"QuechuaTok","organization_count":1,"organizations":["claire_radar"],"rank":2702,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23943"},{"aliases":[],"benchmark_id":"claire-radar-2608-13547","caveat":"Benchmark of 56 one-shot tasks from 14 incident-derived families that measures exact final-state outcomes when LLM coding agents issue Bash commands through serializing/wrapping/reparsing transport, using an added unescaped parser.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13547"],"document_share":0.0002723311546840959,"domain":"General AI","name":"QuoteBench","organization_count":1,"organizations":["claire_radar"],"rank":2703,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13547"},{"aliases":[],"benchmark_id":"claire-radar-2609-22038","caveat":"Evaluates generative AI on Quranic Arabic across multiple dimensions of linguistic complexity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.22038"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"QuranicMMLU","organization_count":1,"organizations":["claire_radar"],"rank":2704,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.22038"},{"aliases":[],"benchmark_id":"claire-radar-2606-32034","caveat":"QVal is a training-free testbed for evaluating dense supervision signals for long-horizon LLM agents. It measures Q-alignment of scores from methods across four environments and seven families, without requiring training runs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.32034"],"document_share":0.0002723311546840959,"domain":"General AI","name":"QVal","organization_count":1,"organizations":["claire_radar"],"rank":2705,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.32034"},{"aliases":[],"benchmark_id":"llm-stats-qvhighlights","caveat":"QVHighlights is a video moment retrieval benchmark for detecting moments and highlights in videos via natural language queries. Given a query, the model must localize the start and end times of relevant moments in the video, evaluated using metrics such as Recall@1 at a 0.5 IoU threshold.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qvhighlights?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"QVHighlights","organization_count":1,"organizations":["llm_stats"],"rank":2706,"released":"2021-07-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qvhighlights?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-28091","caveat":"Qwen-Image-Bench evaluates text-to-image models on five pillars including Real-world Fidelity and Creative Generation, with 1000 prompts and 56 rubric-based facets scored by a trained judge model.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28091"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Qwen-Image-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2707,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28091"},{"aliases":[],"benchmark_id":"claire-radar-huggingface-pxzleo-qwen3-8-27b-inference-benchmark-4090","caveat":"The dataset presents performance and accuracy results for specific serving configurations, but does not define a reusable evaluation protocol or scoring contract for other teams.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/pxzleo/qwen3.8-27b-inference-benchmark-4090"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"qwen3.8-27b-inference-benchmark-4090","organization_count":1,"organizations":["claire_radar"],"rank":2708,"released":"2026-08-21","source":"claire_radar","url":"https://huggingface.co/datasets/pxzleo/qwen3.8-27b-inference-benchmark-4090"},{"aliases":[],"benchmark_id":"llm-stats-qwenclawbench","caveat":"QwenClawBench is a real-user-distribution Claw agent benchmark for evaluating coding agents on realistic developer tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwenclawbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"QwenClawBench","organization_count":1,"organizations":["llm_stats"],"rank":2709,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenclawbench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-qwen-qoder-bench","caveat":"QwenQoderBench is Qwen's internal benchmark for evaluating coding-agent performance.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwen-qoder-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"QwenQoderBench","organization_count":1,"organizations":["llm_stats"],"rank":2710,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-qoder-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-qwen-react-bench","caveat":"QwenReactBench is Qwen's internal React application generation benchmark, reported as a BT/Elo rating.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwen-react-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"QwenReactBench","organization_count":1,"organizations":["llm_stats"],"rank":2711,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-react-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-qwen-svg","caveat":"QwenSVG is Qwen's internal SVG generation benchmark for evaluating front-end and visual code generation. Scores are reported as BT/Elo ratings from auto-rendered outputs judged by a multimodal evaluator.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwen-svg?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"QwenSVG","organization_count":1,"organizations":["llm_stats"],"rank":2712,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-svg?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-qwen-swe-bench","caveat":"QwenSWEBench is Qwen's software-engineering agent benchmark for repository-level issue resolution.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwen-swe-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"QwenSWEBench","organization_count":1,"organizations":["llm_stats"],"rank":2713,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qwen-swe-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-qwenwebbench","caveat":"QwenWebBench is an internal front-end code generation benchmark by Qwen. It is bilingual (EN/CN) and spans 7 categories (Web Design, Web Apps, Games, SVG, Data Visualization, Animation, and 3D), using auto-render plus a multimodal judge for code and visual correctness. Scores are reported as BT/Elo ratings.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwenwebbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"QwenWebBench","organization_count":1,"organizations":["llm_stats"],"rank":2714,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenwebbench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-qwenworldbench","caveat":"QwenWorldBench is Qwen's internal benchmark for evaluating LLMs as world models that simulate agentic environments across Terminal, SWE, MCP, Search, OS, Android, and Web domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/qwenworldbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"QwenWorldBench","organization_count":1,"organizations":["llm_stats"],"rank":2715,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/qwenworldbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-21011","caveat":"Simulation benchmark for robot-to-human object handovers with standardized evaluation protocol and five complementary metrics: planning feasibility, reachability, grasp stability, grasp affordance, and safety. Compares four baseline methods on shared grasp pose prediction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21011"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"R2HandoverSim","organization_count":1,"organizations":["claire_radar"],"rank":2716,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21011"},{"aliases":[],"benchmark_id":"claire-radar-2608-27328","caveat":"Evaluates revisit memory in interactive video world models using relative consistency: for each detected return, compares revisit pair with gap-matched non-revisit and short-range control pairs to compute MemoryGain and Normalized Memory Ratio across appearance, identity, geometry, and state.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27328"],"document_share":0.0002723311546840959,"domain":"General AI","name":"R2M-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2717,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27328"},{"aliases":["$R^3$-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets"],"benchmark_id":"claire-radar-2608-16033","caveat":"Evaluates resource-rational reasoning by measuring how models allocate a shared budget across six-problem suites in mathematics, competitive programming, and abstract reasoning, with scoring based on completed problems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16033"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"R3-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2718,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16033"},{"aliases":[],"benchmark_id":"claire-radar-2608-14391","caveat":"RA-Bench evaluates AI-generated video detection using real crisis-event videos as anchors. It includes 17,886 videos: 1,830 real anchors across 10 social-risk categories and 16,056 generated clips from nine generators. Detection is scored per source via AUC and TPR@5%FPR, with separate tracks for human-deceptive and dissemination-processed videos.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14391"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2719,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14391"},{"aliases":[],"benchmark_id":"opencompass-516-race-high","caveat":"RACE is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students. RACE 是一个大规模的阅读理解数据集，包含超过 28,000 个段落和近 100,000 个问题。该数据集是从中国的英语考试中收集而来，这些考试是为中学和高中学生设计的。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RACE%28High%29"],"document_share":0.0002723311546840959,"domain":"理解","name":"RACE(High)","organization_count":1,"organizations":["opencompass_hub"],"rank":2720,"released":"2017-04-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RACE%28High%29"},{"aliases":[],"benchmark_id":"opencompass-517-race-middle","caveat":"RACE is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students. RACE 是一个大规模的阅读理解数据集，包含超过 28,000 个段落和近 100,000 个问题。该数据集是从中国的英语考试中收集而来，这些考试是为中学和高中学生设计的。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RACE%28Middle%29"],"document_share":0.0002723311546840959,"domain":"理解","name":"RACE(Middle)","organization_count":1,"organizations":["opencompass_hub"],"rank":2721,"released":"2017-04-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RACE%28Middle%29"},{"aliases":[],"benchmark_id":"claire-radar-github-ano-kodokushi-rag-doc-qa","caveat":"Reports a locally developed 20-question evaluation set for a RAG document QA service with hybrid retrieval and cited answers.","document_count":1,"document_ids":["claire_radar:https://github.com/ano-kodokushi/rag-doc-qa"],"document_share":0.0002723311546840959,"domain":"General AI","name":"rag-doc-qa","organization_count":1,"organizations":["claire_radar"],"rank":2722,"released":"2026-09-18","source":"claire_radar","url":"https://github.com/ano-kodokushi/rag-doc-qa"},{"aliases":[],"benchmark_id":"claire-radar-2610-08571","caveat":"Retrieval-Augmented Generation (RAG) systems are vulnerable to prompt-injection attacks embedded in retrieved content.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.08571"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RAG-PIBench","organization_count":1,"organizations":["claire_radar"],"rank":2723,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.08571"},{"aliases":[],"benchmark_id":"claire-radar-github-gauthamrohan-rag-retrieval-benchmark","caveat":"The repository compares seven retrieval configurations on FiQA queries using standard retrieval metrics, but does not self-identify as a named benchmark release.","document_count":1,"document_ids":["claire_radar:https://github.com/gauthamRohan/rag-retrieval-benchmark"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"rag-retrieval-benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2724,"released":"2026-08-23","source":"claire_radar","url":"https://github.com/gauthamRohan/rag-retrieval-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-11758","caveat":"Measurement of the safety impact of retrieval augmentation on LLM outputs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11758"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RAG-Safety-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2725,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11758"},{"aliases":[],"benchmark_id":"claire-radar-2610-04691","caveat":"Retrieval-Augmented Generation (RAG) is typically evaluated under the implicit assumption that the underlying knowledge base (KB) is clean, leaving the behaviour of RAG systems under realistic KB degradation poorly characterised.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.04691"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"RAGStress","organization_count":1,"organizations":["claire_radar"],"rank":2726,"released":"2026-10-03","source":"claire_radar","url":"https://arxiv.org/abs/2610.04691"},{"aliases":[],"benchmark_id":"claire-radar-2606-05874","caveat":"RandomBench evaluates whether multimodal LLMs maintain distributionally neutral behavior when selecting among equivalent options, providing metrics for entropy and distributional bias under explicit random instructions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05874"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RandomBench","organization_count":1,"organizations":["claire_radar"],"rank":2727,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05874"},{"aliases":["RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature"],"benchmark_id":"claire-radar-2608-27394","caveat":"Evaluates retrieval of scientific literature passages relevant to three ideation operations: Address, Broaden, and Specify, using millions of full-text computer science papers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27394"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RATIO","organization_count":1,"organizations":["claire_radar"],"rank":2728,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27394"},{"aliases":[],"benchmark_id":"claire-radar-huggingface-grobestreet-rcv-bench","caveat":"Evaluates research-claim verification by scoring agents on classifying claimed results as reproduced, deviation, fabricated, or fragile using code re-execution and robustness probing.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/GrobeStreet/rcv-bench"],"document_share":0.0002723311546840959,"domain":"Agents","name":"RCV-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2729,"released":"2026-09-28","source":"claire_radar","url":"https://huggingface.co/datasets/GrobeStreet/rcv-bench"},{"aliases":[],"benchmark_id":"opencompass-1918-rdb2g-bench","caveat":"RDB2G-Bench provides comprehensive performance evaluation data for graph neural network models applied to relational database tasks. The dataset contains extensive experiments across multiple graph configurations and architectures. RDB2G-Bench提供了针对关系数据库任务应用的图神经网络模型的全面性能评估数据。该数据集包含了跨多种图配置和架构的广泛实验。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RDB2G-Bench"],"document_share":0.0002723311546840959,"domain":"代码","name":"RDB2G-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2730,"released":"2025-06-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RDB2G-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2607-25392","caveat":"RDVSv2 is a large-scale benchmark for RGB-D video salient object detection, containing 249 video sequences with 29,077 annotated frames. It includes depth maps, optical flow, and eye-tracking-guided salient object masks. The benchmark provides a fixed dataset and evaluation protocol for comparing models on this task.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25392"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RDVSv2","organization_count":1,"organizations":["claire_radar"],"rank":2731,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25392"},{"aliases":[],"benchmark_id":"opencompass-1252-re-bench","caveat":"RE-Bench (Research Engineering Benchmark, v1) consists of 7 challenging, open-ended ML research engineering environments and data from 71 8-hour attempts by 61 distinct human experts. RE-Bench用于评估AI智能体研发的自动化能力，它由61位人类专家71次在7个具有挑战性的开放式ML研究工程环境中的8小时尝试的数据组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RE-Bench"],"document_share":0.0002723311546840959,"domain":"强推理","name":"RE-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2732,"released":"2024-11-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RE-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2605-29579","caveat":"ReactBench evaluates multimodal large language models on cause-driven hallucination through four targeted tasks (Relational Erasure, Counterfactual Attribute, Alteration Tracing, Dense Counting) with exam-style evaluation and chain-of-thought reasoning for sub-cause identification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29579"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ReactBench","organization_count":1,"organizations":["claire_radar"],"rank":2733,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29579"},{"aliases":[],"benchmark_id":"claire-radar-2609-10895","caveat":"Evaluates human-like reactive decision-making in multimodal LLMs by simulating 1,000+ reproducible household hazard scenes with physics-grounded ground truth across 17 event families.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10895"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"ReactHuman","organization_count":1,"organizations":["claire_radar"],"rank":2734,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10895"},{"aliases":[],"benchmark_id":"claire-radar-2606-14058","caveat":"ReactSim-Bench evaluates reactive capability of behavior world model simulators in autonomous driving by decoupling agent and AV control, using collision, map, and kinematic metrics on 2,636 scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14058"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"ReactSim-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2735,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14058"},{"aliases":["Read the Room, Read the Image: Understanding Indirect Speech Acts in Multimodal Visual Contexts"],"benchmark_id":"claire-radar-2608-30270","caveat":"Multimodal benchmark for indirect speech act understanding in visual contexts, supporting English and Korean.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30270"],"document_share":0.0002723311546840959,"domain":"General AI","name":"READI","organization_count":1,"organizations":["claire_radar"],"rank":2736,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30270"},{"aliases":[],"benchmark_id":"opencompass-1755-real","caveat":"Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites 在真实网站的确定性模拟上对自主代理进行基准测试","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/REAL"],"document_share":0.0002723311546840959,"domain":"智能体","name":"REAL","organization_count":1,"organizations":["opencompass_hub"],"rank":2737,"released":"2025-04-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/REAL"},{"aliases":[],"benchmark_id":"claire-radar-2607-13653","caveat":"REAL-Bench evaluates vision-driven embodied agents in open-world mobile manipulation across 241 tasks spanning active exploration, visual distraction, articulated manipulation, and interactive disambiguation. The benchmark provides standardized task definitions and a simulator-based evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.13653"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"REAL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2738,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.13653"},{"aliases":[],"benchmark_id":"claire-radar-2605-24945","caveat":"Evaluates data-driven numerical weather forecasting models under operational conditions, using strictly out-of-distribution test data from 2025 and integrating low-latency operational analysis and large-scale in-situ observations from over 10,000 stations. It provides metrics for global forecasting and for extreme events such as heatwaves, cold surges, and tropical cyclones.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24945"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RealBench","organization_count":1,"organizations":["claire_radar"],"rank":2739,"released":"2026-05-24","source":"claire_radar","url":"https://arxiv.org/abs/2605.24945"},{"aliases":[],"benchmark_id":"claire-radar-2609-03773","caveat":"A proposed benchmark for parametric CAD modeling across 12,632 tasks from 19 factory-automation categories, evaluating text, drawing, picture, and rendered image inputs for Part and Assembly modeling.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03773"],"document_share":0.0002723311546840959,"domain":"CAD","name":"RealCADBench","organization_count":1,"organizations":["claire_radar"],"rank":2740,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03773"},{"aliases":[],"benchmark_id":"claire-radar-2607-16736","caveat":"RealDESED is a real-world domestic sound event detection benchmark with 5,710 recordings from 652 participants, 15 classes, and temporally precise annotations. Includes multi-annotator labeling and rich metadata.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.16736"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RealDESED","organization_count":1,"organizations":["claire_radar"],"rank":2741,"released":"2026-07-18","source":"claire_radar","url":"https://arxiv.org/abs/2607.16736"},{"aliases":[],"benchmark_id":"claire-radar-2606-07401","caveat":"RealDocBench evaluates field-level QA and layout understanding on real regulated documents, with 1,356 field-level questions over 581 documents and 1,500 annotated page images, scored on per-field accuracy and adjacency-aware layout metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07401"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RealDocBench","organization_count":1,"organizations":["claire_radar"],"rank":2742,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07401"},{"aliases":[],"benchmark_id":"claire-radar-2609-32734","caveat":"AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on dataset-specific cues and fail on unfamiliar generators or proce…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.32734"],"document_share":0.0002723311546840959,"domain":"General AI","name":"REALIS","organization_count":1,"organizations":["claire_radar"],"rank":2743,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.32734"},{"aliases":[],"benchmark_id":"claire-radar-2608-12004","caveat":"RealisticTritonBench evaluates LLM-generated Triton kernels using tasks derived from real-world pull requests in popular AI frameworks, with end-to-end integration tests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12004"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"RealisticTritonBench","organization_count":1,"organizations":["claire_radar"],"rank":2744,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.12004"},{"aliases":[],"benchmark_id":"llm-stats-realkie-fcc","caveat":"RealKIE-FCC is a key information extraction benchmark drawn from real enterprise documents (FCC filings), part of the RealKIE suite of five novel datasets for enterprise key information extraction. Models must convert documents to markdown and extract structured fields against a specified JSON schema. Nova 2 reports results on a human-verified version of the dataset.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/realkie-fcc?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"RealKIE-FCC","organization_count":1,"organizations":["llm_stats"],"rank":2745,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/realkie-fcc?top_n=500"},{"aliases":["RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning"],"benchmark_id":"claire-radar-2606-10254","caveat":"A benchmark of authentic high-school exam responses for evaluating how accurately language-model judges assess human mathematical reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10254"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"RealMath-Eval","organization_count":1,"organizations":["claire_radar"],"rank":2746,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.10254"},{"aliases":[],"benchmark_id":"claire-radar-2608-30820","caveat":"Real-world occlusion boundary benchmark with definition-consistent labels and orientation maps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30820"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RealOOB","organization_count":1,"organizations":["claire_radar"],"rank":2747,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30820"},{"aliases":[],"benchmark_id":"opencompass-1124-realtoxicityprompts","caveat":"RealToxicityPrompts is a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widely used toxicity classifier. RealToxicityPrompts 是一个包含 100,000 个自然出现的、句子级提示的数据集，这些提示来自于大量的英语网络文本，并配有来自广泛使用的毒性分类器的毒性评分。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RealToxicityPrompts"],"document_share":0.0002723311546840959,"domain":"安全","name":"RealToxicityPrompts","organization_count":1,"organizations":["opencompass_hub"],"rank":2748,"released":"2020-11-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RealToxicityPrompts"},{"aliases":[],"benchmark_id":"llm-stats-realworldqa","caveat":"RealWorldQA is a benchmark designed to evaluate basic real-world spatial understanding capabilities of multimodal models. The initial release consists of over 700 anonymized images taken from vehicles and other real-world scenarios, each accompanied by a question and easily verifiable answer. Released by xAI as part of their Grok-1.5 Vision preview to test models' ability to understand natural scenes and spatial relationships in everyday visual contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/realworldqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"spatial_reasoning","name":"RealWorldQA","organization_count":1,"organizations":["llm_stats"],"rank":2749,"released":"2024-04-12","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/realworldqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1361-realworldqa","caveat":"RealWorldQA is a benchmark designed for real-world understanding, including 765 images, each accompanied by a question and a verifiable answer. RealWorldQA用于评估多模态模型在现实世界中的空间理解能力，包含765张图像，每张图像都配有一个问题和易于验证的答案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RealworldQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"RealworldQA","organization_count":1,"organizations":["opencompass_hub"],"rank":2750,"released":"2024-04-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RealworldQA"},{"aliases":[],"benchmark_id":"claire-radar-2606-03577","caveat":"ReasonMatch-Bench evaluates wide-baseline matching and spatial reasoning in MLLMs, with benchmarks stratified by viewpoint displacement and matching granularity across indoor, outdoor, and object-centric scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03577"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ReasonMatch-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2751,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03577"},{"aliases":[],"benchmark_id":"claire-radar-2606-06698","caveat":"RECAP evaluates continual-learning phenomena in prompt-based LLM adaptation under evolving constraints, using a proactive adapt-then-test protocol with constraint-level metrics for forgetting, regression, and forward transfer.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06698"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RECAP","organization_count":1,"organizations":["claire_radar"],"rank":2752,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06698"},{"aliases":[],"benchmark_id":"claire-radar-2605-26855","caveat":"Receipt Replay OOD is a small out-of-domain benchmark for screen replay detection. It uses receipts, which share planar geometry, curved corners, wear-and-tear artifacts, and text patterns with identity documents, to evaluate document replay detection models under cross-domain conditions without personally identifiable information constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26855"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Receipt Replay OOD","organization_count":1,"organizations":["claire_radar"],"rank":2753,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26855"},{"aliases":[],"benchmark_id":"claire-radar-2605-22413","caveat":"ReceiptBench evaluates multimodal large language models on visual information extraction from receipts. It includes 10k human-annotated receipts and four hierarchical subtasks: basic perception, format normalization, semantic reasoning, and structure parsing, with defined metrics and scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22413"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ReceiptBench","organization_count":1,"organizations":["claire_radar"],"rank":2754,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22413"},{"aliases":["RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering"],"benchmark_id":"claire-radar-2606-19218","caveat":"RECOM evaluates automatic metrics for open-ended Reddit question answering using authentic replies and a random-derangement noise floor.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19218"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RECOM","organization_count":1,"organizations":["claire_radar"],"rank":2755,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19218"},{"aliases":[],"benchmark_id":"claire-radar-2608-16645","caveat":"Reconstruction is a blind benchmark for recovering research ideas from pre-publication bibliographies, using a judge model to match hypotheses.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16645"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Reconstruction","organization_count":1,"organizations":["claire_radar"],"rank":2756,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16645"},{"aliases":[],"benchmark_id":"opencompass-530-record","caveat":"ReCoRD is a reading comprehension task, which requires to extract the answer from the article based on the given news article and question. ReCoRD是一个阅读理解任务，要求根据给定的新闻文章和问题，从文章中抽取出答案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ReCoRD"],"document_share":0.0002723311546840959,"domain":"推理","name":"ReCoRD","organization_count":1,"organizations":["opencompass_hub"],"rank":2757,"released":"2018-10-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ReCoRD"},{"aliases":[],"benchmark_id":"claire-radar-github-aryanthemscoder-recoveriq","caveat":"The repository describes an internal evaluation harness for a revenue recovery engine, but no formal benchmark name or independent reuse protocol is declared.","document_count":1,"document_ids":["claire_radar:https://github.com/aryanthemscoder/recoveriq"],"document_share":0.0002723311546840959,"domain":"Finance","name":"recoveriq","organization_count":1,"organizations":["claire_radar"],"rank":2758,"released":"2026-08-21","source":"claire_radar","url":"https://github.com/aryanthemscoder/recoveriq"},{"aliases":[],"benchmark_id":"llm-stats-recreationbench","caveat":"RecreationBench is Qwen's long-horizon application-recreation benchmark for hybrid agents across Ubuntu, macOS, Windows, Android, and the web.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/recreationbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"RecreationBench","organization_count":1,"organizations":["llm_stats"],"rank":2759,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/recreationbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-19881","caveat":"REDACT is a multilingual benchmark for PII detection with 13,427 records, 51 entity types, and controlled generation axes. It allows stratified evaluation via metadata fields and includes an evaluation harness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19881"],"document_share":0.0002723311546840959,"domain":"General AI","name":"REDACT","organization_count":1,"organizations":["claire_radar"],"rank":2760,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.19881"},{"aliases":[],"benchmark_id":"claire-radar-2606-18782","caveat":"RedactionBench evaluates contextual redaction of PII across 200 documents and 11 domains, with a character-level R-Score metric that treats semantically similar redactions equally.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18782"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RedactionBench","organization_count":1,"organizations":["claire_radar"],"rank":2761,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18782"},{"aliases":["REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems"],"benchmark_id":"claire-radar-2608-10669","caveat":"Evaluates LLM agent safety through executable red-teaming, adversarial case generation, and verification of harmful effects across 1,661 cases and five service surfaces.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10669"],"document_share":0.0002723311546840959,"domain":"Agents","name":"REDAgentBench","organization_count":1,"organizations":["claire_radar"],"rank":2762,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10669"},{"aliases":[],"benchmark_id":"opencompass-1333-redcode","caveat":"RedCode provides comprehensive and practical evaluations on the safety of code agents, including 4,050 risky test cases covering 25 types of critical vulnerabilities spanning 8 domains and 160 prompts aiming to generate harmful code or software. RedCode旨在为LLM代码智能体的安全性提供全面实用的评估，包括来自8个领域25种关键漏洞的4050个风险测试用例，以及160个生成有害代码的提示。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RedCode"],"document_share":0.0002723311546840959,"domain":"安全","name":"RedCode","organization_count":1,"organizations":["opencompass_hub"],"rank":2763,"released":"2024-11-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RedCode"},{"aliases":[],"benchmark_id":"claire-radar-2605-29893","caveat":"RedundancyBench is a benchmark for detecting redundant steps in agent trajectories. It contains diverse tasks with annotated trajectories where each step is labeled for its contribution to task completion.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29893"],"document_share":0.0002723311546840959,"domain":"Agents","name":"RedundancyBench","organization_count":1,"organizations":["claire_radar"],"rank":2764,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29893"},{"aliases":[],"benchmark_id":"claire-radar-2606-26968","caveat":"RedVox is a multilingual safety and fairness benchmark for speech models, built on real voices. It covers unsafe and unfair stereotypical requests across five languages and evaluates models under naturalistic conditions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26968"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RedVox","organization_count":1,"organizations":["claire_radar"],"rank":2765,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.26968"},{"aliases":[],"benchmark_id":"llm-stats-refcoco-avg","caveat":"RefCOCO-avg measures object grounding accuracy averaged across RefCOCO, RefCOCO+, and RefCOCOg benchmarks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/refcoco-avg?top_n=500"],"document_share":0.0002723311546840959,"domain":"spatial_reasoning","name":"RefCOCO-avg","organization_count":1,"organizations":["llm_stats"],"rank":2766,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/refcoco-avg?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-refcocog","caveat":"RefCOCOg is a referring expression comprehension benchmark that evaluates spatial grounding in images. Given a natural language expression describing an object, the model must localize the correct region, evaluated by accuracy at a 0.5 IoU threshold. It features longer, more descriptive expressions than RefCOCO and RefCOCO+.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/refcocog?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"RefCOCOg","organization_count":1,"organizations":["llm_stats"],"rank":2767,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/refcocog?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-37969","caveat":"High-resolution video generation is expensive, as its cost grows rapidly with the number of spatiotemporal tokens.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37969"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Refiner-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2768,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37969"},{"aliases":[],"benchmark_id":"claire-radar-2608-14379","caveat":"Evaluates vision-language-action models on reaction-critical manipulation across six dynamic tasks, with configurable latency under synchronous and asynchronous inference in a simulated environment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14379"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"ReflexBench","organization_count":1,"organizations":["claire_radar"],"rank":2769,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14379"},{"aliases":[],"benchmark_id":"claire-radar-2606-01223","caveat":"RefMem-Bench is a benchmark for reflective memory in long-horizon dialogue, containing 26K QA instances across eight reflective-memory dimensions and three task formats.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01223"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RefMem-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2770,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01223"},{"aliases":[],"benchmark_id":"claire-radar-2610-03356","caveat":"Large Language Model (LLM) agents are increasingly deployed in high-stakes settings such as industrial maintenance and equipment fault troubleshooting, where workers occupy a variety of roles.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03356"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ReFract","organization_count":1,"organizations":["claire_radar"],"rank":2771,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03356"},{"aliases":[],"benchmark_id":"llm-stats-refspatialbench","caveat":"RefSpatialBench evaluates spatial reference understanding and grounding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/refspatialbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"spatial_reasoning","name":"RefSpatialBench","organization_count":1,"organizations":["llm_stats"],"rank":2772,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/refspatialbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-github-audn-ai-refusal-benchmark","caveat":"Measures LLM refusal rates on a set of harmful prompts via OpenAI-compatible endpoints.","document_count":1,"document_ids":["claire_radar:https://github.com/audn-ai/refusal-benchmark"],"document_share":0.0002723311546840959,"domain":"Finance","name":"refusal-benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2773,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/audn-ai/refusal-benchmark"},{"aliases":["reg-affairs-bench"],"benchmark_id":"claire-radar-huggingface-arca-inc-reg-affairs-bench","caveat":"Evaluates AI systems on regulatory affairs deliverables from drugs, biologics, and medical devices using scenarios and dossier documents, scored against task-specific rubrics by a jury of LLM judges.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/arca-inc/reg-affairs-bench"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Reg Affairs Bench","organization_count":1,"organizations":["claire_radar"],"rank":2774,"released":"2026-10-10","source":"claire_radar","url":"https://huggingface.co/datasets/arca-inc/reg-affairs-bench"},{"aliases":[],"benchmark_id":"claire-radar-2608-06850","caveat":"RegionDet is a benchmark for region detection with eight categories, using COCO-style bounding-box annotations and evaluation protocols. It aims to extend object detection to region-level targets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06850"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RegionDet","organization_count":1,"organizations":["claire_radar"],"rank":2775,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.06850"},{"aliases":["RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models"],"benchmark_id":"claire-radar-2609-16847","caveat":"Intended for region-level retrieval in large multimodal models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16847"],"document_share":0.0002723311546840959,"domain":"General AI","name":"REGMB","organization_count":1,"organizations":["claire_radar"],"rank":2776,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.16847"},{"aliases":[],"benchmark_id":"claire-radar-2607-21143","caveat":"RegretBench evaluates clarification policies in multi-turn conversational LLMs, using hidden-intent tasks and a regret-based objective to measure value loss relative to a reference policy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21143"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RegretBench","organization_count":1,"organizations":["claire_radar"],"rank":2777,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21143"},{"aliases":[],"benchmark_id":"claire-radar-2606-20736","caveat":"ReKey is a live benchmark protocol that regenerates visual keys in VQA images at evaluation time, creating fresh instances with new answers, to combat data leakage and memorization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20736"],"document_share":0.0002723311546840959,"domain":"General AI","name":"REKEY","organization_count":1,"organizations":["claire_radar"],"rank":2778,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.20736"},{"aliases":["RAD: Rule-Augmented Relational Anomaly Detection","RAD"],"benchmark_id":"claire-radar-2608-23468","caveat":"The Relational Anomaly Detection Benchmark evaluates anomaly detection in multi-table relational databases, spanning three settings: LANL cybersecurity events, Amazon user churn, and H&M user churn. It provides standardized tasks and evaluation protocols for relational anomaly detection methods.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23468v1"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"Relational Anomaly Detection Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2779,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23468v1"},{"aliases":[],"benchmark_id":"claire-radar-2607-18438","caveat":"Relay-Bench evaluates LLMs on multi-domain reasoning chains, presenting composite problems that combine subproblems from domains like visual reasoning, coding, math, information extraction, and data analysis. It measures overall accuracy on these complex, text-only tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18438"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Relay-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2780,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.18438"},{"aliases":[],"benchmark_id":"claire-radar-2606-07141","caveat":"REMEDI is a benchmark for machine unlearning in multi-label clinical disease inference, built on MIMIC-III, covering diverse forget sets and tasks with utility and unlearning metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07141"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"REMEDI","organization_count":1,"organizations":["claire_radar"],"rank":2781,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07141"},{"aliases":[],"benchmark_id":"claire-radar-2606-16178","caveat":"ReMemBench is a benchmark with eight diverse household manipulation tasks across four categories of short-term memory, designed to evaluate memory mechanisms in visuomotor policies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16178"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"ReMemBench","organization_count":1,"organizations":["claire_radar"],"rank":2782,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16178"},{"aliases":[],"benchmark_id":"claire-radar-2610-08684","caveat":"Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and motion.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.08684"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RenderBench","organization_count":1,"organizations":["claire_radar"],"rank":2783,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.08684"},{"aliases":[],"benchmark_id":"claire-radar-2606-29937","caveat":"REPAIR-Bench evaluates robot error perception and recovery in human-robot interaction. It includes 214 interaction trials from 41 participants with four induced failure types, synchronized facial action units, head pose, speech transcripts, and post-interaction reports. Three tasks cover failure detection across sessions, failure-type classification, and recovery prediction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29937"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"REPAIR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2784,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.29937"},{"aliases":[],"benchmark_id":"claire-radar-2607-28008","caveat":"RepBench compiles 353 public benchmarks into 46,149 probe texts spanning 94 capabilities, with a taxonomy of 182 clusters in 13 families. It evaluates representation readout methods across 12 models under cross-benchmark transfer, providing a reusable closed-loop pipeline for capability-aligned probing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.28008"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RepBench","organization_count":1,"organizations":["claire_radar"],"rank":2785,"released":"2026-07-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.28008"},{"aliases":[],"benchmark_id":"opencompass-1317-repliqa","caveat":"RepLiQA is suited for question-answering and topic retrieval tasks, including collection of five splits of test sets. Accurate answers can only be generated if a model can find relevant content within the provided document. RepLiQA适用于问答和主题检索任务，集合了是5个测试集；只有当模型可以在提供的文档中找到相关内容时，才能生成准确的答案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RepLiQA"],"document_share":0.0002723311546840959,"domain":"推理","name":"RepLiQA","organization_count":1,"organizations":["opencompass_hub"],"rank":2786,"released":"2024-06-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RepLiQA"},{"aliases":[],"benchmark_id":"llm-stats-repo-env","caveat":"Repo Env evaluates an agent's ability to set up, configure, and run real repositories, including dependency resolution and environment management.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/repo-env?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Repo Env","organization_count":1,"organizations":["llm_stats"],"rank":2787,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/repo-env?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-github-parfait18-repo-context-ambiguity","caveat":"Measures whether code generation models violate implicit repository conventions, using automatic AST oracles for constraint and security violations.","document_count":1,"document_ids":["claire_radar:https://github.com/Parfait18/repo-context-ambiguity"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"repo-context-ambiguity","organization_count":1,"organizations":["claire_radar"],"rank":2788,"released":"2026-08-23","source":"claire_radar","url":"https://github.com/Parfait18/repo-context-ambiguity"},{"aliases":[],"benchmark_id":"llm-stats-repobench","caveat":"RepoBench is a benchmark for evaluating repository-level code auto-completion systems through three interconnected tasks: RepoBench-R (retrieval of relevant code snippets across files), RepoBench-C (code completion with cross-file and in-file context), and RepoBench-P (pipeline combining retrieval and prediction). Supports Python and Java programming languages and addresses the gap in evaluating real-world, multi-file programming scenarios by providing a more complete comparison of performance in auto-completion systems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/repobench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"RepoBench","organization_count":1,"organizations":["llm_stats"],"rank":2789,"released":"2023-06-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/repobench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-26177","caveat":"RepoMirage is a two-stage evaluation suite built on SWE-Bench Verified that applies semantics-preserving repository-level perturbations and extended tasks to probe repository context reasoning in code agents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26177"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"RepoMirage","organization_count":1,"organizations":["claire_radar"],"rank":2790,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26177"},{"aliases":[],"benchmark_id":"claire-radar-2606-06492","caveat":"RepoPeftBench evaluates repository-aware code models on static and software-evolution tracks using assertion-completion tasks derived from Python repositories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06492"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"RepoPeftBench","organization_count":1,"organizations":["claire_radar"],"rank":2791,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06492"},{"aliases":[],"benchmark_id":"claire-radar-2608-04783","caveat":"Evaluates repository-level code understanding via 500 open-ended questions derived from GitHub Discussions across 50 repositories, scored with checklist-based verification for factual correctness and clarity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04783"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"RepoProbe","organization_count":1,"organizations":["claire_radar"],"rank":2792,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04783"},{"aliases":[],"benchmark_id":"llm-stats-repoqa","caveat":"RepoQA is a benchmark for evaluating long-context code understanding capabilities of Large Language Models through the Searching Needle Function (SNF) task, where LLMs must locate specific functions in code repositories using natural language descriptions. The benchmark contains 500 code search tasks spanning 50 repositories across 5 modern programming languages (Python, Java, TypeScript, C++, and Rust), tested on 26 general and code-specific LLMs to assess their ability to comprehend and navigate code repositories.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/repoqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"RepoQA","organization_count":1,"organizations":["llm_stats"],"rank":2793,"released":"2024-06-10","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/repoqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-25996","caveat":"Evaluates long-context LLMs on repository-level code reasoning through Output Prediction and Call Chain Prediction tasks, using dynamic tracing and I/O rewriting to reduce memorization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25996"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"RepoReasoner","organization_count":1,"organizations":["claire_radar"],"rank":2794,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25996"},{"aliases":["Do Coding Agents Reuse Existing Code or Reinvent the Wheel?"],"benchmark_id":"claire-radar-2609-35357","caveat":"Evaluates coding agents on reuse of existing code in real repositories through multi-turn tasks measuring reuse rate, recall, and redundancy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35357"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"RepoReuse","organization_count":1,"organizations":["claire_radar"],"rank":2795,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35357"},{"aliases":[],"benchmark_id":"claire-radar-2610-13022","caveat":"As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13022"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"ReqClarifyBench","organization_count":1,"organizations":["claire_radar"],"rank":2796,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13022"},{"aliases":[],"benchmark_id":"claire-radar-2608-16011","caveat":"ReRef-3D benchmarks language-guided 3D scene rearrangement with 33,826 instructions across 998 scenes, evaluating placement validity and relation satisfaction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16011"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ReRef-3D","organization_count":1,"organizations":["claire_radar"],"rank":2797,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16011"},{"aliases":[],"benchmark_id":"claire-radar-2606-02852","caveat":"RESCAST-100K evaluates cross-domain residential load and indoor temperature forecasting. It provides ~100,000 EnergyPlus-simulated U.S. homes with 15-minute time series for total load, HVAC load, and indoor temperature, plus weather, setpoints, and static covariates. It includes configurable domain axes and integrates five real-world datasets for sim-to-real evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02852"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RESCAST-100K","organization_count":1,"organizations":["claire_radar"],"rank":2798,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02852"},{"aliases":["RESCUE-BENCH"],"benchmark_id":"claire-radar-2609-09657","caveat":"Evaluates LLMs on relation-aware emotional support through six tasks derived from couple and family interview conversations, assessing relational understanding and relation-sensitive support.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09657"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RESCUE","organization_count":1,"organizations":["claire_radar"],"rank":2799,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09657"},{"aliases":[],"benchmark_id":"claire-radar-2606-01848","caveat":"RescueBench evaluates embodied search-and-rescue agents in simulated photo-realistic environments. It comprises a four-stage pipeline: multimodal exploration, target rescue, memory-guided return, and final handoff. Five difficulty levels vary environmental complexity, clue ambiguity, and spatial hierarchy. Automatic episode generation and annotation support scalable evaluation. A unified benchmark framework and runner scripts provide standardized scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01848"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"RescueBench","organization_count":1,"organizations":["claire_radar"],"rank":2800,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01848"},{"aliases":["RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases"],"benchmark_id":"claire-radar-2609-10092","caveat":"Evaluates LLM agents' ability to predict research attention shifts by forecasting paper shares across 278 AI/ML fields over six months using a rolling temporally restricted arXiv corpus.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10092"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Research Attention Prediction (RAP)","organization_count":1,"organizations":["claire_radar"],"rank":2801,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10092"},{"aliases":[],"benchmark_id":"claire-radar-2606-07591","caveat":"ResearchClawBench evaluates autonomous scientific research agents across 40 tasks from 10 domains, each grounded in a published paper with provided literature and raw data. It uses expert-curated multimodal rubrics to score target-paper-level re-discovery.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07591"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ResearchClawBench","organization_count":1,"organizations":["claire_radar"],"rank":2802,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2606.07591"},{"aliases":[],"benchmark_id":"llm-stats-researchclawbench","caveat":"ResearchClawBench evaluates research agents on realistic, tool-using research tasks that require code execution and filesystem workspace interaction.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/researchclawbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"research","name":"ResearchClawBench","organization_count":1,"organizations":["llm_stats"],"rank":2803,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/researchclawbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-11074","caveat":"ResearchQA is a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers across eight domains, designed for citation-grounded evaluation with multiple valid supporting passages and grounded refusal.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11074"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ResearchQA","organization_count":1,"organizations":["claire_radar"],"rank":2804,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11074"},{"aliases":[],"benchmark_id":"claire-radar-2608-04514","caveat":"Evaluates clinical decision-making in respiratory specialty care using 623 multimodal cases, including AECOPD-PIM with 427 open-ended COPD cases and PNBIM with 196 pulmonary nodule cases that combine chest CT with clinical information.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04514"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"RESPClinBench","organization_count":1,"organizations":["claire_radar"],"rank":2805,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04514"},{"aliases":[],"benchmark_id":"claire-radar-2609-00384","caveat":"RestoreBench evaluates LLM agents on restoring AC power-flow convergence through reactive-control maneuvers in non-convergent power-grid snapshots, with success determined by a physics solver and fixed budget.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00384"],"document_share":0.0002723311546840959,"domain":"Agents","name":"RestoreBench","organization_count":1,"organizations":["claire_radar"],"rank":2806,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2609.00384"},{"aliases":[],"benchmark_id":"claire-radar-2606-15862","caveat":"RetailBench is a simulation benchmark evaluating tool-using LLM agents in single-store supermarket operations over a 180-day horizon. It covers pricing, replenishment, supplier selection, assortment, inventory aging, customer feedback, external events, and cash-flow constraints, with a privileged oracle policy for comparison.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15862"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"RetailBench","organization_count":1,"organizations":["claire_radar"],"rank":2807,"released":"2026-06-14","source":"claire_radar","url":"https://arxiv.org/abs/2606.15862"},{"aliases":[],"benchmark_id":"claire-radar-2609-08090","caveat":"RevalExo evaluates inertial and visual locomotion mode recognition on 10.1 hours of frame-level annotated data from 27 participants across older adults, stroke survivors, and probable sarcopenia, spanning 11 locomotion modes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08090"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"RevalExo","organization_count":1,"organizations":["claire_radar"],"rank":2808,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08090"},{"aliases":[],"benchmark_id":"opencompass-1087-reveal","caveat":"REVEAL(ReasoningVerification Evaluation) is a new dataset to benchmark automatic verifiers of complex Chain-ofThought reasoning in open-domain question answering settings. Reveal 是一个用于基准测试开放域问答环境中复杂链式推理自动验证器的新数据集。Reveal 包含关于语言模型答案中每个推理步骤的相关性、证据段落的归因和逻辑正确性的全面标签，涵盖多种数据集和最先进的语言模型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Reveal"],"document_share":0.0002723311546840959,"domain":"推理","name":"Reveal","organization_count":1,"organizations":["opencompass_hub"],"rank":2809,"released":"2024-05-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Reveal"},{"aliases":[],"benchmark_id":"claire-radar-2606-26094","caveat":"RevengeBench is a benchmark for recovering code-space policies from behavioral traces. It includes 75 LLM-generated policies across five game environments, where a learner designs behavioral probes and submits executable hypotheses, evaluated using continuous action-distance metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26094"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RevengeBench","organization_count":1,"organizations":["claire_radar"],"rank":2810,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.26094"},{"aliases":[],"benchmark_id":"claire-radar-2609-17745","caveat":"Benchmark controlling reversibility via a continuous parameter and providing a reset oracle across eight manipulation settings in five physics engines.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17745"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"REVERSAL-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":2811,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17745"},{"aliases":[],"benchmark_id":"claire-radar-2608-27154","caveat":"Evaluates vision-language models on visual text error correction in real-world images, requiring models to identify and fix text errors grounded in visual context.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27154"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ReViCo","organization_count":1,"organizations":["claire_radar"],"rank":2812,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.27154"},{"aliases":[],"benchmark_id":"claire-radar-2609-36161","caveat":"Can coding agents restore software that no longer runs while preserving its underlying methods, and reconstruct industrial software engines from open specifications?","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36161"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"ReviveBench","organization_count":1,"organizations":["claire_radar"],"rank":2813,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.36161"},{"aliases":[],"benchmark_id":"opencompass-1915-rewardbench","caveat":"RewardBench is a benchmark designed to evaluate the capabilities and safety of reward models (including those trained with Direct Preference Optimization, DPO). RewardBench是一个用于评估奖励模型（包括通过直接偏好优化（DPO）训练的模型）能力和安全性的基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RewardBench"],"document_share":0.0002723311546840959,"domain":"指令跟随","name":"RewardBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2814,"released":"2025-06-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RewardBench"},{"aliases":[],"benchmark_id":"opencompass-2033-rexbench","caveat":"RExBench is a benchmark for evaluating large language model agents’ ability to autonomously implement AI research extensions, focusing on code generation, experimental design, and research comprehension. RExBench 是一个评估大型语言模型代理在自动实现 AI 研究扩展能力的基准，涵盖代码生成、实验设计和研究理解等维度。该基准包含12个基于真实论文和代码库的任务，每项任务由领域专家提供扩展指令，并通过自动化基础设施执行代理输出以验证成功标准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RExBench"],"document_share":0.0002723311546840959,"domain":"代码","name":"RExBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2815,"released":"2025-06-27","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RExBench"},{"aliases":[],"benchmark_id":"opencompass-1656-rfuav","caveat":"RFUAV offers a comprehensive benchmark dataset for Radio-Frequency (RF)-based drone detection and identification. RFUAV 提供了一个基于射频（RF）的无人机检测和识别的全面基准数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RFUAV"],"document_share":0.0002723311546840959,"domain":"其他","name":"RFUAV","organization_count":1,"organizations":["opencompass_hub"],"rank":2816,"released":"2025-03-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RFUAV"},{"aliases":[],"benchmark_id":"claire-radar-2609-34455","caveat":"We study reasoning in Rule-Governed Decision Tasks (RGDTs), where models apply external rules to case facts and justify decisions, as required in policy, contract, and compliance settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34455"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RGDT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2817,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34455"},{"aliases":[],"benchmark_id":"claire-radar-2610-10947","caveat":"Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model systems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10947"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RH-Detect","organization_count":1,"organizations":["claire_radar"],"rank":2818,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10947"},{"aliases":["RIDE: An Open Dataset and Benchmark for Train Delay Prediction"],"benchmark_id":"claire-radar-2606-05070","caveat":"RIDE is an open, nationwide-scale benchmark for train-delay prediction on the Belgian railway network, with standardized splits and evaluation using MAE and RMSE.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05070"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RIDE","organization_count":1,"organizations":["claire_radar"],"rank":2819,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05070"},{"aliases":[],"benchmark_id":"claire-radar-2609-17985","caveat":"RideWay is an efficiency-centered benchmark for ridehailing agents in a stateful tool-calling environment, proposing an Efficiency Utility metric that discounts successful trajectories for excess tool calls and user-facing turns relative to reference effort.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17985"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"RideWay","organization_count":1,"organizations":["claire_radar"],"rank":2820,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.17985"},{"aliases":[],"benchmark_id":"claire-radar-2606-23927","caveat":"RIFT-Bench is described as a methodology for dynamic red-teaming of agentic AI systems, using a graph representation to enable unified evaluations across diverse agentic architectures. It operates in two phases: Discovery and Scanning, deploying adaptive adversarial attacks and producing evaluation reports.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23927"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RIFT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2821,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23927"},{"aliases":[],"benchmark_id":"claire-radar-2609-02864","caveat":"Intended to evaluate reasoning-driven image generation across concept-based, transformation-based, pattern and structure, and scenario-based domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02864"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RIG-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":2822,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02864"},{"aliases":[],"benchmark_id":"claire-radar-2608-15555","caveat":"RigidBench evaluates rigid-body physics in video generation models using a simulator-grounded benchmark with 100 examples and five tasks. It provides ten measurements covering motion, geometry, identity, background stability, and appearance, with per-frame data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15555"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RigidBench","organization_count":1,"organizations":["claire_radar"],"rank":2823,"released":"2026-08-16","source":"claire_radar","url":"https://arxiv.org/abs/2608.15555"},{"aliases":[],"benchmark_id":"claire-radar-2606-22678","caveat":"RigorBench evaluates autonomous AI coding agents on engineering process discipline across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity. It includes 30 tasks in five categories and a composite RigorScore metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22678"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"RigorBench","organization_count":1,"organizations":["claire_radar"],"rank":2824,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22678"},{"aliases":[],"benchmark_id":"opencompass-2371-rigorousbench","caveat":"Artificial intelligence is undergoing the paradigm shift from closed language models to interconnected agent systems capable of external perception and information integration. As a representative embodiment, Deep Research Agents (DRAs) systematically exhibit the capabilities for task decomposition, Artificial intelligence is undergoing the paradigm shift from closed language models to interconnected agent systems capable of external perception and information integration. As a representative embodiment, Deep Research Agents (DRAs) systematically exhibit the capabilities for task decomposition,","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RigorousBench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"RigorousBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2825,"released":"2025-10-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RigorousBench"},{"aliases":[],"benchmark_id":"opencompass-2048-risebench","caveat":"RISEBench is a benchmark for evaluating large multimodal models (LMMs) on reasoning-informed visual editing tasks, targeting models with image understanding and generation capabilities. RISEBench 是一个用于评估多模态大模型（LMMs）在推理驱动视觉编辑任务中能力的基准，面向具备图像理解与生成能力的模型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RISEBench"],"document_share":0.0002723311546840959,"domain":"推理","name":"RISEBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2826,"released":"2025-04-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RISEBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-16900","caveat":"Pairs synthetic token-text restoration inputs with human-labeled local web environments for investigation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16900"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RiskChainBench","organization_count":1,"organizations":["claire_radar"],"rank":2827,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.16900"},{"aliases":[],"benchmark_id":"claire-radar-2609-29429","caveat":"Evaluates binary detection of AI alignment failures across ten failure types, covering 44 benchmarks and target-model outputs presented in a standard instance format.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.29429"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RLCDAlignBench","organization_count":1,"organizations":["claire_radar"],"rank":2828,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.29429"},{"aliases":[],"benchmark_id":"claire-radar-2609-34210","caveat":"Coding agents are beginning to move beyond purely digital tasks to tackle physical-world challenges, particularly in robotics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34210"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"RLE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2829,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34210"},{"aliases":[],"benchmark_id":"opencompass-1524-rm-bench","caveat":"RM-Bench, a benchmark dataset for evaluating reward models of language modeling. RM-Bench，一个用于评估语言模型奖励模型的基准数据集","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RM-Bench"],"document_share":0.0002723311546840959,"domain":"语言","name":"RM-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2830,"released":"2024-10-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RM-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2610-05950","caveat":"LLM-driven evolution enables iterative model development, but two practical goals remain underexplored: finding model designs that transfer across related tasks and sustaining improvement when training is expensive.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05950"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RMD-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2831,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.05950"},{"aliases":[],"benchmark_id":"claire-radar-2610-05414","caveat":"Although the advancement of vision-language models (VLMs) has endowed robots with enhanced environmental understanding and task reasoning, a comprehensive evaluation methodology is important to advance the integration of VLMs in robotic na…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05414"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"RMMBench","organization_count":1,"organizations":["claire_radar"],"rank":2832,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05414"},{"aliases":[],"benchmark_id":"claire-radar-2606-19338","caveat":"RNG-Bench evaluates multimodal LLMs in controllable non-Markov games, requiring reconstruction of past observations and acting on them, with two games: Matching Pairs and 3D Maze.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19338"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RNG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2833,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19338"},{"aliases":["rob2-ai-benchmark"],"benchmark_id":"claire-radar-github-coresr-rob2-ai-benchmark","caveat":"Unified frozen benchmark for AI risk-of-bias assessment using Cochrane RoB 2, built from three public expert-annotated sources, scored by agreement with human judgments at signalling, domain, and overall levels.","document_count":1,"document_ids":["claire_radar:https://github.com/CoreSR/rob2-ai-benchmark"],"document_share":0.0002723311546840959,"domain":"Safety & Trustworthiness","name":"RoB 2 AI Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2834,"released":"2026-10-12","source":"claire_radar","url":"https://github.com/CoreSR/rob2-ai-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-36605","caveat":"Understanding ongoing robot manipulation requires models to interpret visual observations in relation to interaction history and task progress.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36605"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"RoboChrono","organization_count":1,"organizations":["claire_radar"],"rank":2835,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36605"},{"aliases":["Exploring the Limits of Combinatorial Generalization in Embodied Manipulation"],"benchmark_id":"claire-radar-2610-15425","caveat":"A benchmark for skill recombination in embodied manipulation without combined skill training trajectories, consisting of 19 compositional tasks with evaluation tracks for different skill execution paradigms.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.15425"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"RoboCombine","organization_count":1,"organizations":["claire_radar"],"rank":2836,"released":"2026-10-11","source":"claire_radar","url":"https://arxiv.org/abs/2610.15425"},{"aliases":[],"benchmark_id":"claire-radar-2607-04434","caveat":"RoboDojo is a unified sim-and-real benchmark for evaluating generalist robot manipulation policies, comprising 42 simulation tasks and 18 real-world tasks across three robot embodiments, covering five capability dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04434"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"RoboDojo","organization_count":1,"organizations":["claire_radar"],"rank":2837,"released":"2026-07-05","source":"claire_radar","url":"https://arxiv.org/abs/2607.04434"},{"aliases":[],"benchmark_id":"claire-radar-2609-25636","caveat":"Diagnostic benchmark for instruction following in embodied agents, featuring shared-scene tasks that require language to disambiguate objectives.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25636"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"RoboFollow","organization_count":1,"organizations":["claire_radar"],"rank":2838,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.25636"},{"aliases":[],"benchmark_id":"claire-radar-2606-22338","caveat":"RoboMME-Interference evaluates robot long-context memory under cross-session interference. The benchmark builds on RoboMME, constructing session histories per query episode with relevant demonstration plus controlled unrelated sessions, and measures task success for memory-augmented vision-language-action models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22338"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"RoboMME-Interference","organization_count":1,"organizations":["claire_radar"],"rank":2839,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22338"},{"aliases":[],"benchmark_id":"claire-radar-2606-13040","caveat":"RoboProcessBench evaluates vision-language models on process-aware understanding in robotic manipulation, with 12 diagnostic question families covering static monitoring and dynamic reasoning over execution traces. The benchmark includes 58k QA pairs across 260 tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13040"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"RoboProcessBench","organization_count":1,"organizations":["claire_radar"],"rank":2840,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13040"},{"aliases":[],"benchmark_id":"claire-radar-2610-10388","caveat":"Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10388"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"RoboQuest","organization_count":1,"organizations":["claire_radar"],"rank":2841,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10388"},{"aliases":[],"benchmark_id":"claire-radar-2609-28952","caveat":"RoboRecover is introduced as a benchmark for robot policy recovery, but no environment, task instances, or scoring procedure are released.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.28952"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"RoboRecover","organization_count":1,"organizations":["claire_radar"],"rank":2842,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.28952"},{"aliases":["Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation"],"benchmark_id":"claire-radar-2609-08209","caveat":"Benchmark for robot learning from observation using real human videos, simulated trajectories, and evaluation environments across ten tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08209"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"RoboReel","organization_count":1,"organizations":["claire_radar"],"rank":2843,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08209"},{"aliases":[],"benchmark_id":"claire-radar-2609-05401","caveat":"Evaluates paraphrase invariance in vision-language reward models for robotic learning using 2,390 real-robot trajectories, ground-truth progress labels, and 21,673 verified paraphrases spanning lexical, syntactic, and action-goal rewrites, measuring stability of predicted progress scores under semantically equivalent instructions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05401"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"ROBORMBENCH","organization_count":1,"organizations":["claire_radar"],"rank":2844,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05401"},{"aliases":[],"benchmark_id":"claire-radar-2606-02277","caveat":"RoboSemanticBench (RSB) is an embodied benchmark evaluating whether vision-language-action models use instruction semantics to select and grasp the correct physical target among candidate blocks in response to math or general-knowledge questions. It includes six suites with four- or ten-choice variants, procedural arithmetic, GSM8K-style, and MMLU-style questions, with diagnostic metrics separating task success from grasp success.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02277"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"RoboSemanticBench","organization_count":1,"organizations":["claire_radar"],"rank":2845,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02277"},{"aliases":["RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?"],"benchmark_id":"claire-radar-2609-05324","caveat":"RoboSPA is a large-scale robotic manipulation benchmark that evaluates Vision-Language-Action models on fine-grained spatial reasoning and long-horizon procedural planning. It includes 10 task categories and 56 base tasks, each instantiated at five difficulty levels to yield 280 variants, with 527K trajectories. Beyond binary success, it reports diagnostic metrics such as ONTA and Progress Score. Code and data are available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05324"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"RoboSPA","organization_count":1,"organizations":["claire_radar"],"rank":2846,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05324"},{"aliases":[],"benchmark_id":"llm-stats-robospatialhome","caveat":"RoboSpatialHome evaluates spatial understanding for robotic home navigation and manipulation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/robospatialhome?top_n=500"],"document_share":0.0002723311546840959,"domain":"robotics","name":"RoboSpatialHome","organization_count":1,"organizations":["llm_stats"],"rank":2847,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/robospatialhome?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-01600","caveat":"RoboTrustBench evaluates the trustworthiness of video world models for robotic manipulation across four scenarios: Normal, Constraint-Sensitive, Counterfactual, and Adversarial. It contains 1,207 expert-validated instruction-image pairs from DROID episodes and a six-dimensional evaluation protocol with 13 fine-grained criteria.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01600"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"RoboTrustBench","organization_count":1,"organizations":["claire_radar"],"rank":2848,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01600"},{"aliases":[],"benchmark_id":"claire-radar-2606-03312","caveat":"RobotValues evaluates household robot planners in 10K value-conflict scenarios with realistic images, testing value preferences and action selection under conflicting human values.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03312"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"RobotValues","organization_count":1,"organizations":["claire_radar"],"rank":2849,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03312"},{"aliases":[],"benchmark_id":"claire-radar-2609-26292","caveat":"Evaluates robot manipulation policies across diverse physical conditions, treating physical-condition diversity as an explicit evaluation dimension.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.26292"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"RoboTwin-Phys","organization_count":1,"organizations":["claire_radar"],"rank":2850,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.26292"},{"aliases":[],"benchmark_id":"claire-radar-2610-10409","caveat":"General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10409"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"RobotWorld","organization_count":1,"organizations":["claire_radar"],"rank":2851,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10409"},{"aliases":[],"benchmark_id":"claire-radar-2609-17843","caveat":"A benchmark dataset for cross-domain video anomaly detection in robotic arm manipulation, featuring 6.9 GB of videos with action and anomaly type splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17843"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RoboVAD","organization_count":1,"organizations":["claire_radar"],"rank":2852,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17843"},{"aliases":[],"benchmark_id":"claire-radar-2607-04610","caveat":"RoboVista evaluates vision-language models on robot question answering with 474 visual question answering instances spanning 39 task types across agricultural, industrial, domestic, surgical, and autonomous driving domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04610"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"RoboVista","organization_count":1,"organizations":["claire_radar"],"rank":2853,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.04610"},{"aliases":[],"benchmark_id":"claire-radar-2605-30326","caveat":"RoboWits is a bi-manual robotic benchmark designed to evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions. It includes 30 seed tasks and 208 mutated tasks with graded difficulty across geometry, material, and assembly-based reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30326"],"document_share":0.0002723311546840959,"domain":"Agents","name":"RoboWits","organization_count":1,"organizations":["claire_radar"],"rank":2854,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30326"},{"aliases":[],"benchmark_id":"llm-stats-robust-if","caveat":"Robust IF evaluates instruction-following robustness on diverse, hard prompts, measuring whether a model reliably adheres to constraints across challenging single-turn and multi-turn scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/robust-if?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Robust IF","organization_count":1,"organizations":["llm_stats"],"rank":2855,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/robust-if?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-16243","caveat":"Evaluates robustness of multimodal small language models for industrial anomaly detection across open-ended queries and visual degradations, using multiple-choice accuracy and LLM-judged open-ended responses.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.16243"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RobustMAD","organization_count":1,"organizations":["claire_radar"],"rank":2856,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2607.16243"},{"aliases":["Rockman"],"benchmark_id":"claire-radar-github-inserloft-rockman","caveat":"An evaluation suite of 220 code and software engineering tasks measuring LLM performance via executable tests, with a primary 44-task private split.","document_count":1,"document_ids":["claire_radar:https://github.com/InserLoft/Rockman"],"document_share":0.0002723311546840959,"domain":"Code","name":"Rockman Benchmark v0.2","organization_count":1,"organizations":["claire_radar"],"rank":2857,"released":"2026-09-20","source":"claire_radar","url":"https://github.com/InserLoft/Rockman"},{"aliases":["MINER: Multi-crop INference-time Enhancement for Rare-Object Retrieval with Frozen Dual Encoders"],"benchmark_id":"claire-radar-2609-27142","caveat":"Evaluates text-to-image retrieval models on finding a named low-salience rare object in cluttered scenes, using re-captioned subsets of Flickr30K and MS COCO.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27142"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ROCS","organization_count":1,"organizations":["claire_radar"],"rank":2858,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.27142"},{"aliases":[],"benchmark_id":"claire-radar-2609-16614","caveat":"Assesses long-horizon role-playing robustness in spoken dialogue systems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16614"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RoleBreak","organization_count":1,"organizations":["claire_radar"],"rank":2859,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.16614"},{"aliases":[],"benchmark_id":"claire-radar-2606-01552","caveat":"RoleCDE evaluates role-playing agents under structured conflicts between role-specific values and alignment-oriented constraints. It comprises approximately 8,000 role profiles and 240,000 dilemma instances across three difficulty levels and eight role categories, with scoring via LLM-as-a-judge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01552"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RoleCDE","organization_count":1,"organizations":["claire_radar"],"rank":2860,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01552"},{"aliases":[],"benchmark_id":"opencompass-1091-rolellm","caveat":"RoleLLM is a role-playing framework of data construction and evaluation (RoleBench), as well as solutions for both closed-source and open-source models (RoleGPT, RoleLLaMA, RoleGLM). We also propose Context-Instruct for long-text knowledge extraction and role-specific knowledge injection. RoleLLM 是一个角色扮演的数据构建和评估框架，同时提供闭源和开源模型的解决方案（RoleGPT、RoleLLaMA、RoleGLM）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RoleLLM"],"document_share":0.0002723311546840959,"domain":"其他","name":"RoleLLM","organization_count":1,"organizations":["opencompass_hub"],"rank":2861,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RoleLLM"},{"aliases":[],"benchmark_id":"claire-radar-github-4esv-roman-numeral-harmony-benchmark","caveat":"Evaluates Roman-numeral analysis, cadence classification, and key identification across four synthetic notation representations with gold deterministic labels.","document_count":1,"document_ids":["claire_radar:https://github.com/4esv/roman-numeral-harmony-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Roman Numeral Harmony Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":2862,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/4esv/roman-numeral-harmony-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-github-vmalguy-rootme-gym","caveat":"Evaluates LLMs on Root-Me CTF challenges in a closed-book setup with a Docker sandbox and deterministic scrypt hash scoring.","document_count":1,"document_ids":["claire_radar:https://github.com/vmalguy/rootme-gym"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"rootme-gym","organization_count":1,"organizations":["claire_radar"],"rank":2863,"released":"2026-09-22","source":"claire_radar","url":"https://github.com/vmalguy/rootme-gym"},{"aliases":[],"benchmark_id":"claire-radar-2606-07641","caveat":"Paired diagnostic benchmark for rotated-outcome prediction in vision-language models, spanning open visual cases and controlled text-image rotations, with accuracy metrics for direct reading and prediction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07641"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RotOutBench","organization_count":1,"organizations":["claire_radar"],"rank":2864,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.07641"},{"aliases":["Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer"],"benchmark_id":"claire-radar-2609-31587","caveat":"A roundtrip benchmark that scores natural-language code descriptions by regenerating code from them and measuring test pass fraction against original tests, along with description metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.31587"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Roundtrip","organization_count":1,"organizations":["claire_radar"],"rank":2865,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.31587"},{"aliases":[],"benchmark_id":"claire-radar-2607-24471","caveat":"ROUTEBENCH evaluates whether transformers can learn latent algorithm routing across four solver families (ridge, lasso, Huber, kNN) in a controlled diagnostic setting.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24471"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ROUTEBENCH","organization_count":1,"organizations":["claire_radar"],"rank":2866,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24471"},{"aliases":[],"benchmark_id":"claire-radar-2609-00918","caveat":"Evaluates LLM-based recommendation assistants on detecting, localizing, and handling faulty premises in natural-language recommendation requests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00918"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RPCBench","organization_count":1,"organizations":["claire_radar"],"rank":2867,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.00918"},{"aliases":[],"benchmark_id":"claire-radar-2606-12071","caveat":"RQ-Bench evaluates novelty of research questions generated by LLMs against author-anchored reference questions from recent arXiv papers, using standalone and comparative LLM judging as well as human expert evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12071"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RQ-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2868,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12071"},{"aliases":[],"benchmark_id":"claire-radar-2607-24810","caveat":"RRS-10K is a benchmark for rare remote sensing image interpretation containing 10,738 military-related images with multiple format question-answer pairs, organized into three capability dimensions and 20 leaf tasks covering perception, reasoning, and robustness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24810"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RRS-10K","organization_count":1,"organizations":["claire_radar"],"rank":2869,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.24810"},{"aliases":[],"benchmark_id":"claire-radar-2607-20197","caveat":"RS-RIE-Bench evaluates reasoning-guided remote sensing image editing across temporal, causal, and spatial reasoning tasks, with metrics for region plausibility, preservation, and quality consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20197"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RS-RIE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2870,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.20197"},{"aliases":[],"benchmark_id":"llm-stats-rsi-index","caveat":"The RSI (Recursive Self-Improvement) Index is OpenAI's aggregate metric across a bundle of internal AI-research evaluations, including debugging research systems, optimizing kernels and training recipes, and improving other models, measuring progress toward recursive self-improvement.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/rsi-index?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"RSI Index","organization_count":1,"organizations":["llm_stats"],"rank":2871,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/rsi-index?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-25886","caveat":"RSIBench-Data evaluates LLM agents as data-centric researchers, where agents iteratively revise training-data strategies for a fixed target model on six benchmarks, with real training and evaluation runs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25886"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"RSIBench-Data","organization_count":1,"organizations":["claire_radar"],"rank":2872,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25886"},{"aliases":[],"benchmark_id":"opencompass-1678-rsmmvp","caveat":"This dataset follows a similar procedure to the original MMVP benchmark on natural images but directed towards the remote sensing domain. Challenging visual patterns are identified based on CLIP blind pairs, accompanied with the correpsonding questions, options and ground-truth answer. RSMMVP遵循与原始 MMVP 基准在自然图像上的类似流程，但针对遥感领域。根据 CLIP 盲对识别具有挑战性的视觉模式，并附带相应的问题、选项和真实答案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RSMMVP"],"document_share":0.0002723311546840959,"domain":"多模态","name":"RSMMVP","organization_count":1,"organizations":["opencompass_hub"],"rank":2873,"released":"2025-03-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RSMMVP"},{"aliases":[],"benchmark_id":"claire-radar-2610-09294","caveat":"Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.09294"],"document_share":0.0002723311546840959,"domain":"Agents","name":"RT-Safe","organization_count":1,"organizations":["claire_radar"],"rank":2874,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.09294"},{"aliases":[],"benchmark_id":"claire-radar-github-aiast1-rtcbench","caveat":"Evaluates LLM-generated controllers on simulated closed-loop control tasks using metrics like CVaR@10%, safety gating, and replayable traces.","document_count":1,"document_ids":["claire_radar:https://github.com/aiast1/rtcbench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RTCbench","organization_count":1,"organizations":["claire_radar"],"rank":2875,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/aiast1/rtcbench"},{"aliases":[],"benchmark_id":"opencompass-528-rte","caveat":"RTE is a natural language inference task, which requires to determine the logical relation between the given sentence pair, with three relations: entailment, contradiction and neutral. RTE是一个自然语言推理任务，要求根据给定的句子对，判断它们之间的逻辑关系，有三种关系：蕴含、矛盾和中立。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RTE"],"document_share":0.0002723311546840959,"domain":"推理","name":"RTE","organization_count":1,"organizations":["opencompass_hub"],"rank":2876,"released":null,"source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RTE"},{"aliases":[],"benchmark_id":"claire-radar-2606-08976","caveat":"RTL-BenchLS evaluates LLMs on RTL design generation and reasoning, containing over 10,000 formally verified Verilog designs. Tasks include specification-to-RTL generation, round-trip reasoning, masked-content reasoning, and repository-issue reasoning. All tasks are verified via formal equivalence checking without manual testbenches.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08976"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"RTL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2877,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.08976"},{"aliases":[],"benchmark_id":"claire-radar-2607-06411","caveat":"Evaluates coding agents on 25 repository-level tasks in Russian, mined from recent fix commits across five open-source projects. Graded by upstream regression tests with withheld oracles. Multiple rounds document model change and contamination audits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06411"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"RuBench","organization_count":1,"organizations":["claire_radar"],"rank":2878,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06411"},{"aliases":["Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models"],"benchmark_id":"claire-radar-2608-20237","caveat":"Benchmark for rule-compliant visual spatial planning in multimodal LLMs, requiring maze navigation under natural-language rules with automated rule generation and validation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.20237"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RuleMaze","organization_count":1,"organizations":["claire_radar"],"rank":2879,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.20237"},{"aliases":[],"benchmark_id":"llm-stats-ruler","caveat":"RULER v1 is a synthetic long-context benchmark for measuring how model quality degrades as input length increases. This packaging follows the public standalone NVIDIA RULER implementation with 13 official tasks spanning retrieval, multi-hop tracing, aggregation, and QA.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"RULER","organization_count":1,"organizations":["llm_stats"],"rank":2880,"released":"2024-04-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ruler-1000k","caveat":"RULER 1000K evaluates the official 13-task RULER v1 suite at a 1048576-token (1M) context budget.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-1000k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"RULER 1000K","organization_count":1,"organizations":["llm_stats"],"rank":2881,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-1000k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ruler-128k","caveat":"RULER 128k evaluates the official 13-task RULER v1 suite at a 131072-token context budget.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-128k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"RULER 128k","organization_count":1,"organizations":["llm_stats"],"rank":2882,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-128k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ruler-2048k","caveat":"RULER 2048K evaluates the official 13-task RULER v1 suite at a 2097152-token (2M) context budget.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-2048k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"RULER 2048K","organization_count":1,"organizations":["llm_stats"],"rank":2883,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-2048k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ruler-512k","caveat":"RULER 512K evaluates the official 13-task RULER v1 suite at a 524288-token context budget.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-512k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"RULER 512K","organization_count":1,"organizations":["llm_stats"],"rank":2884,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-512k?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-ruler-64k","caveat":"RULER 64k evaluates the official 13-task RULER v1 suite at a 65536-token context budget.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/ruler-64k?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"RULER 64k","organization_count":1,"organizations":["llm_stats"],"rank":2885,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/ruler-64k?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-23893","caveat":"RuleShift-Bench is a benchmark for evaluating concept drift under evolving concept definitions, spanning multiple data types and revision types. It assesses the ability of learning systems to adapt to rule-induced concept shifts with high accuracy while minimizing reprocessing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.23893"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"RuleShift-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2886,"released":"2026-08-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.23893"},{"aliases":["RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models"],"benchmark_id":"claire-radar-2608-26832","caveat":"RuleWeaver evaluates rule-centered scenario reasoning through scenario QA instances built from IF-THEN rules, with scoring based on answer correctness, rubric-based quality, rule recall, and rule precision.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26832"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RuleWeaver","organization_count":1,"organizations":["claire_radar"],"rank":2887,"released":"2026-08-27","source":"claire_radar","url":"https://arxiv.org/abs/2608.26832"},{"aliases":["Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models"],"benchmark_id":"claire-radar-2608-22753","caveat":"Evaluates step-level procedural rule reasoning with single-rule, parallel multi-rule, and multi-hop scenarios over large rule pools.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22753v1"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"RuleWorld","organization_count":1,"organizations":["claire_radar"],"rank":2888,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.22753v1"},{"aliases":[],"benchmark_id":"opencompass-1738-rulistening","caveat":"RUListening: Robust Understanding through Listening, an automated QA generation framework for evaluating multimodal perception. RUListening：通过聆听的稳健理解，这是一个用于评估多模态感知的自动问答生成框架。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RUListening"],"document_share":0.0002723311546840959,"domain":"理解","name":"RUListening","organization_count":1,"organizations":["opencompass_hub"],"rank":2889,"released":"2025-04-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RUListening"},{"aliases":[],"benchmark_id":"claire-radar-2609-37543","caveat":"Cross-lingual zero-shot transfer and multilingual fine-tuning are promising approaches for NLP tasks such as Named Entity Recognition (NER) in low-resource languages, but in the absence of target language benchmarks, it is unclear which au…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37543"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RunyaNER","organization_count":1,"organizations":["claire_radar"],"rank":2890,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37543"},{"aliases":[],"benchmark_id":"claire-radar-2609-31245","caveat":"Evaluates demographic bias in LLM-generated economic guidance for Indian settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.31245"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RupeeBias","organization_count":1,"organizations":["claire_radar"],"rank":2891,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.31245"},{"aliases":[],"benchmark_id":"claire-radar-2607-01388","caveat":"RusFinChain is a Russian-language benchmark for verifiable chain-of-thought reasoning in finance, comprising 5,280 parameterized examples generated from executable Python templates across 17 domains and 172 topics. Each example includes a gold-standard reasoning chain with intermediate numeric values for automatic verification, and the dataset, code, and evaluation framework are publicly released.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01388"],"document_share":0.0002723311546840959,"domain":"Finance","name":"RusFinChain","organization_count":1,"organizations":["claire_radar"],"rank":2892,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.01388"},{"aliases":["Rushes: A Human Preference Dataset for Pluralistic Alignment"],"benchmark_id":"claire-radar-2607-20767","caveat":"A dataset and diagnostic benchmark of 44,226 user decisions in AI-generated branching narratives. Models predict each user's choice at the event level.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20767"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Rushes","organization_count":1,"organizations":["claire_radar"],"rank":2893,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.20767"},{"aliases":[],"benchmark_id":"claire-radar-github-wwewtech-russian-it-community-corpus","caveat":"A corpus of Russian IT community discussions spanning 2017-2026, with splits for SFT dialogues, DPO pairs, and RAG knowledge base chunks.","document_count":1,"document_ids":["claire_radar:https://github.com/wwewtech/russian-it-community-corpus"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Russian IT Community Corpus","organization_count":1,"organizations":["claire_radar"],"rank":2894,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/wwewtech/russian-it-community-corpus"},{"aliases":[],"benchmark_id":"claire-radar-2606-03318","caveat":"RUT-Bench evaluates LLM tool-use in realistic user interactions with 1,638 test samples across 59 executable environments, measuring success rate, informational honesty, and tool discipline.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03318"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RUT-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2895,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03318"},{"aliases":[],"benchmark_id":"claire-radar-2606-29920","caveat":"RuVerBench evaluates LLM-as-a-judge reliability for rubric verification in agentic scenarios. It includes 2,458 instances across deep research and agentic coding, each with a model-generated output, a rubric, and a human-annotated label indicating rubric satisfaction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29920"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RuVerBench","organization_count":1,"organizations":["claire_radar"],"rank":2896,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.29920"},{"aliases":[],"benchmark_id":"claire-radar-2607-14846","caveat":"The Real World Voice EQ Bench evaluates voice AI systems across TTS, STS, SU, and ASR, focusing on how well models use acoustic information beyond text. It assesses dimensions like naturalness, expressiveness, identity stability, reliability, vocal affect use, and robustness to real-world conditions such as accent, emotion, noise, and conversation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14846"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RW-Voice-EQ Bench","organization_count":1,"organizations":["claire_radar"],"rank":2897,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14846"},{"aliases":[],"benchmark_id":"claire-radar-2606-24894","caveat":"Evaluates related work generation as citation-centric scholarly positioning. Uses 100 peer-reviewed papers and a 1.09M-document retrieval corpus, with metrics for citation selection, contextual appropriateness, organization, and discourse structure.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24894"],"document_share":0.0002723311546840959,"domain":"General AI","name":"RWGBench","organization_count":1,"organizations":["claire_radar"],"rank":2898,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.24894"},{"aliases":[],"benchmark_id":"opencompass-1707-rxrx3-core","caveat":"RxRx3-core dataset is a challenge dataset in phenomics optimized for the research\ncommunity. RxRx3-core数据集是Recursion为研究社区优化的表型组学挑战数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/RXRX3-CORE"],"document_share":0.0002723311546840959,"domain":"其他","name":"RXRX3-CORE","organization_count":1,"organizations":["opencompass_hub"],"rank":2899,"released":"2025-03-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/RXRX3-CORE"},{"aliases":[],"benchmark_id":"opencompass-1003-s-eval","caveat":"S-Eval is a new comprehensive, multi-dimensional and open-ended safety evaluation benchmark for LLMs consisting of 220,000 evaluation prompts (still in active expansion) across 102 risk subcategories and 10 advanced jailbreak attacks. S-Eval 是一个针对 LLM 的全新全面、多维、开放式安全评估基准，包含 102 个风险子类别的 220,000 个评估提示（仍在积极扩展中）和 10 个高级越狱攻击。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/S-Eval"],"document_share":0.0002723311546840959,"domain":"安全","name":"S-Eval","organization_count":1,"organizations":["opencompass_hub"],"rank":2900,"released":"2024-05-23","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/S-Eval"},{"aliases":[],"benchmark_id":"opencompass-1772-s1-bench","caveat":"S1-Bench is a novel benchmark designed to evaluate the performance of LRMs on simple tasks that are more aligned with intuitive System 1 thinking, rather than deliberate System 2 reasoning. S1-Bench offers a set of simple, diverse, and naturally clear questions across multiple domains and languages. S1-Bench是一个新颖的基准，旨在评估大模型在简单任务中的表现，这些任务更倾向于直观的系统1思维，而非深思熟虑的系统2推理。尽管大模型在复杂推理任务中通过明确的思维链取得了显著突破，但它们对深度分析思维的依赖可能限制了其系统1思维能力。此外，目前缺乏评估大模型在需要此类能力的任务中表现的基准。为了填补这一空白，S1-Bench提供了一组简单、多样且自然清晰的问题，涵盖多个领域和语言，专门设计用于评估大模型在此类任务中的表现。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/S1-Bench"],"document_share":0.0002723311546840959,"domain":"强推理","name":"S1-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2901,"released":"2025-04-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/S1-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2608-00610","caveat":"S2M-Bench evaluates the task of reconstructing mind maps from lecture slides, comprising 12,774 slide pages from 24 university courses with expert-annotated mind maps. The evaluation framework integrates ground-truth comparison, structure conformity analysis, and VLM-as-a-Judge scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00610"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"S2M-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2902,"released":"2026-08-01","source":"claire_radar","url":"https://arxiv.org/abs/2608.00610"},{"aliases":[],"benchmark_id":"claire-radar-2608-11562","caveat":"S2R-Bench evaluates video reflection removal, supporting full-reference and human perceptual assessment, and is used to validate the S2R-Removal model.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11562"],"document_share":0.0002723311546840959,"domain":"General AI","name":"S2R-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2903,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.11562"},{"aliases":["S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?"],"benchmark_id":"claire-radar-2608-31100","caveat":"Interactive benchmark for evaluating LLM self-improvement via self-testing, self-judging, and self-improvement in text-based games.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.31100"],"document_share":0.0002723311546840959,"domain":"General AI","name":"S3Gym","organization_count":1,"organizations":["claire_radar"],"rank":2904,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.31100"},{"aliases":["SA-Bench"],"benchmark_id":"claire-radar-2608-24252","caveat":"SA-Bench (SemanticAlign-Bench) evaluates semantic alignment in LLM-based paper reproduction across 30 papers from top conferences. It decomposes paper specifications into atomic verifiable claims (SAUs) and evaluates repositories along four diagnostic dimensions (numerical, methodological, protocol, ordering drift). Includes 1,491 SAUs across five ML domains and evaluates 12 generator configurations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24252"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2905,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24252"},{"aliases":[],"benchmark_id":"claire-radar-2606-01317","caveat":"SABER evaluates operational safety of LLM coding agents in stateful project workspaces. Agents perform realistic tasks, and safety is scored from the final environment state after action sequences. Violations are categorized by cause, enabling model-specific safety profile analysis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01317"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SABER","organization_count":1,"organizations":["claire_radar"],"rank":2906,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01317"},{"aliases":[],"benchmark_id":"claire-radar-2606-29894","caveat":"SABER-Math evaluates information retrieval for mathematical queries, with about 283K problems and tasks for reranking based on fine-grained relevance. Scoring uses preference tournament ratings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29894"],"document_share":0.0002723311546840959,"domain":"Information retrieval","name":"SABER-Math","organization_count":1,"organizations":["claire_radar"],"rank":2907,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.29894"},{"aliases":[],"benchmark_id":"opencompass-2083-saebench","caveat":"SAEBench is a comprehensive benchmark designed to evaluate and compare the performance of language model sparse autoencoders (SAEs). This benchmark provides over 200 SAE models covering seven architectures for systematic comparison, and has open-source code and models. SAEBench 是一个旨在评估和比较语言模型稀疏自动编码器（SAEs）性能的综合性基准。该基准提供超过200个SAE模型，涵盖七种架构，以实现系统性比较，并且开源了代码和模型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SAEBench"],"document_share":0.0002723311546840959,"domain":"理解","name":"SAEBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2908,"released":"2025-03-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SAEBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-09113","caveat":"Evaluates AI agents performing sparse autoencoder feature interpretation on Gemma-2-9B-IT, measuring feature rank, activation separation, and steering ability across 20 tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09113"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"SAEScientist-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2909,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.09113"},{"aliases":["SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment"],"benchmark_id":"claire-radar-2609-20584","caveat":"Industrial benchmark for LLM-assisted automotive hazard analysis and risk assessment under ISO 26262.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.20584"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SAFARI","organization_count":1,"organizations":["claire_radar"],"rank":2910,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.20584"},{"aliases":[],"benchmark_id":"claire-radar-2608-29098","caveat":"Evaluates multimodal safety moderation via 5,000-instance held-out set with five-level ordered risk predictions and continuous scores across 15 harm categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29098"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SafeAtlas-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2911,"released":"2026-08-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.29098"},{"aliases":[],"benchmark_id":"claire-radar-2608-00068","caveat":"A benchmark for evaluating multimodal large language models on construction safety hazard identification and description, with 3,314 expert-verified task instances from over 3,000 images.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00068"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SafeBuild-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2912,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.00068"},{"aliases":["Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens"],"benchmark_id":"claire-radar-2606-30755","caveat":"SafeClawArena evaluates security failures in Claw-like agents using 406 adversarial tasks spanning four attack surfaces in containerized platform replicas.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30755"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SafeClawArena","organization_count":1,"organizations":["claire_radar"],"rank":2913,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30755"},{"aliases":[],"benchmark_id":"claire-radar-2606-18356","caveat":"SafeClawBench is a staged benchmark for tool-using LLM agent security with 600 adversarial tasks across six attack families, reporting three endpoints: semantic attack acceptance, audit-visible harm evidence, and sandbox-observed tool/state harm.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18356"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SafeClawBench","organization_count":1,"organizations":["claire_radar"],"rank":2914,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18356"},{"aliases":[],"benchmark_id":"claire-radar-2606-01481","caveat":"Evaluates safety of conditional text-to-video generation using selected start frames and text prompts across 10 malicious categories, measuring unsafety scores and guardrail effectiveness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01481"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SafeGen-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2915,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01481"},{"aliases":[],"benchmark_id":"claire-radar-2608-16081","caveat":"SafeGesture evaluates vision-language models on scenario-conditioned safety interpretation of hand gestures, pairing 6 gestures with 8 scenarios for 4,800 items.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16081"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SafeGesture","organization_count":1,"organizations":["claire_radar"],"rank":2916,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16081"},{"aliases":[],"benchmark_id":"claire-radar-2606-29887","caveat":"SafePyramid evaluates in-context policy guardrailing across 1,000 multi-turn conversations and 3,000 application-specific policies containing 61,699 natural-language rules, organized into three hierarchical capability levels (L0, L1, L2). Scoring is based on violated-rule set prediction with metrics RMR and RDR, and the evaluation harness supports any API or local model.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29887"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SafePyramid","organization_count":1,"organizations":["claire_radar"],"rank":2917,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.29887"},{"aliases":[],"benchmark_id":"claire-radar-2607-14543","caveat":"SafeRelBench evaluates VLM-driven embodied agents on household tasks with process-level safety constraints, focusing on spatial relations such as support, containment, and proximity. It includes 507 executable samples (248 spatial-relation, 259 control) and measures whether agents satisfy safety conditions before risk-prone actions, alongside task success.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14543"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"SafeRelBench","organization_count":1,"organizations":["claire_radar"],"rank":2918,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14543"},{"aliases":["SafeRun: Enabling Determinism in LLM Planning for Running"],"benchmark_id":"claire-radar-2606-09027","caveat":"A benchmark for evaluating language-model running plans under physiological, safety, and instruction-following constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09027"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SafeRun","organization_count":1,"organizations":["claire_radar"],"rank":2919,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09027"},{"aliases":[],"benchmark_id":"claire-radar-2608-09230","caveat":"SafeSceneReason evaluates multimodal industrial-safety reasoning with 123,695 question-answer pairs covering compliance, hazard interaction, accident mechanisms, and prevention recommendations across scene-centric and report-centric pipelines.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09230"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"SafeSceneReason","organization_count":1,"organizations":["claire_radar"],"rank":2920,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09230"},{"aliases":[],"benchmark_id":"claire-radar-2609-21223","caveat":"Evaluates manipulation safety in vision-language-conditioned robot tasks across lifecycle stages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21223"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"SafeStage","organization_count":1,"organizations":["claire_radar"],"rank":2921,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21223"},{"aliases":[],"benchmark_id":"opencompass-1073-safetybench","caveat":"SafetyBench, a comprehensive benchmark for evaluating the safety of LLMs, which comprises 11,435 diverse multiple choice questions spanning across 7 distinct categories of safety concerns. Notably, SafetyBench also incorporates both Chinese and English data. SafetyBench 是一个全面的基准，用于评估大型语言模型（LLMs）的安全性，包含 11,435 道多样化的选择题，涵盖 7 个不同的安全关注类别。SafetyBench 还包含中文和英文的数据，方便双语评估。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SafetyBench"],"document_share":0.0002723311546840959,"domain":"安全","name":"SafetyBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2922,"released":"2024-06-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SafetyBench"},{"aliases":["SafeGuard: A Multi-Agent Perception-Reasoning Framework for Social-Risk AI-Generated Video Detection"],"benchmark_id":"claire-radar-2607-03069","caveat":"SafeVid is a 20K-video benchmark for detecting AI-generated videos across 10 social risk categories. It tests physical plausibility, structural consistency, and whether social behavior makes sense.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03069"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SafeVid","organization_count":1,"organizations":["claire_radar"],"rank":2923,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.03069"},{"aliases":["SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models"],"benchmark_id":"claire-radar-2606-00773","caveat":"SafeVLA-Bench evaluates safety-relevant trajectory behavior in simulator-based vision-language-action tasks using task success, unsafe-success frequency, and violation severity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00773"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"SafeVLA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2924,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00773"},{"aliases":[],"benchmark_id":"claire-radar-2609-07823","caveat":"Evaluates MLLMs on 193K multiple-choice VQA over 83K fire and smoke images across 20 scenarios and 10 reasoning dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07823"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SAFIRE","organization_count":1,"organizations":["claire_radar"],"rank":2925,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07823"},{"aliases":[],"benchmark_id":"claire-radar-2606-03301","caveat":"SagaQA is a long-form video benchmark for multi-hop reasoning over full-length TV series, requiring reasoning across episodes and multimodal narrative understanding.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03301"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SagaQA","organization_count":1,"organizations":["claire_radar"],"rank":2926,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03301"},{"aliases":[],"benchmark_id":"claire-radar-2609-31082","caveat":"Evaluates species distribution models on 5771 plant species using GBIF training data and sPlotOpen presence-absence plots, with species grouped by sampling effort and relative prevalence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.31082"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SAGE: A sampling-aware global evaluation benchmark for species distribution modeling","organization_count":1,"organizations":["claire_radar"],"rank":2927,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.31082"},{"aliases":["SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models"],"benchmark_id":"claire-radar-2606-29520","caveat":"SAKE is a standardized multiple-choice evaluation of software-architecture knowledge in language models, covering eight architectural categories and four context-length levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29520"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"SAKE","organization_count":1,"organizations":["claire_radar"],"rank":2928,"released":"2026-06-28","source":"claire_radar","url":"https://arxiv.org/abs/2606.29520"},{"aliases":[],"benchmark_id":"opencompass-1077-salad-bench","caveat":"SALAD-Bench, a safety benchmark specifically designed for evaluating LLMs, attack, and defense methods. Distinguished by its breadth, SALAD-Bench transcends conventional benchmarks through its large scale, rich diversity, intricate taxonomy\nspanning three levels, and versatile functionalities. SALAD-Bench 是一个专门用于评估大型语言模型（LLMs）、攻击和防御方法的安全基准。SALAD-Bench 的特点在于其广泛性，超越了传统基准，具有大规模、丰富的多样性、复杂的三层分类法以及多功能性。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SALAD-Bench"],"document_share":0.0002723311546840959,"domain":"安全","name":"SALAD-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2929,"released":"2024-02-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SALAD-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2606-12671","caveat":"SalArt-VQA is a closed-set visual question answering benchmark for fine-grained salient artifact understanding in AI-generated images. It includes 950 images and 3,681 human-authored multiple-choice questions covering artifact images, matched real references, and paired generated references. Four aligned question types evaluate presence detection, semantic localization, spatial grounding, and evidence-grounded defect identification, with reference splits for calibration and abstention testing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12671"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SalArt-VQA","organization_count":1,"organizations":["claire_radar"],"rank":2930,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12671"},{"aliases":["SambaGraph: Action-Reaction Spatio-Temporal Graphs for Soccer Tactical Response Modeling"],"benchmark_id":"claire-radar-2609-25569","caveat":"Soccer tactical response modeling dataset and benchmark built from 2022 FIFA World Cup event and tracking data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25569"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SambaGraph","organization_count":1,"organizations":["claire_radar"],"rank":2931,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.25569"},{"aliases":[],"benchmark_id":"claire-radar-github-srcollins785-samuel-collins-cv-benchmarking","caveat":"A public function for benchmarking classical ML and neural-network image classifiers on labeled image datasets provided in folder, CSV, JSON/JSONL, or NumPy array form.","document_count":1,"document_ids":["claire_radar:https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking"],"document_share":0.0002723311546840959,"domain":"Software & AI Compute","name":"Samuel_Collins_CV_Benchmarking","organization_count":1,"organizations":["claire_radar"],"rank":2932,"released":"2026-09-06","source":"claire_radar","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking"},{"aliases":[],"benchmark_id":"llm-stats-sat-math","caveat":"SAT Math benchmark from AGIEval containing standardized mathematics questions from the College Board SAT examination, designed to evaluate mathematical reasoning capabilities of foundation models using human-centric assessment methods.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sat-math?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"SAT Math","organization_count":1,"organizations":["llm_stats"],"rank":2933,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/sat-math?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-31507","caveat":"Evaluates long-horizon UAV vision-language navigation from satellite imagery.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.31507"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SatNav","organization_count":1,"organizations":["claire_radar"],"rank":2934,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.31507"},{"aliases":[],"benchmark_id":"claire-radar-2610-10624","caveat":"Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10624"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SAVU-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":2935,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10624"},{"aliases":[],"benchmark_id":"claire-radar-github-cosbi-research-sbmllm-bench","caveat":"Evaluates LLM reconstruction of executable systems-biology models from scientific papers using metrics like simulation ratio, species/reaction recovery, stoichiometric error, and AAFE reproducibility.","document_count":1,"document_ids":["claire_radar:https://github.com/cosbi-research/SBMLLM-Bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SBMLLM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2936,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/cosbi-research/SBMLLM-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2606-07656","caveat":"SC3 is a multi-solvent solubility benchmark built on BigSolDB v2.1 with 101,535 measurements over 1,327 solutes and 206 solvents. It features a reproducible curation pipeline, nested Gold/Silver/Bronze consensus tiers, leakage-checked splits, and a multi-solvent metric suite (PS-RMSE, Z-RMSE).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07656"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"SC3","organization_count":1,"organizations":["claire_radar"],"rank":2937,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.07656"},{"aliases":["Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context"],"benchmark_id":"claire-radar-2608-25655","caveat":"SCALE-QA evaluates interleaved conversational memory using 3,000 audited multiple-choice questions across 10 domains, where correct answers depend on causally related evidence from earlier turns in flat unsegmented threads.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25655"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SCALE-QA","organization_count":1,"organizations":["claire_radar"],"rank":2938,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25655"},{"aliases":[],"benchmark_id":"opencompass-1732-scam","caveat":"SCAM is the largest and most diverse dataset of real-world typographic attack images to date, containing 1,162 images across hundreds of object categories and attack words. SCAM，是迄今为止规模最大、多样性最丰富的真实世界排版攻击图像数据集，包含数百个对象类别和攻击词汇的1,162张图像。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SCAM"],"document_share":0.0002723311546840959,"domain":"多模态","name":"SCAM","organization_count":1,"organizations":["opencompass_hub"],"rank":2939,"released":"2025-04-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SCAM"},{"aliases":[],"benchmark_id":"claire-radar-2606-26563","caveat":"scBench-Long evaluates long-horizon single-cell biology reasoning. Agents must recover scientific conclusions from raw or near-raw data without prescribed methods. It contains 21 evaluations spanning diverse biological contexts, with deterministic grading and trajectory rubrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26563"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"scBench-Long","organization_count":1,"organizations":["claire_radar"],"rank":2940,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.26563"},{"aliases":[],"benchmark_id":"claire-radar-2610-12920","caveat":"LLM-generated service-call code is increasingly important for service computing, where developers rely on large language models (LLMs) to compose services and invoke third-party services.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12920"],"document_share":0.0002723311546840959,"domain":"Code","name":"SCCGBench","organization_count":1,"organizations":["claire_radar"],"rank":2941,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.12920"},{"aliases":[],"benchmark_id":"claire-radar-2607-22393","caveat":"A benchmark for visually conditioned action on complete multi-object 3D scenes across five tasks under a unified agent-environment loop, scored against hidden geometric ground truth.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22393"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SceneActBench","organization_count":1,"organizations":["claire_radar"],"rank":2942,"released":"2026-07-24","source":"claire_radar","url":"https://arxiv.org/abs/2607.22393"},{"aliases":[],"benchmark_id":"claire-radar-2609-16233","caveat":"A benchmark of 966 photorealistic 3D scenes with hierarchical annotations for vision-language spatial reasoning tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16233"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SceneBench","organization_count":1,"organizations":["claire_radar"],"rank":2943,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.16233"},{"aliases":[],"benchmark_id":"claire-radar-2608-00991","caveat":"SCHEDBench evaluates LLM constraint faithfulness in natural-language combinatorial scheduling. It includes 1,132 instances across JSP, RCPSP, nurse rostering, and timetabling, with templated NL variations and solver-verified references.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00991"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SCHEDBench","organization_count":1,"organizations":["claire_radar"],"rank":2944,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.00991"},{"aliases":[],"benchmark_id":"claire-radar-2608-22390","caveat":"Evaluates controllable GUI generation via deterministic function-call references synthesized from parameterized schemas across six bilingual scenarios.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22390v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SchemaGUI","organization_count":1,"organizations":["claire_radar"],"rank":2945,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22390v1"},{"aliases":["SchemeArena: Factorized Stress Testing of Scheming in LLM Agents"],"benchmark_id":"claire-radar-2609-08126","caveat":"Evaluates scheming in LLM agents across 400 factorized scenarios, with a monitor that scores covert misaligned behavior along five rubric dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08126"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SchemeArena","organization_count":1,"organizations":["claire_radar"],"rank":2946,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08126"},{"aliases":[],"benchmark_id":"claire-radar-2609-11243","caveat":"Evaluation of multi-step evidence-grounded scientific reasoning in multimodal agents using structured argument graphs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11243"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Sci-MMR","organization_count":1,"organizations":["claire_radar"],"rank":2947,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11243"},{"aliases":[],"benchmark_id":"claire-radar-2608-09873","caveat":"Sci-VBench evaluates text-to-video generation across 1,253 expert-annotated examples in 60 scientific subjects. It requires temporally rich videos demonstrating scientific reasoning and knowledge-grounded synthesis. Scoring covers four dimensions: prompt grounding, scientific correctness, spatiotemporal consistency, and low-level perceptual fidelity, using a rubric-based protocol with MLLM judges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09873"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Sci-VBench","organization_count":1,"organizations":["claire_radar"],"rank":2948,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09873"},{"aliases":["Benchmarking AI Agents for Addressing Scientific Challenges Across Scales"],"benchmark_id":"claire-radar-2606-12736","caveat":"Interactive, agent-agnostic environment with ~200 scientific research tasks evaluated via stepwise verification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12736"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciAgentArena","organization_count":1,"organizations":["claire_radar"],"rank":2949,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12736"},{"aliases":[],"benchmark_id":"artificial-analysis-scicode","caveat":"Coding","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/scicode"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"SciCode","organization_count":1,"organizations":["artificial_analysis"],"rank":2950,"released":"2024-07-18","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/scicode"},{"aliases":[],"benchmark_id":"llm-stats-scicode","caveat":"SciCode is a research coding benchmark curated by scientists that challenges language models to code solutions for scientific problems. It contains 338 subproblems decomposed from 80 challenging main problems across 16 natural science sub-fields including mathematics, physics, chemistry, biology, and materials science. Problems require knowledge recall, reasoning, and code synthesis skills.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/scicode?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"SciCode","organization_count":1,"organizations":["llm_stats"],"rank":2951,"released":"2024-07-18","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/scicode?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-11337","caveat":"SciConBench evaluates AI agents on open-domain scientific conclusion synthesis using 9,110 questions and expert-written conclusions from systematic reviews. The evaluation decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11337"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciConBench","organization_count":1,"organizations":["claire_radar"],"rank":2952,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11337"},{"aliases":[],"benchmark_id":"claire-radar-2609-05141","caveat":"SciDocBench evaluates scientific document understanding with 124 expert-authored questions across seven capability groups and 19 subtasks in five scientific domains. Each question is tested under four conditions (English vs Chinese and all-images-first vs interleaved), yielding 496 instances. Performance is reported on a 100-point scale. Data and pipeline are released via a public repository.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05141"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciDocBench","organization_count":1,"organizations":["claire_radar"],"rank":2953,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05141"},{"aliases":[],"benchmark_id":"claire-radar-2606-28406","caveat":"SciDraw-Bench is a benchmark for scientific figure generation, with 32 tasks across eight figure types and ten disciplines. Each task pairs a natural-language prompt with a machine-checkable specification, and evaluation uses four dimensions: Text Fidelity, Semantic Correctness, Structural Quality, and Convention Adherence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.28406"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciDraw-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2954,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.28406"},{"aliases":["Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery"],"benchmark_id":"claire-radar-2608-06931","caveat":"Science Edge Evaluation (SEE) is a multimodal benchmark of expert-curated questions grounded in peer-reviewed literature and experimental practice in chemistry, biology, and materials science, evaluated via accuracy across MLLMs and tool-use agents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06931"],"document_share":0.0002723311546840959,"domain":"AI Scientist","name":"Science Edge Evaluation","organization_count":1,"organizations":["claire_radar"],"rank":2955,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.06931"},{"aliases":["ScienceArena: Benchmarking LLMs on Latest Scientific Olympiad Competitions"],"benchmark_id":"claire-radar-2608-30517","caveat":"Olympiad-style benchmark with open-ended physics, chemistry, and biology problems scored via rubrics and LLM judges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30517"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"ScienceArena","organization_count":1,"organizations":["claire_radar"],"rank":2956,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30517"},{"aliases":[],"benchmark_id":"llm-stats-scienceqa","caveat":"ScienceQA is the first large-scale multimodal science question answering benchmark with 21,208 multiple-choice questions covering 3 subjects (natural science, language science, social science), 26 topics, 127 categories, and 379 skills. The benchmark includes both text and image modalities, featuring detailed explanations and Chain-of-Thought reasoning to diagnose multi-hop reasoning ability.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/scienceqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"ScienceQA","organization_count":1,"organizations":["llm_stats"],"rank":2957,"released":"2022-09-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1100-scienceqa","caveat":"SCIENCEQA is a new benchmark that consists of ∼21k multimodal multiple choice questions with diverse science topics and annotations of their answers with corresponding lectures and explanations. SCIENCEQA 包含约 21,000 道多模态选择题，涵盖多种科学主题，并附有相应的讲座和解释的答案注释。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ScienceQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"ScienceQA","organization_count":1,"organizations":["opencompass_hub"],"rank":2958,"released":"2022-10-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ScienceQA"},{"aliases":[],"benchmark_id":"llm-stats-scienceqa-visual","caveat":"ScienceQA Visual is a multimodal science question answering benchmark consisting of 21,208 multiple-choice questions from elementary and high school science curricula. The dataset covers 3 subjects (natural science, language science, social science), 26 topics, 127 categories, and 379 skills. 48.7% of questions include image context requiring multimodal reasoning. Questions are annotated with lectures (83.9%) and explanations (90.5%) to support chain-of-thought reasoning for science question answering.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/scienceqa-visual?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ScienceQA Visual","organization_count":1,"organizations":["llm_stats"],"rank":2959,"released":"2022-09-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/scienceqa-visual?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-20926","caveat":"SciExplore evaluates scientific information-seeking and reasoning in LLMs and agents with four task types: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, spanning 103 expert-curated tasks across more than ten scientific disciplines.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20926"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciExplore","organization_count":1,"organizations":["claire_radar"],"rank":2960,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.20926"},{"aliases":[],"benchmark_id":"opencompass-1323-scifibench","caveat":"SciFIBench is a scientific figure interpretation benchmark for LMMs, consisting of 2000 questions split between two tasks across 8 categories. SciFIBench用于评估多模态大模型的科学图表解释能力，由2000个问题组成，涵盖8个类别的2种任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SciFIBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"SciFIBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2961,"released":"2024-05-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SciFIBench"},{"aliases":[],"benchmark_id":"claire-radar-2608-13267","caveat":"SciFigBench is a diagnostic VLM benchmark for scientific figure understanding, covering perception, reasoning, and behavioral reliability under uncertainty.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13267"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciFigBench","organization_count":1,"organizations":["claire_radar"],"rank":2962,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13267"},{"aliases":[],"benchmark_id":"claire-radar-2607-29124","caveat":"SciFigPlag-Bench evaluates provenance-aware reasoning for scientific figure plagiarism detection. It includes 2,582 positive and 2,541 negative pairs, with tasks for pairwise detection, source attribution, reuse-type classification, and reuse correspondence localization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.29124"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciFigPlag-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2963,"released":"2026-07-31","source":"claire_radar","url":"https://arxiv.org/abs/2607.29124"},{"aliases":[],"benchmark_id":"claire-radar-2607-27084","caveat":"A benchmark for evaluating scientific figure quality across five dimensions with full-manuscript context, using 6,308 expert-rated images from top CS conferences and a fixed eval1200 test split.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27084"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciFigQual-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2964,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27084"},{"aliases":[],"benchmark_id":"claire-radar-2609-08155","caveat":"Evaluates generation of editable Python programs that preserve presentation of scientific figure panels, using a 337-panel test set across 31 chart subtypes, five domains, and three complexity levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08155"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciFigure2Code","organization_count":1,"organizations":["claire_radar"],"rank":2965,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08155"},{"aliases":[],"benchmark_id":"claire-radar-2607-18665","caveat":"SciHazard evaluates LLMs on scientific safety risks across 12 disciplines with 2,400 hazardous and 600 oversafety questions grounded in regulated entities. The DeHarm-Score decomposes harm into executability and net-new risk, providing a detailed scoring contract.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18665"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciHazard","organization_count":1,"organizations":["claire_radar"],"rank":2966,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.18665"},{"aliases":[],"benchmark_id":"claire-radar-2605-29468","caveat":"SciIntBench is an adversarial benchmark of 810 prompts across ten responsible-conduct-of-research categories and three scientific domains. Each scenario appears in overt, covert, and benign versions to measure framing-sensitive refusal of misconduct.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29468"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciIntBench","organization_count":1,"organizations":["claire_radar"],"rank":2967,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.29468"},{"aliases":[],"benchmark_id":"claire-radar-2606-30124","caveat":"SciIR-Bench evaluates text-to-image models on scientific image reasoning across three semiotic-aligned tracks: entity structure, scientific process, and scientific law. It uses an atomic checklist to convert scientific accuracy into verifiable fine-grained questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30124"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciIR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2968,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30124"},{"aliases":[],"benchmark_id":"claire-radar-2609-01552","caveat":"A benchmark for scientific law discovery using published research and real data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01552"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SCILAWS-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":2969,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01552"},{"aliases":[],"benchmark_id":"claire-radar-2608-25973","caveat":"Evaluates instruction following of MLLMs across five scientific disciplines with a taxonomy of 10 constraint groups.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25973"],"document_share":0.0002723311546840959,"domain":"Materials & Chemistry","name":"SciMIF","organization_count":1,"organizations":["claire_radar"],"rank":2970,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25973"},{"aliases":[],"benchmark_id":"claire-radar-2606-13020","caveat":"SciR evaluates LLMs on deduction, induction, and causal abduction in scientific settings, with tasks generated from formal objects and rendered into multi-document scientific discourse. Difficulty is controlled along extraction and inference axes, with verifiable answers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13020"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciR","organization_count":1,"organizations":["claire_radar"],"rank":2971,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13020"},{"aliases":[],"benchmark_id":"claire-radar-2606-18936","caveat":"SciRisk-Bench evaluates AI4Science safety across 7 disciplines and 10 risk dimensions, assessing whether models recognize and avoid risks in scientific contexts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18936"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciRisk-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2972,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18936"},{"aliases":[],"benchmark_id":"claire-radar-2608-01666","caveat":"SciStyleBench diagnoses stylistic bias in LLM-based idea evaluation through controlled stylistic perturbations, metrics, and a mitigation extractor.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01666"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SciStyleBench","organization_count":1,"organizations":["claire_radar"],"rank":2973,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01666"},{"aliases":[],"benchmark_id":"claire-radar-2608-13990","caveat":"SCOPE-Bench evaluates short-video recommendation systems by quantifying content depth using the Content Depth Score (CDS), a seven-level scale based on cognitive psychology. It provides CDS annotations for 150K videos from an open-source dataset, enabling systematic assessment of recommenders from a cognitive-content perspective.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13990"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SCOPE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2974,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.13990"},{"aliases":[],"benchmark_id":"claire-radar-2606-15242","caveat":"SCR-Bench is a benchmark for evaluating security risks in composed LLM agent skill workflows. It includes three sub-benchmarks (SCR-CapFlow, SCR-TrustLift, SCR-AuthBlur) that measure attack success rates, trust transfer, and authorization confusion in sandboxed environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15242"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SCR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2975,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2606.15242"},{"aliases":[],"benchmark_id":"claire-radar-2608-02358","caveat":"ScrambleToolBench evaluates agent behavioral reasoning in an interactive terminal environment with obfuscated tools and dynamic challenges like mapping drift and stochastic failures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02358"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"ScrambleToolBench","organization_count":1,"organizations":["claire_radar"],"rank":2976,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02358"},{"aliases":[],"benchmark_id":"claire-radar-2606-31689","caveat":"ScratchWorld is an offline diagnostic benchmark for world models, using Scratch projects as executable worlds. It evaluates next-state prediction, long-horizon tracking, causal attribution, and counterfactual prediction with replay-verified transitions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31689"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"ScratchWorld","organization_count":1,"organizations":["claire_radar"],"rank":2977,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31689"},{"aliases":[],"benchmark_id":"llm-stats-screenspot","caveat":"ScreenSpot is the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. The dataset comprises over 1,200 instructions from iOS, Android, macOS, Windows and Web environments, along with annotated element types (text and icon/widget), designed to evaluate visual GUI agents' ability to accurately locate screen elements based on natural language instructions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/screenspot?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ScreenSpot","organization_count":1,"organizations":["llm_stats"],"rank":2978,"released":"2024-01-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-screenspot-pro","caveat":"ScreenSpot-Pro is a novel GUI grounding benchmark designed to rigorously evaluate the grounding capabilities of multimodal large language models (MLLMs) in professional high-resolution computing environments. The benchmark comprises 1,581 instructions across 23 applications spanning 5 industries and 3 operating systems, featuring authentic high-resolution images from professional domains with expert annotations. Unlike previous benchmarks that focus on cropped screenshots in consumer applications, ScreenSpot-Pro addresses the complexity and diversity of real-world professional software scenarios, revealing significant performance gaps in current MLLM GUI perception capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/screenspot-pro?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ScreenSpot Pro","organization_count":1,"organizations":["llm_stats"],"rank":2979,"released":"2025-04-04","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/screenspot-pro?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-03906","caveat":"scTranslation evaluates single-cell multi-omics modality translation across RNA, ATAC, and ADT modalities, with diverse datasets, six representative models, and comprehensive metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03906"],"document_share":0.0002723311546840959,"domain":"General AI","name":"scTranslation","organization_count":1,"organizations":["claire_radar"],"rank":2980,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03906"},{"aliases":[],"benchmark_id":"claire-radar-2607-11079","caveat":"SDABench evaluates LLMs' scientific data analysis capabilities across six capabilities (descriptive, exploratory, inferential, predictive, causal, mechanistic) and five domains, with 527 real and 6000 synthetic instances in multiple-choice and open-ended formats.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11079"],"document_share":0.0002723311546840959,"domain":"AI Scientist","name":"SDABench","organization_count":1,"organizations":["claire_radar"],"rank":2981,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11079"},{"aliases":[],"benchmark_id":"claire-radar-2609-01548","caveat":"Evaluates stigma detection and response generation in dyadic and group dialogues.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01548"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"SDARE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2982,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01548"},{"aliases":[],"benchmark_id":"claire-radar-2605-21919","caveat":"SDGBiasBench is a benchmark suite for evaluating vision-language models on Sustainable Development Goals (SDG) reasoning, covering 500k multiple-choice questions and 50k regression tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.21919"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SDGBiasBench","organization_count":1,"organizations":["claire_radar"],"rank":2983,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.21919"},{"aliases":[],"benchmark_id":"claire-radar-2607-20471","caveat":"SDR-Bench evaluates LLM personalization in a two-party framework. It includes 6,279 customer success stories across 22 industries and ~200 enterprises, with a temporally constrained simulation to prevent data leakage. Scoring compares model-generated outreach against human outcomes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20471"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"SDR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2984,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.20471"},{"aliases":[],"benchmark_id":"claire-radar-2609-09672","caveat":"Evaluates speech understanding across 11 Southeast Asian languages via 97,194 samples covering speech processing, paralinguistic analysis, and temporal understanding tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09672"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"SEA-SpeechBench","organization_count":1,"organizations":["claire_radar"],"rank":2985,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09672"},{"aliases":[],"benchmark_id":"claire-radar-2609-35596","caveat":"Self-evolving LLM agents have gained prominence for their ability to improve after deployment by modifying their harness, including their controller instructions, memory management protocols, and reusable tools and skills, in response to u…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35596"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SEABench","organization_count":1,"organizations":["claire_radar"],"rank":2986,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35596"},{"aliases":[],"benchmark_id":"llm-stats-seal-0","caveat":"Seal-0 is a benchmark for evaluating agentic search capabilities, testing models' ability to navigate and retrieve information using tools.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/seal-0?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Seal-0","organization_count":1,"organizations":["llm_stats"],"rank":2987,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/seal-0?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-22725","caveat":"SEAM-Bench is a double-blind continuity storyboarding benchmark for evaluating visual continuity in short-drama generation. It assesses cross-episode continuity recall and generalizes across six mainstream text models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.22725"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SEAM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":2988,"released":"2026-08-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.22725"},{"aliases":[],"benchmark_id":"llm-stats-openai-search-function-calling","caveat":"Search and Function-Calling is an OpenAI internal production benchmark measuring reliable search-tool use and function calling in agentic workflows, reported as a pass rate.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/openai-search-function-calling?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Search and Function-Calling","organization_count":1,"organizations":["llm_stats"],"rank":2989,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/openai-search-function-calling?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-05212","caveat":"SearchAuditBench evaluates LLM auditors on localizing, attributing, and repairing failures in long-horizon search agent trajectories using expert-annotated error steps, root causes, and repair rubrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05212"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SearchAuditBench","organization_count":1,"organizations":["claire_radar"],"rank":2990,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.05212"},{"aliases":[],"benchmark_id":"claire-radar-2605-26548","caveat":"SEC-bench Pro measures long-horizon vulnerability discovery in software security. It includes 344 verified vulnerabilities across V8, SpiderMonkey, and Linux kernel, each paired with instructions for reproducing a working proof-of-concept. Grading uses an LLM-based judge to classify generated PoCs against vulnerable, fixed, and latest images.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26548"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"SEC-bench","organization_count":1,"organizations":["claire_radar"],"rank":2991,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26548"},{"aliases":[],"benchmark_id":"opencompass-1986-sec-bench","caveat":"SEC-bench is a benchmark designed to evaluate large language model (LLM) agents on real-world software security tasks. SEC-bench 是一个面向大型语言模型（LLM）智能体的软件安全任务评测基准，旨在自动化评估模型在真实漏洞环境中的能力。 该基准通过多智能体框架自动构建代码仓库、复现漏洞并生成修复补丁，涵盖漏洞验证（PoC 生成）和补丁修复两个关键任务，包含数百个真实安全案例。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SEC-bench"],"document_share":0.0002723311546840959,"domain":"安全","name":"SEC-bench","organization_count":1,"organizations":["opencompass_hub"],"rank":2992,"released":"2025-06-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SEC-bench"},{"aliases":[],"benchmark_id":"llm-stats-sec-bench-pro","caveat":"SEC-bench Pro is a self-evolving software-security benchmark that measures agent bug hunting on critical, high-complexity systems. It instantiates validated vulnerabilities across the V8 and SpiderMonkey JavaScript engines as reproducible vulnerability-discovery and proof-of-concept-generation tasks with oracle-based validation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sec-bench-pro?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"SEC-bench Pro","organization_count":1,"organizations":["llm_stats"],"rank":2993,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/sec-bench-pro?top_n=500"},{"aliases":["SEC-Bench Pro"],"benchmark_id":"sec_bench_pro","caveat":"Security engineering benchmark; evaluates both vulnerability detection and secure coding practices.","document_count":1,"document_ids":["model_reports:deepseek_v4_1_flash_technical_report"],"document_share":0.0002723311546840959,"domain":"security","name":"SEC-Bench Pro","organization_count":1,"organizations":["DeepSeek"],"rank":2994,"released":null,"source":"model_reports","url":"https://github.com/sec-bench/sec-bench"},{"aliases":[],"benchmark_id":"claire-radar-github-zidaniel123-secagent-bench","caveat":"Evaluates LLM providers and self-hosted models on security analysis tasks: SAST source code vulnerability detection, DAST HTTP request/response analysis, and prompt-injection robustness in code comments and HTTP responses.","document_count":1,"document_ids":["claire_radar:https://github.com/zidaniel123/secagent-bench"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"secagent-bench","organization_count":1,"organizations":["claire_radar"],"rank":2995,"released":"2026-09-22","source":"claire_radar","url":"https://github.com/zidaniel123/secagent-bench"},{"aliases":[],"benchmark_id":"opencompass-948-secbench","caveat":"Tencent Zhuque Lab and Tencent Security Keen Lab, together with Tencent Huyuan Team, Professor Jiang Yong/Professor Xia Shutao's team from Tsinghua University, Professor Luo Xiapu's research team from Hong Kong Polytechnic University, and OpenCompass team from Shanghai Artificial Intelligence Laboratory, have jointly built a safety benchmark, SecBench. We provide fair, impartial, objective, and comprehensive evaluation capabilities and promote the construction of large model in security dimension. 腾讯朱雀实验室和腾讯安全科恩实验室联合腾讯混元大模型团队、清华大学江勇教授/夏树涛教授团队、香港理工大学罗夏朴教授研究团队以及上海人工智能实验室OpenCompass团队，通过建设安全大模型评测基准SecBench，为安全大模型研发提供公平、公正、客观、全面的评测能力，推动安全大模型建设。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SecBench"],"document_share":0.0002723311546840959,"domain":"安全","name":"SecBench","organization_count":1,"organizations":["opencompass_hub"],"rank":2996,"released":"2024-01-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SecBench"},{"aliases":[],"benchmark_id":"llm-stats-seccodebench","caveat":"SecCodeBench evaluates LLM coding agents on secure code generation and vulnerability detection, testing the ability to produce code that is both functional and free from security vulnerabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/seccodebench?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"SecCodeBench","organization_count":1,"organizations":["llm_stats"],"rank":2997,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/seccodebench?top_n=500"},{"aliases":["SecCodeBench"],"benchmark_id":"seccodebench","caveat":"Measures secure-coding behaviour, not exploitation capability.","document_count":1,"document_ids":["model_reports:qwen3_5_model_card"],"document_share":0.0002723311546840959,"domain":"security","name":"SecCodeBench","organization_count":1,"organizations":["Qwen"],"rank":2998,"released":"2025-10-14","source":"model_reports","url":"https://github.com/alibaba/SecCodeBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-25225","caveat":"SecDrift is a framework measuring sector-conditioned security drift in AI-generated code, evaluating vulnerability rates from LLMs when prompted with industry contexts versus neutral baselines across CISA sectors and CWE categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25225"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"SecDrift","organization_count":1,"organizations":["claire_radar"],"rank":2999,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25225"},{"aliases":[],"benchmark_id":"claire-radar-2607-26791","caveat":"A benchmark for evaluating LLM agents on post-compromise incident-response workflows using forensic disk snapshots and host security reports across 10 cyber ranges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26791"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"SecRespond","organization_count":1,"organizations":["claire_radar"],"rank":3000,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.26791"},{"aliases":["Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxies"],"benchmark_id":"claire-radar-2609-19844","caveat":"ArXiv paper on auditing AI-generated RTL verification plans with a benchmark, but no artifact evidence is provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19844"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"SecTB-RTL","organization_count":1,"organizations":["claire_radar"],"rank":3001,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.19844"},{"aliases":[],"benchmark_id":"opencompass-1359-seed-bench","caveat":"SEED-Bench aims at the evaluation of generative comprehension in MLLMs, consisting of 19K multiple choice questions, which spans 12 evaluation dimensions including the comprehension of both the image and video modality. SEED-Bench用于评估多模态大模型的理解能力，包括对图像和视频的理解，由跨越12个评估维度的19K道多项选择题组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SEED-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"SEED-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":3002,"released":"2023-07-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench"},{"aliases":[],"benchmark_id":"opencompass-1363-seed-bench-2","caveat":"SEED-Bench-2 assesses both text and image generation of MLLMs. It spans 27 evaluation dimensions, featuring 24K multiple-choice questions with precise human annotations. SEED-Bench-2用于评估多模态大模型的文本和图像生成能力，包括跨越27个维度的24K道多选题及准确的人工注释。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2"],"document_share":0.0002723311546840959,"domain":"多模态","name":"SEED-Bench-2","organization_count":1,"organizations":["opencompass_hub"],"rank":3003,"released":"2023-11-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2"},{"aliases":[],"benchmark_id":"opencompass-1395-seed-bench-2-plus","caveat":"SEED-Bench-2-Plus, a benchmark specifically designed for evaluating text-rich visual comprehension of MLLMs. The benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs. SEED-Bench-2-Plus, a benchmark specifically designed for evaluating text-rich visual comprehension of MLLMs. The benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2-Plus"],"document_share":0.0002723311546840959,"domain":"多模态","name":"SEED-Bench-2-Plus","organization_count":1,"organizations":["opencompass_hub"],"rank":3004,"released":"2024-04-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SEED-Bench-2-Plus"},{"aliases":[],"benchmark_id":"llm-stats-seedclawbench","caveat":"SeedClawBench is an agentic coding benchmark measuring overall model performance on real-world, tool-using software development tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/seedclawbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"SeedClawBench","organization_count":1,"organizations":["llm_stats"],"rank":3005,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/seedclawbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-03983","caveat":"SeGaBench is an executable benchmark containing 120 cases (100 synthetic, 20 source-backed) to test whether LLMs can recover semantic optimization opportunities that compilers miss, with hidden enabling semantics, oracle artifacts, and validators.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03983"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"SeGaBench","organization_count":1,"organizations":["claire_radar"],"rank":3006,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03983"},{"aliases":[],"benchmark_id":"claire-radar-2607-20090","caveat":"SelectBench evaluates selective evidence adoption in retrieval-augmented language models, focusing on rejecting deceptive content. The benchmark includes a 325-example test set and rule- or judge-based reward scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20090"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SelectBench","organization_count":1,"organizations":["claire_radar"],"rank":3007,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.20090"},{"aliases":["Semantic-ITC: A Frame-wise Indoor Mobile Laser Scanning Dataset and Benchmark for Semantic Segmentation"],"benchmark_id":"claire-radar-2609-18493","caveat":"Frame-wise indoor mobile laser scanning semantic segmentation dataset and benchmark.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18493"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Semantic-ITC","organization_count":1,"organizations":["claire_radar"],"rank":3008,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18493"},{"aliases":["SemCityLoc: Aerial 6DoF Localization Using Semantic 3D City Models"],"benchmark_id":"claire-radar-2606-27444","caveat":"A real-world aerial 6DoF localization benchmark pairing centimeter-accurate UAV poses with standardized semantic city models and low-altitude imagery.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27444"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SemCityLockeD","organization_count":1,"organizations":["claire_radar"],"rank":3009,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.27444"},{"aliases":["When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates"],"benchmark_id":"claire-radar-2606-13218","caveat":"A benchmark for evaluating Arabic–Hebrew cognate identification and semantic disambiguation across multiple input representations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13218"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SemCog Bench","organization_count":1,"organizations":["claire_radar"],"rank":3010,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13218"},{"aliases":["Learning Materials Properties from Scarce Labels and Unlabeled Crystals"],"benchmark_id":"claire-radar-2608-30682","caveat":"Evaluates semi-supervised materials property regression across six tasks with fixed splits, four graph backbones, and normalized MAE, using labeled and unlabeled crystal data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30682"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SemiMat","organization_count":1,"organizations":["claire_radar"],"rank":3011,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30682"},{"aliases":[],"benchmark_id":"claire-radar-2609-11180","caveat":"A benchmark of LLM version-constraint resolution semantics across npm, PEP 440, and Cargo, with 240 machine-checkable items.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11180"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SemVerBench","organization_count":1,"organizations":["claire_radar"],"rank":3012,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11180"},{"aliases":[],"benchmark_id":"claire-radar-2606-31875","caveat":"SENSE-VAD is a synthetic video anomaly detection dataset for autonomous driving, generated with CARLA and Unreal Engine. It includes socially complex anomalies across five categories with per-frame binary labels, plus real-world videos for sim-to-real transfer.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31875"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"SENSE-VAD","organization_count":1,"organizations":["claire_radar"],"rank":3013,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31875"},{"aliases":["Sentinel: Embodied Cooperative Spatial Reasoning and Planning"],"benchmark_id":"claire-radar-2605-26239","caveat":"Sentinel Challenge evaluates decentralized embodied agents that must communicate, select a meeting point, and navigate around dynamic patrols in city-scale environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26239"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Sentinel Challenge","organization_count":1,"organizations":["claire_radar"],"rank":3014,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26239"},{"aliases":[],"benchmark_id":"claire-radar-2606-05342","caveat":"SentinelBench is an open-source benchmark for time-evolving monitoring tasks. It contains 100 tasks across 10 synthetic web environments (email, calendars, finance, etc.) with scripted events, measuring task completion, reaction time, and resource use.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05342"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SentinelBench","organization_count":1,"organizations":["claire_radar"],"rank":3015,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05342"},{"aliases":[],"benchmark_id":"claire-radar-2607-15606","caveat":"Seq2Synth evaluates temporal fidelity of synthetic sequential tabular data across timestamp, cross-sectional, longitudinal, structural, and privacy dimensions, using a taxonomy to determine applicable metrics. It spans seven core datasets and multiple generators.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15606"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Seq2Synth","organization_count":1,"organizations":["claire_radar"],"rank":3016,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.15606"},{"aliases":[],"benchmark_id":"claire-radar-2608-30451","caveat":"3D visual grounding benchmark with sequence-aligned RGB observations and human-verified expressions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30451"],"document_share":0.0002723311546840959,"domain":"Geometric reasoning","name":"SeqAlign3DVG","organization_count":1,"organizations":["claire_radar"],"rank":3017,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30451"},{"aliases":[],"benchmark_id":"claire-radar-2607-27056","caveat":"A benchmark for evaluating memory-augmented personalized agents on hierarchical user understanding from heterogeneous data, with four levels of user understanding and psychometrics-based synthetic data.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27056"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Setoka","organization_count":1,"organizations":["claire_radar"],"rank":3018,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27056"},{"aliases":[],"benchmark_id":"claire-radar-2606-13757","caveat":"SEVRA-BENCH is a benchmark for measuring how often LLM-based code review agents approve adversarial pull requests with social-engineering framings, built from vulnerability-fixing commits. It includes a challenge split of roughly 1,000 adversarial PRs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13757"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SEVRA-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":3019,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13757"},{"aliases":[],"benchmark_id":"opencompass-1961-sfe","caveat":"The Scientists' First Exam (SFE) benchmark, designed to comprehensively evaluate the scientific cognitive capabilities of MLLMs through three cognitive levels (cog-levels):Scientific Signal Perception、Scientific Attribute Understanding 、Scientific Comparative Reasoning. The Scientists' First Exam (SFE) 基准测试旨在通过三个认知层级——科学信号感知、科学属性理解 和 科学对比推理，全面评估多模态大语言模型（MLLMs）的科学认知能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SFE"],"document_share":0.0002723311546840959,"domain":"多模态","name":"SFE","organization_count":1,"organizations":["opencompass_hub"],"rank":3020,"released":"2025-06-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SFE"},{"aliases":[],"benchmark_id":"opencompass-1330-sg-bench","caveat":"SG-Bench assess LLM safety across various tasks and prompt types. It integrates both generative and discriminative evaluation tasks and includes extended data to examine the impact of prompt engineering and jailbreak. SG-Bench用于评估LLM在不同任务和提示下的安全性，整合了生成性和判别性评估任务，并包含扩展数据以度量提示工程和越狱对安全性的影响。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SG-Bench"],"document_share":0.0002723311546840959,"domain":"安全","name":"SG-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":3021,"released":"2024-10-29","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SG-Bench"},{"aliases":[],"benchmark_id":"opencompass-2325-sgi-bench","caveat":"SGI-Bench operationalizes this definition via four scientist-aligned task families: deep research, idea generation, dry/wet experiments, and multimodal experimental reasoning. The benchmark spans 10 disciplines and more than 1,000 expert-curated samples inspired by Science's 125 Big Questions. 科学通用智能（SGI）被定义为一种人工智能系统，其能够以接近人类科学家的通用性与熟练程度，自主地贯穿并迭代完整的科学研究流程，包括审思、构思、行动与感知四个阶段。SGI-Bench 通过四类与科学家工作流程对齐的任务对上述定义进行操作化刻画，具体包括：深度研究、思想与假设生成、干/湿实验、以及多模态实验推理。该评测基准覆盖10 个科学学科领域，包含1,000 余个由领域专家精心策划的样本，其设计灵感来源于 Science 杂志提出的 125 个重大科学问题（125 Big Questions）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SGI-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"SGI-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":3022,"released":"2025-12-22","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SGI-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2605-22219","caveat":"SGR-Bench evaluates search agents on state-gated retrieval tasks. It includes 100 expert-curated tasks across 12 public data ecosystems, requiring agents to configure site-specific filters, views, hierarchies, or scopes to retrieve structured answers. Tasks come in goal-oriented and constraint-guided formulations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22219"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SGR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3023,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22219"},{"aliases":[],"benchmark_id":"claire-radar-2608-29270","caveat":"Evaluates autoformalization systems on 178 Lean 4 problems using the SA-Pass metric, which checks compile, forward, and backward semantic alignment via shadows.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29270"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ShadowBench","organization_count":1,"organizations":["claire_radar"],"rank":3024,"released":"2026-08-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.29270"},{"aliases":[],"benchmark_id":"claire-radar-2606-06717","caveat":"ShallowBench is a curated benchmark of 5,780 shallow-pocket targets for evaluating generative drug design models. Targets are extracted from CrossDocked2020 based on low concavity and sufficient surface area.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06717"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ShallowBench","organization_count":1,"organizations":["claire_radar"],"rank":3025,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06717"},{"aliases":[],"benchmark_id":"opencompass-2348-shell","caveat":"Targeting overlooked implicit risks in vertical domains (Education, Finance, Management), we introduce an implicit risk benchmark and the MENTOR framework. By leveraging Rule Evolution and Activation Steering, MENTOR effectively detects and mitigates these subtle hazards. Shell由华东师范大学Shell@Educhat团队和上海人工智能实验室联合推出。当下，确保垂直领域任务中大模型的安全性至关重要。虽然目前的对齐工作主要针对偏见和暴力等显性风险，但往往忽略了更深层次的特定领域隐性风险。研发团队推出了一个包含大量隐式风险查询的基准测试集，将风险分为引导、反思、禁止三类，以及 MENTOR 框架。该框架利用规则演化循环（REC）和激活引导（RV）技术，能够有效发现并缓解这些不易察觉的潜在风险。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Shell"],"document_share":0.0002723311546840959,"domain":"安全","name":"Shell","organization_count":1,"organizations":["opencompass_hub"],"rank":3026,"released":"2025-12-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Shell"},{"aliases":[],"benchmark_id":"claire-radar-2607-07844","caveat":"Shift & Drift evaluates closed-loop motion planners on two tracks: Semantic Shift, which uses a conversion pipeline to transform the DeepScenario Open 3D dataset into nuPlan for zero-shot testing on 1,182 scenarios across German cities and San Francisco, and State-Distribution Drift, which injects stochastic perturbations into ego-vehicle dynamics. Scoring is based on safety and progress metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.07844"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"Shift & Drift","organization_count":1,"organizations":["claire_radar"],"rank":3027,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.07844"},{"aliases":[],"benchmark_id":"claire-radar-2606-12608","caveat":"Assesses multi-turn conversational shopping assistants across 525 expert-authored missions with importance-weighted binary rubrics. Evaluates reasoning across five categories covering preference refinement, trade-off analysis, and compatibility assessment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12608"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Shopping Reasoning Bench","organization_count":1,"organizations":["claire_radar"],"rank":3028,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12608"},{"aliases":[],"benchmark_id":"opencompass-1321-shoppingmmlu","caveat":"Shopping MMLU is a diverse multi-task online shopping benchmark derived from real-world Amazon data. It consists of 57 tasks covering 4 major shopping skills: concept understanding, knowledge reasoning, user behavior alignment, and multi-linguality. Shopping MMLU是一个基于真实亚马逊数据的多样化多任务在线购物基准测试，由57项任务组成，涵盖概念理解、知识推理、用户行为对齐和多语言4大购物场景技能。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ShoppingMMLU"],"document_share":0.0002723311546840959,"domain":"其他","name":"ShoppingMMLU","organization_count":1,"organizations":["opencompass_hub"],"rank":3029,"released":"2024-10-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ShoppingMMLU"},{"aliases":[],"benchmark_id":"claire-radar-2609-31002","caveat":"Evaluates e-commerce reranking with contamination-limited private-traffic preference pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.31002"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ShopRank-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3030,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.31002"},{"aliases":[],"benchmark_id":"claire-radar-2606-30201","caveat":"SHOVIR evaluates vision shortcut learning in radiology report generation by extending MIMIC-CXR and PadChest-GR with per-box CheXpert labels. It defines image-level and disease-level occlusion experiments that compare model predictions on clean images against localized perturbations to isolate direct and contextual shortcut failures at the disease-class level.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30201"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"SHOVIR","organization_count":1,"organizations":["claire_radar"],"rank":3031,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30201"},{"aliases":["ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations"],"benchmark_id":"claire-radar-2609-25467","caveat":"A benchmark protocol and public dataset for comprehension after narrated business demonstrations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25467"],"document_share":0.0002723311546840959,"domain":"Finance","name":"ShowTellArena","organization_count":1,"organizations":["claire_radar"],"rank":3032,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.25467"},{"aliases":[],"benchmark_id":"claire-radar-2606-16953","caveat":"SidewalkBench evaluates visual navigation models on urban sidewalks using GPU-accelerated simulation with procedurally generated and real-world scanned scenes, including pedestrian-reactive and long-horizon scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16953"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SidewalkBench","organization_count":1,"organizations":["claire_radar"],"rank":3033,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16953"},{"aliases":[],"benchmark_id":"llm-stats-sifo","caveat":"SIFO (Simple Instruction Following) evaluates how well language models follow simple, explicit instructions. It tests fundamental instruction-following capabilities across various task types.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sifo?top_n=500"],"document_share":0.0002723311546840959,"domain":"structured_output","name":"SIFO","organization_count":1,"organizations":["llm_stats"],"rank":3034,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-sifo-multiturn","caveat":"SIFO-Multiturn evaluates instruction following capabilities in multi-turn conversational settings, testing how well models maintain context and follow instructions across multiple exchanges.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sifo-multiturn?top_n=500"],"document_share":0.0002723311546840959,"domain":"structured_output","name":"SIFO-Multiturn","organization_count":1,"organizations":["llm_stats"],"rank":3035,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/sifo-multiturn?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-04244","caveat":"SIGNPOST-Bench evaluates text-vision conflict resolution in multimodal large language models via a counterfactual benchmark of image variants (Original, Blank, Similar, Random, Adversarial) for visual geolocation. It includes 5,111 groups and 25,555 variants, with metrics for localization error and target-directed shifts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04244"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SIGNPOST-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3036,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.04244"},{"aliases":[],"benchmark_id":"claire-radar-github-hemachandra666-silent-misresolution-benchmark","caveat":"A benchmark for silent misresolution in dictation cleanup, where a system deletes spoken content without leaving a trace in the output; it includes a corpus with revision sites and a scorer for detecting such failures.","document_count":1,"document_ids":["claire_radar:https://github.com/hemachandra666/silent-misresolution-benchmark"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Silent Misresolution","organization_count":1,"organizations":["claire_radar"],"rank":3037,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/hemachandra666/silent-misresolution-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-github-json604-silentbug-bench","caveat":"Evaluates agents on detecting silent ML training defects across 18 CPU-only PyTorch tasks, scoring patch application, metric recovery, localization, minimality, and false positives.","document_count":1,"document_ids":["claire_radar:https://github.com/Json604/silentbug-bench"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"silentbug","organization_count":1,"organizations":["claire_radar"],"rank":3038,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/Json604/silentbug-bench"},{"aliases":[],"benchmark_id":"claire-radar-github-p20030920p-sim2real-algobench","caveat":"Sim2Real-AlgoBench evaluates interchangeable navigation algorithms on a three-wheeled omnidirectional robot performing a map-search task: find a green wall marker, approach it, drive onto the pad, and hold still for 3 s, scored in simulation and on hardware.","document_count":1,"document_ids":["claire_radar:https://github.com/p20030920p/Sim2Real-AlgoBench"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"Sim2Real-AlgoBench","organization_count":1,"organizations":["claire_radar"],"rank":3039,"released":"2026-09-11","source":"claire_radar","url":"https://github.com/p20030920p/Sim2Real-AlgoBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-01676","caveat":"Evaluates sim-to-real transfer for reinforcement learning traffic signal control under decomposed observation, action, transition, and reward gaps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01676"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Sim2Signal","organization_count":1,"organizations":["claire_radar"],"rank":3040,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01676"},{"aliases":[],"benchmark_id":"claire-radar-2606-14574","caveat":"SIMMER evaluates latent failures in LLM-generated plans for kitchen-domain tasks using a curated symbolic world model with 77 actions and 262 objects, scoring error-free plans and latent hazard detection.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14574"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SIMMER","organization_count":1,"organizations":["claire_radar"],"rank":3041,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14574"},{"aliases":[],"benchmark_id":"claire-radar-2606-08278","caveat":"SIMPLE is a simulation testbed for humanoid loco-manipulation with 60 tasks, 50 scenes, and over 1,000 objects, integrating data generation pipelines and benchmarking policies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08278"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"SIMPLE","organization_count":1,"organizations":["claire_radar"],"rank":3042,"released":"2026-06-06","source":"claire_radar","url":"https://arxiv.org/abs/2606.08278"},{"aliases":[],"benchmark_id":"llm-stats-simpleqa","caveat":"SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/simpleqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SimpleQA","organization_count":1,"organizations":["llm_stats"],"rank":3043,"released":"2024-10-30","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-simpleqa-verified","caveat":"SimpleQA Verified is a curated, reliability-focused subset of SimpleQA that addresses label noise and redundancy in the original benchmark, measuring short-form parametric factual accuracy of large language models on fact-seeking questions with single, indisputable answers.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/simpleqa-verified?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SimpleQA Verified","organization_count":1,"organizations":["llm_stats"],"rank":3044,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/simpleqa-verified?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-simplevqa","caveat":"SimpleVQA is a visual question answering benchmark focused on simple queries.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/simplevqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"SimpleVQA","organization_count":1,"organizations":["llm_stats"],"rank":3045,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/simplevqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-22873","caveat":"A multimodal guardrail evaluation covering multimodal question answering, adversarial attacks, dynamic policy rules, and cross-modal joint risks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22873"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"SingGuard-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3046,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.22873"},{"aliases":["Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth"],"benchmark_id":"claire-radar-2607-03870","caveat":"SALT is a procedurally generated benchmark for evaluating correctness and uncertainty estimates in long-form language-model outputs against deterministic ground truth.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03870"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Single-answer Atomic Long-form Target (SALT)","organization_count":1,"organizations":["claire_radar"],"rank":3047,"released":"2026-07-04","source":"claire_radar","url":"https://arxiv.org/abs/2607.03870"},{"aliases":[],"benchmark_id":"opencompass-533-siqa","caveat":"SIQA is a social interaction question answering task, which requires to select the most reasonable behavior based on the given scenario and three possible subsequent behaviors. This task is designed to test the model's knowledge in social commonsense. This dataset consists of 38,963 training samples, 1,951 development samples and 1,960 test samples, all on English text. SIQA 是一个社会交互问答任务，要求根据给定的场景和三个可能的后续行为，选择最合理的行为。这个任务是为了测试模型在社会常识方面的知识。这个数据集包含了 38963 个训练样本，1951 个开发样本和 1960 个测试样本，所有的文本都是英文文本。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SIQA"],"document_share":0.0002723311546840959,"domain":"推理","name":"SIQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3048,"released":"2019-09-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SIQA"},{"aliases":[],"benchmark_id":"llm-stats-siren-agentdojo-attack-success","caveat":"Siren AgentDojo evaluates tool-using agents under adversarial prompt-injection attacks. This metric is the attack success rate; lower is better.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-attack-success?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"Siren AgentDojo Attack Success Rate","organization_count":1,"organizations":["llm_stats"],"rank":3049,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-attack-success?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-siren-agentdojo-utility","caveat":"Siren AgentDojo evaluates tool-using agents under adversarial prompt-injection attacks. This metric reports utility on the assigned tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-utility?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"Siren AgentDojo Utility","organization_count":1,"organizations":["llm_stats"],"rank":3050,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/siren-agentdojo-utility?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-24094","caveat":"SIREN-Bench-v1 is a benchmark for emergency-vehicle (EMV) interactions, built on a SUMO-CARLA co-simulation platform. It includes seven parameterized interaction templates across emergency levels L1-L3 and three behavior families, with synchronized sensor observations and simulator-native annotations. Supports tasks such as 3D object detection, trajectory prediction, and vision-language risk understanding.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24094"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SIREN-Bench-v1","organization_count":1,"organizations":["claire_radar"],"rank":3051,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24094"},{"aliases":[],"benchmark_id":"claire-radar-2607-12477","caveat":"SIS-Bench evaluates embodied spatial intelligence in UAV scenarios across two dimensions (spatial cognition and self-awareness) and three cognitive levels (perception, memory, reasoning), with 4,856 QA pairs from 1,646 real-world UAV videos.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.12477"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"SIS-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3052,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.12477"},{"aliases":[],"benchmark_id":"claire-radar-2610-09152","caveat":"Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.09152"],"document_share":0.0002723311546840959,"domain":"General AI","name":"sk-bench","organization_count":1,"organizations":["claire_radar"],"rank":3053,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.09152"},{"aliases":[],"benchmark_id":"claire-radar-2608-04828","caveat":"Skill-Use evaluates skill use in agentic harnesses through 79 real skills and 177 executable tasks across nine domains, measuring trigger, compliance, and boundary adherence with a combined SU score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04828"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Skill-Use","organization_count":1,"organizations":["claire_radar"],"rank":3054,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04828"},{"aliases":[],"benchmark_id":"claire-radar-2607-22015","caveat":"SkillAdeqBench evaluates whether Cloud Skill test suites exercise their specified operational obligations and identifies uncovered obligations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22015"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"SkillAdeqBench","organization_count":1,"organizations":["claire_radar"],"rank":3055,"released":"2026-07-24","source":"claire_radar","url":"https://arxiv.org/abs/2607.22015"},{"aliases":[],"benchmark_id":"claire-radar-2606-17266","caveat":"SkillChain-Gym evaluates production-inventory control policies with reskilling dynamics, including skill certification, forgetting, and training constraints. Metrics cover operations, resilience, capability growth, and training access across seeded disruption scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17266"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"SkillChain-Gym","organization_count":1,"organizations":["claire_radar"],"rank":3056,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.17266"},{"aliases":[],"benchmark_id":"claire-radar-2605-24117","caveat":"SkillEvolBench evaluates whether LLM agents can distill episodic experience into reusable procedural skills. It includes 180 tasks across six environments, with acquisition tasks and frozen deployment tasks testing context shift, adversarial shortcuts, and composition. Scoring uses success rates and other agent metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24117"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SkillEvolBench","organization_count":1,"organizations":["claire_radar"],"rank":3057,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.24117"},{"aliases":[],"benchmark_id":"claire-radar-2606-02540","caveat":"Benchmark of skill-based attacks across the skill-use lifecycle, with 879 attack samples across 71 skills, evaluating Fixed-Payload Poisoning and Self-Mutating Poisoning scenarios across 12 risk types, with attack success rate as primary metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02540"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SkillHarm","organization_count":1,"organizations":["claire_radar"],"rank":3058,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02540"},{"aliases":["Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agents"],"benchmark_id":"claire-radar-2608-12851","caveat":"Lifecycle-aware benchmark for persistent safety failures in self-improving LLM agents, consisting of 25 episodes with malicious demonstrations, benign twins, and fresh-session probes, scored with nine metrics including carryover ASR and unsafe retrieval.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12851"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SkillMisevo-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3059,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.12851"},{"aliases":[],"benchmark_id":"claire-radar-2608-08640","caveat":"SkillReason-Bench is a retrieval benchmark with 3,729 queries and 61,228 skills across nine domains, but the paper focuses on proposing a retrieval method (SkillReason) rather than the benchmark itself.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08640"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SkillReason-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3060,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08640"},{"aliases":[],"benchmark_id":"claire-radar-2606-10388","caveat":"SkillResolve-Bench 1.0 evaluates agent skill retrieval under same-capability ambiguity, pairing helpful skills with risky siblings. It includes 661 pairs, a 7,982-candidate pool, disjoint splits, and reports Recall@K and harmful sibling rate (HSR@K).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10388"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SkillResolve-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3061,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.10388"},{"aliases":[],"benchmark_id":"claire-radar-2608-08542","caveat":"SkillSafe-Bench evaluates skill-merged LLMs on static refusal, adaptive jailbreak robustness, and capability retention using a two-judge AND rule, across multiple open-weight bases and attack types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08542"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SkillSafe-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3062,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08542"},{"aliases":[],"benchmark_id":"llm-stats-skillsbench","caveat":"SkillsBench evaluates coding agents on self-contained programming tasks, measuring practical engineering skills across diverse software development scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/skillsbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"SkillsBench","organization_count":1,"organizations":["llm_stats"],"rank":3063,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/skillsbench?top_n=500"},{"aliases":["SkillsBench","SkillsBench V1","Skills Bench"],"benchmark_id":"skillsbench","caveat":"Measures whether curated Agent Skills raise agent pass rates, not raw model capability; the meaningful figure is the paired no-Skills versus curated-Skills gap. The release date follows the paper's first submission.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"agent","name":"SkillsBench","organization_count":1,"organizations":["Tencent"],"rank":3064,"released":"2026-02-13","source":"model_reports","url":"https://skillsbench.ai/"},{"aliases":[],"benchmark_id":"claire-radar-2608-05573","caveat":"Evaluates whether a judge can verify long-horizon agent executions using original instructions, normalized trajectories, task-time skills, and inspectable artifacts in a runnable environment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05573"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SkillTV-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3065,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05573"},{"aliases":[],"benchmark_id":"claire-radar-2606-00925","caveat":"SkillVetBench is a two-stage security vetting benchmark for open agentic skill ecosystems, evaluating detection of malicious skills via semantic analysis and runtime verification in a sandbox. The benchmark is built from confirmed malicious skills in the OpenClaw ecosystem.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00925"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SkillVetBench","organization_count":1,"organizations":["claire_radar"],"rank":3066,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00925"},{"aliases":[],"benchmark_id":"claire-radar-2606-15899","caveat":"SkillVetBench evaluates security risks in open-source LLM agent skills through multidimensional LLM-based judging and a weighted agentic-risk score.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15899"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SkillVetBench","organization_count":1,"organizations":["claire_radar"],"rank":3067,"released":"2026-06-14","source":"claire_radar","url":"https://arxiv.org/abs/2606.15899"},{"aliases":[],"benchmark_id":"claire-radar-2608-07382","caveat":"SkySeaLand is a satellite object detection dataset with 1,307 high-resolution images and 19,101 bounding boxes across four classes. It provides COCO and YOLO annotations, a common split, and COCO metrics for detector evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07382"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SkySeaLand","organization_count":1,"organizations":["claire_radar"],"rank":3068,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07382"},{"aliases":[],"benchmark_id":"llm-stats-slakevqa","caveat":"A semantically-labeled knowledge-enhanced dataset for medical visual question answering. Contains 642 radiology images (CT scans, MRI scans, X-rays) covering five body parts and 14,028 bilingual English-Chinese question-answer pairs annotated by experienced physicians. Features comprehensive semantic labels and a structural medical knowledge base with both vision-only and knowledge-based questions requiring external medical knowledge reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/slakevqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"SlakeVQA","organization_count":1,"organizations":["llm_stats"],"rank":3069,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/slakevqa?top_n=500"},{"aliases":["SLAMSqueezeBench: Comparing SLAM Systems under Resource Constraints"],"benchmark_id":"claire-radar-2609-19533","caveat":"Benchmark for comparing SLAM systems under resource constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19533"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"SLAMSqueezeBench","organization_count":1,"organizations":["claire_radar"],"rank":3070,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.19533"},{"aliases":[],"benchmark_id":"claire-radar-2607-15517","caveat":"SLAPBench evaluates multimodal LLMs on four-finger SLAP fingerprint verification using NIST SD302b, with 7,832 image pairs. It assesses identity verification under zero-shot, task-description, and similarity-scoring prompts, reporting metrics like AUC and FAR.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15517"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SLAPBench","organization_count":1,"organizations":["claire_radar"],"rank":3071,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.15517"},{"aliases":[],"benchmark_id":"claire-radar-2609-33905","caveat":"SlopBench evaluates repetitive AI writing across 112 hand-written tasks in email, social posts, essays, and workplace chat, scoring four surface behaviors: word count, opener repetition, paragraph rhythm, and fixed lexical constructions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33905"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SlopBench","organization_count":1,"organizations":["claire_radar"],"rank":3072,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33905"},{"aliases":[],"benchmark_id":"claire-radar-2609-19876","caveat":"Egocentric floor plan localization benchmark for large buildings using Aria glasses recordings and aligned floor plans.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19876"],"document_share":0.0002723311546840959,"domain":"CAD","name":"SlugTrails","organization_count":1,"organizations":["claire_radar"],"rank":3073,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.19876"},{"aliases":[],"benchmark_id":"claire-radar-2607-11312","caveat":"SLVMBench evaluates video-LLMs' ability to learn skills from long video memory and apply them to real-time tasks, using 2-3 hour streams with embedded tutorials and human-annotated questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11312"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SLVMBench","organization_count":1,"organizations":["claire_radar"],"rank":3074,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11312"},{"aliases":["4b-agent-model-benchmark"],"benchmark_id":"claire-radar-github-majorminorstudio-4b-agent-model-benchmark","caveat":"Tests six approximately 3B–4B models on seven deterministic simulated operations tasks involving tool use, state tracking, and objective completion.","document_count":1,"document_ids":["claire_radar:https://github.com/MAJORminorStudio/4b-agent-model-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Small-Model Operational Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3075,"released":"2026-09-19","source":"claire_radar","url":"https://github.com/MAJORminorStudio/4b-agent-model-benchmark"},{"aliases":[],"benchmark_id":"opencompass-2217-smartbench","caveat":"SmartBench is the first benchmark specifically designed to evaluate the capabilities of on-device large language models in smartphone scenarios. SmartBench 是首个面向智能手机终端大模型能力评估的基准，基于手机厂商提供的功能将其划分为五类共20项任务，涵盖文本摘要、问答、信息抽取、内容创作和通知管理等场景。它提供高质量数据集与定制化评估标准，旨在推动终端大模型在移动应用中的标准化评估与发展。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SmartBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"SmartBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3076,"released":"2025-09-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SmartBench"},{"aliases":[],"benchmark_id":"claire-radar-2610-12975","caveat":"We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12975"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"SmartSalesBench","organization_count":1,"organizations":["claire_radar"],"rank":3077,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.12975"},{"aliases":[],"benchmark_id":"claire-radar-2606-05574","caveat":"SmellBench is a code refactoring benchmark that proactively injects code smells into clean code snippets. It contains 294 cases across 7 smell types, 3 difficulty levels, and 2 instruction settings, with evaluation covering functional correctness, localization, and refactoring quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05574"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"SmellBench","organization_count":1,"organizations":["claire_radar"],"rank":3078,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05574"},{"aliases":[],"benchmark_id":"claire-radar-2606-01912","caveat":"Benchmark for LLM agents in smart-home environments, with 1,100 tasks across 7 categories and 22 subcategories, built on the executable HomeEnv simulator, stratified by home complexity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01912"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"SMH-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3079,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01912"},{"aliases":[],"benchmark_id":"opencompass-2029-smmile","caveat":"SMMILE is a benchmark for evaluating multimodal large language models (MLLMs) in medical in-context learning tasks, constructed by medical experts. SMMILE 是一个由医学专家主导构建的多模态医疗上下文学习评测基准，旨在评估多模态大语言模型（MLLMs）在医学任务中的上下文学习能力。该基准包含111个问题（共517个图文问答三元组），涵盖6个医学专科和13种影像模态，另提供扩展版本 SMMILE++，包含1038个变换问题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SMMILE"],"document_share":0.0002723311546840959,"domain":"多模态","name":"SMMILE","organization_count":1,"organizations":["opencompass_hub"],"rank":3080,"released":"2025-06-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SMMILE"},{"aliases":[],"benchmark_id":"claire-radar-github-geodiegodrone-sms-spam-classifier","caveat":"A reproducible SMS spam NLP workflow using TF-IDF, logistic regression, and MLflow.","document_count":1,"document_ids":["claire_radar:https://github.com/geodiegodrone/sms-spam-classifier"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"sms-spam-classifier","organization_count":1,"organizations":["claire_radar"],"rank":3081,"released":"2026-09-22","source":"claire_radar","url":"https://github.com/geodiegodrone/sms-spam-classifier"},{"aliases":[],"benchmark_id":"claire-radar-2608-29607","caveat":"Evaluates multimodal retrieval on 1,145 queries and 9,085 gallery items under 53 controlled image and text corruption conditions, with clean and noisy matching metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29607"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SnapBench","organization_count":1,"organizations":["claire_radar"],"rank":3082,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29607"},{"aliases":[],"benchmark_id":"claire-radar-2609-17855","caveat":"Masked-concept recommendation task using SNOMED CT Entity Linking Challenge data derived from MIMIC-IV-Note.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17855"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"SNOMED CT Concept Recommendation from Masked Clinical Context","organization_count":1,"organizations":["claire_radar"],"rank":3083,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17855"},{"aliases":[],"benchmark_id":"claire-radar-2610-09742","caveat":"Refereeing decisions in professional soccer remain inconsistent because referees cannot easily compare a contentious foul against similar past cases.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.09742"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SoccerNet-FoulRet","organization_count":1,"organizations":["claire_radar"],"rank":3084,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.09742"},{"aliases":[],"benchmark_id":"llm-stats-social-iqa","caveat":"The first large-scale benchmark for commonsense reasoning about social situations. Contains 38,000 multiple choice questions probing emotional and social intelligence in everyday situations, testing commonsense understanding of social interactions and theory of mind reasoning about the implied emotions and behavior of others.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/social-iqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"psychology","name":"Social IQa","organization_count":1,"organizations":["llm_stats"],"rank":3085,"released":"2019-04-22","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/social-iqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-26654","caveat":"SocialPersona evaluates multimodal LLMs on recovering revealed preferences from longitudinal social-media timelines and using them in dialogue. It includes 171 user timelines, 2,597 human-verified preference tags across seven domains, and supports profile construction and response generation tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26654"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SocialPersona","organization_count":1,"organizations":["claire_radar"],"rank":3086,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.26654"},{"aliases":[],"benchmark_id":"claire-radar-2608-04009","caveat":"SocietyBench is a benchmark for forecasting counterfactual social-world evolution, collecting web news and social-media posts to build timelines and generating forecasting questions scored on probability calibration and temporal accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04009"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"SocietyBench","organization_count":1,"organizations":["claire_radar"],"rank":3087,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.04009"},{"aliases":[],"benchmark_id":"claire-radar-2605-31597","caveat":"SOCO is a benchmark for semantic object correspondence with keypoint annotations across 100 categories and over 1M pairs. The evaluation is implemented within OmniProbe, which includes correspondence tasks among others.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31597"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SOCO","organization_count":1,"organizations":["claire_radar"],"rank":3088,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31597"},{"aliases":[],"benchmark_id":"claire-radar-2606-05563","caveat":"SoCRATES is a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It contains 600 conflict scenarios across eight domains, probing five socio-cognitive adaptation axes, with topic-localized scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05563"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SoCRATES","organization_count":1,"organizations":["claire_radar"],"rank":3089,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05563"},{"aliases":[],"benchmark_id":"claire-radar-2606-11447","caveat":"SocSci-Repro-Bench evaluates AI coding agents on reproducing social science findings from 221 tasks across four disciplines and 13 domains, using studies with known reproducibility outcomes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11447"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SocSci-Repro-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3090,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11447"},{"aliases":[],"benchmark_id":"claire-radar-2607-04234","caveat":"SoftVTBench is a safety-aware visuo-tactile benchmark for deformable object manipulation in Isaac Sim, with FEM-simulated objects, multi-view RGB, tactile sensing, proprioception, language instructions, and separate Goal Success and Safety Success metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04234"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"SoftVTBench","organization_count":1,"organizations":["claire_radar"],"rank":3091,"released":"2026-07-05","source":"claire_radar","url":"https://arxiv.org/abs/2607.04234"},{"aliases":[],"benchmark_id":"claire-radar-2608-18701","caveat":"SoftVTBench is a visuo-tactile dataset and benchmark for deformable-object manipulation, providing expert demonstrations with synchronized sensor data and finite-element ground truth, with a closed-loop evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.18701"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation","organization_count":1,"organizations":["claire_radar"],"rank":3092,"released":"2026-08-19","source":"claire_radar","url":"https://arxiv.org/abs/2608.18701"},{"aliases":[],"benchmark_id":"claire-radar-2607-08681","caveat":"A physics-constrained benchmark for evaluating economic agents in decentralized energy markets, with a Gymnasium-compatible MDP and an LLM-based Planner/Auditor layer. It evaluates market utility, physical safety, slippage, action smoothness, spatial fairness, and auditability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08681"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"SolarChain-Eval","organization_count":1,"organizations":["claire_radar"],"rank":3093,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08681"},{"aliases":[],"benchmark_id":"claire-radar-2606-19988","caveat":"SolidityBench contains 5,470 repository-level Solidity smart contracts with natural language descriptions, plus SolidityScore, a semantic metric emphasizing domain-critical constructs. It evaluates code generation models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19988"],"document_share":0.0002723311546840959,"domain":"Code","name":"SolidityBench","organization_count":1,"organizations":["claire_radar"],"rank":3094,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.19988"},{"aliases":[],"benchmark_id":"claire-radar-2607-23740","caveat":"SoMBench evaluates social intelligence in large language models across 3 primary dimensions, 17 secondary dimensions, and 71 task paradigms, with 284 shared scenarios and 3,481 expert-verified instances.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23740"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SoMBench","organization_count":1,"organizations":["claire_radar"],"rank":3095,"released":"2026-07-26","source":"claire_radar","url":"https://arxiv.org/abs/2607.23740"},{"aliases":[],"benchmark_id":"claire-radar-2606-06784","caveat":"SopriBench evaluates user-level privacy leakage from social media posts across text, images, and metadata in 50 synthetic profiles with 1,569 images, scoring via the Privacy Exposure Score (PES) that weights value granularity by contextual sensitivity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06784"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SopriBench","organization_count":1,"organizations":["claire_radar"],"rank":3096,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.06784"},{"aliases":[],"benchmark_id":"claire-radar-2605-30329","caveat":"SoundnessBench evaluates LLMs' ability to judge the methodological soundness of research proposals. It contains 1,099 machine-learning research proposals reconstructed from ICLR submissions, labeled with reviewer soundness sub-scores and audited against source papers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30329"],"document_share":0.0002723311546840959,"domain":"AI Scientist","name":"SoundnessBench","organization_count":1,"organizations":["claire_radar"],"rank":3097,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30329"},{"aliases":[],"benchmark_id":"claire-radar-2607-02814","caveat":"SovereignNegotiation-Bench evaluates personal agents in delegated bargaining scenarios, measuring agreement success alongside user utility, privacy, consent, evidence grounding, concession discipline, escalation, and auditability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02814"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SovereignNegotiation-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3098,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02814"},{"aliases":[],"benchmark_id":"claire-radar-2607-05363","caveat":"SovereignPA-Bench evaluates user-owned personal agents on 120 sovereignty stress scenarios, measuring task success, alignment, privacy, consent, evidence, manipulation, burden, and auditability in evolving intent and platform mediation contexts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05363"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"SovereignPA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3099,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.05363"},{"aliases":[],"benchmark_id":"claire-radar-2606-24235","caveat":"SP-Mind is an autonomous AI agent for spatial proteomics analysis, and SP-Bench is introduced for evaluation with 102 tasks across 18 categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24235"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SP-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3100,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24235"},{"aliases":[],"benchmark_id":"claire-radar-2606-18952","caveat":"SP-TransientBench (STB) is a real-captured multi-task benchmark for single-photon perception. It comprises 10 diverse scenes and 10,297 views captured with a solid-state single-photon LiDAR at 256×192 resolution, providing full time-of-flight histograms with multi-return behavior, calibrated camera poses, and standardized metadata. The benchmark evaluates depth estimation, multi-view reconstruction, and 3D semantic understanding, with 13-class semantic annotations for selected scenes. Dedicated data splits and evaluation protocols are provided for each task.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18952"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SP-TransientBench","organization_count":1,"organizations":["claire_radar"],"rank":3101,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18952"},{"aliases":["Human-Centric Image Captioning with Subject-Centered Spatial Understanding"],"benchmark_id":"claire-radar-2609-08300","caveat":"SPACE evaluates subject-centered spatial understanding in human-centric image captioning, focusing on egocentric laterality and anatomical-object bindings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08300"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SPACE","organization_count":1,"organizations":["claire_radar"],"rank":3102,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08300"},{"aliases":[],"benchmark_id":"claire-radar-2610-12402","caveat":"Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12402"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SpaceCast-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3103,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.12402"},{"aliases":["SPADE: SPaT Attack Detection from the Connected Vehicle's Perspective"],"benchmark_id":"claire-radar-2609-02741","caveat":"Provides a labeled, multimodal, simulation-based dataset for deep learning intrusion detection systems focused on SPaT attack detection from the connected vehicle's onboard perspective.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02741"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"SPADE","organization_count":1,"organizations":["claire_radar"],"rank":3104,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02741"},{"aliases":[],"benchmark_id":"claire-radar-2606-02380","caveat":"Benchmark for evaluating spontaneous plan-action divergence in agents, integrating actual tool execution and controlled pressure scenarios to distinguish strategic deception from hallucination.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02380"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SPADE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3105,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02380"},{"aliases":[],"benchmark_id":"claire-radar-2605-25764","caveat":"SpaPath-Bench evaluates pathology foundation models on spatial domain identification using paired whole slide images and spatial transcriptomics data from 42 public slides, measuring partition quality via unsupervised spatial coherence, transcriptomics-referenced agreement, and expert-referenced agreement.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25764"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"SpaPath-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3106,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25764"},{"aliases":[],"benchmark_id":"claire-radar-2608-28092","caveat":"Evaluates spatial reasoning in medical vision encoders via eight probes over multi-organ abdominal CT covering coordinate localization, relational reasoning, and spatial queries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.28092"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"SPAR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3107,"released":"2026-08-28","source":"claire_radar","url":"https://arxiv.org/abs/2608.28092"},{"aliases":[],"benchmark_id":"opencompass-2092-spatial457","caveat":"Spatial457 systematically introduces four key capabilities—multi-object understanding, 2D and 3D localization, and 3D orientation—across five difficulty levels and seven question types, from simple object recognition to complex 6DoF spatial reasoning tasks. Spatial457 聚焦于空间推理的四项核心能力：多物体理解、二维位置识别、三维位置识别，以及三维朝向判断。这些能力对现实世界中复杂场景的认知和理解至关重要。我们构建了一种层级递进的评估结构，将问题划分为 7 类问题类型，覆盖从基础的单物体识别任务，到我们首次提出的更具挑战性的 6DoF 空间推理任务。整个数据集按 5 个难度等级组织，帮助系统性地评估模型在不同层次空间理解任务中的表现。\n\nSpatial457 不仅提升了空间推理评测的覆盖面和挑战性，也为后续研究提供了统一的基准，有助于推动多模态模型在真实世界空间认知任务中的发展。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Spatial457"],"document_share":0.0002723311546840959,"domain":"多模态","name":"Spatial457","organization_count":1,"organizations":["opencompass_hub"],"rank":3108,"released":"2025-04-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Spatial457"},{"aliases":[],"benchmark_id":"claire-radar-2605-27367","caveat":"SpatialBench is a deterministic, density-aware benchmark for spatial foundation models, spanning 19 datasets, 546 scenes, and five spatial domains. It evaluates 41 models across six paradigms on five task suites—depth, camera pose, trajectory, point-cloud reconstruction, and long-sequence streaming—under four input density settings with precomputed and pinned test frames.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27367"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SpatialBench","organization_count":1,"organizations":["claire_radar"],"rank":3109,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27367"},{"aliases":[],"benchmark_id":"claire-radar-2605-28065","caveat":"SpatialBench-Long evaluates AI agents on long-horizon spatial biology tasks, requiring recovery of biological claims from raw data across 24 evaluations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28065"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SpatialBench-Long","organization_count":1,"organizations":["claire_radar"],"rank":3110,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28065"},{"aliases":[],"benchmark_id":"claire-radar-2607-21072","caveat":"ProVisE is a framework for evaluating image-generation models on spatial benchmarks by converting visual answers into structured predictions. SpatialGen-Bench is a diagnostic dataset of 470 samples across 14 spatial subtasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.21072"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SpatialGen-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3111,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.21072"},{"aliases":["SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Models"],"benchmark_id":"claire-radar-2608-01709","caveat":"A benchmark of over one million RGB-only question-answer pairs from 200 indoor scenes for closest-instance distance queries in vision-language models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01709"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SPATIALQUERY-1M","organization_count":1,"organizations":["claire_radar"],"rank":3112,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01709"},{"aliases":[],"benchmark_id":"claire-radar-2606-27876","caveat":"Evaluates spatial intelligence in low-altitude UAV scenarios across 14 task types covering semantic discrimination, spatial relations, collaboration, and motion understanding, with 7 input configurations and 9 answer formats.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27876"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SpatialUAV","organization_count":1,"organizations":["claire_radar"],"rank":3113,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.27876"},{"aliases":[],"benchmark_id":"claire-radar-2606-09669","caveat":"SpatialWorld evaluates multimodal agents on interactive spatial reasoning in 760 real-world tasks across eight simulation backends. Agents operate under vision-only partial observability, using a unified text-based action interface. Performance is measured via terminal-state verifiers for task success rate and step efficiency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09669"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"SpatialWorld","organization_count":1,"organizations":["claire_radar"],"rank":3114,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09669"},{"aliases":[],"benchmark_id":"claire-radar-2607-05365","caveat":"SPEARBench evaluates naturalness in streaming speech-to-speech language models via question-answer interactions, measuring latency, interruptions, speech quality, ASR robustness, language consistency, emotional naturalness, and interpersonal stance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05365"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SPEARBench","organization_count":1,"organizations":["claire_radar"],"rank":3115,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.05365"},{"aliases":[],"benchmark_id":"claire-radar-2610-04253","caveat":"Generating an executable program does not necessarily mean that it correctly implements the behavioral requirements specified in natural language.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.04253"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Spec2Game","organization_count":1,"organizations":["claire_radar"],"rank":3116,"released":"2026-10-03","source":"claire_radar","url":"https://arxiv.org/abs/2610.04253"},{"aliases":[],"benchmark_id":"claire-radar-2609-33699","caveat":"SpecRead evaluates specification comprehension via 383 questions over 10 OpenTitan IP blocks: exact retrieval, cross-section reasoning, contradiction detection in mutated specs, and spec-RTL consistency checking.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33699"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"SpecRead","organization_count":1,"organizations":["claire_radar"],"rank":3117,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33699"},{"aliases":[],"benchmark_id":"claire-radar-2606-17339","caveat":"SpeechDx evaluates clinical speech AI across 12 datasets and 27 tasks covering various conditions, structured by speech production stages. Scoring includes classification accuracy and zero-shot transfer performance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.17339"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"SpeechDx","organization_count":1,"organizations":["claire_radar"],"rank":3118,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.17339"},{"aliases":[],"benchmark_id":"claire-radar-2610-08076","caveat":"Frontier LLM agents have been shown to be capable of solving increasingly complex tasks for which humans have measurable solutions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.08076"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"SpeedrunBench","organization_count":1,"organizations":["claire_radar"],"rank":3119,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.08076"},{"aliases":[],"benchmark_id":"llm-stats-spider","caveat":"A large-scale, complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students. Contains 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables, covering 138 different domains. Requires models to generalize to both new SQL queries and new database schemas, making it distinct from previous semantic parsing tasks that use single databases.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/spider?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Spider","organization_count":1,"organizations":["llm_stats"],"rank":3120,"released":"2018-09-24","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/spider?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1133-spider","caveat":"Spider is a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases. Spider 是一个大规模、复杂且跨领域的语义解析和文本到 SQL 数据集。Spider 挑战的目标是开发自然语言接口以访问跨领域数据库。该数据集包含 10,181 个问题和 5,693 个独特的复杂 SQL 查询，涵盖 200 个包含多个表的数据库，涉及 138 个不同的领域。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Spider"],"document_share":0.0002723311546840959,"domain":"智能体","name":"Spider","organization_count":1,"organizations":["opencompass_hub"],"rank":3121,"released":"2019-02-02","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Spider"},{"aliases":[],"benchmark_id":"claire-radar-2607-06229","caveat":"Extends text-to-SQL to AI-native SQL workflows with 465 instances across 125 real-world databases on Snowflake. Tasks require using AI functions like classification and sentiment analysis. Evaluates execution accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06229"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Spider 2.0-AIFunc","organization_count":1,"organizations":["claire_radar"],"rank":3122,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06229"},{"aliases":[],"benchmark_id":"opencompass-1270-spider2-v","caveat":"Spider2-V is the first multimodal agent benchmark focusing on professional data science and engineering workflows, featuring 494 real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. Spider2-V是第一个专注于专业数据科学和工程工作流程的多模态代理基准测试，整合了20 个企业级专业应用程序，包含来自真实计算机环境的494 个真实任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Spider2-V"],"document_share":0.0002723311546840959,"domain":"智能体","name":"Spider2-V","organization_count":1,"organizations":["opencompass_hub"],"rank":3123,"released":"2024-07-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Spider2-V"},{"aliases":[],"benchmark_id":"claire-radar-2608-10692","caveat":"SPIEval is a human-curated benchmark for evaluating large language models as mobile assistants that retrieve and reason over personal information scattered across multiple apps. It comprises 250 tasks across five cognitive capabilities, 4,335 fictional personal records in 10 simulated apps, and supports multi-turn interaction through 21 tools. Models receive underspecified user instructions and must search records and invoke tools; final execution calls are compared to human-annotated gold calls at the parameter level.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10692"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SPIEval","organization_count":1,"organizations":["claire_radar"],"rank":3124,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10692"},{"aliases":[],"benchmark_id":"claire-radar-2608-02684","caveat":"SPIKE-Bench evaluates LLM biosecurity risks using 631 curated toxin-design prompts and a three-stage funnel (compliance, plausibility, predicted toxicity) producing the Functional Harmfulness Rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02684"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"SPIKE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3125,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02684"},{"aliases":["Measuring LLM Sycophancy under Sustained Multi-Turn Pressure"],"benchmark_id":"claire-radar-2609-09090","caveat":"SPINE evaluates LLM sycophancy by having an adaptive LLM proxy persistently challenge a target model across up to 25 turns on false-presupposition and unethical-query items, measuring collapse rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09090"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SPINE","organization_count":1,"organizations":["claire_radar"],"rank":3126,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.09090"},{"aliases":[],"benchmark_id":"claire-radar-2607-11844","caveat":"SportMV-Bench evaluates multi-view sports video understanding with 1022 multi-view bundles and 3015 QA pairs across 10 sports, covering perception, rule-aware event interpretation, and adjudicative reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11844"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"SportMV-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3127,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11844"},{"aliases":[],"benchmark_id":"opencompass-1152-sportqa","caveat":"SportQA aims to evaluate LLMs in the context of sports understanding. SportQA encompasses over 70,000 multiple-choice questions across three distinct difficulty levels, each targeting different aspects of sports knowledge from basic historical facts to intricate, scenariobased reasoning tasks. SportQA 专门用于评估大型语言模型（LLMs）在体育理解方面的能力。SportQA 包含超过 70,000 道多项选择题，分为三个不同的难度级别，针对从基本历史事实到复杂情境推理任务的各种体育知识。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SportQA"],"document_share":0.0002723311546840959,"domain":"理解","name":"SportQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3128,"released":"2024-06-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SportQA"},{"aliases":[],"benchmark_id":"claire-radar-2609-10356","caveat":"A human-verified benchmark for grounded image difference captioning of long-term changes, with captions and spatial masks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10356"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SPOT-THE-SHIFT","organization_count":1,"organizations":["claire_radar"],"rank":3129,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10356"},{"aliases":["Graph Reduction in Multirelational Networks: A Spreading-Oriented Reduction Benchmark"],"benchmark_id":"claire-radar-2606-12581","caveat":"An open-source framework for evaluating how graph sparsification and coarsening affect influence-maximisation performance on single-layer and multilayer real-world networks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12581"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Spreading-Oriented Reduction Benchmark (SORB)","organization_count":1,"organizations":["claire_radar"],"rank":3130,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.12581"},{"aliases":[],"benchmark_id":"claire-radar-2606-29955","caveat":"SpreadsheetBench 2 evaluates spreadsheet agents on end-to-end business workflows across generation, debugging, and visualization tasks. It includes 321 tasks from authentic business data, with multi-sheet workbooks requiring cross-sheet reasoning. The benchmark provides a unified multi-turn agent scaffold and evaluation scripts for reproducibility.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29955"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"SpreadsheetBench","organization_count":1,"organizations":["claire_radar"],"rank":3131,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.29955"},{"aliases":[],"benchmark_id":"opencompass-1267-spreadsheetbench","caveat":"SpreadsheetBench is a challenging spreadsheet manipulation benchmark built from 912 real questions gathered from online Excel forums. SpreadsheetBench是一个具有挑战性的电子表格操作基准测试，包含912个来自在线Excel论坛的真实问题","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SpreadsheetBench"],"document_share":0.0002723311546840959,"domain":"其他","name":"SpreadsheetBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3132,"released":"2024-06-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SpreadsheetBench"},{"aliases":[],"benchmark_id":"llm-stats-spreadsheetbench-2","caveat":"SpreadsheetBench 2 evaluates office automation agents on spreadsheet analysis, reasoning, and manipulation tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-2?top_n=500"],"document_share":0.0002723311546840959,"domain":"productivity","name":"SpreadsheetBench 2","organization_count":1,"organizations":["llm_stats"],"rank":3133,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-spreadsheetbench-v1","caveat":"SpreadSheetBench-v1 evaluates office automation agents on spreadsheet reasoning and manipulation tasks, measuring the ability to analyze, transform, and operate on spreadsheet data through tools.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-v1?top_n=500"],"document_share":0.0002723311546840959,"domain":"productivity","name":"SpreadSheetBench-v1","organization_count":1,"organizations":["llm_stats"],"rank":3134,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/spreadsheetbench-v1?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-23123","caveat":"SQBench v1.0 evaluates language-model agents on 220 production-oriented tasks organized into L1 atomic capabilities, L2 composite skills, and L3 business scenarios. Tasks require processing input assets, using tools, and producing a specified deliverable. Scoring computes Completion, Risk Penalty, and Performance from a 10D Risk Matrix; Strict Pass requires Completion=1 and Risk Penalty=0.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23123"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SQBench","organization_count":1,"organizations":["claire_radar"],"rank":3135,"released":"2026-07-25","source":"claire_radar","url":"https://arxiv.org/abs/2607.23123"},{"aliases":[],"benchmark_id":"llm-stats-squality","caveat":"SQuALITY (Summarization-format QUestion Answering with Long Input Texts, Yes!) is a long-document summarization dataset built by hiring highly-qualified contractors to read public-domain short stories (3000-6000 words) and write original summaries from scratch. Each document has five summaries: one overview and four question-focused summaries. Designed to address limitations in existing summarization datasets by providing high-quality, faithful summaries.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/squality?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"SQuALITY","organization_count":1,"organizations":["llm_stats"],"rank":3136,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/squality?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-37576","caveat":"With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture fine-grained alignment and generative artifacts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37576"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SQUARE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3137,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37576"},{"aliases":[],"benchmark_id":"claire-radar-2610-13461","caveat":"AI systems are making rapid progress in mathematics, but this progress is hard to measure.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13461"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SqueezeBench","organization_count":1,"organizations":["claire_radar"],"rank":3138,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13461"},{"aliases":[],"benchmark_id":"claire-radar-2608-11469","caveat":"SRE-Bench evaluates AI agents on reverse engineering of binaries compiled from 19 private real-world-scale C programs (average 16.9K LoC) with 44 anti-analysis primitives, yielding 262 binary instances and 1572 deterministically graded tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11469"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"SRE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3139,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.11469"},{"aliases":[],"benchmark_id":"claire-radar-2609-33023","caveat":"Benchmarks for site reliability engineering (SRE) agents are typically episodic: one fault is injected, the agent receives an incident task, and its response is scored.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33023"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"SRE-Marathon","organization_count":1,"organizations":["claire_radar"],"rank":3140,"released":"2026-09-26","source":"claire_radar","url":"https://arxiv.org/abs/2609.33023"},{"aliases":[],"benchmark_id":"claire-radar-2606-25634","caveat":"SSMNBench is a diagnostic benchmark for cross-view human and human-object understanding, comprising 3,300 QA pairs categorized into Single-View Sufficiency (SVS) and Multi-View Necessity (MVN) tasks. It evaluates multimodal LLMs by perturbing view availability to assess distraction robustness and cross-view evidence integration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25634"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SSMNBench","organization_count":1,"organizations":["claire_radar"],"rank":3141,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25634"},{"aliases":[],"benchmark_id":"claire-radar-2606-09746","caveat":"ST-Bench is a verification benchmark for evaluating certified robustness of spatio-temporal neural networks on autonomous driving (Udacity) and activity recognition (UCF-101) tasks, using spatio-temporal perturbation constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09746"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"ST-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3142,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09746"},{"aliases":[],"benchmark_id":"claire-radar-2610-07763","caveat":"The rapid progress of LLM-based multi-agent systems (MAS) has shown that they largely outperform single agents on coding, math, and QA tasks, where executable tests provide a binary success signal.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.07763"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ST-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3143,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.07763"},{"aliases":["Evidence-Backed Video Question Answering"],"benchmark_id":"claire-radar-2607-11862","caveat":"ST-Evidence evaluates video question answering that must return both semantic answers and dense spatio-temporal visual evidence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11862"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ST-Evidence","organization_count":1,"organizations":["claire_radar"],"rank":3144,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11862"},{"aliases":[],"benchmark_id":"claire-radar-2607-20558","caveat":"StabilityBench is a benchmark operator that converts single-turn benchmark queries into multi-turn interaction histories with injected user simulations, evaluating LLM performance stability across demographic proxies and sycophantic baits on existing benchmarks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20558"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"StabilityBench","organization_count":1,"organizations":["claire_radar"],"rank":3145,"released":"2026-07-17","source":"claire_radar","url":"https://arxiv.org/abs/2607.20558"},{"aliases":[],"benchmark_id":"opencompass-1084-stabletoolbench","caveat":"StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. StableToolBench 是一个从 ToolBench 发展而来的基准，提出了一个虚拟 API 服务器和稳定的评估系统。虚拟 API 服务器包含一个缓存系统和 API 模拟器，这些组件相辅相成，以缓解 API 状态变化带来的影响。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StableToolBench"],"document_share":0.0002723311546840959,"domain":"其他","name":"StableToolBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3146,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/StableToolBench"},{"aliases":[],"benchmark_id":"claire-radar-2606-10394","caveat":"STAGE-Claw is an automated framework for building and evaluating personal-agent tasks in state-based computing environments, with a benchmark of 40 tasks. Evaluations measure final system state correctness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10394"],"document_share":0.0002723311546840959,"domain":"Agents","name":"STAGE-Claw","organization_count":1,"organizations":["claire_radar"],"rank":3147,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.10394"},{"aliases":[],"benchmark_id":"claire-radar-2606-13385","caveat":"StakeBench is a stakeholder-centric benchmark for prompt-injection attacks in web agents for online shopping. It decomposes risk into 12 attack objectives across three stakeholder classes, with 264 adversarial cases across 12 product categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13385"],"document_share":0.0002723311546840959,"domain":"Agents","name":"StakeBench","organization_count":1,"organizations":["claire_radar"],"rank":3148,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13385"},{"aliases":[],"benchmark_id":"claire-radar-2606-19613","caveat":"StaminaBench stress-tests coding agents over 100 interaction turns of change requests, measuring how many consecutive turns they handle before failing, with programmatically generated tasks and black-box HTTP evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19613"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"StaminaBench","organization_count":1,"organizations":["claire_radar"],"rank":3149,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19613"},{"aliases":[],"benchmark_id":"claire-radar-2607-22658","caveat":"StanceBench evaluates interpersonal stance in conversational speech across 9 dimensions using LLM-as-a-judge on the Seamless Interaction corpus.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.22658"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StanceBench","organization_count":1,"organizations":["claire_radar"],"rank":3150,"released":"2026-06-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.22658"},{"aliases":[],"benchmark_id":"claire-radar-2607-24191","caveat":"StanceFlip is a benchmark for multimodal conversational stance flipping forecasting with two subtasks: sextuple extraction and flip attribution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.24191"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StanceFlip","organization_count":1,"organizations":["claire_radar"],"rank":3151,"released":"2026-07-27","source":"claire_radar","url":"https://arxiv.org/abs/2607.24191"},{"aliases":[],"benchmark_id":"opencompass-1845-stark-10k","caveat":"STARK is a comprehensive benchmark designed to systematically evaluate large language models (LLMs) and large reasoning models (LRMs) on spatiotemporal reasoning tasks, particularly for applications in cyber-physical systems (CPS) such as robotics, autonomous vehicles, and smart city infrastructure. STARK 是一个全面的基准测试套件，旨在系统评估大语言模型（LLMs）和大推理模型（LRMs）在时空推理任务中的表现，特别是在网络物理系统（CPS）中的应用，如机器人、自动驾驶和智能城市基础设施。该基准包含 26 种不同的时空任务，涵盖状态估计、时空关系推理和世界知识感知推理三个层次。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/STARK_10k"],"document_share":0.0002723311546840959,"domain":"多模态","name":"STARK_10k","organization_count":1,"organizations":["opencompass_hub"],"rank":3152,"released":"2025-05-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/STARK_10k"},{"aliases":[],"benchmark_id":"claire-radar-2608-17800","caveat":"StartupBench is a benchmark for general-purpose agents on end-to-end workflows derived from market-validated AI startup products, with deliverable-oriented tasks and fine-grained rubrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.17800"],"document_share":0.0002723311546840959,"domain":"Agents","name":"StartupBench","organization_count":1,"organizations":["claire_radar"],"rank":3153,"released":"2026-08-18","source":"claire_radar","url":"https://arxiv.org/abs/2608.17800"},{"aliases":[],"benchmark_id":"claire-radar-2606-22977","caveat":"StatABench evaluates LLMs' statistical analysis capabilities through two components: Stat-Closed, 404 questions across 18 topics in multiple formats, and Stat-Open, 30 complex modeling tasks from competitions, scored via LLM-as-judge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22977"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StatABench","organization_count":1,"organizations":["claire_radar"],"rank":3154,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.22977"},{"aliases":[],"benchmark_id":"claire-radar-2609-03673","caveat":"Evaluates world-state reasoning in video continuation through 200 cross-segment tasks across past-visible, occluded-process, and complex-transition state categories, with scoring based on state equivalence, state correctness, and hallucination rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.03673"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Statebench","organization_count":1,"organizations":["claire_radar"],"rank":3155,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.03673"},{"aliases":["nsr-product-use-benchmark"],"benchmark_id":"claire-radar-github-stateset-nsr-product-use-benchmark","caveat":"Eleven synthetic cases evaluate how an agent uses StateSet NSR MCP tools before a consequential action, with a harness scoring tool calls and a final structured JSON for passed cases and unsafe actions.","document_count":1,"document_ids":["claire_radar:https://github.com/stateset/nsr-product-use-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"StateSet NSR Product Use Benchmark v1","organization_count":1,"organizations":["claire_radar"],"rank":3156,"released":"2026-09-26","source":"claire_radar","url":"https://github.com/stateset/nsr-product-use-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-01982","caveat":"Evaluates language models on statistical problem formulation tasks drawn from five statistics textbooks and a data science case library, covering diverse problem types, data representations, and scenario styles.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01982"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StatFormBench","organization_count":1,"organizations":["claire_radar"],"rank":3157,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.01982"},{"aliases":[],"benchmark_id":"claire-radar-2607-26367","caveat":"A benchmark of six Ising-type problems for evaluating LLM agents' ability to discover statistical mechanical mappings from raw partition functions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26367"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StatMechBench-v0","organization_count":1,"organizations":["claire_radar"],"rank":3158,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.26367"},{"aliases":[],"benchmark_id":"claire-radar-2607-26314","caveat":"StealthBench measures operational stealth of autonomous offensive-security agents across six OPSEC dimensions, using 14 dockerized task scenarios and a three-model judge panel, with metrics like safe success rate and Stealth@Solve.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26314"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"StealthBench","organization_count":1,"organizations":["claire_radar"],"rank":3159,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.26314"},{"aliases":[],"benchmark_id":"claire-radar-github-qbalhkim-steam-review-sentiment","caveat":"Evaluates sentence-level sentiment classification on a balanced Steam review test set across six models, plus aspect-based sentiment extraction over eight game-review aspects using spaCy, VADER and a DeBERTa ABSA model.","document_count":1,"document_ids":["claire_radar:https://github.com/qbalhkim/steam-review-sentiment"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"steam-review-sentiment","organization_count":1,"organizations":["claire_radar"],"rank":3160,"released":"2026-09-10","source":"claire_radar","url":"https://github.com/qbalhkim/steam-review-sentiment"},{"aliases":[],"benchmark_id":"claire-radar-2607-05264","caveat":"STEELBENCH evaluates vision-language models on per-worker activity recognition and safety-rule reasoning in industrial CCTV footage. It includes 1,345 clips with dense annotations and a provenance-aware audit protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05264"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SteelBench","organization_count":1,"organizations":["claire_radar"],"rank":3161,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.05264"},{"aliases":[],"benchmark_id":"claire-radar-2609-12623","caveat":"SteerBench evaluates full-duplex spoken steerability using 390 spoken prompts and 1,067 human-authored binary audio and text rubrics spanning tone, persona, style or accent, and speed or length.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12623"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SteerBench","organization_count":1,"organizations":["claire_radar"],"rank":3162,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12623"},{"aliases":[],"benchmark_id":"claire-radar-2608-12654","caveat":"SteerBench-Work evaluates agent steering decisions at action boundaries in workplace scenarios across seven domains, with 106 incident-anchored scenarios and evidence-reversed mirrors, scored on correct proceed/hold boundaries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12654"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SteerBench-Work","organization_count":1,"organizations":["claire_radar"],"rank":3163,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.12654"},{"aliases":[],"benchmark_id":"llm-stats-stem","caveat":"A comprehensive multimodal benchmark dataset with 448 skills and 1,073,146 questions spanning all STEM subjects (Science, Technology, Engineering, Mathematics), designed to test neural models' vision-language STEM skills based on K-12 curriculum. Unlike existing datasets that focus on expert-level ability, this dataset includes fundamental skills designed around educational standards.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/stem?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"STEM","organization_count":1,"organizations":["llm_stats"],"rank":3164,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/stem?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-00148","caveat":"StemBind is a diagnostic benchmark with shared-stem questions to attribute failures in abstract visual reasoning. It includes perception, rule, and full tasks with stage annotations but no official code or dataset release.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00148"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StemBind","organization_count":1,"organizations":["claire_radar"],"rank":3165,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.00148"},{"aliases":[],"benchmark_id":"claire-radar-2606-29592","caveat":"STEMGym is an open-source Gymnasium benchmark of 15 physics-simulated STEM worlds across five materials and four characterization tasks, scored by DEC-AUC.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29592"],"document_share":0.0002723311546840959,"domain":"General AI","name":"STEMGym","organization_count":1,"organizations":["claire_radar"],"rank":3166,"released":"2026-06-28","source":"claire_radar","url":"https://arxiv.org/abs/2606.29592"},{"aliases":[],"benchmark_id":"claire-radar-2608-06477","caveat":"StepJack evaluates computer-use agents against multi-step indirect prompt injection attacks, where adversarial goals are decomposed into innocuous sub-steps across a chain of pages. It comprises 480 test examples across platforms and instruction types, with attack success rate as the primary metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06477"],"document_share":0.0002723311546840959,"domain":"Agents","name":"StepJack","organization_count":1,"organizations":["claire_radar"],"rank":3167,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06477"},{"aliases":[],"benchmark_id":"claire-radar-2605-23237","caveat":"StereoGenBench is a synthetic multi-camera benchmark for stereo generation, rendered in Unreal Engine with a rigid six-camera array. It provides calibrated view pairs across baseline regimes, along with RGB, metric depth, intrinsics, and poses for each scene.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.23237"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StereoGenBench","organization_count":1,"organizations":["claire_radar"],"rank":3168,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.23237"},{"aliases":[],"benchmark_id":"claire-radar-2608-22309","caveat":"StocBench evaluates generative models for probabilistic forecasting of stochastic fluid flows, using a two-dimensional Kolmogorov flow with stochastic forcing, measuring one-step distributional accuracy and preservation of the enstrophy spectrum during autoregressive rollouts.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22309v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StocBench","organization_count":1,"organizations":["claire_radar"],"rank":3169,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22309v1"},{"aliases":[],"benchmark_id":"claire-radar-2609-09264","caveat":"A Lean 4 benchmark of 450 graduate stochastic-processes problems at varying abstraction levels, each paired with its natural-language source.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09264"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"StochBench","organization_count":1,"organizations":["claire_radar"],"rank":3170,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.09264"},{"aliases":[],"benchmark_id":"claire-radar-2610-10942","caveat":"Reinforcement learning environments are now a primary lever for improving large language model (LLM) capabilities in post-training, yet most agentic benchmarks remain static: the world moves only when the agent acts, the reward is a termin…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10942"],"document_share":0.0002723311546840959,"domain":"Agents","name":"StoreBench","organization_count":1,"organizations":["claire_radar"],"rank":3171,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10942"},{"aliases":[],"benchmark_id":"claire-radar-2609-30981","caveat":"Presents a benchmark comprising 5,736 questions across 630 episodes for online video QA, but no artifact evidence is available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30981"],"document_share":0.0002723311546840959,"domain":"General AI","name":"STORM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3172,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.30981"},{"aliases":[],"benchmark_id":"claire-radar-2605-28073","caveat":"STORYLENSBENCH benchmarks preference-aligned story rewriting with structured story books and reader profiles, plus reward model and rewriting model.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28073"],"document_share":0.0002723311546840959,"domain":"General AI","name":"STORYLENSBENCH","organization_count":1,"organizations":["claire_radar"],"rank":3173,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28073"},{"aliases":[],"benchmark_id":"claire-radar-2608-12549","caveat":"StrAD is a benchmark for long-form audio description generation on full-length videos across diverse genres, reformulating the task as streaming dense video captioning without ground-truth timestamps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12549"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StrAD","organization_count":1,"organizations":["claire_radar"],"rank":3174,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.12549"},{"aliases":[],"benchmark_id":"claire-radar-github-movahedi-ca-strata-bench","caveat":"Evaluates AI agents on compiling longitudinal market intelligence from fragmented, geographically nested, and temporally incomplete evidence, with scoring across temporal, source, spatial, statistical, visual, and narrative dimensions.","document_count":1,"document_ids":["claire_radar:https://github.com/movahedi-ca/strata-bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"STRATA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3175,"released":"2026-09-19","source":"claire_radar","url":"https://github.com/movahedi-ca/strata-bench"},{"aliases":[],"benchmark_id":"claire-radar-github-derrtaderr-strategy-benchmark","caveat":"Evaluates strategy documents by measuring claim survival rate under adversarial scrutiny, comparing gated-evidence and naive LLM-generated strategies.","document_count":1,"document_ids":["claire_radar:https://github.com/derrtaderr/strategy-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"strategy-benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3176,"released":"2026-09-07","source":"claire_radar","url":"https://github.com/derrtaderr/strategy-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2608-23475","caveat":"StrategyBench is a benchmark for evaluating explicit strategy induction in large language models. It selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23475v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StrategyBench","organization_count":1,"organizations":["claire_radar"],"rank":3177,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23475v1"},{"aliases":[],"benchmark_id":"opencompass-1107-strategyqa","caveat":"STRATEGYQA is a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy STRATEGYQA 是一个问答基准，其中所需的推理步骤在问题中是隐含的，可以通过策略进行推断。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StrategyQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"StrategyQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3178,"released":"2021-01-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/StrategyQA"},{"aliases":[],"benchmark_id":"claire-radar-2608-05703","caveat":"StreamArena evaluates hour-scale streaming video understanding across 243 full-length videos (avg 88.8 min) with 3,646 open-ended QA pairs, covering real-time perception, historical retrospection, proactive interaction, and multimodal tool use. Includes a standardized runner and LLM-as-judge scorer.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05703"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StreamArena","organization_count":1,"organizations":["claire_radar"],"rank":3179,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05703"},{"aliases":[],"benchmark_id":"claire-radar-2606-14571","caveat":"StreamMemBench is a streaming benchmark for evaluating agent memory with two-step task sequences around evidence anchors from EgoLife egocentric streams, with four diagnostic metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.14571"],"document_share":0.0002723311546840959,"domain":"Agents","name":"StreamMemBench","organization_count":1,"organizations":["claire_radar"],"rank":3180,"released":"2026-06-12","source":"claire_radar","url":"https://arxiv.org/abs/2606.14571"},{"aliases":[],"benchmark_id":"claire-radar-2605-25758","caveat":"StreamProfileBench evaluates LLMs on fine-grained streaming user profiling. Models maintain a rolling persona summary from a stream of user posts and predict which tags from a candidate pool the user will engage with next. Includes over 120,000 posts from 7,000+ users across five Chinese platforms with metrics for recall, novelty, stability, and error rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25758"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StreamProfileBench","organization_count":1,"organizations":["claire_radar"],"rank":3181,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25758"},{"aliases":[],"benchmark_id":"opencompass-1542-structflowbench","caveat":"StructFlowBench is a structurally annotated multi-turn benchmark that leverages a structure-driven generation paradigm to enhance the simulation of complex dialogue scenarios. StructFlowBench，这是一个包含155条数据的结构化标注多轮基准，它利用结构驱动生成范式来增强复杂对话场景的模拟。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StructFlowBench"],"document_share":0.0002723311546840959,"domain":"指令跟随","name":"StructFlowBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3182,"released":"2025-02-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/StructFlowBench"},{"aliases":[],"benchmark_id":"opencompass-2082-structtokenbench","caveat":"StructTokenBench is a benchmark framework designed to comprehensively evaluate the quality and efficiency of protein structure tokenization methods, particularly focusing on fine-grained local substructures. StructTokenBench 是一个评估蛋白质结构标记化方法质量效率的基准。它关注细粒度局部子结构，评测维度包括下游有效性、敏感性、独特性和码本利用效率。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StructTokenBench"],"document_share":0.0002723311546840959,"domain":"其他","name":"StructTokenBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3183,"released":"2025-02-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/StructTokenBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-28470","caveat":"Evaluates large language model tutors via GRE learning gains, comparing AI vs human tutoring and assessing teaching quality and cost efficiency across seven GRE domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.28470"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StudentBench","organization_count":1,"organizations":["claire_radar"],"rank":3184,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.28470"},{"aliases":[],"benchmark_id":"opencompass-1082-studenteval","caveat":"STUDENTEVAL contains 1,749 prompts written by 80 students who have only completed one introductory Python course. STUDENTEVAL contains numerous non-expert prompts describing the same problem, enabling exploration of key factors in prompt success. StudentEval 包含 1,749 个由 80 名仅完成一门入门 Python 课程的学生撰写的提示。StudentEval 中包含许多非专家提示，描述相同的问题，使得探索提示成功的关键因素成为可能。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StudentEval"],"document_share":0.0002723311546840959,"domain":"代码","name":"StudentEval","organization_count":1,"organizations":["opencompass_hub"],"rank":3185,"released":"2024-08-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/StudentEval"},{"aliases":[],"benchmark_id":"claire-radar-2609-00787","caveat":"Evaluates language models on hard physics textbook exercises (Application Set) and olympiad-level theory problems (Transfer Set), reporting pass@k accuracy for absorption and transfer of textbook knowledge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00787"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"StudyBench","organization_count":1,"organizations":["claire_radar"],"rank":3186,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.00787"},{"aliases":[],"benchmark_id":"claire-radar-2608-30584","caveat":"Benchmark for compositional spatio-temporal video grounding with difficulty levels based on temporal complexity and spatial interference.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30584"],"document_share":0.0002723311546840959,"domain":"General AI","name":"STVG-CompBench","organization_count":1,"organizations":["claire_radar"],"rank":3187,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30584"},{"aliases":[],"benchmark_id":"opencompass-1744-stylerec","caveat":"A Benchmark Dataset for Prompt Recovery in Writing Style Transformation. 基于写作风格转换的提示词恢复的评测集","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/StyleRec"],"document_share":0.0002723311546840959,"domain":"理解","name":"StyleRec","organization_count":1,"organizations":["opencompass_hub"],"rank":3188,"released":"2025-04-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/StyleRec"},{"aliases":[],"benchmark_id":"claire-radar-2606-20527","caveat":"StylisticBias is a benchmark of 25,000 images with controlled single-attribute variations to evaluate attribute-level social bias in multimodal LLMs. It measures how visual cues shift model judgments in 25 scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20527"],"document_share":0.0002723311546840959,"domain":"General AI","name":"StylisticBias","organization_count":1,"organizations":["claire_radar"],"rank":3189,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.20527"},{"aliases":[],"benchmark_id":"claire-radar-2606-05761","caveat":"SubtleMemory evaluates fine-grained relational memory discrimination in long-horizon AI agents. It contains 1,522 evaluation instances over 10 long histories, grounded in 1,090 relation-controlled memory-variant sets, spanning user-related and non-user-related queries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05761"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SubtleMemory","organization_count":1,"organizations":["claire_radar"],"rank":3190,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05761"},{"aliases":[],"benchmark_id":"claire-radar-2605-27911","caveat":"SuiChat-CN is a Chinese group-chat benchmark for contextual suicide risk assessment, with 13,312 segments from 1,406 users.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27911"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SuiChat-CN","organization_count":1,"organizations":["claire_radar"],"rank":3191,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27911"},{"aliases":[],"benchmark_id":"llm-stats-summscreenfd","caveat":"SummScreenFD is the ForeverDreaming subset of the SummScreen dataset for abstractive screenplay summarization, comprising pairs of TV series transcripts and human-written recaps from 88 different shows. The dataset provides a challenging testbed for abstractive summarization where plot details are often expressed indirectly in character dialogues and scattered across the entirety of the transcript, requiring models to find and integrate these details to form succinct plot descriptions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/summscreenfd?top_n=500"],"document_share":0.0002723311546840959,"domain":"long_context","name":"SummScreenFD","organization_count":1,"organizations":["llm_stats"],"rank":3192,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/summscreenfd?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-sunrgbd","caveat":"SUNRGBD evaluates RGB-D scene understanding and 3D grounding capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/sunrgbd?top_n=500"],"document_share":0.0002723311546840959,"domain":"spatial_reasoning","name":"SUNRGBD","organization_count":1,"organizations":["llm_stats"],"rank":3193,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/sunrgbd?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-superchem","caveat":"SuperChem is a benchmark of advanced chemistry problems requiring expert-level domain knowledge and reasoning.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/superchem?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SuperChem","organization_count":1,"organizations":["llm_stats"],"rank":3194,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/superchem?top_n=500"},{"aliases":[],"benchmark_id":"superchem","caveat":"Multimodal chemical reasoning; image and formula parsing quality moves the reported figure.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"science","name":"SUPERChem","organization_count":1,"organizations":["Tencent"],"rank":3195,"released":null,"source":"model_reports","url":"https://github.com/catalystforyou/SUPERChem_eval"},{"aliases":[],"benchmark_id":"llm-stats-superglue","caveat":"SuperGLUE is a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard. It includes 8 primary tasks: BoolQ (Boolean Questions), CB (CommitmentBank), COPA (Choice of Plausible Alternatives), MultiRC (Multi-Sentence Reading Comprehension), ReCoRD (Reading Comprehension with Commonsense Reasoning), RTE (Recognizing Textual Entailment), WiC (Word-in-Context), and WSC (Winograd Schema Challenge). The benchmark evaluates diverse language understanding capabilities including reading comprehension, commonsense reasoning, causal reasoning, coreference resolution, textual entailment, and word sense disambiguation across multiple domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/superglue?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SuperGLUE","organization_count":1,"organizations":["llm_stats"],"rank":3196,"released":"2019-05-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/superglue?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-supergpqa","caveat":"SuperGPQA is a comprehensive benchmark that evaluates large language models across 285 graduate-level academic disciplines. The benchmark contains 25,957 questions covering 13 broad disciplinary areas including Engineering, Medicine, Science, and Law, with specialized fields in light industry, agriculture, and service-oriented domains. It employs a Human-LLM collaborative filtering mechanism with over 80 expert annotators to create challenging questions that assess graduate-level knowledge and reasoning capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/supergpqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"SuperGPQA","organization_count":1,"organizations":["llm_stats"],"rank":3197,"released":"2025-02-20","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/supergpqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1538-supergpqa","caveat":"SuperGPQA, a comprehensive benchmark designed to evaluate the knowledge and reasoning abilities of Large Language Models (LLMs) across 285 graduate-level disciplines. SuperGPQA features at least 50 questions per discipline, covering a broad spectrum of graduate-level topics. SuperGPQA，这是一个旨在评估大型语言模型在 285 个研究生学科领域的知识和推理能力的全面基准。SuperGPQA 每个学科至少包含 50 个问题，涵盖广泛的硕士研究生学科主题。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SuperGPQA"],"document_share":0.0002723311546840959,"domain":"学科","name":"SuperGPQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3198,"released":"2025-02-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SuperGPQA"},{"aliases":[],"benchmark_id":"claire-radar-2608-07862","caveat":"SurakshaEval evaluates the safety of LLMs across ten major Indian languages and English, using human-written prompts covering generic and region-specific scenarios, with a structured scoring protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07862"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SurakshaEval","organization_count":1,"organizations":["claire_radar"],"rank":3199,"released":"2026-08-08","source":"claire_radar","url":"https://arxiv.org/abs/2608.07862"},{"aliases":[],"benchmark_id":"llm-stats-surds","caveat":"SURDS is a benchmark for spatial understanding and reasoning in autonomous-driving scenes.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/surds?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"SURDS","organization_count":1,"organizations":["llm_stats"],"rank":3200,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/surds?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-27139","caveat":"SurgHiBench is a hierarchy-aware evaluation suite for surgical video understanding with three tasks measuring recognition, consistency, and severity across granularity levels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27139"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SurgHiBench","organization_count":1,"organizations":["claire_radar"],"rank":3201,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.27139"},{"aliases":[],"benchmark_id":"claire-radar-2606-29247","caveat":"Evaluates vision-language-action models in laparoscopic surgical robotics across 8 tasks (atomic, conditional, composite) on the SurRoL simulator, using action accuracy and semantic consistency metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29247"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"SurgVLA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3202,"released":"2026-06-28","source":"claire_radar","url":"https://arxiv.org/abs/2606.29247"},{"aliases":[],"benchmark_id":"claire-radar-2608-08070","caveat":"Evaluates surgical world models on short-horizon instrument motion prediction and dynamics stability from intraoperative image sequences, focusing on geometric accuracy and temporal coherence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08070"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SurgWMBench","organization_count":1,"organizations":["claire_radar"],"rank":3203,"released":"2026-08-08","source":"claire_radar","url":"https://arxiv.org/abs/2608.08070"},{"aliases":["SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators"],"benchmark_id":"claire-radar-2608-07641","caveat":"SurveyReview provides a dataset of 675 survey papers with 1,630 review reports annotated into four-dimensional scores, along with standardized splits and an evaluation protocol for measuring alignment between automatic evaluators and human reviewers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07641"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SurveyReview","organization_count":1,"organizations":["claire_radar"],"rank":3204,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07641"},{"aliases":[],"benchmark_id":"opencompass-1113-svamp","caveat":"SVAMP includes one-unknown arithmetic word problems with grade level up to 4 by applying simple variations over word problems in an existing dataset. SVAMP further highlights the brittle nature of existing models when trained on these benchmark datasets. SVAMP 是一个包含算术文字问题的数据集，最高适用于四年级的学生，是通过对现有数据集中的文字问题应用简单变体而生成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SVAMP"],"document_share":0.0002723311546840959,"domain":"数学","name":"SVAMP","organization_count":1,"organizations":["opencompass_hub"],"rank":3205,"released":"2021-04-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SVAMP"},{"aliases":[],"benchmark_id":"llm-stats-svg-bench","caveat":"SVG-Bench is an internal benchmark that comprehensively evaluates SVG generation performance. It accepts text and image inputs across build-from-scratch and edit-based tasks, using a VLM to verify rendering accuracy of the generated outputs.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/svg-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"SVG-Bench","organization_count":1,"organizations":["llm_stats"],"rank":3206,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/svg-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-01977","caveat":"SVGEval is a vision-grounded benchmark for human-aligned SVG quality assessment, with multi-round labeled annotations, evaluating models on semantic, aesthetic, geometry, and layout judgments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.01977"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SVGEval","organization_count":1,"organizations":["claire_radar"],"rank":3207,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.01977"},{"aliases":[],"benchmark_id":"claire-radar-2606-02642","caveat":"SVHalluc evaluates speech-vision hallucination in audio-visual large language models, focusing on semantic and temporal alignment between speech content and visual signals.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02642"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SVHalluc","organization_count":1,"organizations":["claire_radar"],"rank":3208,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.02642"},{"aliases":[],"benchmark_id":"claire-radar-2605-31529","caveat":"SVI-Bench evaluates vision-language models on strategic video intelligence using sports as a microworld. It includes 9 tasks across 4 pillars: perception, reasoning, simulation, and agency, using basketball, soccer, and hockey videos with annotated actions and reports.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31529"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SVI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3209,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31529"},{"aliases":["SWE Atlas","SWE-Atlas"],"benchmark_id":"swe_atlas","caveat":"Reports three splits (Codebase Q&A, Test Writing, Refactoring) as separate figures; the split is part of the instrument.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"coding_agent","name":"SWE Atlas","organization_count":1,"organizations":["Tencent"],"rank":3210,"released":"2026-05-08","source":"model_reports","url":"https://github.com/scaleapi/SWE-Atlas"},{"aliases":[],"benchmark_id":"llm-stats-swe-atlas-codebase-qna","caveat":"SWE Atlas - Codebase QnA evaluates a model's ability to answer questions about real codebases, measuring repository-level comprehension and the ability to reason about code structure, behavior, and intent across an entire project.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-codebase-qna?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"SWE Atlas - Codebase QnA","organization_count":1,"organizations":["llm_stats"],"rank":3211,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-codebase-qna?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-swe-atlas-test-writing","caveat":"SWE Atlas - Test Writing evaluates a model's ability to author meaningful tests for real-world software projects, measuring how well agents can understand code and produce correct, useful test coverage.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-test-writing?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"SWE Atlas - Test Writing","organization_count":1,"organizations":["llm_stats"],"rank":3212,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas-test-writing?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-23564","caveat":"SWE Refactor Bench is a benchmark for evaluating coding agents on whole-repository stack migrations. It comprises 20 migrations covering 4 types of technical debt, with a three-stage evaluation protocol measuring migration completeness and behavioral correctness: Migration Audit, Behavioral Tests, and Agentic Verification.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23564v1"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"SWE Refactor Bench","organization_count":1,"organizations":["claire_radar"],"rank":3213,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23564v1"},{"aliases":["SWE-Refactor-Bench"],"benchmark_id":"swe_refactor_bench","caveat":"Composite score is not full-acceptance rate. One attempt per task, per-task budgets of 6–30 hours, different native clients and reasoning efforts, model-based audit and a finite six-agent verifier panel constrain comparisons.","document_count":1,"document_ids":["model_reports:einsia_swe_refactor_bench_v1_table3"],"document_share":0.0002723311546840959,"domain":"coding","name":"SWE Refactor Bench","organization_count":1,"organizations":["Navers Lab / Einsia"],"rank":3214,"released":"2026-08-24","source":"model_reports","url":"https://lab.einsia.ai/swe-refactor-bench/"},{"aliases":[],"benchmark_id":"llm-stats-swe-atlas","caveat":"SWE-Atlas is a software engineering benchmark focused on debugging, evaluating a model's ability to localize and fix bugs in real-world codebases.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"SWE-Atlas","organization_count":1,"organizations":["llm_stats"],"rank":3215,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-atlas?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-swe-bench-multilingual","caveat":"A multilingual benchmark for issue resolving in software engineering that covers Java, TypeScript, JavaScript, Go, Rust, C, and C++. Contains 1,632 high-quality instances carefully annotated from 2,456 candidates by 68 expert annotators, designed to evaluate Large Language Models across diverse software ecosystems beyond Python.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multilingual?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SWE-bench Multilingual","organization_count":1,"organizations":["llm_stats"],"rank":3216,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multilingual?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-swe-bench-multimodal","caveat":"SWE-Bench Multimodal extends SWE-Bench to evaluate language models on software engineering tasks that involve visual inputs such as screenshots, UI mockups, and diagrams alongside code understanding.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multimodal?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"SWE-Bench Multimodal","organization_count":1,"organizations":["llm_stats"],"rank":3217,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-multimodal?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-swe-bench-pro","caveat":"SWE-Bench Pro is an advanced version of SWE-Bench that evaluates language models on complex, real-world software engineering tasks requiring extended reasoning and multi-step problem solving.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-pro?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SWE-Bench Pro","organization_count":1,"organizations":["llm_stats"],"rank":3218,"released":"2025-09-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-pro?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-08149","caveat":"A verified version of SWE-Bench Pro for evaluating software engineering agents on repository-level coding tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08149"],"document_share":0.0002723311546840959,"domain":"Code","name":"SWE-Bench Pro Verified","organization_count":1,"organizations":["claire_radar"],"rank":3219,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08149"},{"aliases":[],"benchmark_id":"claire-radar-2608-09802","caveat":"A multilingual code refactoring benchmark with 170 instances drawn from real commits across seven programming languages. Evaluates AI agents on large-scale refactoring tasks averaging 11.4 modified files and 261.6 lines of code, using manually curated issue descriptions and test suites.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09802"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"SWE-Bench ProMax","organization_count":1,"organizations":["claire_radar"],"rank":3220,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09802"},{"aliases":[],"benchmark_id":"claire-radar-2608-19799","caveat":"A repository-level benchmark with 119 tasks from 98 GitHub repositories across 20 scientific domains, organized into issue-driven, expert-exploratory, and engineering-integration paradigms.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.19799"],"document_share":0.0002723311546840959,"domain":"Code","name":"SWE-bench Science","organization_count":1,"organizations":["claire_radar"],"rank":3221,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.19799"},{"aliases":["SWE-bench Science","SWE-Bench-Science","SWE Bench Science"],"benchmark_id":"swe_bench_science","caveat":"Repository-level scientific software engineering benchmark from the OpenMOSS team (Fudan): 119 tasks across 98 GitHub repositories in 20 scientific domains, organized into Issue-driven, Expert-exploratory, and Engineering-integration paradigms. Best reported pass@1 is below 50% (Claude Code with Opus-5 max), and the paper identifies four recurring failure mechanisms, so scores are highly sensitive to scientific-domain knowledge and tool access, not only to coding ability.","document_count":1,"document_ids":["model_reports:swe_bench_science_results_table"],"document_share":0.0002723311546840959,"domain":"coding_agent","name":"SWE-bench Science","organization_count":1,"organizations":["OpenMOSS (Fudan University)"],"rank":3222,"released":"2026-08-20","source":"model_reports","url":"https://arxiv.org/abs/2608.19799"},{"aliases":[],"benchmark_id":"llm-stats-swe-bench-verified","caveat":"A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SWE-Bench Verified","organization_count":1,"organizations":["llm_stats"],"rank":3223,"released":"2024-08-13","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-swe-bench-verified-agentic-coding","caveat":"SWE-bench Verified is a human-filtered subset of 500 software engineering problems drawn from real GitHub issues across 12 popular Python repositories. Given a codebase and an issue description, language models are tasked with generating patches that resolve the described problems. This benchmark evaluates AI's real-world agentic coding skills by requiring models to navigate complex codebases, understand software engineering problems, and coordinate changes across multiple functions, classes, and files to fix well-defined issues with clear descriptions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentic-coding%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SWE-bench Verified (Agentic Coding)","organization_count":1,"organizations":["llm_stats"],"rank":3224,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentic-coding%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-swe-bench-verified-agentless","caveat":"A human-validated subset of SWE-bench that evaluates language models' ability to resolve real-world GitHub issues using an agentless approach. The benchmark tests models on software engineering problems requiring understanding and coordinating changes across multiple functions, classes, and files simultaneously.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentless%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SWE-bench Verified (Agentless)","organization_count":1,"organizations":["llm_stats"],"rank":3225,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28agentless%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-swe-bench-verified-multiple-attempts","caveat":"SWE-bench Verified is a human-validated subset of 500 test samples from the original SWE-bench dataset that evaluates AI systems' ability to automatically resolve real GitHub issues in Python repositories. Given a codebase and issue description, models must edit the code to successfully resolve the problem, requiring understanding and coordination of changes across multiple functions, classes, and files. The Verified version provides more reliable evaluation through manual validation of test samples.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28multiple-attempts%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SWE-bench Verified (Multiple Attempts)","organization_count":1,"organizations":["llm_stats"],"rank":3226,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-bench-verified-%28multiple-attempts%29?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1984-swe-bench-live","caveat":"SWE-bench-Live is a live-updatable benchmark designed for evaluating large language models (LLMs) and agents on real-world software issue resolution tasks. SWE-bench-Live 是一个面向大语言模型（LLMs）和智能体的实时可更新评测基准，专注于真实世界软件开发中的问题修复任务。 该基准从 2024 年以来的 GitHub 活跃仓库中自动收集了 1,319 个问题修复任务，涵盖 93 个项目，并为每个任务提供可复现的 Docker 执行环境。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SWE-bench-Live"],"document_share":0.0002723311546840959,"domain":"代码","name":"SWE-bench-Live","organization_count":1,"organizations":["opencompass_hub"],"rank":3227,"released":"2025-06-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SWE-bench-Live"},{"aliases":[],"benchmark_id":"claire-radar-2610-13626","caveat":"ArXiv paper introducing a benchmark for deceptive behaviors of LLMs across the software development lifecycle, with abstract stating tasks and results, but no public artifact available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13626"],"document_share":0.0002723311546840959,"domain":"Code","name":"SWE-DeceptionBench","organization_count":1,"organizations":["claire_radar"],"rank":3228,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13626"},{"aliases":[],"benchmark_id":"claire-radar-2606-07297","caveat":"SWE-Explore evaluates repository exploration by coding agents: given a repository and an issue, an explorer returns a ranked list of relevant code regions under a fixed line budget. Ground truth is line-level, derived from successful repair trajectories. Coverage, ranking, and context-efficiency metrics are scored.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07297"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"SWE-Explore","organization_count":1,"organizations":["claire_radar"],"rank":3229,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07297"},{"aliases":[],"benchmark_id":"opencompass-1989-swe-factory","caveat":"SWE-Factory is a benchmark for evaluating large language models on software issue fixing tasks. SWE-Factory 是一个面向大型语言模型的软件问题修复评测基准，旨在提升构建效率与评估准确性。该基准集成多智能体系统 SWE-Builder 自动搭建任务环境，采用退出码自动评分，并通过 fail2pass 流程验证修复有效性，确保评测可靠。SWE-Factory 覆盖四种语言共 671 个问题，支持高效、自动化、可扩展的 LLM 评估流程。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SWE-Factory"],"document_share":0.0002723311546840959,"domain":"代码","name":"SWE-Factory","organization_count":1,"organizations":["opencompass_hub"],"rank":3230,"released":"2025-06-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SWE-Factory"},{"aliases":[],"benchmark_id":"llm-stats-swe-fficiency","caveat":"SWE-fficiency is an open-source benchmark and workflow that evaluates language models on optimizing the runtime efficiency of real-world software engineering tasks, measuring how well agents can improve code performance autonomously.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-fficiency?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"SWE-fficiency","organization_count":1,"organizations":["llm_stats"],"rank":3231,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-fficiency?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-33678","caveat":"We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.33678"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SWE-Game","organization_count":1,"organizations":["claire_radar"],"rank":3232,"released":"2026-09-27","source":"claire_radar","url":"https://arxiv.org/abs/2609.33678"},{"aliases":[],"benchmark_id":"claire-radar-2609-04167","caveat":"Evaluates repository-level repair agents on both functional correctness and adherence to review-derived constraints, using 303 instances with separate functional and constraint tests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.04167"],"document_share":0.0002723311546840959,"domain":"Code","name":"SWE-Gate","organization_count":1,"organizations":["claire_radar"],"rank":3233,"released":"2026-09-03","source":"claire_radar","url":"https://arxiv.org/abs/2609.04167"},{"aliases":["SWE-InfraBench: Evaluating Language Models on Cloud Infrastructure Code"],"benchmark_id":"claire-radar-2606-05249","caveat":"A benchmark of incremental infrastructure-as-code modifications in real-world AWS CDK repositories, evaluated through provided test cases.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05249"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"SWE-InfraBench","organization_count":1,"organizations":["claire_radar"],"rank":3234,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05249"},{"aliases":[],"benchmark_id":"claire-radar-2606-30573","caveat":"SWE-Interact evaluates coding agents on multi-turn, interactive software engineering tasks where a simulated user provides vague instructions, reveals requirements progressively, and gives feedback. The benchmark comprises 75 tasks and measures agents' ability to discover user intent, adapt to evolving requirements, and build on prior work.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.30573"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"SWE-INTERACT","organization_count":1,"organizations":["claire_radar"],"rank":3235,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.30573"},{"aliases":[],"benchmark_id":"claire-radar-2610-11559","caveat":"Coding assistants such as Claude Code and Codex have become a major application of LLM agents, yet existing benchmarks remain far from real-world use, particularly in task horizon and interaction length.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11559"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SWE-Journey","organization_count":1,"organizations":["claire_radar"],"rank":3236,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11559"},{"aliases":[],"benchmark_id":"llm-stats-swe-lancer","caveat":"A benchmark for evaluating large language models on real-world freelance software engineering tasks from Upwork. Contains over 1,400 tasks valued at $1 million USD total, ranging from $50 bug fixes to $32,000 feature implementations. Includes both independent engineering tasks graded via end-to-end tests and managerial tasks assessed against original engineering managers' choices.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SWE-Lancer","organization_count":1,"organizations":["llm_stats"],"rank":3237,"released":"2025-02-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer?top_n=500"},{"aliases":["SWE-Lancer"],"benchmark_id":"swe_lancer","caveat":"Software engineering freelancer tasks from Upwork; real-world constraints and client communication are part of the task.","document_count":1,"document_ids":["model_reports:bytedance_seed2_0_model_card"],"document_share":0.0002723311546840959,"domain":"coding_agent","name":"SWE-Lancer","organization_count":1,"organizations":["ByteDance"],"rank":3238,"released":"2025-02-17","source":"model_reports","url":"https://github.com/openai/frontier-evals/tree/main/project/swelancer"},{"aliases":[],"benchmark_id":"llm-stats-swe-lancer-ic-diamond-subset","caveat":"SWE-Lancer (IC-Diamond subset) is a benchmark of real-world freelance software engineering tasks from Upwork, ranging from $50 bug fixes to $32,000 feature implementations. It evaluates AI models on independent engineering tasks using end-to-end tests triple-verified by experienced software engineers, and includes managerial tasks where models choose between technical implementation proposals.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer-%28ic-diamond-subset%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"SWE-Lancer (IC-Diamond subset)","organization_count":1,"organizations":["llm_stats"],"rank":3239,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-lancer-%28ic-diamond-subset%29?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-07682","caveat":"SWE-Marathon evaluates AI agents on 20 ultra-long-horizon software engineering tasks, each with a unique executable environment, a human-written reference solution, and a multi-layer verification suite. Tasks average 27.2M tokens per logged agent attempt, requiring sustained progress over hours and millions of tokens.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07682"],"document_share":0.0002723311546840959,"domain":"Code","name":"SWE-Marathon","organization_count":1,"organizations":["claire_radar"],"rank":3240,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07682"},{"aliases":[],"benchmark_id":"llm-stats-swe-marathon","caveat":"SWE-Marathon is an ultra-long-horizon software engineering benchmark covering tasks such as building compilers, optimizing kernels, and developing production-grade services. It measures whether agents can sustain quality across extremely long engineering trajectories.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-marathon?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"SWE-Marathon","organization_count":1,"organizations":["llm_stats"],"rank":3241,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-marathon?top_n=500"},{"aliases":["SWE-Marathon","SWE Marathon"],"benchmark_id":"swe_marathon","caveat":"Ultra long-horizon software engineering tasks; the score depends on the agent's loop budget as much as on coding skill.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"coding_agent","name":"SWE-Marathon","organization_count":1,"organizations":["Tencent"],"rank":3242,"released":null,"source":"model_reports","url":"https://github.com/abundant-ai/swe-marathon"},{"aliases":[],"benchmark_id":"llm-stats-swe-mm","caveat":"SWE-MM evaluates software-engineering agents on repository tasks that require understanding both source code and visual evidence.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-mm?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"SWE-MM","organization_count":1,"organizations":["llm_stats"],"rank":3243,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-mm?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-22175","caveat":"SWE-Mutation evaluates LLM-generated test suites in software engineering by using 2,636 mutated variants derived from 800 original instances across nine programming languages, measuring verification and detection rates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22175"],"document_share":0.0002723311546840959,"domain":"Code","name":"SWE-Mutation","organization_count":1,"organizations":["claire_radar"],"rank":3244,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22175"},{"aliases":[],"benchmark_id":"claire-radar-2607-27409","caveat":"A benchmark of 188 tasks for evaluating coding agents on non-functional improvements in Python projects, with 92 executable rules combining functional correctness and rule-based evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27409"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"SWE-NFI","organization_count":1,"organizations":["claire_radar"],"rank":3245,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27409"},{"aliases":[],"benchmark_id":"llm-stats-swe-perf","caveat":"Software Engineering Performance benchmark measuring code optimization capabilities","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-perf?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"SWE-Perf","organization_count":1,"organizations":["llm_stats"],"rank":3246,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-perf?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-29754","caveat":"SWE-PolyVision is described as an executable benchmark of 92 real tasks with public and private holdouts, but no access to tasks, scorer, or submission mechanism is provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.29754"],"document_share":0.0002723311546840959,"domain":"Code","name":"SWE-PolyVision","organization_count":1,"organizations":["claire_radar"],"rank":3247,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.29754"},{"aliases":[],"benchmark_id":"claire-radar-2609-29465","caveat":"SWE-Prometheus is presented as a benchmark for improving repository engineering governance, but no dataset, scoring rules, or evaluation interface are available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.29465"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"SWE-Prometheus","organization_count":1,"organizations":["claire_radar"],"rank":3248,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.29465"},{"aliases":[],"benchmark_id":"llm-stats-swe-review","caveat":"Software Engineering Review benchmark evaluating code review capabilities","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swe-review?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"SWE-Review","organization_count":1,"organizations":["llm_stats"],"rank":3249,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swe-review?top_n=500"},{"aliases":["A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents"],"benchmark_id":"claire-radar-2608-09072","caveat":"A repository-level dataset and protocol evaluating issue resolution through requirement clarification, implementation planning, code generation, and artifact submission, with 163 tasks across 31 Python and Java repositories and validated intermediate ground-truth references.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09072"],"document_share":0.0002723311546840959,"domain":"Code","name":"SWE-RPG","organization_count":1,"organizations":["claire_radar"],"rank":3250,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09072"},{"aliases":[],"benchmark_id":"claire-radar-2609-26777","caveat":"Pending verification of task definition and scoring contract for production inference serving agent evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.26777"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"SWE-Serve","organization_count":1,"organizations":["claire_radar"],"rank":3251,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.26777"},{"aliases":[],"benchmark_id":"claire-radar-2606-29957","caveat":"SWE-Together evaluates coding agents in multi-turn interactive user sessions reconstructed from real user-agent interactions. It comprises 109 repository-level tasks with a reactive LLM-based user simulator, measuring final repository correctness and the number of corrective feedback turns.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29957"],"document_share":0.0002723311546840959,"domain":"Agents","name":"SWE-Together","organization_count":1,"organizations":["claire_radar"],"rank":3252,"released":"2026-06-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.29957"},{"aliases":[],"benchmark_id":"opencompass-1608-swiltra-bench","caveat":"SwiLTra-Bench is a comprehensive multilingual benchmark of over 180K aligned Swiss legal translation pairs comprising laws, headnotes, and press releases across all Swiss languages along with English, designed to evaluate LLM-based translation systems. SwiLTra-Bench是一个包含超过 18 万对对齐的瑞士法律翻译语料库的全面多语言基准，包括所有瑞士语言以及英语的法律、摘要和新闻稿，旨在评估基于LLM的翻译系统。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/SwiLTra-Bench"],"document_share":0.0002723311546840959,"domain":"语言","name":"SwiLTra-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":3253,"released":"2025-03-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/SwiLTra-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2608-09497","caveat":"Evaluates crop mapping models on national-scale multi-year Sentinel-2 data with fine-grained crop taxonomy and non-crop classes under leave-one-year-out protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09497"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SwissCrop25","organization_count":1,"organizations":["claire_radar"],"rank":3254,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09497"},{"aliases":["SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection"],"benchmark_id":"claire-radar-2609-09349","caveat":"A benchmark of syntactically correct but factually incorrect statements across eight languages for evaluating multilingual factual error rejection.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09349"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"SWORD","organization_count":1,"organizations":["claire_radar"],"rank":3255,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.09349"},{"aliases":[],"benchmark_id":"llm-stats-swt-bench","caveat":"Software Test Benchmark evaluating LLM ability to write tests for software repositories","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/swt-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"SWT-Bench","organization_count":1,"organizations":["llm_stats"],"rank":3256,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/swt-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-15216","caveat":"SymbalBench evaluates automated detection of systematic misalignments in MLLM-generated image captions. It comprises 420 vision-language datasets (1.7 million image-text pairs) from natural and medical domains, each annotated with known systematic misalignments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.15216"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"SymbalBench","organization_count":1,"organizations":["claire_radar"],"rank":3257,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.15216"},{"aliases":[],"benchmark_id":"claire-radar-2609-35113","caveat":"Symbolic regression (SR) seeks concise and interpretable mathematical expressions from data for scientific equation discovery.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35113"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SymbolicArena","organization_count":1,"organizations":["claire_radar"],"rank":3258,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35113"},{"aliases":[],"benchmark_id":"claire-radar-2606-03348","caveat":"SynCred-Bench evaluates detection of AI-generated visual misinformation with synthetic credibility, using 600 AI-generated images and FP450 real-image negatives.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03348"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SynCred-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3259,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03348"},{"aliases":[],"benchmark_id":"claire-radar-2610-03329","caveat":"Large language models are increasingly used where small syntactic errors matter, yet character-level reasoning is still evaluated mostly through isolated probes and aggregate accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.03329"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SyntaxBench","organization_count":1,"organizations":["claire_radar"],"rank":3260,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.03329"},{"aliases":[],"benchmark_id":"claire-radar-2607-07469","caveat":"Presents a synthetic labeling pipeline for e-commerce attribute extraction with human validation via multi-LLM arena voting.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.07469"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SynthAVE","organization_count":1,"organizations":["claire_radar"],"rank":3261,"released":"2026-07-08","source":"claire_radar","url":"https://arxiv.org/abs/2607.07469"},{"aliases":[],"benchmark_id":"claire-radar-2607-10400","caveat":"SynthDocBench evaluates vision-language models on 1,788 questions over 200 synthetic long-context documents (avg. 51.1 pages) with 3,340 charts. It varies document length, layout archetype, modality composition, and question type as independent controlled factors, across chart, cross-modal, and complex subsets with deterministic ground truth.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10400"],"document_share":0.0002723311546840959,"domain":"General AI","name":"SynthDocBench","organization_count":1,"organizations":["claire_radar"],"rank":3262,"released":"2026-07-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.10400"},{"aliases":["synthetic-data-benchmark"],"benchmark_id":"claire-radar-github-mervecaliskann-synthetic-data-benchmark","caveat":"The project evaluates synthetic tabular data generators on fidelity, utility, and privacy metrics using the Give Me Some Credit dataset, comparing Gaussian Copula, CTGAN, and an LLM generator over a fixed evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://github.com/Mervecaliskann/synthetic-data-benchmark"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Synthetic Data Generator Benchmark (Credit Risk)","organization_count":1,"organizations":["claire_radar"],"rank":3263,"released":"2026-09-07","source":"claire_radar","url":"https://github.com/Mervecaliskann/synthetic-data-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-30027","caveat":"Synthetic Hospital is proposed as an open, synthetic, physician-validated longitudinal EHR benchmark, but no released task inputs, scoring code, or public data are provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30027"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Synthetic Hospital","organization_count":1,"organizations":["claire_radar"],"rank":3264,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.30027"},{"aliases":["system-one-bench"],"benchmark_id":"claire-radar-github-rssr25-system-one-bench","caveat":"sys1bench evaluates typed System One decision models on calibration, framing sensitivity, selective prediction, ordinal fidelity, interference, and robustness through a pip-installable benchmark suite.","document_count":1,"document_ids":["claire_radar:https://github.com/rssr25/system-one-bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"sys1bench","organization_count":1,"organizations":["claire_radar"],"rank":3265,"released":"2026-09-22","source":"claire_radar","url":"https://github.com/rssr25/system-one-bench"},{"aliases":[],"benchmark_id":"opencompass-540-t-eval","caveat":"T-Eval evaluates the tool utilization capabilities of LLMs and decomposing them into instruction following, planning, reasoning, retrieval, understanding, and review. T-Eval 评估了 LLM 的工具使用能力，并将其分解为指令遵循、规划、推理、检索、理解和审查等子能力","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/T-Eval"],"document_share":0.0002723311546840959,"domain":"智能体","name":"T-Eval","organization_count":1,"organizations":["opencompass_hub"],"rank":3266,"released":"2024-01-15","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/T-Eval"},{"aliases":[],"benchmark_id":"claire-radar-2606-22339","caveat":"T-IMPACT evaluates models on severity-aware detection of manipulated news-style image-text pairs, with 98,786 examples covering pristine, image-only, text-only, and joint manipulations, alongside calibrated continuous severity scores, coarse labels, and grounding metadata.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.22339"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"T-IMPACT","organization_count":1,"organizations":["claire_radar"],"rank":3267,"released":"2026-06-21","source":"claire_radar","url":"https://arxiv.org/abs/2606.22339"},{"aliases":[],"benchmark_id":"claire-radar-2606-11070","caveat":"T1-Bench evaluates agentic systems through realistic customer-facing, multi-domain tasks. Scenarios involve multi-turn user-assistant interactions across 25 domains with varying difficulty, measuring structured reasoning, tool utilization, and conversational quality. Automatic evaluation is complemented by human judgments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11070"],"document_share":0.0002723311546840959,"domain":"General AI","name":"T1-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3268,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11070"},{"aliases":[],"benchmark_id":"llm-stats-t2-bench","caveat":"t2-bench is a benchmark for evaluating agentic tool use capabilities, measuring how well models can select, sequence, and utilize tools to solve complex tasks. It tests autonomous planning and execution in multi-step scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/t2-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"t2-bench","organization_count":1,"organizations":["llm_stats"],"rank":3269,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/t2-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-24145","caveat":"T2D-Bench evaluates LLM outputs for type 2 diabetes against evidence constraints using a multi-layer clinical-lifestyle knowledge graph, covering diagnosis, medication safety, and lifestyle conflicts across 100 structured vignettes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24145"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"T2D-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3270,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24145"},{"aliases":[],"benchmark_id":"claire-radar-2605-29054","caveat":"T2J-Bench benchmarks codebase conversion by transferring PyTorch code to JAX under a fixed equivalence contract with three ordered verification stages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29054"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"T2J-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3271,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.29054"},{"aliases":[],"benchmark_id":"claire-radar-2609-02255","caveat":"Evaluates localized semantic control in text-to-image generation using 50 seed subjects and 1,200 prompt cases per model, scoring 7,160 images across six models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02255"],"document_share":0.0002723311546840959,"domain":"General AI","name":"T2LSC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3272,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02255"},{"aliases":[],"benchmark_id":"claire-radar-2609-07441","caveat":"A living benchmark of 43 high-dimensional biomedical tables with a standardized cross-validation protocol and Elo-based ranking of ML models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07441"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"TabBench-Bio","organization_count":1,"organizations":["claire_radar"],"rank":3273,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07441"},{"aliases":["Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language"],"benchmark_id":"claire-radar-2605-27886","caveat":"Tabero evaluates language-conditioned gentle manipulation using vision, touch, task success, and physical-interaction quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27886"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"Tabero","organization_count":1,"organizations":["claire_radar"],"rank":3274,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.27886"},{"aliases":[],"benchmark_id":"opencompass-1132-tabfact","caveat":"TabFac consists of 117,854 manually annotated statements with regard to 16,573 Wikipedia tables, their relations are classified as ENTAILED and REFUTED. TabFac 包含 117,854 条手动标注的语句，涉及 16,573 个维基百科表格，是第一个评估结构化数据上语言推理的数据集，涉及在符号和语言两个方面的混合推理能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TabFact"],"document_share":0.0002723311546840959,"domain":"智能体","name":"TabFact","organization_count":1,"organizations":["opencompass_hub"],"rank":3275,"released":"2020-06-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TabFact"},{"aliases":[],"benchmark_id":"opencompass-2025-tableeval","caveat":"TableEval is the first cross-lingual benchmark for tabular question answering, supporting Simplified Chinese, Traditional Chinese, and English. It is designed to evaluate model performance on real-world, complex table understanding tasks across multiple languages. TableEval 是首个支持简体中文、繁体中文和英文的跨语言表格问答基准数据集，旨在系统评估大模型在真实复杂表格理解任务中的表现。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TableEval"],"document_share":0.0002723311546840959,"domain":"推理","name":"TableEval","organization_count":1,"organizations":["opencompass_hub"],"rank":3276,"released":"2025-06-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TableEval"},{"aliases":[],"benchmark_id":"claire-radar-2607-03926","caveat":"TabQueryBench evaluates synthetic tabular data generators using SQL-shaped analytical queries as structural assessors, providing 44 reusable query templates across 49 datasets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03926"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TabQueryBench","organization_count":1,"organizations":["claire_radar"],"rank":3277,"released":"2026-07-04","source":"claire_radar","url":"https://arxiv.org/abs/2607.03926"},{"aliases":[],"benchmark_id":"claire-radar-2606-09578","caveat":"TABVERSE is a controlled multimodal benchmark that aligns the same table content across HTML, Markdown, LaTeX, and rendered images, with question category and difficulty tags. It evaluates LLMs and VLMs on question answering, structural understanding, and structure reconstruction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09578"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TABVERSE","organization_count":1,"organizations":["claire_radar"],"rank":3278,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09578"},{"aliases":[],"benchmark_id":"claire-radar-2607-09190","caveat":"TactiDex is a real-world tactile-guided benchmark for dexterous manipulation, aligning whole-hand tactile signals with kinematic and object states. It provides standardized evaluation metrics and a framework for tactile-driven transfer, including experiments on single and bimanual tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09190"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"TactiDex","organization_count":1,"organizations":["claire_radar"],"rank":3279,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.09190"},{"aliases":[],"benchmark_id":"claire-radar-2606-25877","caveat":"TacVerse is a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception, containing 106,800 tactile images from seven vision-based tactile sensors. It supports shape classification, grating classification, and force regression tasks, with evaluation under within-sensor, zero-shot cross-sensor, and few-shot adaptation settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25877"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TacVerse","organization_count":1,"organizations":["claire_radar"],"rank":3280,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25877"},{"aliases":[],"benchmark_id":"claire-radar-2606-02624","caveat":"TadA-Bench is a fixed-data replay benchmark derived from 31 wet-lab rounds of TadA directed evolution. Models rank ~1M protein, DNA, or RNA sequence variants appearing only in later rounds, with scores as Spearman, Recall@10%, and nDCG@10%.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02624"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"TadA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3281,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.02624"},{"aliases":[],"benchmark_id":"claire-radar-2608-10258","caveat":"TAF-MED is a physician-reviewed benchmark of 500 fixed three-turn medical safety scenarios, evaluating LLM responses for unsafe guidance across multi-turn dialogues.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10258"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"TAF-MED","organization_count":1,"organizations":["claire_radar"],"rank":3282,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.10258"},{"aliases":[],"benchmark_id":"claire-radar-2606-24256","caveat":"Tailor-Bench evaluates visual world models on simulating irregular physical interactions with three scenario modes (regular, unconventional, impossible) and predictive/descriptive generation settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.24256"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Tailor-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3283,"released":"2026-06-23","source":"claire_radar","url":"https://arxiv.org/abs/2606.24256"},{"aliases":[],"benchmark_id":"claire-radar-2606-23643","caveat":"TailorBench evaluates personalized multimodal content generation using data from three platforms and five stated evaluation dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23643"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TailorBench","organization_count":1,"organizations":["claire_radar"],"rank":3284,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23643"},{"aliases":[],"benchmark_id":"claire-radar-2608-30068","caveat":"TAKE 85 evaluates directorial-intent understanding across 398 short films using question-answer pairs covering visual and audio intent, with synthesis code available for model predictions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30068"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TAKE 85","organization_count":1,"organizations":["claire_radar"],"rank":3285,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.30068"},{"aliases":[],"benchmark_id":"claire-radar-2609-38867","caveat":"Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38867"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Talk2Agent","organization_count":1,"organizations":["claire_radar"],"rank":3286,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.38867"},{"aliases":[],"benchmark_id":"claire-radar-2609-01810","caveat":"A unified Farsi dialogue benchmark with knowledge-grounded generation, dialogue acts, emotions, and sentiment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01810"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"TalkFa","organization_count":1,"organizations":["claire_radar"],"rank":3287,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01810"},{"aliases":[],"benchmark_id":"claire-radar-2608-11452","caveat":"TangPoetryBench evaluates text-to-image models on illustrating classical Chinese Tang poems across ten human-annotated dimensions, with a rubric-conditioned evaluator (PAE).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.11452"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TangPoetryBench","organization_count":1,"organizations":["claire_radar"],"rank":3288,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.11452"},{"aliases":[],"benchmark_id":"claire-radar-2610-10374","caveat":"A key challenge for multimodal large language models (MLLMs) is moving beyond visual recognition to constraint-aware cross-modal reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.10374"],"document_share":0.0002723311546840959,"domain":"Code","name":"TaoD2C-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3289,"released":"2026-10-07","source":"claire_radar","url":"https://arxiv.org/abs/2610.10374"},{"aliases":["TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views"],"benchmark_id":"claire-radar-2609-01899","caveat":"Evaluates long-term 3D point tracking across multiple synchronized views with camera motion using 284 sequences and over 100K point tracks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01899"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TAPVid-MV","organization_count":1,"organizations":["claire_radar"],"rank":3290,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01899"},{"aliases":[],"benchmark_id":"claire-radar-2608-10317","caveat":"Evaluates video-language models on ten traffic anomaly reasoning tasks using 960 human-curated test annotations over 80 held-out clips.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10317"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TAR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3291,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.10317"},{"aliases":[],"benchmark_id":"opencompass-1164-taskbench","caveat":"TaskBench aims to evaluate the capability of LLMs in task automation, containing 28,271 samples spanning 3 critical stages: task decomposition, tool invocation, and parameter prediction. TaskBench旨在评估LLM在任务自动化方面的能力，包含面向任务分解、工具调用和参数预测三个关键阶段的28271个样本。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TaskBench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"TaskBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3292,"released":"2023-11-30","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TaskBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-25804","caveat":"Evaluates LLM agents' judgment at decision forks in long-horizon engineering and research tasks using 502 trajectory-derived taste questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25804"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Taste-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3293,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.25804"},{"aliases":[],"benchmark_id":"claire-radar-2610-06824","caveat":"We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.06824"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TasteVal","organization_count":1,"organizations":["claire_radar"],"rank":3294,"released":"2026-10-05","source":"claire_radar","url":"https://arxiv.org/abs/2610.06824"},{"aliases":[],"benchmark_id":"claire-radar-2608-05699","caveat":"Evaluates anomaly instance tracking and fine-grained anomaly understanding jointly on 1,118 videos with track-centric annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05699"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TAU-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3295,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05699"},{"aliases":[],"benchmark_id":"llm-stats-tau-bench","caveat":"τ-bench: A benchmark for tool-agent-user interaction in real-world domains. Tests language agents' ability to interact with users and follow domain-specific rules through dynamic conversations using API tools and policy guidelines across retail and airline domains. Evaluates consistency and reliability of agent behavior over multiple trials.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Tau-bench","organization_count":1,"organizations":["llm_stats"],"rank":3296,"released":"2024-06-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau-bench-airline","caveat":"Part of τ-bench (TAU-bench), a benchmark for Tool-Agent-User interaction in real-world domains. The airline domain evaluates language agents' ability to interact with users through dynamic conversations while following domain-specific rules and using API tools. Agents must handle airline-related tasks and policies reliably.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-airline?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"TAU-bench Airline","organization_count":1,"organizations":["llm_stats"],"rank":3297,"released":"2024-06-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-airline?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau-bench-retail","caveat":"A benchmark for evaluating tool-agent-user interaction in retail environments. Tests language agents' ability to handle dynamic conversations with users while using domain-specific API tools and following policy guidelines. Evaluates agents on tasks like order cancellations, address changes, and order status checks through multi-turn conversations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-retail?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"TAU-bench Retail","organization_count":1,"organizations":["llm_stats"],"rank":3298,"released":"2024-06-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau-bench-retail?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau2-airline","caveat":"TAU2 airline domain benchmark for evaluating conversational agents in dual-control environments where both AI agents and users interact with tools in airline customer service scenarios. Tests agent coordination, communication, and ability to guide user actions in tasks like flight booking, modifications, cancellations, and refunds.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau2-airline?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Tau2 Airline","organization_count":1,"organizations":["llm_stats"],"rank":3299,"released":"2025-06-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-airline?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau2-retail","caveat":"τ²-bench retail domain evaluates conversational AI agents in customer service scenarios within a dual-control environment where both agent and user can interact with tools. Tests tool-agent-user interaction, rule adherence, and task consistency in retail customer support contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau2-retail?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Tau2 Retail","organization_count":1,"organizations":["llm_stats"],"rank":3300,"released":"2025-06-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-retail?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau2-telecom","caveat":"τ²-Bench telecom domain evaluates conversational agents in a dual-control environment modeled as a Dec-POMDP, where both agent and user use tools in shared telecommunications troubleshooting scenarios that test coordination and communication capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau2-telecom?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Tau2 Telecom","organization_count":1,"organizations":["llm_stats"],"rank":3301,"released":"2025-06-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau2-telecom?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau3-airline","caveat":"τ³-Bench airline domain evaluates agentic models on multi-turn, tool-using customer-support scenarios in a simulated airline booking and reservations environment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-airline?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Tau3 Airline","organization_count":1,"organizations":["llm_stats"],"rank":3302,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-airline?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau3-banking","caveat":"τ³-Bench banking domain evaluates agentic models on multi-turn, tool-using customer-support scenarios in a simulated retail banking environment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-banking?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Tau3 Banking","organization_count":1,"organizations":["llm_stats"],"rank":3303,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-banking?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau3-retail","caveat":"τ³-Bench retail domain evaluates agentic models on multi-turn, tool-using customer-support scenarios in a simulated online retail environment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-retail?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Tau3 Retail","organization_count":1,"organizations":["llm_stats"],"rank":3304,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-retail?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau3-telecom","caveat":"τ³-Bench telecom domain evaluates agentic models on multi-turn, tool-using customer-support and troubleshooting scenarios in a simulated telecommunications environment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-telecom?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Tau3 Telecom","organization_count":1,"organizations":["llm_stats"],"rank":3305,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-telecom?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-tau3-bench","caveat":"TAU3-Bench is a benchmark for evaluating general-purpose agent capabilities, testing models on multi-turn interactions with simulated user models, retrieval, and complex decision-making scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tau3-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"TAU3-Bench","organization_count":1,"organizations":["llm_stats"],"rank":3306,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tau3-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-24782","caveat":"A benchmark dataset for tropical cyclone research with an automated construction pipeline, used to probe scientific alignment of vision foundation models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24782"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3307,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24782"},{"aliases":[],"benchmark_id":"claire-radar-2607-01667","caveat":"TCA-Bench is a diagnostic benchmark for evaluating audiovisual binding and temporal relational reasoning in video captioning models, using a decoupled evaluation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01667"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TCA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3308,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.01667"},{"aliases":[],"benchmark_id":"claire-radar-2607-17742","caveat":"TCR-Bench is a diagnostic benchmark for table content-level answerability in RAG, focusing on sibling tables with similar schemas but content differences. It evaluates dense retrievers' ability to identify answerable tables.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.17742"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TCR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3309,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.17742"},{"aliases":[],"benchmark_id":"claire-radar-2608-09538","caveat":"Evaluates LLMs on research-level theoretical computer science proof generation, using theorem-proving tasks from papers at STOC, FOCS, and SODA. Provides context for self-contained proofs and uses a verification agent to check correctness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09538"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"TCS-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":3310,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09538"},{"aliases":[],"benchmark_id":"claire-radar-2609-35606","caveat":"Large language models perform strongly on competition mathematics, but their research-level reasoning remains difficult to evaluate systematically.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35606"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"TCSAlgBench","organization_count":1,"organizations":["claire_radar"],"rank":3311,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35606"},{"aliases":["Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents"],"benchmark_id":"claire-radar-2608-08852","caveat":"The Teaching Monster Challenge evaluates AI systems on generating instructional videos for a given topic and learner persona, with submissions screened by an LLM judge, ranked by crowd pairwise voting, and finalized by an expert panel.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08852"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"Teaching Monster Challenge","organization_count":1,"organizations":["claire_radar"],"rank":3312,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08852"},{"aliases":[],"benchmark_id":"claire-radar-2606-02060","caveat":"Evaluates span-level error localization in deep-research agent trajectories. TELBench comprises 1,000 instances with annotations of harmful error spans among normal exploration, failed searches, tentative hypotheses, and harmless noise, scored by span-level localization and first-error accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02060"],"document_share":0.0002723311546840959,"domain":"Agents","name":"TELBench","organization_count":1,"organizations":["claire_radar"],"rank":3313,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02060"},{"aliases":[],"benchmark_id":"claire-radar-2608-07899","caveat":"TelemetrySuffBench evaluates models on fault-origin diagnosis in agent telemetry, using controlled multi-component traces with delayed-binding faults, paired coarse views, seven-factor telemetry masks, and exact-equal ambiguous origin pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07899"],"document_share":0.0002723311546840959,"domain":"Agents","name":"TelemetrySuffBench","organization_count":1,"organizations":["claire_radar"],"rank":3314,"released":"2026-08-08","source":"claire_radar","url":"https://arxiv.org/abs/2608.07899"},{"aliases":[],"benchmark_id":"claire-radar-2606-05001","caveat":"TeleSWEBench is a commit-driven benchmark with 734 questions derived from real developer commits in the srsRAN 5G repository. It evaluates LLM-powered software engineering agents in the telecommunications domain using executable unit tests and a hierarchical LLM-as-a-Judge framework across three difficulty tiers.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05001"],"document_share":0.0002723311546840959,"domain":"Agents","name":"TeleSWEBench","organization_count":1,"organizations":["claire_radar"],"rank":3315,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05001"},{"aliases":[],"benchmark_id":"claire-radar-2609-01515","caveat":"A video cloze benchmark for visual temporal reasoning in Video-LLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01515"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TempCloze","organization_count":1,"organizations":["claire_radar"],"rank":3316,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01515"},{"aliases":[],"benchmark_id":"llm-stats-tempcompass","caveat":"TempCompass is a comprehensive benchmark for evaluating temporal perception capabilities of Video Large Language Models (Video LLMs). It constructs conflicting videos that share identical static content but differ in specific temporal aspects to prevent models from exploiting single-frame bias. The benchmark evaluates multiple temporal aspects including action, motion, speed, temporal order, and attribute changes across diverse task formats including multi-choice QA, yes/no QA, caption matching, and caption generation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tempcompass?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"TempCompass","organization_count":1,"organizations":["llm_stats"],"rank":3317,"released":"2024-03-01","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tempcompass?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-20911","caveat":"Multi-domain coding-agent benchmark with reverse-engineered tasks across Code, Web, Office, and Security. Each subset has its own scoring instrument; scores are not comparable across subsets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20911"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Tencent WorkBuddy Bench","organization_count":1,"organizations":["claire_radar"],"rank":3318,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.20911"},{"aliases":[],"benchmark_id":"claire-radar-2606-05570","caveat":"TensorBench is a benchmark of 199 feature-addition and refactoring tasks on an open-source compiler-based tensor framework extending PyTorch. It grades agents by applying patches and running the framework's test suite.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05570"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"TensorBench","organization_count":1,"organizations":["claire_radar"],"rank":3319,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05570"},{"aliases":[],"benchmark_id":"llm-stats-terminal-bench","caveat":"Terminal-Bench is a benchmark for testing AI agents in real terminal environments. It evaluates how well agents can handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, security tasks, data science workflows, and cybersecurity vulnerabilities. The benchmark consists of a dataset of ~100 hand-crafted, human-verified tasks and an execution harness that connects language models to a terminal sandbox.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Terminal-Bench","organization_count":1,"organizations":["llm_stats"],"rank":3320,"released":"2025-01-17","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-terminal-bench-2","caveat":"Terminal-Bench 2.0 is an updated benchmark for testing AI agents' tool use ability to operate a computer via terminal. It evaluates how well models can handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, security tasks, data science workflows, and cybersecurity vulnerabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Terminal-Bench 2.0","organization_count":1,"organizations":["llm_stats"],"rank":3321,"released":"2025-09-25","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-terminal-bench-2-1","caveat":"Terminal-Bench 2.1 is an updated release of the Terminal-Bench benchmark that tests AI agents' ability to operate a computer via the terminal. It evaluates how well models handle real-world, end-to-end tasks autonomously, including compiling code, training models, setting up servers, system administration, data science workflows, and security tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2.1?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Terminal-Bench 2.1","organization_count":1,"organizations":["llm_stats"],"rank":3322,"released":"2026-05-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-2.1?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-terminal-bench-3-0","caveat":"Terminal-Bench 3.0 is a release of the Terminal-Bench benchmark that tests AI agents' ability to operate a computer via the terminal on real-world, end-to-end tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-3.0?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Terminal-Bench 3.0","organization_count":1,"organizations":["llm_stats"],"rank":3323,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-3.0?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-terminal-bench-hard","caveat":"Agentic coding & terminal use","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/terminalbench-hard"],"document_share":0.0002723311546840959,"domain":"coding","name":"Terminal-Bench Hard","organization_count":1,"organizations":["artificial_analysis"],"rank":3324,"released":"2025-09-02","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/terminalbench-hard"},{"aliases":[],"benchmark_id":"llm-stats-terminal-bench-hard","caveat":"Terminal-Bench Hard is a harder terminal-agent benchmark variant evaluated with the Terminus-2 harness in Cohere's Command A+ and North Mini Code releases.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-hard?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Terminal-Bench Hard","organization_count":1,"organizations":["llm_stats"],"rank":3325,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/terminal-bench-hard?top_n=500"},{"aliases":[],"benchmark_id":"artificial-analysis-terminal-bench-v2-1","caveat":"Agentic coding & terminal use","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/terminalbench-v2-1"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"Terminal-Bench v2.1","organization_count":1,"organizations":["artificial_analysis"],"rank":3326,"released":"2026-05-06","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/terminalbench-v2-1"},{"aliases":[],"benchmark_id":"llm-stats-terminus","caveat":"Terminal-Bench is a benchmark for testing AI agents in real terminal environments, evaluating how well agents can handle real-world, end-to-end tasks autonomously. The benchmark includes tasks spanning coding, system administration, security, data science, model training, file operations, version control, and web development. Terminus is the neutral test-bed agent designed to work with Terminal-Bench, operating purely through tmux sessions without dedicated tools.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/terminus?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Terminus","organization_count":1,"organizations":["llm_stats"],"rank":3327,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/terminus?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-13148","caveat":"TerraBench is a benchmark for grounded Earth-science reasoning, built on TerraAgent, a ReAct-style framework that couples LLM planning with scientific tools. It includes 403 tasks across three tracks and eight domains with 24,500 verified steps.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13148"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TerraBench","organization_count":1,"organizations":["claire_radar"],"rank":3328,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13148"},{"aliases":[],"benchmark_id":"claire-radar-2607-12497","caveat":"TerraLogic evaluates hierarchical geospatial reasoning in Earth observation through 545 scenario-driven tasks spanning optical, SAR, and infrared imagery. Tasks include hazard vulnerability assessment, urban heat island analysis, and forest fragmentation dynamics. Evaluation uses tool-augmented agents with verifiable multi-step workflows and scored by step-wise and end-to-end metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.12497"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"TerraLogic","organization_count":1,"organizations":["claire_radar"],"rank":3329,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.12497"},{"aliases":[],"benchmark_id":"claire-radar-2607-02469","caveat":"TestEvo-Bench evaluates test and code co-evolution tasks from real commit histories. It includes two tracks: test generation (write new tests for changed behavior) and test update (adapt failing tests). Tasks are packaged with environment configurations for execution-grounded metrics like pass rate, coverage, and mutation score. The live component uses timestamps to enable post-training-cutoff evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02469"],"document_share":0.0002723311546840959,"domain":"Agents","name":"TestEvo-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3330,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02469"},{"aliases":[],"benchmark_id":"llm-stats-community-fd462fc2-283c-4967-bd7d-b39d7c661807","caveat":null,"document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/community%3Afd462fc2-283c-4967-bd7d-b39d7c661807?top_n=500"],"document_share":0.0002723311546840959,"domain":"other","name":"testing","organization_count":1,"organizations":["llm_stats"],"rank":3331,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/community%3Afd462fc2-283c-4967-bd7d-b39d7c661807?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-07617","caveat":"TeXFix-Bench evaluates LLM-based full-source document repair across LaTeX, Typst, and Markdown. It provides 10,437 repair instances derived from 743 openly licensed seeds, with a fixed zero-shot protocol and provider-pinned routing.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07617"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TeXFix-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3332,"released":"2026-08-07","source":"claire_radar","url":"https://arxiv.org/abs/2608.07617"},{"aliases":[],"benchmark_id":"opencompass-1574-text2world","caveat":"Text2World, based on planning domain definition language (PDDL), featuring hundreds of diverse domains and employing multi-criteria, execution-based metrics for a more robust evaluation. Text2World 基于规划领域定义语言 (PDDL)，拥有数百个不同的领域，并采用多标准、基于执行的衡量标准来进行更稳健的评估。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Text2World"],"document_share":0.0002723311546840959,"domain":"其他","name":"Text2World","organization_count":1,"organizations":["opencompass_hub"],"rank":3333,"released":"2025-02-18","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Text2World"},{"aliases":[],"benchmark_id":"claire-radar-2606-19259","caveat":"TextRich evaluates detection of AI-generated text-rich images across six categories including posters, charts, receipts, tables, and UI screenshots, with a released dataset on Hugging Face.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19259"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TextRich","organization_count":1,"organizations":["claire_radar"],"rank":3334,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19259"},{"aliases":[],"benchmark_id":"claire-radar-2610-13412","caveat":"ArXiv paper introducing a benchmark for text rendering and preservation in video generation, with abstract stating two tracks and scoring metrics, but no public artifact available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.13412"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TextVBench","organization_count":1,"organizations":["claire_radar"],"rank":3335,"released":"2026-10-09","source":"claire_radar","url":"https://arxiv.org/abs/2610.13412"},{"aliases":[],"benchmark_id":"llm-stats-textvqa","caveat":"TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Introduced to benchmark VQA models' ability to read and reason about text within images, particularly for assistive technologies for visually impaired users. The dataset addresses the gap where existing VQA datasets had few text-based questions or were too small.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/textvqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"TextVQA","organization_count":1,"organizations":["llm_stats"],"rank":3336,"released":"2019-04-18","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/textvqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-23375","caveat":"TF-RefusalBench is a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings. It contains 5,200 prompts across French, German, Italian, and English, designed to trigger refusals in LLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23375"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TF-RefusalBench","organization_count":1,"organizations":["claire_radar"],"rank":3337,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23375"},{"aliases":[],"benchmark_id":"opencompass-1846-tglg","caveat":"Temporally-Grounded Language Generation (TGLG) is a benchmark for real-time vision-language models (VLMs) that focus on two key capabilities: perceptual updating and contingency awareness. 基于时间的语言生成（TGLG）是实时视觉语言模型（VLM）的基准，侧重于两个关键功能：感知更新和应急意识。该存储库还包含TGLG的基线实时VLM代码，即具有时间同步交织的视觉语言模型（VLM-TSI）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TGLG"],"document_share":0.0002723311546840959,"domain":"多模态","name":"TGLG","organization_count":1,"organizations":["opencompass_hub"],"rank":3338,"released":"2025-05-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TGLG"},{"aliases":[],"benchmark_id":"opencompass-1734-thai-local-benchmark","caveat":"Thai local dialect benchmark covers Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. 这是一个涵盖泰国北部（兰纳）、东北部（伊森）和南部（丹布罗）方言的泰国地方方言基准测试，评估大型语言模型在五项自然语言处理任务上的表现：总结、问答、翻译、对话以及与食物相关的任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Thai_local_benchmark"],"document_share":0.0002723311546840959,"domain":"语言","name":"Thai_local_benchmark","organization_count":1,"organizations":["opencompass_hub"],"rank":3339,"released":"2025-04-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Thai_local_benchmark"},{"aliases":["b2b-saas-ai-visibility-benchmark-2026"],"benchmark_id":"claire-radar-github-hamzawamiqhq-b2b-saas-ai-visibility-benchmark-2026","caveat":"Evaluates which of 94 B2B SaaS brands are named by ChatGPT, Claude, Perplexity, Google Gemini and Google AI Overviews on 50 real buying questions across five G2 categories.","document_count":1,"document_ids":["claire_radar:https://github.com/hamzawamiqhq/b2b-saas-ai-visibility-benchmark-2026"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"The B2B SaaS AI Visibility Benchmark 2026","organization_count":1,"organizations":["claire_radar"],"rank":3340,"released":"2026-10-13","source":"claire_radar","url":"https://github.com/hamzawamiqhq/b2b-saas-ai-visibility-benchmark-2026"},{"aliases":[],"benchmark_id":"claire-radar-2606-29278","caveat":"Complexity Ceiling Benchmark evaluates sequential reasoning decay with depth scaling across grounded spatial state-tracking, symbolic pointer manipulation, and transitive relational inference.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29278"],"document_share":0.0002723311546840959,"domain":"General AI","name":"The Complexity Ceiling Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3341,"released":"2026-06-28","source":"claire_radar","url":"https://arxiv.org/abs/2606.29278"},{"aliases":[],"benchmark_id":"claire-radar-2609-24555","caveat":"Measures mathematical progress on fourteen parameterised construction families.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24555"],"document_share":0.0002723311546840959,"domain":"General AI","name":"The Endless Exam","organization_count":1,"organizations":["claire_radar"],"rank":3342,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24555"},{"aliases":[],"benchmark_id":"claire-radar-2607-14865","caveat":"The Energy Society is a multi-agent simulation environment where LLM agents operate under energy constraints tied to token generation, completing jobs and donating energy to survive. The evaluation measures emergent cooperative and competitive behavior across different model sizes and incentive settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14865"],"document_share":0.0002723311546840959,"domain":"Agents","name":"The Energy Society","organization_count":1,"organizations":["claire_radar"],"rank":3343,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14865"},{"aliases":[],"benchmark_id":"claire-radar-2606-01901","caveat":"A benchmark for iterative multimodal dialogue where a vision-language model issues corrective instructions to an image generator over multiple turns, with accumulated common ground observable as a rendered image.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01901"],"document_share":0.0002723311546840959,"domain":"General AI","name":"The Image Reconstruction Game","organization_count":1,"organizations":["claire_radar"],"rank":3344,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01901"},{"aliases":[],"benchmark_id":"claire-radar-2608-22301","caveat":"Evaluates robot imitation across four levels of increasing divergence between demonstration and robot scene, plus an open platform for blind A/B human evaluation.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22301v1"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"The Imitator Game","organization_count":1,"organizations":["claire_radar"],"rank":3345,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22301v1"},{"aliases":[],"benchmark_id":"claire-radar-github-rudycelekli-the-value-engine-benchmark","caveat":"An environment for evaluating and training LLM agents on multi-touch enterprise-sales negotiation, grading episodes with a sales-methodology rubric to produce scalar rewards and structured diagnostics.","document_count":1,"document_ids":["claire_radar:https://github.com/rudycelekli/the-value-engine-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"The Value Engine Benchmark (VEB)","organization_count":1,"organizations":["claire_radar"],"rank":3346,"released":"2026-08-20","source":"claire_radar","url":"https://github.com/rudycelekli/the-value-engine-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2606-09450","caveat":"Evaluates LLMs on theorem proving in Lean4 using classical theorems, with two versions: main and premised. Includes metrics for theorem-level coverage and token efficiency to assess partial progress and proof structure.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09450"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"TheoremBench","organization_count":1,"organizations":["claire_radar"],"rank":3347,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09450"},{"aliases":[],"benchmark_id":"llm-stats-theoremqa","caveat":"A theorem-driven question answering dataset containing 800 high-quality questions covering 350+ theorems from Math, Physics, EE&CS, and Finance. Designed to evaluate AI models' capabilities to apply theorems to solve challenging university-level science problems.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/theoremqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"TheoremQA","organization_count":1,"organizations":["llm_stats"],"rank":3348,"released":"2023-05-21","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/theoremqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1134-theoremqa","caveat":"TheoremQA is the first theorem-driven question-answering dataset designed to evaluate AI models’ capabilities to apply theorems to solve challenging science problems. It is curated by domain experts containing 800 high-quality questions covering 350 theorems from Math, Physics, EE&CS, and Finance. TheoremQA 是第一个基于定理的问题回答数据集，旨在评估 AI 模型应用定理解决复杂科学问题的能力。该数据集由领域专家精心策划，包含 800 个高质量问题，涵盖来自数学、物理、电气与计算机科学以及金融的 350 个定理。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TheoremQA"],"document_share":0.0002723311546840959,"domain":"推理","name":"TheoremQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3349,"released":"2023-12-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TheoremQA"},{"aliases":[],"benchmark_id":"claire-radar-2606-19245","caveat":"TxBench-PP evaluates AI agents on small-molecule preclinical pharmacology tasks including mechanism-of-action, pharmacodynamics, and safety reasoning, using realistic workflow snapshots and deterministic scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19245"],"document_share":0.0002723311546840959,"domain":"Agents","name":"TherapeuticsBench","organization_count":1,"organizations":["claire_radar"],"rank":3350,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19245"},{"aliases":["One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows"],"benchmark_id":"claire-radar-2608-19741","caveat":"Evaluates LLM agents on 507 stateful business workflows in an executable sandbox, with task-specific checks that accept valid trajectories and reject wrong, missing, or extra effects.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.19741"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Thinkingbox-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3351,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.19741"},{"aliases":[],"benchmark_id":"claire-radar-2607-06052","caveat":"Presents a benchmark for force-aware humanoid interaction using demonstrations with synchronized motion and force data. Includes Force-Aware Tracking Score and a simulation protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06052"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ThorArena","organization_count":1,"organizations":["claire_radar"],"rank":3352,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06052"},{"aliases":[],"benchmark_id":"claire-radar-2606-18338","caveat":"ThousandWorlds is a benchmark for climate emulation of potentially habitable exoplanets. It provides a dataset of approximately 1800 simulations from five global climate models, mapping eight planet parameters to 3D atmospheric fields. It includes three nested benchmark subsets: single-simulator regression, multi-simulator regression with complete observations, and multi-simulator regression with structured missingness. Two evaluation protocols are provided: one for ranking methods and one measuring performance relative to inter-model disagreement.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18338"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"ThousandWorlds","organization_count":1,"organizations":["claire_radar"],"rank":3353,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18338"},{"aliases":[],"benchmark_id":"opencompass-2391-threat-signature-eval","caveat":"Threat Signature Eval is a domain-specific benchmark for evaluating video-language models on real-world physical security / surveillance footage. It measures how well a model can recognize and classify security-relevant events into a 10-category threat taxonomy (e.g., Fire & Smoke, Fighting & Violen Threat Signature Eval 是一个面向物理安防场景的领域基准，用于评测视频语言模型在真实监控摄像头画面中的理解与事件识别能力。该基准要求模型在安全相关的视频片段中识别并分类事件（10 类威胁签名，如火灾烟雾、斗殴暴力、非法入侵等），相关介绍与结果在 Ambient.ai 的 Pulsar VLM 发布主题演讲中给出。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Threat-Signature_Eval"],"document_share":0.0002723311546840959,"domain":"视觉定位","name":"Threat-Signature_Eval","organization_count":1,"organizations":["opencompass_hub"],"rank":3354,"released":"2025-11-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Threat-Signature_Eval"},{"aliases":[],"benchmark_id":"opencompass-2050-thunder","caveat":"THUNDER is a benchmark designed to evaluate digital pathology foundation models on tile-level image understanding tasks, facilitating comparative analysis across various downstream tasks and models. THUNDER 是一个用于评估数字病理学基础模型在切片级图像理解任务中表现的基准，旨在支持多种下游任务和模型的对比分析。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/THUNDER"],"document_share":0.0002723311546840959,"domain":"理解","name":"THUNDER","organization_count":1,"organizations":["opencompass_hub"],"rank":3355,"released":"2025-07-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/THUNDER"},{"aliases":["Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment"],"benchmark_id":"claire-radar-2609-02573","caveat":"Evaluates multimodal LLM capability to integrate text-image clues and recover facts within deeply interleaved contexts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02573"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TIC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3356,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02573"},{"aliases":[],"benchmark_id":"claire-radar-2606-15684","caveat":"Evaluates multi-agent collaboration in Minecraft-based tasks with time-sensitive complementary collaboration, requiring agent heterogeneity, mandatory collaboration, and dynamic environments. The framework generates diverse tasks via YAML specifications and filters invalid configurations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15684"],"document_share":0.0002723311546840959,"domain":"Agents","name":"TickingCollabBench","organization_count":1,"organizations":["claire_radar"],"rank":3357,"released":"2026-06-14","source":"claire_radar","url":"https://arxiv.org/abs/2606.15684"},{"aliases":[],"benchmark_id":"claire-radar-2609-23363","caveat":"The proposed benchmark would evaluate coding agents on RTL-level timing closure under post-place-and-route evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23363"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"TicTacBench","organization_count":1,"organizations":["claire_radar"],"rank":3358,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23363"},{"aliases":["Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding"],"benchmark_id":"claire-radar-2608-08512","caveat":"Evaluates LLM performance on 3,050 expert-verified QA pairs about temporally evolving official documents, requiring identification of the document version in force on a queried date across eight task types and three knowledge-access settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08512"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TIDE","organization_count":1,"organizations":["claire_radar"],"rank":3359,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08512"},{"aliases":[],"benchmark_id":"claire-radar-2608-29543","caveat":"Evaluates conversational text-to-SQL on 1,542 samples built from 514 BIRD SQLs, measuring chain identification, drift recognition-resolution, and execution accuracy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29543"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TIDE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3360,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29543"},{"aliases":[],"benchmark_id":"claire-radar-2608-04222","caveat":"A 3D incompressible-turbulence benchmark with multiple configurations, independent ensembles, controlled generalization splits, standardized baselines, and physical-fidelity evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04222"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TIDE: A Physically Diverse 3D Turbulence Benchmark Dataset for Advancing Scientific Machine Learning","organization_count":1,"organizations":["claire_radar"],"rank":3361,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.04222"},{"aliases":[],"benchmark_id":"claire-radar-2609-05117","caveat":"Evaluates LLM safety behaviors across four risk domains and four threat levels using a six-label behavior scale judged by two LLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05117"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TIER","organization_count":1,"organizations":["claire_radar"],"rank":3362,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05117"},{"aliases":["How Good are Foundation Models in Longitudinal MRI Disease Progression Reasoning?"],"benchmark_id":"claire-radar-2608-13309","caveat":"The benchmark comprises 3,920 expert-verified QA pairs from 890 patients across longitudinal MRI timepoints, evaluating temporal reasoning, disease progression, structured localization, sequence ordering, and change localization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13309"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"Time-Aware Multi-View MRI Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3363,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13309"},{"aliases":["PRISM Edit: One Vector for All Temporal Answers"],"benchmark_id":"claire-radar-2607-11327","caveat":"TimeConflict evaluates whether model edits preserve historically correct answers while making updated facts current in the appropriate temporal contexts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.11327"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TimeConflict","organization_count":1,"organizations":["claire_radar"],"rank":3364,"released":"2026-07-13","source":"claire_radar","url":"https://arxiv.org/abs/2607.11327"},{"aliases":[],"benchmark_id":"claire-radar-2609-35143","caveat":"AI agents increasingly carry out long-horizon professional work, but their evaluations rarely require a finished creative deliverable.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35143"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Timeline-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3365,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35143"},{"aliases":[],"benchmark_id":"claire-radar-2608-14270","caveat":"TimeSage-EV evaluates LLM agents on time series analysis tasks in evolving environments, using 60 institutional scenarios across 6 domains with 1,485 scenario-period QA pairs. Agents receive data and reports, with withheld target releases as ground truth, assessing state identification, data summarization, and outlook reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14270"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TimeSage-EV","organization_count":1,"organizations":["claire_radar"],"rank":3366,"released":"2026-08-14","source":"claire_radar","url":"https://arxiv.org/abs/2608.14270"},{"aliases":[],"benchmark_id":"claire-radar-2606-01498","caveat":"TimeSage-MT evaluates agentic time series reasoning in multi-turn dialogues, covering 240 tasks and 2,680 turns across 8 domains, with a protocol and leaderboard for comparing systems.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01498"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TimeSage-MT","organization_count":1,"organizations":["claire_radar"],"rank":3367,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01498"},{"aliases":[],"benchmark_id":"opencompass-1668-timetravel","caveat":"TimeTravel Taxonomy maps artifacts from 10 civilizations, 266 cultures, and 10k+ verified samples for AI-driven historical analysis. 时间旅行分类将来自 10 个文明、266 个文化以及 10k+个验证样本的文物映射，用于 AI 驱动的历史分析。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TimeTravel"],"document_share":0.0002723311546840959,"domain":"多模态","name":"TimeTravel","organization_count":1,"organizations":["opencompass_hub"],"rank":3368,"released":"2025-02-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TimeTravel"},{"aliases":[],"benchmark_id":"claire-radar-2606-16173","caveat":"TimeVista is a benchmark for evaluating time series forecasting using Vision-Language Models (VLMs) as judges, with 5563 samples and rubrics for micro- and macro-level judgments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16173"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TimeVista","organization_count":1,"organizations":["claire_radar"],"rank":3369,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16173"},{"aliases":[],"benchmark_id":"opencompass-1839-tiny-qa-benchmark-pp","caveat":"TinyQA is a benchmark suite designed to evaluate the reasoning abilities of large language models (LLMs). It focuses on assessing LLMs through natural language question-answer pairs, covering various types of reasoning tasks such as causal, logical, and commonsense reasoning. TinyQA是一个用于评估大语言模型（LLMs）推理能力的基准测试套件。该基准专注于通过自然语言问题和答案对来衡量LLMs的推理能力，涵盖了多种类型的推理任务，包括因果推理、逻辑推理和常识推理。TinyQA提供了多样化的数据集，旨在挑战LLMs的推理深度和广度。通过严格的评估，TinyQA能够帮助研究人员更好地理解LLMs在处理复杂语言任务时的表现，并为改进模型提供方向。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/tiny_qa_benchmark_pp"],"document_share":0.0002723311546840959,"domain":"推理","name":"tiny_qa_benchmark_pp","organization_count":1,"organizations":["opencompass_hub"],"rank":3370,"released":"2025-05-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/tiny_qa_benchmark_pp"},{"aliases":[],"benchmark_id":"claire-radar-github-sver0411-tinyedgebench","caveat":"Compares four lightweight classification methods on a synthetic sensor-window classification task with four classes, scoring accuracy, macro F1, and compiled flash size for ESP32-class devices.","document_count":1,"document_ids":["claire_radar:https://github.com/Sver0411/TinyEdgeBench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TinyEdgeBench","organization_count":1,"organizations":["claire_radar"],"rank":3371,"released":"2026-09-15","source":"claire_radar","url":"https://github.com/Sver0411/TinyEdgeBench"},{"aliases":[],"benchmark_id":"llm-stats-tir-bench","caveat":"A tool-calling and multimodal interaction benchmark for testing visual instruction following and execution reliability.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tir-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"TIR-Bench","organization_count":1,"organizations":["llm_stats"],"rank":3372,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tir-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-23425","caveat":"TLA+-Bench evaluates natural-language to TLA+ specification generation by executing each specification in the TLA+ model checker across the full reachable state space. The dataset includes 403 model-checked gold and 897 parse-only silver specifications, with multiple descriptions and difficulty labels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23425"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"TLA+-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3373,"released":"2026-07-26","source":"claire_radar","url":"https://arxiv.org/abs/2607.23425"},{"aliases":[],"benchmark_id":"llm-stats-tldr9-test","caveat":"A large-scale summarization dataset containing over 9 million training instances extracted from Reddit, designed for extreme summarization (generating one-sentence summaries with high compression and abstraction). More than twice larger than previously proposed datasets.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tldr9%2B-%28test%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"summarization","name":"TLDR9+ (test)","organization_count":1,"organizations":["llm_stats"],"rank":3374,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tldr9%2B-%28test%29?top_n=500"},{"aliases":["token-xray"],"benchmark_id":"claire-radar-github-dustin621408-pixel-token-xray","caveat":"Measures quality-preserving reductions in repeated AI model execution by comparing baseline and governed lanes.","document_count":1,"document_ids":["claire_radar:https://github.com/dustin621408-pixel/token-xray"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Token X-Ray","organization_count":1,"organizations":["claire_radar"],"rank":3375,"released":"2026-09-13","source":"claire_radar","url":"https://github.com/dustin621408-pixel/token-xray"},{"aliases":["TomaMMU: A Comprehensive Multimodal Understanding Benchmark for Tomato Leaf Diseases"],"benchmark_id":"claire-radar-2608-08727","caveat":"Evaluates vision-language models on tomato leaf disease understanding through seven tasks spanning basic perception, pathology understanding, and expert diagnosis using multimodal QA pairs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.08727"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"TomaBench","organization_count":1,"organizations":["claire_radar"],"rank":3376,"released":"2026-08-09","source":"claire_radar","url":"https://arxiv.org/abs/2608.08727"},{"aliases":[],"benchmark_id":"llm-stats-tomato","caveat":"TOMATO (Temporal Reasoning Multimodal Evaluation) assesses multimodal models on motion and temporal perception in video, testing understanding of actions, motion, and changes over time.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tomato?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"TOMATO","organization_count":1,"organizations":["llm_stats"],"rank":3377,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tomato?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-14285","caveat":"The evaluation object is a set of 128 scenarios across 16 domains for tool-calling LLM agents in regulated industries, assessing conflicts between safety-aligned values and deployment instructions. The task involves processing confidential documents and measuring override behavior.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14285"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"ToolAlignBench","organization_count":1,"organizations":["claire_radar"],"rank":3378,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.14285"},{"aliases":[],"benchmark_id":"llm-stats-toolathlon","caveat":"Tool Decathlon is a comprehensive benchmark for evaluating AI agents' ability to use multiple tools across diverse task categories. It measures proficiency in tool selection, sequencing, and execution across ten different tool-use scenarios.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/toolathlon?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Toolathlon","organization_count":1,"organizations":["llm_stats"],"rank":3379,"released":"2025-10-29","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/toolathlon?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-25819","caveat":"ToolBench-X evaluates tool-using agents on multi-step tasks with executable tools and automatic scoring, in environments that include recoverable reliability hazards such as specification drift, invocation errors, execution failures, output drift, and cross-source conflicts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25819"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"ToolBench-X","organization_count":1,"organizations":["claire_radar"],"rank":3380,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25819"},{"aliases":[],"benchmark_id":"claire-radar-2607-04686","caveat":"ToolFailBench evaluates tool-use failures in LLM agents across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. It labels traces with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and LLM judges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.04686"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"ToolFailBench","organization_count":1,"organizations":["claire_radar"],"rank":3381,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.04686"},{"aliases":[],"benchmark_id":"opencompass-1712-toolhop","caveat":"ToolHop is a dataset specifically designed for rigorous evaluation of multi-hop tool use, which ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach. ToolHop是一个通过查询驱动构建的数据集，专门用于评测大模型的多跳工具使用能力，具备多样化的查询、有意义的相互依赖关系、本地可执行的工具、详细的反馈和可验证的答案五大特征。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ToolHop"],"document_share":0.0002723311546840959,"domain":"智能体","name":"ToolHop","organization_count":1,"organizations":["opencompass_hub"],"rank":3382,"released":"2025-01-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ToolHop"},{"aliases":["When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents"],"benchmark_id":"claire-radar-2606-05806","caveat":"A benchmark that tests how tool-using LLM agents replan and recover from tool failures. It varies task structure (DAG-based complexity) and failure type (explicit/implicit, transient/permanent).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05806"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ToolMaze","organization_count":1,"organizations":["claire_radar"],"rank":3383,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05806"},{"aliases":[],"benchmark_id":"claire-radar-2606-15508","caveat":"ToolMenuBench is a benchmark for evaluating tool-menu filtering strategies in multi-step LLM agents. It varies tool-menu size, distractor type, state-dependent structure, and risk exposure, and reports filter-level and downstream metrics such as task success, tool calls, and token usage.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15508"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ToolMenuBench","organization_count":1,"organizations":["claire_radar"],"rank":3384,"released":"2026-06-13","source":"claire_radar","url":"https://arxiv.org/abs/2606.15508"},{"aliases":[],"benchmark_id":"opencompass-1607-toolret","caveat":"ToolRet is a heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks, and a corpus of 43k tools, collected from existing datasets. ToolRet是一个包含 7.6k 个不同检索任务的异构工具检索基准，以及从现有数据集中收集的 43k 个工具语料库。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ToolRet"],"document_share":0.0002723311546840959,"domain":"其他","name":"ToolRet","organization_count":1,"organizations":["opencompass_hub"],"rank":3385,"released":"2025-03-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ToolRet"},{"aliases":[],"benchmark_id":"claire-radar-2608-23635","caveat":"Evaluates tool-calling agents under perturbations across the tool-use pipeline, attributing failures to selection, grounding, argument binding, and feedback handling.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.23635"],"document_share":0.0002723311546840959,"domain":"Agents","name":"ToolRobustBench","organization_count":1,"organizations":["claire_radar"],"rank":3386,"released":"2026-08-23","source":"claire_radar","url":"https://arxiv.org/abs/2608.23635"},{"aliases":[],"benchmark_id":"claire-radar-2609-29740","caveat":"TopU-LBVS is a multi-target ligand-based virtual screening benchmark with 93 protein targets and hard-negative decoys, scored by enrichment factor metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.29740"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"TopU-LBVS","organization_count":1,"organizations":["claire_radar"],"rank":3387,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.29740"},{"aliases":["Tables Decoded: DELTA for Structure, TARQA for Understanding"],"benchmark_id":"claire-radar-2609-17458","caveat":"A proposed dataset named TORQUE for table reconstruction and table visual question answering, with a supporting table structure representation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17458"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TORQUE","organization_count":1,"organizations":["claire_radar"],"rank":3388,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17458"},{"aliases":[],"benchmark_id":"claire-radar-2605-31196","caveat":"TouchSafeBench is a physics-grounded benchmark with 2,940 simulated episodes for evaluating collision grounding in vision-language models for safe human-robot collaboration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31196"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"TouchSafeBench","organization_count":1,"organizations":["claire_radar"],"rank":3389,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31196"},{"aliases":[],"benchmark_id":"claire-radar-2607-26849","caveat":"A benchmark of roughly 800 backdoored LLMs across attack objectives, trigger mechanisms, poisoning rates, model scales, and training mechanisms. Evaluates whether a defender can recover a planted trigger given white-box access and behavior of concern.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26849"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ToxScreen","organization_count":1,"organizations":["claire_radar"],"rank":3390,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.26849"},{"aliases":[],"benchmark_id":"claire-radar-2608-24232","caveat":"TRACE is an evidence-grounded safety evaluation benchmark covering prompts, reasoning traces, and final responses of Large Reasoning Models (LRMs). It includes prompts in two languages, nine risk categories, ten attack strategies, and per-component safety annotations with supporting evidence. Evaluates 18 guardrail models on safety judgment and evidence extraction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24232"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TRACE","organization_count":1,"organizations":["claire_radar"],"rank":3391,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24232"},{"aliases":[],"benchmark_id":"claire-radar-2608-16765","caveat":"Evaluates multi-reference image generation by decomposing prompts into atomic operators (Anchor, Disentangle, Apply, Compose) and scoring operator-aligned capabilities and diagnostic failure localization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.16765"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TRACE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3392,"released":"2026-08-17","source":"claire_radar","url":"https://arxiv.org/abs/2608.16765"},{"aliases":[],"benchmark_id":"claire-radar-2609-30670","caveat":"Streaming video understanding evaluation with annotated timing and trigger conditions, scored on accuracy, timeliness, response behavior, workload, completion, and reliability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30670"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding","organization_count":1,"organizations":["claire_radar"],"rank":3393,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.30670"},{"aliases":[],"benchmark_id":"claire-radar-2609-12439","caveat":"A diagnostic benchmark for auditing citation-like artifacts in retrieval-augmented generation and agent-workflow evaluation, including content-equivalent pairs, citation ablations, correctness conflicts, quality gaps, prompt-strength ladders, decoupled judging, and workflow ranking.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12439"],"document_share":0.0002723311546840959,"domain":"Agents","name":"TraceJudgeBench","organization_count":1,"organizations":["claire_radar"],"rank":3394,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12439"},{"aliases":[],"benchmark_id":"claire-radar-2608-06549","caveat":"TradeVerse evaluates LLMs on longitudinal political trade negotiation understanding using reconstructed minutes of 1170 WTO meetings across 5 groups and 89 product groups, with three tasks: predicting HS codes, identifying responding countries, and generating final statements.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06549"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TradeVerse","organization_count":1,"organizations":["claire_radar"],"rank":3395,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06549"},{"aliases":[],"benchmark_id":"llm-stats-trae-code-gen","caveat":"Trae Code Gen is a component of Trae Agent Bench that evaluates implementing new functionality across multiple programming languages in containerized, runnable repositories.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/trae-code-gen?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Trae Code Gen","organization_count":1,"organizations":["llm_stats"],"rank":3396,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-code-gen?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-trae-error-fix","caveat":"Trae Error Fix is a component of Trae Agent Bench that evaluates fixing existing code across multiple programming languages in containerized, runnable repositories.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/trae-error-fix?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Trae Error Fix","organization_count":1,"organizations":["llm_stats"],"rank":3397,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/trae-error-fix?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-30722","caveat":"Introduces a benchmark for counterfactual roadside traffic video generation, but no artifact evidence is provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30722"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TrafficImag","organization_count":1,"organizations":["claire_radar"],"rank":3398,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.30722"},{"aliases":[],"benchmark_id":"claire-radar-2608-06346","caveat":"Evaluates critical error detection in long-horizon agent trajectories using 486 annotated failed trajectories with error lifecycle labels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06346"],"document_share":0.0002723311546840959,"domain":"Agents","name":"TrajErrBench","organization_count":1,"organizations":["claire_radar"],"rank":3399,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.06346"},{"aliases":[],"benchmark_id":"opencompass-1855-transbench","caveat":"TransBench is the first industry-oriented comprehensive multilingual translation evaluation system designed for industrial applications. It quantifies translation model performance across diverse industries and linguistic environments through meticulously curated datasets aligned with standards. TransBench is the first industry-oriented comprehensive multilingual translation evaluation system designed for industrial applications. It quantifies translation model performance across diverse industries and linguistic environments through meticulously curated datasets aligned with standards.","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TransBench"],"document_share":0.0002723311546840959,"domain":"语言","name":"TransBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3400,"released":"2025-05-20","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TransBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-11399","caveat":"TransClean is a controlled benchmark of 9,900 noisy-and-clean translation output pairs for detecting and extracting clean translations from large language model outputs, including 8,800 synthetic and 1,100 manually curated instances.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11399"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TransClean","organization_count":1,"organizations":["claire_radar"],"rank":3401,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11399"},{"aliases":[],"benchmark_id":"llm-stats-translation-en-set1-comet22","caveat":"COMET-22 is an ensemble machine translation evaluation metric combining a COMET estimator model trained with Direct Assessments and a multitask model that predicts sentence-level scores and word-level OK/BAD tags. It demonstrates improved correlations compared to state-of-the-art metrics and increased robustness to critical errors.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-comet22?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"Translation en→Set1 COMET22","organization_count":1,"organizations":["llm_stats"],"rank":3402,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-comet22?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-translation-en-set1-spbleu","caveat":"Translation evaluation using spBLEU (SentencePiece BLEU), a BLEU metric computed over text tokenized with a language-agnostic SentencePiece subword model. Introduced in the FLORES-101 evaluation benchmark for low-resource and multilingual machine translation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-spbleu?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"Translation en→Set1 spBleu","organization_count":1,"organizations":["llm_stats"],"rank":3403,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-en%E2%86%92set1-spbleu?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-translation-set1-en-comet22","caveat":"COMET-22 is a neural machine translation evaluation metric that uses an ensemble of two models: a COMET estimator trained with Direct Assessments and a multitask model that predicts sentence-level scores and word-level OK/BAD tags. It provides improved correlations with human judgments and increased robustness to critical errors compared to previous metrics.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-comet22?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"Translation Set1→en COMET22","organization_count":1,"organizations":["llm_stats"],"rank":3404,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-comet22?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-translation-set1-en-spbleu","caveat":"spBLEU (SentencePiece BLEU) evaluation metric for machine translation quality assessment, using language-agnostic SentencePiece tokenization with BLEU scoring. Part of the FLORES-101 evaluation benchmark for low-resource and multilingual machine translation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-spbleu?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"Translation Set1→en spBleu","organization_count":1,"organizations":["llm_stats"],"rank":3405,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/translation-set1%E2%86%92en-spbleu?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2039-translaw","caveat":"本文提出TransLaw——专为香港判例翻译设计的协同交互式多智能体框架。该框架创新性地将传统翻译流程拆解为翻译、错误标注及校对修正三大子任务，并分配三个智能体协同执行。为评估框架性能，我们构建了大规模双语基准数据集BJC Judgments，对13个开源与商业大语言模型（作为智能体）展开评测。实验结果验证了协同策略的有效性：在多智能体协作显著提升效果的同时，提供了具有参考价值的LLM性能横向对比。通过错误类型学分析，本研究进一步揭示了亟待解决的关键翻译挑战。未来工作将聚焦于优化智能体架构以应对这些挑战，同时开发更全面、低成本的评估基准。 本文提出TransLaw——专为香港判例翻译设计的协同交互式多智能体框架。该框架创新性地将传统翻译流程拆解为翻译、错误标注及校对修正三大子任务，并分配三个智能体协同执行。为评估框架性能，我们构建了大规模双语基准数据集BJC Judgments，对13个开源与商业大语言模型（作为智能体）展开评测。实验结果验证了协同策略的有效性：在多智能体协作显著提升效果的同时，提供了具有参考价值的LLM性能横向对比。通过错误类型学分析，本研究进一步揭示了亟待解决的关键翻译挑战。未来工作将聚焦于优化智能体架构以应对这些挑战，同时开发更全面、低成本的评估基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TransLaw"],"document_share":0.0002723311546840959,"domain":"语言","name":"TransLaw","organization_count":1,"organizations":["opencompass_hub"],"rank":3406,"released":"2025-07-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TransLaw"},{"aliases":[],"benchmark_id":"claire-radar-2608-13167","caveat":"TRAPSBench is a procedurally generated video benchmark with 1,404 physics pairs to test epistemic restraint in VLMs, using Penalized Epistemic Calibration Score (PECS).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13167"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TRAPSBench","organization_count":1,"organizations":["claire_radar"],"rank":3407,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13167"},{"aliases":[],"benchmark_id":"claire-radar-2606-01046","caveat":"TravelEval evaluates LLM-powered travel planning agents in a realistic sandbox with accommodation pricing and intercity transport data, using six dimensions: accuracy, compliance, temporality, spatiality, economy, and utility. It simulates complete plans with API-integrated geographic information and queuing time.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01046"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TravelEval","organization_count":1,"organizations":["claire_radar"],"rank":3408,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01046"},{"aliases":["Locating Hidden Failures Makes Long-Horizon Agents More Reliable"],"benchmark_id":"claire-radar-2609-17930","caveat":"Benchmark of human-verified annotations for locating hidden failures in long-horizon agent trajectories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17930"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Traverse","organization_count":1,"organizations":["claire_radar"],"rank":3409,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17930"},{"aliases":[],"benchmark_id":"claire-radar-2608-07540","caveat":"TREAT evaluates LLMs' ability to recognize theorem identities from equivalence-preserving formula transformations, with 737 identities and 29,480 transformed rows.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07540"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"TREAT","organization_count":1,"organizations":["claire_radar"],"rank":3410,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2608.07540"},{"aliases":[],"benchmark_id":"llm-stats-treebench","caveat":"TreeBench evaluates visual grounded reasoning, requiring models to localize and reason about fine-grained visual details.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/treebench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"TreeBench","organization_count":1,"organizations":["llm_stats"],"rank":3411,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/treebench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-00640","caveat":"TreeProbe is a dataset of 4,719 expert-adjudicated items covering 467 diseases and 10 subtasks across the Tibetan Tree of Medicine framework, used to evaluate cultural bias in LLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00640"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"TreeProbe","organization_count":1,"organizations":["claire_radar"],"rank":3412,"released":"2026-08-01","source":"claire_radar","url":"https://arxiv.org/abs/2608.00640"},{"aliases":["TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning"],"benchmark_id":"claire-radar-2607-26977","caveat":"TREK evaluates agents on synthesizing feasible travel itineraries under interacting entity, route, time, budget, and persona constraints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.26977"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TREK","organization_count":1,"organizations":["claire_radar"],"rank":3413,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.26977"},{"aliases":[],"benchmark_id":"claire-radar-2609-14762","caveat":"A benchmark that evaluates open-weight LLMs and a DeepLog baseline for root cause analysis using four public log datasets, with three retrieval-augmented strategies, on a local GPU setup.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.14762"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TriCalRAG","organization_count":1,"organizations":["claire_radar"],"rank":3414,"released":"2026-09-13","source":"claire_radar","url":"https://arxiv.org/abs/2609.14762"},{"aliases":[],"benchmark_id":"claire-radar-2606-23459","caveat":"TriggerBench is a benchmark for evaluating prospective memory (PM) in LLMs, spanning five dimensions across daily assistant and professional workflow scenarios, with matched retrospective memory (RM) controls, contrastive variants, and overloaded triggers, measuring proactive recall, false-alarm rate, and attentional robustness.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.23459"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TriggerBench","organization_count":1,"organizations":["claire_radar"],"rank":3415,"released":"2026-06-22","source":"claire_radar","url":"https://arxiv.org/abs/2606.23459"},{"aliases":[],"benchmark_id":"claire-radar-2608-29577","caveat":"Evaluates video highlight detection across three perspectives—Event, Emotion, and Nature—using precomputed CLIP features and corresponding annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.29577"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TRINITY","organization_count":1,"organizations":["claire_radar"],"rank":3416,"released":"2026-08-30","source":"claire_radar","url":"https://arxiv.org/abs/2608.29577"},{"aliases":["From Context-Aware to Conflict-Aware: Generalizing Contrastive Decoding for Knowledge Conflict in LLMs"],"benchmark_id":"claire-radar-2606-10298","caveat":"TriState-Bench evaluates how language models handle conflicts between retrieved context and parametric knowledge after calibrating each model's prior knowledge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10298"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"TriState-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3417,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.10298"},{"aliases":[],"benchmark_id":"llm-stats-triviaqa","caveat":"A large-scale reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents (six per question on average) that provide high quality distant supervision for answering the questions. The dataset features relatively complex, compositional questions with considerable syntactic and lexical variability, requiring cross-sentence reasoning to find answers.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/triviaqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"TriviaQA","organization_count":1,"organizations":["llm_stats"],"rank":3418,"released":"2017-05-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/triviaqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-512-triviaqa","caveat":"TriviaqQA is a reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaqQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. TriviaqQA是一个阅读理解数据集，包含超过65万个问题-答案-证据三元组。其包括95K个问答对，由冷知识爱好者和独立收集的事实性文档撰写，平均每个问题6个，为回答问题提供高质量的远程监督。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TriviaQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"TriviaQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3419,"released":"2017-05-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TriviaQA"},{"aliases":[],"benchmark_id":"claire-radar-2606-26029","caveat":"TriViewBench evaluates multimodal LLMs on multi-view structural reasoning using synthetic 3D scenes with controlled object count and occlusion. It comprises 1,923 scenes and over 14,000 QA pairs across four complexity levels and three reasoning categories: Local Decision, Object Counting, and Global Recovery.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26029"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TriViewBench","organization_count":1,"organizations":["claire_radar"],"rank":3420,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.26029"},{"aliases":[],"benchmark_id":"claire-radar-2609-26314","caveat":"Evaluates embodied world models on synchronized triple-view video generation using 19 metrics across consistency, alignment, physics, motion, temporal and visual quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.26314"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"TriWorldBench","organization_count":1,"organizations":["claire_radar"],"rank":3421,"released":"2026-09-22","source":"claire_radar","url":"https://arxiv.org/abs/2609.26314"},{"aliases":[],"benchmark_id":"claire-radar-2606-09323","caveat":"TRL-Bench evaluates tabular encoders at the representation level by exporting row-, column-, or table-level embeddings through each encoder's supported wrapper and probing them with shared lightweight heads across three suites: TRL-CTbench (column/table), TRL-Rbench (row), and TRL-DLTE (compositional Data-Lake Table Enrichment) covering 16 tasks and 20 models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09323"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TRL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3422,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09323"},{"aliases":["trust-me-bro-benchmark"],"benchmark_id":"claire-radar-github-netraruntime-trust-me-bro-benchmark","caveat":"A statistical toolkit that compares answer distributions of API endpoints claiming to serve the same model using matched prompts and a two-sample permutation test.","document_count":1,"document_ids":["claire_radar:https://github.com/NetraRuntime/trust-me-bro-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Trust Me Bro Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3423,"released":"2026-09-27","source":"claire_radar","url":"https://github.com/NetraRuntime/trust-me-bro-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2608-24145","caveat":"Evaluates LLM reliability and robustness for structured data analysis using 2,340 human-verified perturbed instances derived from evidence-path perturbations, scoring refusal behavior and resistance to table representation changes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24145"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TrustDABench","organization_count":1,"organizations":["claire_radar"],"rank":3424,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24145"},{"aliases":[],"benchmark_id":"llm-stats-truthfulqa","caveat":"TruthfulQA is a benchmark to measure whether language models are truthful in generating answers to questions. It comprises 817 questions that span 38 categories, including health, law, finance and politics. The questions are crafted such that some humans would answer falsely due to a false belief or misconception, testing models' ability to avoid generating false answers learned from human texts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/truthfulqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"TruthfulQA","organization_count":1,"organizations":["llm_stats"],"rank":3425,"released":"2021-09-08","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/truthfulqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1096-truthfulqa","caveat":"TruthfulQA is  a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. TruthfulQA 用于测量语言模型在回答问题时的真实度。该基准包含 817 个问题，涵盖 38 个类别，包括健康、法律、金融和政治。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TruthfulQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"TruthfulQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3426,"released":"2022-05-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TruthfulQA"},{"aliases":[],"benchmark_id":"claire-radar-2609-05079","caveat":"TruthInsightBench evaluates open-ended scientific discovery agents on 40 blind tasks across 10 scientific domains. Each task presents a neutral research objective with frozen data while source conclusions are withheld. A fixed LLM-based judge scores agents' claims along six dimensions with 29 artifact-grounded items, and automated aggregation produces overall scores. Data and scoring code are publicly available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05079"],"document_share":0.0002723311546840959,"domain":"AI Scientist","name":"TruthInsightBench","organization_count":1,"organizations":["claire_radar"],"rank":3427,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05079"},{"aliases":[],"benchmark_id":"claire-radar-2606-18539","caveat":"Evaluates time series forecasting models under four explicit structural fault modes (time-warped shock, dependency-fracture shock, regime-transition missingness, cascading sensor-to-system failure) injected into lookback windows, with paired clean/corrupt protocol and five difficulty levels across nine datasets and six domains.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18539"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TS-Fault","organization_count":1,"organizations":["claire_radar"],"rank":3428,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18539"},{"aliases":[],"benchmark_id":"claire-radar-2605-24703","caveat":"A controlled benchmark for evaluating analytical skills in time-series question answering: temporal scale selection, temporal localization, and cross-interval integration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.24703"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TS-Skill","organization_count":1,"organizations":["claire_radar"],"rank":3429,"released":"2026-05-23","source":"claire_radar","url":"https://arxiv.org/abs/2605.24703"},{"aliases":[],"benchmark_id":"claire-radar-2607-09528","caveat":"TSAI-MetaFraud is a multimodal, multi-task benchmark dataset for fraud detection in virtual economies, integrating behavioral, transactional, and graph-structured data. It defines tasks including fraud detection, node classification, temporal link prediction, and weakly supervised fraud detection, with baseline evaluations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09528"],"document_share":0.0002723311546840959,"domain":"Finance","name":"TSAI-MetaFraud","organization_count":1,"organizations":["claire_radar"],"rank":3430,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.09528"},{"aliases":[],"benchmark_id":"claire-radar-2606-05635","caveat":"TSC-Bench is a benchmark for triple-shot composition, generating establishing, medium, and close-up crops from human-centric images with shot descriptions. It contains 1.2k expert-annotated test cases.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05635"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TSC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3431,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05635"},{"aliases":[],"benchmark_id":"claire-radar-2605-31113","caveat":"TSM-Bench is a multilingual, multi-generator, multi-task benchmark for evaluating machine-generated text detectors on real-world Wikipedia editing tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31113"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TSM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3432,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31113"},{"aliases":[],"benchmark_id":"claire-radar-2608-13221","caveat":"TsuGO is a process-level reasoning benchmark for search efficiency in LLMs using Go life-and-death problems, parsing CoT into search trees and reporting diagnostics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13221"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TsuGO","organization_count":1,"organizations":["claire_radar"],"rank":3433,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13221"},{"aliases":["TSWAP: A Multilingual Retrieval-Augmented Thai Wellness Advisor"],"benchmark_id":"claire-radar-2608-22917","caveat":"TSWAP Thai Wellness Benchmark is a retrieval-augmented question-answering benchmark for Thai traditional medicine and wellness advice. It includes 50 questions with gold document IDs, evaluated via Recall@5, and also releases production QA logs and a no-retrieval probe.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22917v1"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"TSWAP Thai Wellness Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3434,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.22917v1"},{"aliases":[],"benchmark_id":"claire-radar-2606-28480","caveat":"Evaluates terminal-use agents on 120 real-world tasks across five families (document editing, email, web info, scientific/engineering workflows) using execution-based scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.28480"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"TUA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3435,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.28480"},{"aliases":[],"benchmark_id":"claire-radar-2606-01322","caveat":"TUKABENCH is a benchmark for jailbreak evaluation in seven African languages, extending JailbreakBench with human translations, cultural adaptations, and code-switched prompts. It assesses LLM safety in low-resource languages using metrics like Refused, Jailbroken, and Deflection, with human validation of LLM-as-a-judge.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01322"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TukaBench","organization_count":1,"organizations":["claire_radar"],"rank":3436,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01322"},{"aliases":[],"benchmark_id":"claire-radar-github-kadirnar-turkish-llm-benchmark","caveat":"Evaluates Turkish language quality, instruction following, latency, API cost, local throughput, and speculative decoding across OpenAI-compatible LLM endpoints.","document_count":1,"document_ids":["claire_radar:https://github.com/kadirnar/turkish-llm-benchmark"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Turkish LLM Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3437,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/kadirnar/turkish-llm-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2608-25218","caveat":"Evaluates end-of-turn and interruption detection in dyadic human conversation across six interaction styles using a 30-hour hand-labeled corpus and standardized metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25218"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TurnBench","organization_count":1,"organizations":["claire_radar"],"rank":3438,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.25218"},{"aliases":[],"benchmark_id":"claire-radar-2606-03626","caveat":"TurtleAI is a benchmark of 823 tasks for visual programming in Turtle Graphics, evaluating models on perceiving geometric patterns and synthesizing Python code.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03626"],"document_share":0.0002723311546840959,"domain":"Code generation","name":"TurtleAI","organization_count":1,"organizations":["claire_radar"],"rank":3439,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03626"},{"aliases":[],"benchmark_id":"llm-stats-tvbench","caveat":"TVBench is a temporal video understanding benchmark evaluating reasoning over actions, events, and temporal dynamics in videos.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tvbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"TVBench","organization_count":1,"organizations":["llm_stats"],"rank":3440,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tvbench?top_n=500"},{"aliases":["Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?"],"benchmark_id":"claire-radar-2606-01247","caveat":"A simulated indoor benchmark in which foundation models must actively move and turn to reproduce the viewpoint shown in a target image.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01247"],"document_share":0.0002723311546840959,"domain":"Agents","name":"TVRBench","organization_count":1,"organizations":["claire_radar"],"rank":3441,"released":"2026-05-31","source":"claire_radar","url":"https://arxiv.org/abs/2606.01247"},{"aliases":[],"benchmark_id":"claire-radar-huggingface-opentwbench-tw-bench","caveat":"Taiwan benchmark dataset with claimed leaderboard and methodology on an external site.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/OpenTWBench/tw-bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"tw-bench","organization_count":1,"organizations":["claire_radar"],"rank":3442,"released":"2026-08-22","source":"claire_radar","url":"https://huggingface.co/datasets/OpenTWBench/tw-bench"},{"aliases":[],"benchmark_id":"claire-radar-huggingface-opentwbench-tw-exam-bench","caveat":"Taiwan exam benchmark dataset with claimed leaderboard and methodology on an external site.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/OpenTWBench/tw-exam-bench"],"document_share":0.0002723311546840959,"domain":"General AI","name":"tw-exam-bench","organization_count":1,"organizations":["claire_radar"],"rank":3443,"released":"2026-08-22","source":"claire_radar","url":"https://huggingface.co/datasets/OpenTWBench/tw-exam-bench"},{"aliases":[],"benchmark_id":"claire-radar-2609-15028","caveat":"A diagnostic benchmark pairing text and image versions of shape tasks to compare in-context learning across modalities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15028"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TwinICL","organization_count":1,"organizations":["claire_radar"],"rank":3444,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15028"},{"aliases":[],"benchmark_id":"llm-stats-tydiqa","caveat":"A multilingual question answering benchmark covering 11 typologically diverse languages with 204K question-answer pairs. Questions are written by people seeking genuine information and data is collected directly in each language without translation to test model generalization across diverse linguistic structures.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/tydiqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"TydiQA","organization_count":1,"organizations":["llm_stats"],"rank":3445,"released":"2020-03-10","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/tydiqa?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-508-tydiqa","caveat":"TyDi QA is a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology -- the set of linguistic features that each language expresses. TyDi QA 是一个涵盖 11 种不同语言的问题回答数据集，包含 20.4 万个问题-答案对。TyDi QA 的语言种类多样，涵盖了语言学特征的各种类型。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/TyDiQA"],"document_share":0.0002723311546840959,"domain":"知识","name":"TyDiQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3446,"released":"2020-03-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/TyDiQA"},{"aliases":[],"benchmark_id":"claire-radar-2610-11392","caveat":"System One models output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes, via a non-generative interface.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11392"],"document_share":0.0002723311546840959,"domain":"General AI","name":"TypedBench","organization_count":1,"organizations":["claire_radar"],"rank":3447,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11392"},{"aliases":[],"benchmark_id":"opencompass-1742-u-niah","caveat":"U-NIAH is a framework unifying RAG and LLM for needle-in-a-haystack tasks, based on the fictional Starlight Academy dataset. It eliminates interference from pre-trained knowledge and supports diverse, complex scenarios (e.g., multi-needle, long-needle, \"needle-in-needle\"). U-NIAH是将 RAG 和 LLM 统一映射在大海捞针任务中的框架。所有任务基于一个虚构背景下的数据集Starlight Academy，涵盖了魔法系统、学术课程、校园生活、等多个方面，旨在消除预训练知识的干扰，从而能够独立于 LLMs 的先验知识。框架包含多种评估场景，支持多针（3、7、15个针）和长针（400-500 token）配置，还引入了“针中针”结构，进一步增加了复杂性。该数据集通过多样化的场景和合成生成的内容，能够从多个维度分析模型在长文本场景下的性能。同时通过模块化设计，U-NIAH可持续注入新的挑战场景。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/U-NIAH"],"document_share":0.0002723311546840959,"domain":"长文本","name":"U-NIAH","organization_count":1,"organizations":["opencompass_hub"],"rank":3448,"released":"2025-03-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/U-NIAH"},{"aliases":[],"benchmark_id":"claire-radar-2605-29170","caveat":"UA-Legal-Bench is a five-task benchmark for evaluating LLMs on Ukrainian legal reasoning using court decisions, covering classification, outcome prediction, and norm extraction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.29170"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UA-Legal-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3449,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.29170"},{"aliases":[],"benchmark_id":"claire-radar-2605-25615","caveat":"UAV-OVO is an out-of-viewpoint generalization benchmark for UAV action recognition. It evaluates models on low-depression viewpoint training data and tests on high-depression viewpoint out-of-distribution data, with class-matched ID/OOD splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25615"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UAV-OVO","organization_count":1,"organizations":["claire_radar"],"rank":3450,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25615"},{"aliases":[],"benchmark_id":"claire-radar-2608-06404","caveat":"UAV3DCrop is a benchmark of repeated multi-angle UAV crop surveys with 88,830 images across 91 scenes, evaluating 3D reconstruction methods on appearance, geometry, and canopy height.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.06404"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UAV3DCrop","organization_count":1,"organizations":["claire_radar"],"rank":3451,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.06404"},{"aliases":[],"benchmark_id":"opencompass-1625-ubuntu-osworld","caveat":"OSWorld is a first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across operating systems. OSWorld 是一个首创的、可扩展的、真实计算机环境，用于多模态智能体，支持操作系统跨平台的任务设置、基于执行的评估和交互式学习。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ubuntu_osworld"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ubuntu_osworld","organization_count":1,"organizations":["opencompass_hub"],"rank":3452,"released":"2024-04-11","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ubuntu_osworld"},{"aliases":["Learning to Deny: Action Denial in Multimodal Large Language Models"],"benchmark_id":"claire-radar-2606-31187","caveat":"A paired-video benchmark evaluating whether multimodal language models can distinguish action presence from action denial under matched contextual and motion cues.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31187"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UCF101-AD","organization_count":1,"organizations":["claire_radar"],"rank":3453,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31187"},{"aliases":[],"benchmark_id":"claire-radar-2607-13621","caveat":"UESF-Bench evaluates embodied agents on unified language-guided human seeking and following in dynamic environments, covering semantic-guided exploration, behavior switching, and identity grounding across single- and multi-person settings. Scoring uses success metrics for both phases.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.13621"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"UESF-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3454,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2607.13621"},{"aliases":[],"benchmark_id":"claire-radar-2608-30688","caveat":"Face recognition bias evaluation using public videos of Brazilian politicians with race/color labels, tested across verification and identification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30688"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UFPR-PEs","organization_count":1,"organizations":["claire_radar"],"rank":3455,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30688"},{"aliases":[],"benchmark_id":"opencompass-1088-uhgeval","caveat":"UHGEval(Unconstrained Hallucination Generation Evaluation) benchmark contains hallucinations generated by LLMs with minimal restrictions. UHGEval 基准，包含由限制条件最小的大语言模型（LLMs）生成的幻觉。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/UHGEval"],"document_share":0.0002723311546840959,"domain":"知识","name":"UHGEval","organization_count":1,"organizations":["opencompass_hub"],"rank":3456,"released":"2024-05-24","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/UHGEval"},{"aliases":[],"benchmark_id":"claire-radar-2608-23857","caveat":"UHI-Bench is a benchmark for dual-source urban heat island (UHI) modeling that integrates dynamic meteorological drivers and static urban morphology. It evaluates over 20 baselines from four model families across five tasks on 20 cities in nine Köppen climate classes, covering land surface temperature UHI and near-surface air temperature UHI.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.23857"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UHI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3457,"released":"2026-08-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.23857"},{"aliases":[],"benchmark_id":"claire-radar-2607-06306","caveat":"Benchmarks visual interaction inference in executable web application generation. Contains 327 screenshots in 45 sets, evaluating executability, navigation reachability, visual fidelity, and interaction inference via the IIS metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06306"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"UI2App","organization_count":1,"organizations":["claire_radar"],"rank":3458,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06306"},{"aliases":[],"benchmark_id":"claire-radar-2609-24613","caveat":"A benchmark for paragraph-level precedent retrieval in UK case law.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24613"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UK-PRBENCH","organization_count":1,"organizations":["claire_radar"],"rank":3459,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24613"},{"aliases":[],"benchmark_id":"claire-radar-2609-28610","caveat":"Ultrasound vision foundation model benchmark; details withheld but no artifacts available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.28610"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UltraBench 2","organization_count":1,"organizations":["claire_radar"],"rank":3460,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.28610"},{"aliases":[],"benchmark_id":"claire-radar-2609-30928","caveat":"Evaluates vision-language models on pixel-level evidence grounding in ultrasound via segmentation, VQA, and report generation tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30928"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"UltraG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3461,"released":"2026-09-25","source":"claire_radar","url":"https://arxiv.org/abs/2609.30928"},{"aliases":[],"benchmark_id":"claire-radar-2606-05576","caveat":"UltraVR is a diagnostic benchmark for evidence-grounded visual reasoning over ultra-resolution images, spanning four domains: CCTV surveillance, remote sensing, whole-slide pathology, and industrial anomaly detection. It includes structured ground-truth chain-of-thought annotations for process-level diagnosis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05576"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"UltraVR","organization_count":1,"organizations":["claire_radar"],"rank":3462,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05576"},{"aliases":[],"benchmark_id":"claire-radar-2606-10382","caveat":"UMI-Bench 1.0 is a benchmark for tabletop robotic manipulation policies using the Universal Manipulation Interface, with a unified protocol for data collection, reset, execution, and logging.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10382"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"UMI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3463,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.10382"},{"aliases":[],"benchmark_id":"claire-radar-2609-27510","caveat":"Evaluates base language models on newly collected internet data using compression-based negative log likelihood.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27510"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Uncheatable Eval","organization_count":1,"organizations":["claire_radar"],"rank":3464,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27510"},{"aliases":["A Benchmark for Vehicle Attribute Classification in Cross-Domain Surveillance Scenarios"],"benchmark_id":"claire-radar-2609-01584","caveat":"Evaluates vehicle attribute classification for front/rear orientation, VMMR suitability, and color clarity using binary annotations across images from seven Brazilian datasets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01584"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Unconstrained Vehicle Identification Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3465,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01584"},{"aliases":[],"benchmark_id":"claire-radar-2607-02294","caveat":"UnderSpecBench evaluates coding agents on DevOps tasks under varying instruction underspecification, measuring action-boundary violations such as wrong-target or over-scope actions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.02294"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"UnderSpecBench","organization_count":1,"organizations":["claire_radar"],"rank":3466,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.02294"},{"aliases":[],"benchmark_id":"claire-radar-2610-05622","caveat":"Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05622"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"UndoBench","organization_count":1,"organizations":["claire_radar"],"rank":3467,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05622"},{"aliases":[],"benchmark_id":"opencompass-1332-unibench","caveat":"UniBench is meant for evaluating VLMs' reasoning abilities. It is a unified implementation of 50+ VLM benchmarks spanning a comprehensive range of carefully categorized capabilities from object recognition to spatial awareness, counting, and much more. UniBench旨在评估VLM的推理能力，包括50个基准测试，涵盖对象识别、空间感知、计数等任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/UniBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"UniBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3468,"released":"2024-08-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/UniBench"},{"aliases":[],"benchmark_id":"claire-radar-2606-05058","caveat":"UniCAD is a benchmark for multi-modal CAD learning covering point-to-CAD reconstruction, text/image-to-CAD generation, and CAD question answering. It includes a universal model, UniCAD-MLLM, and reports state-of-the-art results on UniCAD and Fusion360 benchmarks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05058"],"document_share":0.0002723311546840959,"domain":"CAD","name":"UniCAD","organization_count":1,"organizations":["claire_radar"],"rank":3469,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.05058"},{"aliases":[],"benchmark_id":"claire-radar-2607-08768","caveat":"A capability-driven benchmark for proactive agents in real-world tasks, with 400 bilingual tasks across five capabilities. It evaluates agents in Docker containers using step-by-step checkpoints and a closed-loop strategy with executor, supervisor, and user agents.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08768"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UniClawBench","organization_count":1,"organizations":["claire_radar"],"rank":3470,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08768"},{"aliases":[],"benchmark_id":"llm-stats-uniform-bar-exam","caveat":"The Uniform Bar Examination (UBE) benchmark evaluates language models on the complete bar exam including multiple-choice Multistate Bar Examination (MBE), open-ended Multistate Essay Exam (MEE), and Multistate Performance Test (MPT) components. Used to assess legal reasoning capabilities across seven subject areas including Evidence, Torts, Constitutional Law, Contracts, Criminal Law and Procedure, Real Property, and Civil Procedure.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/uniform-bar-exam?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"Uniform Bar Exam","organization_count":1,"organizations":["llm_stats"],"rank":3471,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/uniform-bar-exam?top_n=500"},{"aliases":["Unified Hallucination Fuzzing for Multimodal Large Language Models"],"benchmark_id":"claire-radar-2608-07525","caveat":"UniHall evaluates multimodal-model hallucinations across object, instruction, and knowledge dimensions, alongside adaptive fuzzing stress tests.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07525"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UniHall","organization_count":1,"organizations":["claire_radar"],"rank":3472,"released":"2026-07-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.07525"},{"aliases":["Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs"],"benchmark_id":"claire-radar-2606-00477","caveat":"UniKE evaluates whether textual knowledge edits in unified multimodal models transfer to image generation, using 2,971 subjects covering attribute and relation edits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00477"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"UniKE","organization_count":1,"organizations":["claire_radar"],"rank":3473,"released":"2026-05-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.00477"},{"aliases":[],"benchmark_id":"claire-radar-2606-08018","caveat":"UniQL is a human-verified benchmark for cross-dialect text-to-SQL, aligning 1,534 questions with executable SQL across 16 dialects, with dialect-specific evaluation tracks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08018"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UniQL","organization_count":1,"organizations":["claire_radar"],"rank":3474,"released":"2026-06-06","source":"claire_radar","url":"https://arxiv.org/abs/2606.08018"},{"aliases":[],"benchmark_id":"claire-radar-2606-26984","caveat":"Evaluates unified multimodal models on joint understanding and generation across four dimensions: internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement, using 2,169 task samples.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26984"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Unison","organization_count":1,"organizations":["claire_radar"],"rank":3475,"released":"2026-06-25","source":"claire_radar","url":"https://arxiv.org/abs/2606.26984"},{"aliases":[],"benchmark_id":"claire-radar-2606-06622","caveat":"UnpredictaBench evaluates whether language-model samples reproduce specified target distributions across statistical, program-induced, and natural-language random processes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06622"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UnpredictaBench","organization_count":1,"organizations":["claire_radar"],"rank":3476,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06622"},{"aliases":[],"benchmark_id":"claire-radar-2606-12864","caveat":"UOJ-Bench is a benchmark for evaluating LLMs in code generation, hacking, and repair, built from real-world submissions on the Universal Online Judge and evaluated through UOJ's judging infrastructure.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.12864"],"document_share":0.0002723311546840959,"domain":"Code","name":"UOJ-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3477,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.12864"},{"aliases":[],"benchmark_id":"claire-radar-2608-00915","caveat":"A benchmark that evaluates uplift (conditional-average-treatment-effect) estimators across seven dataset families under a fixed, test-isolated, multi-objective protocol, with a public living leaderboard.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00915"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"UpliftBench","organization_count":1,"organizations":["claire_radar"],"rank":3478,"released":"2026-08-02","source":"claire_radar","url":"https://arxiv.org/abs/2608.00915"},{"aliases":["TransGaze-Object: Transformer Based Driver Gaze Object Prediction Framework in Real Driving"],"benchmark_id":"claire-radar-2609-10139","caveat":"A driver gaze dataset with synchronized face and scene images, object bounding boxes, and gaze labels used to evaluate a proposed model.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10139"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Urban Driving-Face Scene Gaze (UD-FSG)","organization_count":1,"organizations":["claire_radar"],"rank":3479,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10139"},{"aliases":[],"benchmark_id":"opencompass-1636-urbanvideo-bench","caveat":"The benchmark is designed to evaluate whether video-large language models (Video-LLMs) can naturally process continuous first-person visual observations like humans, enabling recall, perception, reasoning, and navigation. UrbanVideo-Bench旨在评估视频大型语言模型（Video-LLMs）是否能够像人类一样自然地处理连续的第一人称视觉观察，实现回忆、感知、推理和导航。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/UrbanVideo-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"UrbanVideo-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":3480,"released":"2025-03-08","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/UrbanVideo-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2606-15890","caveat":"UrbanWell provides a dataset and evaluation protocol for assessing spatio-temporal reasoning in multimodal large language models using satellite and street view imagery across 38 cities, covering environmental, accessibility, urban form, vitality, and subjective perception indicators with tasks in static prediction, forecasting, and trend classification.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.15890"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UrbanWell","organization_count":1,"organizations":["claire_radar"],"rank":3481,"released":"2026-06-14","source":"claire_radar","url":"https://arxiv.org/abs/2606.15890"},{"aliases":[],"benchmark_id":"claire-radar-2606-07167","caveat":"UrduMMLU evaluates Urdu language understanding through 26,431 multiple-choice questions across 26 subjects and five domains, sourced from native educational materials. Accuracy under zero-shot and few-shot prompting protocols serves as the primary scoring metric.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07167"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UrduMMLU","organization_count":1,"organizations":["claire_radar"],"rank":3482,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07167"},{"aliases":[],"benchmark_id":"llm-stats-usamo-2026","caveat":"USAMO 2026 evaluates models on the six problems from the 2026 United States of America Mathematical Olympiad, requiring rigorous proof-based reasoning. Following the MathArena methodology, proofs are graded against human expert rubrics by dual strong judge models, with the minimum of the two scores taken as final. Maximum score is 42 points.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/usamo-2026?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"USAMO 2026","organization_count":1,"organizations":["llm_stats"],"rank":3483,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo-2026?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-usamo25","caveat":"The 2025 United States of America Mathematical Olympiad (USAMO) benchmark consists of six challenging mathematical problems requiring rigorous proof-based reasoning. USAMO is the most prestigious high school mathematics competition in the United States, serving as the final round of the American Mathematics Competitions series. This benchmark evaluates models on mathematical problem-solving capabilities beyond simple numerical computation, focusing on formal mathematical reasoning and proof generation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/usamo25?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"USAMO25","organization_count":1,"organizations":["llm_stats"],"rank":3484,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/usamo25?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-38043","caveat":"Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38043"],"document_share":0.0002723311546840959,"domain":"Agents","name":"UserProxyBench","organization_count":1,"organizations":["claire_radar"],"rank":3485,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.38043"},{"aliases":[],"benchmark_id":"claire-radar-2608-10042","caveat":"UserToolBench evaluates personalized decision making in tool-use LLMs through inference of latent user preferences, clarification need, and user-aligned tool-call trajectories, built from privacy-sanitized interaction traces with 10 user profiles, 36 tool sets, and 1,065 turns.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10042"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UserToolBench","organization_count":1,"organizations":["claire_radar"],"rank":3486,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.10042"},{"aliases":[],"benchmark_id":"opencompass-2014-utboost","caveat":"UTBoost generates unit tests using LLMs to augment the test cases for certain instances in SWE-Bench, enabling a more rigorous use of SWE-Bench to evaluate the performance of Code Agents. UTBoost通过LLM生成的单元测试，增强了SWE-Bench中一些instances的测试用例，能严谨的使用SWE-Bench来评估Code Agents的表现。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/UTBoost"],"document_share":0.0002723311546840959,"domain":"代码","name":"UTBoost","organization_count":1,"organizations":["opencompass_hub"],"rank":3487,"released":"2025-06-10","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/UTBoost"},{"aliases":[],"benchmark_id":"claire-radar-2608-30508","caveat":"Evaluates long-term conversational memory systems on evidence utilization through 1,717 instances across five domains, testing reasoning, implicit retrieval, synthesis, and interference resistance.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.30508"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UTILMEM","organization_count":1,"organizations":["claire_radar"],"rank":3488,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2608.30508"},{"aliases":[],"benchmark_id":"claire-radar-2606-09570","caveat":"UXBench evaluates user experience in AI assistants through three tasks: UX Judge (binary classification of response quality), UX Eval (response generation), and UX Recovery (repairing failed interactions). The dataset contains 7,400 test instances from 70K+ real interaction logs, covering 8 scenarios and 83 domains. Scoring uses accuracy for Judge and GRM-rated quality for generation tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09570"],"document_share":0.0002723311546840959,"domain":"General AI","name":"UXBench","organization_count":1,"organizations":["claire_radar"],"rank":3489,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09570"},{"aliases":[],"benchmark_id":"claire-radar-2606-16262","caveat":"UXBench evaluates LLM-generated UX critiques through local web fixtures, coverage-gated exploration, and a downstream repair agent, measuring report actionability across seven rubric dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.16262"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"UXBench","organization_count":1,"organizations":["claire_radar"],"rank":3490,"released":"2026-06-15","source":"claire_radar","url":"https://arxiv.org/abs/2606.16262"},{"aliases":[],"benchmark_id":"llm-stats-v-star","caveat":"A visual reasoning benchmark evaluating multimodal inference under challenging spatial and grounded tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/v-star?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"V*","organization_count":1,"organizations":["llm_stats"],"rank":3491,"released":"2023-12-21","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/v-star?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-15683","caveat":"Evaluates video in-context learning for multimodal agents across 342 interactive tasks in 37 environments.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15683"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"V-ICAL Bench","organization_count":1,"organizations":["claire_radar"],"rank":3492,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15683"},{"aliases":[],"benchmark_id":"opencompass-1654-v-star","caveat":"V-STaR is a spatio-temporal reasoning benchmark for Video-LLMs, evaluating Video-LLM’s spatio-temporal reasoning ability in answering questions explicitly in the context of “when”, “where”, and “what”. V-STaR 是一个针对 Video-LLMs的空间时间推理基准，评估 Video-LLM在“何时”、“何地”和“何物”的上下文中明确回答问题的空间时间推理能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/V-STaR"],"document_share":0.0002723311546840959,"domain":"多模态","name":"V-STaR","organization_count":1,"organizations":["opencompass_hub"],"rank":3493,"released":"2025-03-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/V-STaR"},{"aliases":[],"benchmark_id":"claire-radar-2606-05665","caveat":"V2V-Bench evaluates video-to-video generation models across 11 dimensions in five categories: temporal alignment, structural fidelity, transformation quality, video quality, and semantic alignment. It pairs source videos with editing tasks and scores models on these dimensions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.05665"],"document_share":0.0002723311546840959,"domain":"General AI","name":"V2V-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3494,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.05665"},{"aliases":["V\\={a}kQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering"],"benchmark_id":"claire-radar-2609-19879","caveat":"Telugu spoken factoid question answering benchmark with 2,001 QA pairs and speech audio.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19879"],"document_share":0.0002723311546840959,"domain":"General AI","name":"V\\={a}kQA","organization_count":1,"organizations":["claire_radar"],"rank":3495,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.19879"},{"aliases":[],"benchmark_id":"claire-radar-2608-04587","caveat":"Evaluates long-form video understanding agents across eight video distributions with separate evolution and held-out splits.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.04587"],"document_share":0.0002723311546840959,"domain":"Agents","name":"VA-EvoBench","organization_count":1,"organizations":["claire_radar"],"rank":3496,"released":"2026-08-05","source":"claire_radar","url":"https://arxiv.org/abs/2608.04587"},{"aliases":[],"benchmark_id":"claire-radar-2608-18607","caveat":"VA-Judger-Bench evaluates reward models for joint video-audio generation. It contains paired comparisons of generated video-audio samples with human preference labels, covering in-domain and out-of-domain model outputs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.18607"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VA-Judger-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3497,"released":"2026-08-19","source":"claire_radar","url":"https://arxiv.org/abs/2608.18607"},{"aliases":[],"benchmark_id":"claire-radar-2609-19554","caveat":"Proposed benchmark for evaluating embodied spatial intelligence through visual demonstrations, active perception, and metric control.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.19554"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"VABench","organization_count":1,"organizations":["claire_radar"],"rank":3498,"released":"2026-09-17","source":"claire_radar","url":"https://arxiv.org/abs/2609.19554"},{"aliases":["VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies"],"benchmark_id":"claire-radar-2608-12282","caveat":"Evaluates multi-hop tool-use agents across live executable APIs and retrieval, scoring via policy adherence, exact match, and groundedness judges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.12282"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VAKRA","organization_count":1,"organizations":["claire_radar"],"rank":3499,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.12282"},{"aliases":[],"benchmark_id":"claire-radar-2609-01788","caveat":"A pragmatic competence benchmark for Hindi, Punjabi, Tamil, and Malayalam.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01788"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VakyArth","organization_count":1,"organizations":["claire_radar"],"rank":3500,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01788"},{"aliases":[],"benchmark_id":"claire-radar-2607-27453","caveat":"VAmoS Bench evaluates complete voice-agent systems in a stateful customer-support task, with 100 scenarios, a simulated caller, real SQL tools, and binary assertions assessed against full interaction traces.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.27453"],"document_share":0.0002723311546840959,"domain":"Agents","name":"VAmoS Bench","organization_count":1,"organizations":["claire_radar"],"rank":3501,"released":"2026-07-29","source":"claire_radar","url":"https://arxiv.org/abs/2607.27453"},{"aliases":[],"benchmark_id":"claire-radar-2607-12545","caveat":"VanillaBench evaluates the clean accuracy gap between adversarially trained models and vanilla (non-robust) reference models across four threat models. It defines a protocol for comparing robustness-accuracy trade-offs using model accuracy on standard benchmarks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.12545"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VanillaBench","organization_count":1,"organizations":["claire_radar"],"rank":3502,"released":"2026-07-14","source":"claire_radar","url":"https://arxiv.org/abs/2607.12545"},{"aliases":[],"benchmark_id":"claire-radar-2609-09396","caveat":"Evaluates vision-language models on infrastructure camera video across logistics, transportation, and smart spaces using eight spatio-temporal tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09396"],"document_share":0.0002723311546840959,"domain":"Logistics & Operations","name":"VANTAGE-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":3503,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.09396"},{"aliases":[],"benchmark_id":"claire-radar-2608-15600","caveat":"VARM-Bench evaluates verifiable structured reasoning in Chinese abusive-speech moderation. It uses field-anchored chain-of-thought rationales with six decision fields, and a deterministic protocol assessing field correctness, alignment, output validity, and record errors.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15600"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VARM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3504,"released":"2026-08-16","source":"claire_radar","url":"https://arxiv.org/abs/2608.15600"},{"aliases":[],"benchmark_id":"claire-radar-github-drivelineresearch-vast-benchmarking","caveat":"Measures GPU throughput, effective CPU concurrency, memory bandwidth, and disk speed of rented Vast.ai containers under a fixed wall-clock budget.","document_count":1,"document_ids":["claire_radar:https://github.com/drivelineresearch/vast-benchmarking"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Vast Benchmarking","organization_count":1,"organizations":["claire_radar"],"rank":3505,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/drivelineresearch/vast-benchmarking"},{"aliases":[],"benchmark_id":"llm-stats-vatex","caveat":"VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research. Contains over 41,250 videos and 825,000 captions in both English and Chinese, with over 206,000 English-Chinese parallel translation pairs. Supports multilingual video captioning and video-guided machine translation tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vatex?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VATEX","organization_count":1,"organizations":["llm_stats"],"rank":3506,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vatex?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1375-vbench","caveat":"VBench is a comprehensive benchmark evaluates video generation quality. It comprises 16 dimensions in video generation, and also provides a dataset of human preference annotations. VBench用于评估多模态大模型的视频生成质量，包含16个视频生成维度及1个人类偏好注释数据集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"VBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3507,"released":"2023-11-29","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VBench"},{"aliases":[],"benchmark_id":"claire-radar-2608-26105","caveat":"Provides 300 procedurally generated tasks for native visual reasoning through generation, with verifiable reward scorers and controlled modality comparisons.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26105"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VBVR-Pro","organization_count":1,"organizations":["claire_radar"],"rank":3508,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.26105"},{"aliases":[],"benchmark_id":"claire-radar-2606-04588","caveat":"Evaluates complex instruction following for video understanding across content, format, style, and structure constraints with 306 test instructions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.04588"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VCIFBench","organization_count":1,"organizations":["claire_radar"],"rank":3509,"released":"2026-06-03","source":"claire_radar","url":"https://arxiv.org/abs/2606.04588"},{"aliases":[],"benchmark_id":"claire-radar-2609-34687","caveat":"Spatial reasoning is fundamental to embodied agents, yet it remains unclear whether spatial understanding can be carried forward to guide sequential interactions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.34687"],"document_share":0.0002723311546840959,"domain":"Reasoning","name":"VCN-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3510,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.34687"},{"aliases":[],"benchmark_id":"llm-stats-vcr-en-easy","caveat":"Visual Commonsense Reasoning (VCR) benchmark that tests higher-order cognition and commonsense reasoning beyond simple object recognition. Models must answer challenging questions about images and provide rationales justifying their answers. The benchmark measures the ability to infer people's actions, goals, and mental states from visual context.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vcr-en-easy?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"VCR_en_easy","organization_count":1,"organizations":["llm_stats"],"rank":3511,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vcr-en-easy?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-19056","caveat":"Vector-Bench evaluates instruction-based SVG code editing with 40 repair tasks, deterministic specification rewards, and metrics including repair progress and unintended change rate, across 34 model endpoints.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.19056"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Vector-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3512,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2607.19056"},{"aliases":[],"benchmark_id":"claire-radar-2607-18181","caveat":"VEHBench is a diagnostic benchmark for LLM-assisted vibration energy harvester design, featuring 763 tasks across four design roles: specification triage, verifier-guided search, corrupted-state recovery, and policy-conditioned selection. It uses an analytical physical oracle for scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18181"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VEHBench","organization_count":1,"organizations":["claire_radar"],"rank":3513,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.18181"},{"aliases":[],"benchmark_id":"llm-stats-vending-bench-2","caveat":"Vending-Bench 2 tests longer horizon planning capabilities by evaluating how well AI models can manage a simulated vending machine business over extended periods. The benchmark measures a model's ability to maintain consistent tool usage and decision-making for a full simulated year of operation, driving higher returns without drifting off task.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vending-bench-2?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Vending-Bench 2","organization_count":1,"organizations":["llm_stats"],"rank":3514,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vending-bench-2?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-06254","caveat":"Evaluates deepfake image detectors across three paradigms—commercial APIs, vision LLMs, and open-source detectors—on a fixed 100-image adversarial corpus. Uses a unified output schema and scores primarily by Matthews correlation coefficient with ROC-AUC.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06254"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VendorBench-100","organization_count":1,"organizations":["claire_radar"],"rank":3515,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06254"},{"aliases":[],"benchmark_id":"claire-radar-2610-11694","caveat":"Existing reaction models primarily learn molecular transformations, whereas enzy- matic reactions depend jointly on molecular structure and catalytic function.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11694"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"VenusRX-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3516,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11694"},{"aliases":[],"benchmark_id":"claire-radar-2607-01793","caveat":"Vera-Bench is an executable safety benchmark for tool-using agents, with deterministic, artifact-grounded verification across diverse risks and execution settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01793"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Vera-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3517,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.01793"},{"aliases":["VeriBugBench: An Empirically Grounded Framework for Constructing Verilog RTL Debugging Benchmarks"],"benchmark_id":"claire-radar-2609-18022","caveat":"VeriBugBench is a framework for constructing Verilog RTL debugging benchmarks using empirically grounded fault construction, LLM-based testbench enhancement, and execution-based retention.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.18022"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"VeriBugBench","organization_count":1,"organizations":["claire_radar"],"rank":3518,"released":"2026-09-16","source":"claire_radar","url":"https://arxiv.org/abs/2609.18022"},{"aliases":[],"benchmark_id":"claire-radar-2609-39568","caveat":"Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable guarantees.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.39568"],"document_share":0.0002723311546840959,"domain":"Code","name":"VeriCodeBench","organization_count":1,"organizations":["claire_radar"],"rank":3519,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.39568"},{"aliases":[],"benchmark_id":"claire-radar-2609-00232","caveat":"Evaluates OCR-grounded task verification with 1,800 human-verified samples including 1,600 trap-injected invalid tasks and 200 controls, scored via TVR, DA, and ORR.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00232"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VeriOCRBench","organization_count":1,"organizations":["claire_radar"],"rank":3520,"released":"2026-08-31","source":"claire_radar","url":"https://arxiv.org/abs/2609.00232"},{"aliases":["To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech"],"benchmark_id":"claire-radar-2609-30227","caveat":"A probe benchmark for speech-based fact verification in Large Audio Language Models, derived from KVQA and Wikidata, with synthetic false claims and veracity labels.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.30227"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VeriSpeak","organization_count":1,"organizations":["claire_radar"],"rank":3521,"released":"2026-09-24","source":"claire_radar","url":"https://arxiv.org/abs/2609.30227"},{"aliases":[],"benchmark_id":"claire-radar-2605-28683","caveat":"VeriTrip benchmarks travel planning agents on evidence-grounded reasoning over unstructured web corpora, with a verifiable knowledge base for cell-wise verification of factual reliability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.28683"],"document_share":0.0002723311546840959,"domain":"Agents","name":"VeriTrip","organization_count":1,"organizations":["claire_radar"],"rank":3522,"released":"2026-05-27","source":"claire_radar","url":"https://arxiv.org/abs/2605.28683"},{"aliases":[],"benchmark_id":"claire-radar-2608-13522","caveat":"Vero evaluates repository-level verified code generation in Lean 4, with 43 multi-module instances, formal specifications, and proof-only and code-and-proof evaluation modes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.13522"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Vero","organization_count":1,"organizations":["claire_radar"],"rank":3523,"released":"2026-08-13","source":"claire_radar","url":"https://arxiv.org/abs/2608.13522"},{"aliases":[],"benchmark_id":"claire-radar-2605-25784","caveat":"VertiCue-Bench evaluates multimodal large language models on geospatial reasoning using canopy height models to resolve 2D ambiguity in remote sensing natural scenes. It includes 1,534 instances across 17 tasks, testing height perception and semantic reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25784"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VertiCue-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3524,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25784"},{"aliases":["Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization"],"benchmark_id":"claire-radar-2605-26457","caveat":"Verus-SpecBench has 581 tasks in which agents write formal Verus specifications for Codeforces problems. Specs are executed and tested against official tests and hack cases.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26457"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Verus-SpecBench","organization_count":1,"organizations":["claire_radar"],"rank":3525,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.26457"},{"aliases":[],"benchmark_id":"claire-radar-2608-22516","caveat":"Tests long-horizon video understanding with 600 Temporal Ordering and Event Counting questions, auditing whether decoded frames cover all evidence intervals.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22516v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VES-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3526,"released":"2026-08-23","source":"claire_radar","url":"http://arxiv.org/abs/2608.22516v1"},{"aliases":[],"benchmark_id":"claire-radar-2609-08040","caveat":"Consists of 75 real-world software supply chain vulnerability cases across Python, Java, and Go for evaluating LLM agents on exploitability assessment tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08040"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"VEX-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3527,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.08040"},{"aliases":[],"benchmark_id":"claire-radar-2609-35028","caveat":"A unified benchmark for evaluating verification complexity of LLM-generated misinformation, covering multiple dimensions and using LLM-as-judge validated by content analysis.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35028"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"VEX-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3528,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35028"},{"aliases":[],"benchmark_id":"claire-radar-2606-29445","caveat":"VG-GUI-Bench evaluates MLLM-based GUI agents on following video tutorials to complete interactive tasks, with 1,000 long-horizon test cases and four metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.29445"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"VG-GUIBench","organization_count":1,"organizations":["claire_radar"],"rank":3529,"released":"2026-06-28","source":"claire_radar","url":"https://arxiv.org/abs/2606.29445"},{"aliases":[],"benchmark_id":"claire-radar-2608-25452","caveat":"Evaluates video generation quality and aesthetic value with 1,016 prompts, 60,000 videos, and 36,000 task-level annotations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25452"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VGA-BenchV2","organization_count":1,"organizations":["claire_radar"],"rank":3530,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25452"},{"aliases":[],"benchmark_id":"claire-radar-2605-22570","caveat":"VGenST-Bench evaluates spatio-temporal reasoning in multimodal large language models using 1,200 procedurally generated videos with controlled scene composition, camera trajectory, and reasoning targets. It covers 12 reasoning tasks across three spatial scales and 12 QA types across three reasoning levels, with multiple choice and open-ended question variants.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22570"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VGenST-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3531,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22570"},{"aliases":[],"benchmark_id":"claire-radar-2608-19583","caveat":"Contains 27 tasks and 810 instances organized by task domains and skill tags to evaluate visual reasoning capabilities of video generation models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.19583"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VGI-BENCH","organization_count":1,"organizations":["claire_radar"],"rank":3532,"released":"2026-08-20","source":"claire_radar","url":"https://arxiv.org/abs/2608.19583"},{"aliases":[],"benchmark_id":"claire-radar-2610-05380","caveat":"Large Language Models (LLMs) are increasingly applied in hardware design automation, demonstrating strong potential in generating and understanding hardware description languages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.05380"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"VHDL-REPOBENCH","organization_count":1,"organizations":["claire_radar"],"rank":3533,"released":"2026-10-04","source":"claire_radar","url":"https://arxiv.org/abs/2610.05380"},{"aliases":[],"benchmark_id":"claire-radar-2606-13735","caveat":"VHDLSuite is a benchmark-centered infrastructure for VHDL generation evaluation, integrating automated benchmark synthesis, executable validation, and multi-model diagnostic analysis. It includes VHDLBench with over 200 VHDL problems with validated testbenches.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13735"],"document_share":0.0002723311546840959,"domain":"Chip Design & EDA","name":"VHDLBench","organization_count":1,"organizations":["claire_radar"],"rank":3534,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13735"},{"aliases":[],"benchmark_id":"claire-radar-2609-08079","caveat":"VI-Bench is a benchmark for evaluating video prompt inversion from AIGC videos, using 16.1 million real-user prompts and 900 human-verified videos across three difficulty settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.08079"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3535,"released":"2026-09-08","source":"claire_radar","url":"https://arxiv.org/abs/2609.08079"},{"aliases":[],"benchmark_id":"claire-radar-2607-14660","caveat":"VIABench evaluates multimodal large language models on three tasks from first-person videos of visually impaired individuals: proactive reminder, visual question answering, and vision-guided interaction. It includes 761 videos, 46.9 hours, and 14,526 annotations, with protocols for online and offline settings.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.14660"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VIABench","organization_count":1,"organizations":["claire_radar"],"rank":3536,"released":"2026-07-16","source":"claire_radar","url":"https://arxiv.org/abs/2607.14660"},{"aliases":["A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation"],"benchmark_id":"claire-radar-2605-31351","caveat":"A benchmark of over 300K judgment samples that evaluates VLM judges for visually impaired assistance on effectiveness, impartiality, and stability.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.31351"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VIABLE","organization_count":1,"organizations":["claire_radar"],"rank":3537,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2605.31351"},{"aliases":[],"benchmark_id":"claire-radar-2608-10414","caveat":"Evaluates LLM robustness to Vietnamese dialectal rewrites across emotion recognition, natural language inference, question answering, and multiple-choice QA, measuring performance degradation across six dialect groups.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10414"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VialectBench","organization_count":1,"organizations":["claire_radar"],"rank":3538,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10414"},{"aliases":[],"benchmark_id":"claire-radar-2608-03810","caveat":"VIBE is a benchmark for entity-centered affective profiling of LLM outputs in Valence-Arousal-Dominance space, introducing a measurement contract with scalar favorability, response-level and target-directed VAD, and an Affective Passport reporting format.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03810"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VIBE","organization_count":1,"organizations":["claire_radar"],"rank":3539,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03810"},{"aliases":[],"benchmark_id":"llm-stats-vibe","caveat":"Visual Interface Building Evaluation benchmark for UI/app generation","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"VIBE","organization_count":1,"organizations":["llm_stats"],"rank":3540,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1875-vibe","caveat":"SVRPBench is an open and extensible benchmark for the Stochastic Vehicle Routing Problem (SVRP). It includes 500+ instances spanning small to large scales (10–1000 customers), designed to evaluate algorithms under realistic urban logistics conditions with uncertainty and operational constraints. SVRPBench是一个针对随机车辆路径问题（SVRP）的开放且可扩展的基准测试平台。它包含500多个实例，涵盖小到大规模（10-1000个客户），旨在评估算法在具有不确定性和操作约束的现实城市物流条件下的表现。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VIBE"],"document_share":0.0002723311546840959,"domain":"其他","name":"VIBE","organization_count":1,"organizations":["opencompass_hub"],"rank":3541,"released":"2025-05-29","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VIBE"},{"aliases":[],"benchmark_id":"llm-stats-vibe-android","caveat":"VIBE benchmark subset for Android application generation","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-android?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"VIBE Android","organization_count":1,"organizations":["llm_stats"],"rank":3542,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-android?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vibe-backend","caveat":"VIBE benchmark subset for backend service generation","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-backend?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"VIBE Backend","organization_count":1,"organizations":["llm_stats"],"rank":3543,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-backend?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vibe-ios","caveat":"VIBE benchmark subset for iOS application generation","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-ios?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"VIBE iOS","organization_count":1,"organizations":["llm_stats"],"rank":3544,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-ios?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vibe-simulation","caveat":"VIBE benchmark subset for simulation code generation","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-simulation?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"VIBE Simulation","organization_count":1,"organizations":["llm_stats"],"rank":3545,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-simulation?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vibe-web","caveat":"VIBE benchmark subset for web application generation","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-web?top_n=500"],"document_share":0.0002723311546840959,"domain":"code","name":"VIBE Web","organization_count":1,"organizations":["llm_stats"],"rank":3546,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-web?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-00921","caveat":"Evaluates personalized large language models on cross-concept preference reasoning using psychology-grounded tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.00921"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VIBE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3547,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.00921"},{"aliases":[],"benchmark_id":"llm-stats-vibe-eval","caveat":"VIBE-Eval is a hard evaluation suite for measuring progress of multimodal language models, consisting of 269 visual understanding prompts with gold-standard responses authored by experts. The benchmark has dual objectives: vibe checking multimodal chat models for day-to-day tasks and rigorously testing frontier models, with the hard set containing >50% questions that all frontier models answer incorrectly.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-eval?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Vibe-Eval","organization_count":1,"organizations":["llm_stats"],"rank":3548,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-eval?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vibe-pro","caveat":"VIBE-Pro is an advanced version of the VIBE (Visual & Interactive Benchmark for Execution) benchmark that evaluates LLMs on professional-grade full-stack application development tasks. It measures model performance across complex real-world development scenarios including web, mobile, and backend applications with higher difficulty than the standard VIBE benchmark.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-pro?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"VIBE-Pro","organization_count":1,"organizations":["llm_stats"],"rank":3549,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-pro?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vibe-v2","caveat":"VIBE-V2 is an internal benchmark covering pure front-end and full-stack Web, Android, and iOS projects with build-from-scratch tasks. It uses an Agent-as-a-Verifier paradigm to automatically verify program interaction logic and visual output, scoring models through a unified pipeline that includes a requirement set, containerized deployment, and a dynamic interaction environment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vibe-v2?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"VIBE-V2","organization_count":1,"organizations":["llm_stats"],"rank":3550,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vibe-v2?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-10875","caveat":"VibeLifeBench evaluates LLM agents on 200 long-horizon, multi-week tasks across 10 everyday-life domains in a simulated world of 22 mock services. Agents must manage silent world changes and implicit constraints, with fine-grained weighted checks on end state, timeliness, and constraint adherence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10875"],"document_share":0.0002723311546840959,"domain":"Agents","name":"VibeLifeBench","organization_count":1,"organizations":["claire_radar"],"rank":3551,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10875"},{"aliases":[],"benchmark_id":"claire-radar-2609-23570","caveat":"Evaluates memory systems for coding agents on 111 coding targets from 90 SWE-rebench V2 repositories using 3,634 history trajectories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.23570"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"VibeMemBench","organization_count":1,"organizations":["claire_radar"],"rank":3552,"released":"2026-09-20","source":"claire_radar","url":"https://arxiv.org/abs/2609.23570"},{"aliases":["ViBench"],"benchmark_id":"vibench","caveat":"End-to-end vibe-coding benchmark by authors at Replit and Georgian AI Lab, scoring web applications from the user's perspective. Tasks are derived from anonymized Replit production traces, so the benchmark is first-party to one of the agents it measures. It reaches this registry through a customer testimonial in Anthropic's Claude Fable 5 / Mythos 5 launch post, where it is described qualitatively and never scored in the comparison table. Distinct from VBench, an unrelated video-generation benchmark.","document_count":1,"document_ids":["model_reports:anthropic_claude_fable_5_mythos_5"],"document_share":0.0002723311546840959,"domain":"coding_agent","name":"ViBench","organization_count":1,"organizations":["Anthropic"],"rank":3553,"released":"2026-05-26","source":"model_reports","url":"https://vibench.ai/"},{"aliases":[],"benchmark_id":"claire-radar-2609-17248","caveat":"Benchmark for reasoning with spatio-temporal audio-visual evidence in long videos.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.17248"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Video-HolmesV2","organization_count":1,"organizations":["claire_radar"],"rank":3554,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.17248"},{"aliases":[],"benchmark_id":"claire-radar-2608-25529","caveat":"Evaluates instruction following of multimodal LLMs in video understanding with 1.5K samples across constraint categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25529"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Video-IFBench","organization_count":1,"organizations":["claire_radar"],"rank":3555,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25529"},{"aliases":[],"benchmark_id":"llm-stats-video-mme","caveat":"Video-MME is the first-ever comprehensive evaluation benchmark of Multi-modal Large Language Models (MLLMs) in video analysis. It features 900 videos totaling 254 hours with 2,700 human-annotated question-answer pairs across 6 primary visual domains (Knowledge, Film & Television, Sports Competition, Life Record, Multilingual, and others) and 30 subfields. The benchmark evaluates models across diverse temporal dimensions (11 seconds to 1 hour), integrates multi-modal inputs including video frames, subtitles, and audio, and uses rigorous manual labeling by expert annotators for precise assessment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/video-mme?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Video-MME","organization_count":1,"organizations":["llm_stats"],"rank":3556,"released":"2024-05-31","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1358-video-mme","caveat":"Video-MME is an evaluation benchmark of multi-modal LLMs in video analysis, including 900 videos in various duration with a total of 254 hours which spans 6 primary visual domains with 30 subfields. Video-MME用于评估多模态大模型的视频分析能力，包含900个不同长度的视频，来自6个主要视觉领域和30个子领域，总时长达254小时。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Video-MME"],"document_share":0.0002723311546840959,"domain":"多模态","name":"Video-MME","organization_count":1,"organizations":["opencompass_hub"],"rank":3557,"released":"2024-03-31","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Video-MME"},{"aliases":[],"benchmark_id":"llm-stats-video-mme-long-no-subtitles","caveat":"Video-MME is the first-ever comprehensive evaluation benchmark for Multi-modal Large Language Models (MLLMs) in video analysis. This variant focuses on long-term videos (30min-60min) without subtitle inputs, testing robust contextual dynamics across 6 primary visual domains with 30 subfields including knowledge, film & television, sports competition, life record, and multilingual content.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/video-mme-%28long%2C-no-subtitles%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Video-MME (long, no subtitles)","organization_count":1,"organizations":["llm_stats"],"rank":3558,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/video-mme-%28long%2C-no-subtitles%29?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-27828","caveat":"Evaluates video temporal-logical reasoning in multimodal LLMs across 25 fine-grained task categories, with difficulty-controlled final-answer scoring and intermediate-state diagnostics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.27828"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Video-MME-Logical","organization_count":1,"organizations":["claire_radar"],"rank":3559,"released":"2026-06-26","source":"claire_radar","url":"https://arxiv.org/abs/2606.27828"},{"aliases":[],"benchmark_id":"claire-radar-2609-36691","caveat":"Manipulation behaviors vary widely across objects and scenes, but they share a small set of reusable skills, and planning with these skills helps embodied agents generalize to new tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.36691"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"Video2Skill","organization_count":1,"organizations":["claire_radar"],"rank":3560,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.36691"},{"aliases":[],"benchmark_id":"claire-radar-2609-38966","caveat":"We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38966"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"Video2SwimFish","organization_count":1,"organizations":["claire_radar"],"rank":3561,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.38966"},{"aliases":[],"benchmark_id":"claire-radar-2608-05485","caveat":"Evaluates video generation and editing across five settings using agent-generated rubrics that score candidates with criterion-specific evidence and diagnostic reports.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05485"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VideoArgus-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3562,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05485"},{"aliases":[],"benchmark_id":"claire-radar-2605-30256","caveat":"VideoFDB is a benchmark presented for evaluating full-duplex audio-visual conversational agents. It includes 237 dyadic clips with 11 nonverbal conversational dynamics from real-world video calls, along with a taxonomy and rubric-based LM-as-judge evaluation framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30256"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"VideoFDB","organization_count":1,"organizations":["claire_radar"],"rank":3563,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30256"},{"aliases":[],"benchmark_id":"claire-radar-2608-14718","caveat":"VideoGAIA evaluates agentic video understanding through multi-turn, tool-augmented interactions where models must iteratively perceive videos, invoke external tools, and integrate multimodal evidence. It contains 271 human-verified tasks across diverse real-world scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.14718"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VideoGAIA","organization_count":1,"organizations":["claire_radar"],"rank":3564,"released":"2026-08-12","source":"claire_radar","url":"https://arxiv.org/abs/2608.14718"},{"aliases":[],"benchmark_id":"opencompass-1277-videogui","caveat":"VideoGUI is designed to evaluate GUI assistants on visual-centric GUI tasks. Sourced from high-quality web instructional videos, it focuses on tasks involving professional and novel software and complex activities (e.g., video editing). VideoGUI旨在评估以视觉为中心的GUI任务上的GUI助手，来自高质量的网络教学视频，侧重于涉及专业和新颖软件和复杂活动（例如视频编辑）的任务。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VideoGUI"],"document_share":0.0002723311546840959,"domain":"创作","name":"VideoGUI","organization_count":1,"organizations":["opencompass_hub"],"rank":3565,"released":"2024-06-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VideoGUI"},{"aliases":[],"benchmark_id":"llm-stats-videoholmes","caveat":"VideoHolmes evaluates video understanding and reasoning capabilities in multimodal models.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videoholmes?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VideoHolmes","organization_count":1,"organizations":["llm_stats"],"rank":3566,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/videoholmes?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1926-videomathqa","caveat":"VideoMathQA is a benchmark designed to evaluate mathematical reasoning in real-world educational videos. It requires models to interpret and integrate information from three modalities, visuals, audio, and text, across time. VideoMathQA是一个旨在评估实际教育视频中数学推理能力的基准。它要求模型解释和整合来自三种模态(视觉、音频和文本)随时间变化的信息。该基准解决了\"多模态针堆\"问题,即关键信息稀疏且分散在视频的不同模态和时刻。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VideoMathQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"VideoMathQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3567,"released":"2025-06-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VideoMathQA"},{"aliases":[],"benchmark_id":"llm-stats-videomme-w-sub","caveat":"The first-ever comprehensive evaluation benchmark of Multi-modal LLMs in Video analysis. Features 900 videos (254 hours) with 2,700 question-answer pairs covering 6 primary visual domains and 30 subfields. Evaluates temporal understanding across short (11 seconds) to long (1 hour) videos with multi-modal inputs including video frames, subtitles, and audio.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-sub.?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VideoMME w sub.","organization_count":1,"organizations":["llm_stats"],"rank":3568,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-sub.?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-videomme-w-o-sub","caveat":"Video-MME is a comprehensive evaluation benchmark for multi-modal large language models in video analysis. It features 900 videos across 6 primary visual domains with 30 subfields, ranging from 11 seconds to 1 hour in duration, with 2,700 question-answer pairs. The benchmark evaluates MLLMs' capabilities in processing sequential visual data and multi-modal content including video frames, subtitles, and audio.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-o-sub.?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VideoMME w/o sub.","organization_count":1,"organizations":["llm_stats"],"rank":3569,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/videomme-w-o-sub.?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-videommmu","caveat":"Video-MMMU evaluates Large Multimodal Models' ability to acquire knowledge from expert-level professional videos across six disciplines through three cognitive stages: perception, comprehension, and adaptation. Contains 300 videos and 900 human-annotated questions spanning Art, Business, Science, Medicine, Humanities, and Engineering.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videommmu?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VideoMMMU","organization_count":1,"organizations":["llm_stats"],"rank":3570,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/videommmu?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-22907","caveat":"VideoOdyssey evaluates models on ultra-long-context video understanding using videos averaging 109 minutes across 11 domains, with two subsets for visual and audio-visual understanding. Tasks include question answering with continuous certificates averaging 16 and 12.8 minutes respectively.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.22907"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VideoOdyssey","organization_count":1,"organizations":["claire_radar"],"rank":3571,"released":"2026-05-21","source":"claire_radar","url":"https://arxiv.org/abs/2605.22907"},{"aliases":[],"benchmark_id":"opencompass-1900-videoreasonbench","caveat":"VideoReasonBench is designed to evaluate vision-centric complex video reasoning. VideoReasonBench是一个用于评测视觉为中心、复杂视频推理的基准。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VideoReasonBench"],"document_share":0.0002723311546840959,"domain":"强推理","name":"VideoReasonBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3572,"released":"2025-06-05","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VideoReasonBench"},{"aliases":[],"benchmark_id":"claire-radar-2608-23329","caveat":"VideoRover-Bench is a benchmark for open-world video reasoning that combines video understanding with deep research, stratified by video duration and research difficulty. It evaluates the capability of models to locate sparse visual evidence and acquire external knowledge through coordinated video cropping, multimodal search, and webpage browsing.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23329v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VideoRover-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3573,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23329v1"},{"aliases":[],"benchmark_id":"llm-stats-videosimpleqa","caveat":"VideoSimpleQA evaluates factual knowledge grounded in video content, measuring how accurately models answer short, fact-seeking questions about videos.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/videosimpleqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VideoSimpleQA","organization_count":1,"organizations":["llm_stats"],"rank":3574,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/videosimpleqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-09573","caveat":"Evaluates video-grounded diagnostic understanding of one-shot website generation. Approximately 1.7K Video QA instances from 6,338 verified failures across semantic-logical, visual-motion, structural-temporal, and functional categories. Scoring is based on model accuracy in multi-agent system V2Lens compared against baseline Video MLLMs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.09573"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VideoVIBE","organization_count":1,"organizations":["claire_radar"],"rank":3575,"released":"2026-08-10","source":"claire_radar","url":"https://arxiv.org/abs/2608.09573"},{"aliases":[],"benchmark_id":"claire-radar-2606-08091","caveat":"VideoWeaver is an agent harness and benchmark for long video generation, with 16 task categories and 285 cases, evaluating agents via evidence-grounded judge on process and output.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08091"],"document_share":0.0002723311546840959,"domain":"Agents","name":"VideoWeaver","organization_count":1,"organizations":["claire_radar"],"rank":3576,"released":"2026-06-06","source":"claire_radar","url":"https://arxiv.org/abs/2606.08091"},{"aliases":["Can We Trust Video Hallucination Detectors? VidHalLoc for Evaluating the Evaluators"],"benchmark_id":"claire-radar-2609-09895","caveat":"A benchmark with 2,000 adversarial video hallucination samples for evaluating hallucination detectors.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.09895"],"document_share":0.0002723311546840959,"domain":"Software & AI Compute","name":"VidHalLoc","organization_count":1,"organizations":["claire_radar"],"rank":3577,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.09895"},{"aliases":[],"benchmark_id":"claire-radar-2606-03635","caveat":"VidMsg is a benchmark for implicit message inference in short videos, containing 400 clips across 9 topic areas with retrieval and multiple-choice QA tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03635"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VidMsg","organization_count":1,"organizations":["claire_radar"],"rank":3578,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03635"},{"aliases":[],"benchmark_id":"claire-radar-2609-21521","caveat":"Evaluates fine-grained video understanding by verifying events in dense video captions across complexity and duration.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.21521"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VidOmni-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3579,"released":"2026-09-18","source":"claire_radar","url":"https://arxiv.org/abs/2609.21521"},{"aliases":[],"benchmark_id":"claire-radar-2606-13427","caveat":"VietFashion is a benchmark for sketch-text composed image retrieval centered on the Ao Dai, a traditional Vietnamese garment. It includes 650 sketches expanded to over 21,000 photorealistic images with captions, and adopts a multi-target retrieval setting.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13427"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VietFashion","organization_count":1,"organizations":["claire_radar"],"rank":3580,"released":"2026-06-11","source":"claire_radar","url":"https://arxiv.org/abs/2606.13427"},{"aliases":[],"benchmark_id":"claire-radar-2609-37709","caveat":"Recent multimodal image generation models can take multiple images and textual instructions as input, enabling reference-based generation guided not only by text but also by visual instructions such as layouts, arrows, and pose cues.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37709"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VIF-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3581,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37709"},{"aliases":[],"benchmark_id":"opencompass-1717-vilbench","caveat":"ViLBench is a benchmark designed to evaluate vision-language models. It features 600 examples from 5 datasets, selected based on the criterion that process reward models offer greater improvements over output reward models in guiding generations. ViLBench 是一项旨在评估视觉-语言模型的数据集，其强调对模型进行细粒度的逐步推理能力测试。该基准共包含600个经过严格筛选的样本，来源于五个不同的视觉-语言数据集，筛选标准是在模型答案选择过程中，过程奖励模型（process-reward model）相较于输出奖励模型（output-reward model）具有更显著的性能提升效果。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ViLBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ViLBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3582,"released":"2025-03-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ViLBench"},{"aliases":[],"benchmark_id":"claire-radar-2608-26382","caveat":"VIPER evaluates vision-language models on toxicologic pathology via 1,251 questions over 419 H&E-stained rat histology images across organ systems, covering multiple-choice, KPrim, and free-text formats with domain-expert validation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.26382"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"VIPER","organization_count":1,"organizations":["claire_radar"],"rank":3583,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.26382"},{"aliases":[],"benchmark_id":"claire-radar-2609-02462","caveat":"Evaluates cooperative autonomous driving in vehicle-to-infrastructure settings using pseudo-simulation scenarios.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02462"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"VIPS","organization_count":1,"organizations":["claire_radar"],"rank":3584,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02462"},{"aliases":[],"benchmark_id":"claire-radar-2605-25388","caveat":"ViroBench evaluates nucleotide foundation models on viral genomics tasks across two axes: biological understanding (taxonomy classification, host prediction) and latent biosecurity risk (genome modeling, CDS completion). It includes 18 tasks, 58,314 viral samples, multiple split strategies (genus-disjoint, temporal), and 11 metrics, with a unified evaluation protocol and leaderboard.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25388"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"ViroBench","organization_count":1,"organizations":["claire_radar"],"rank":3585,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25388"},{"aliases":[],"benchmark_id":"llm-stats-vct","caveat":"Virology Capabilities Test (VCT) is an expert-level multiple-choice benchmark measuring the capability to troubleshoot complex virology laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vct?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"Virology Capabilities Test","organization_count":1,"organizations":["llm_stats"],"rank":3586,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vct?top_n=500"},{"aliases":["virtualv_llm"],"benchmark_id":"claire-radar-github-virtuanalytica-virtualv-llm","caveat":"Hardware-aware evaluation suite for GGUF/llama.cpp and 1Cat-vLLM workloads on heterogeneous GPU pools.","document_count":1,"document_ids":["claire_radar:https://github.com/virtuanalytica/virtualv_llm"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"VirtualV LLM Testsuite","organization_count":1,"organizations":["claire_radar"],"rank":3587,"released":"2026-09-22","source":"claire_radar","url":"https://github.com/virtuanalytica/virtualv_llm"},{"aliases":[],"benchmark_id":"claire-radar-2605-30344","caveat":"VisAnomBench is a benchmark assembled from public time-series datasets for anomaly detection, augmented with natural-language explanations selected from large vision-language models. It supports fine-tuning a parameter-efficient VLM called VisAnomReasoner for grounded anomaly detection decisions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30344"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VisAnomBench","organization_count":1,"organizations":["claire_radar"],"rank":3588,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30344"},{"aliases":[],"benchmark_id":"opencompass-2063-visco","caveat":"VISCO aims to evaluate the critique and correction capabilities of VLMs, which are two essential building blocks towards VLM self-improvement. VISCO requires VLMs to critique the correctness of each step in CoT, provide natural language explanation, and corrects the CoT based on the critique. VISCO 旨在评估 VLM 的评判（critique）和纠正（correction）能力，这两个能力是 VLM 自主提升推理性能的基础。对于一个视觉推理问题，给定模型生成的 CoT，VISCO 评测集要求 VLM 评判 CoT 中每个步骤的正确性，提供自然语言解释，并根据评判结果对 CoT 进行修正。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VISCO"],"document_share":0.0002723311546840959,"domain":"多模态","name":"VISCO","organization_count":1,"organizations":["opencompass_hub"],"rank":3589,"released":"2024-12-03","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VISCO"},{"aliases":[],"benchmark_id":"claire-radar-2608-10408","caveat":"VisEditBench is a benchmark comprising 1,395 human-annotated visualization code-editing tasks across two settings: feedback-guided repair and reference-guided restyling. Models are evaluated on their ability to revise existing visualization code based on multimodal feedback such as buggy or marked charts with textual instructions, and target chart images. Scoring is based on pass rates of generated code executions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10408"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VisEditBench","organization_count":1,"organizations":["claire_radar"],"rank":3590,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10408"},{"aliases":[],"benchmark_id":"llm-stats-visfactor","caveat":"VisFactor is a benchmark evaluating fine-grained visual factor perception and reasoning over images.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/visfactor?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VisFactor","organization_count":1,"organizations":["llm_stats"],"rank":3591,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/visfactor?top_n=500"},{"aliases":["Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoning"],"benchmark_id":"claire-radar-2609-28851","caveat":"Proposed benchmark framework to evaluate robustness of vision-language reasoning via flip-direction steering, constructing evaluation groups for a target model and visual variation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.28851"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"VisFlip","organization_count":1,"organizations":["claire_radar"],"rank":3592,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.28851"},{"aliases":[],"benchmark_id":"claire-radar-2609-07151","caveat":"Evaluates forecasting of vishing call trajectories toward terminal harms using a 35,340-utterance synthetic Chinese call corpus labeled with a 6-tactic taxonomy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07151"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Vishing-Tactics-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3593,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07151"},{"aliases":[],"benchmark_id":"llm-stats-vision2web","caveat":"Vision2Web evaluates multimodal models on converting visual designs and screenshots into functional web pages, measuring end-to-end design-to-code capability.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vision2web?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"Vision2Web","organization_count":1,"organizations":["llm_stats"],"rank":3594,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vision2web?top_n=500"},{"aliases":["VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents"],"benchmark_id":"claire-radar-2605-26144","caveat":"A benchmark for LLM coding agents that build web apps from visual specs. It covers five input conditions and scores structure, behavior, and visual fidelity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26144"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"VISTA (VIsual Spec-To-App Benchmark)","organization_count":1,"organizations":["claire_radar"],"rank":3595,"released":"2026-05-22","source":"claire_radar","url":"https://arxiv.org/abs/2605.26144"},{"aliases":[],"benchmark_id":"claire-radar-2609-37287","caveat":"Image translation is a fundamental capability of multimodal models for multilingual applications, requiring visual understanding and meaning preservation across languages.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.37287"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VISTA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3596,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.37287"},{"aliases":[],"benchmark_id":"claire-radar-2606-03273","caveat":"VistaHop is a benchmark for long-horizon Visual DeepSearch, evaluating repeated image inspection, visual-anchor grounding, and evidence traversal across 600 tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.03273"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VistaHop","organization_count":1,"organizations":["claire_radar"],"rank":3597,"released":"2026-06-02","source":"claire_radar","url":"https://arxiv.org/abs/2606.03273"},{"aliases":[],"benchmark_id":"claire-radar-2608-28062","caveat":"VisTarget-Bench is a 150-task human-verified benchmark pairing each multimodal search question with a held-out target image to separate image-retrieval failures from visual-perception failures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.28062"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VisTarget-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3598,"released":"2026-08-28","source":"claire_radar","url":"https://arxiv.org/abs/2608.28062"},{"aliases":[],"benchmark_id":"opencompass-1993-vistorybench","caveat":"ViStoryBench is a benchmark designed to rigorously evaluate the capabilities of multimodal generative models (e.g., diffusion models, LLM-based agents) in synthesizing visually coherent image sequences from textual narratives and reference images. ViStoryBench 是一个面向故事可视化任务的综合性评测基准，旨在评估多模态生成模型（如扩散模型视频生成模型等）根据给定叙事文本和参考图像生成视觉连贯且情节一致的图像序列的能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ViStoryBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ViStoryBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3599,"released":"2025-06-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ViStoryBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-20868","caveat":"An evaluation suite of 1,340 video QA pairs across 15 subtasks assessing MLLM qualitative reasoning in dynamic scenes, covering motion perception, spatial relations, outcome prediction, and physical dynamics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20868"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ViSTR-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3600,"released":"2026-07-23","source":"claire_radar","url":"https://arxiv.org/abs/2607.20868"},{"aliases":["visual-rag-benchmark"],"benchmark_id":"claire-radar-github-james-a-watson-visual-rag-benchmark","caveat":"Visual RAG Benchmark evaluates end-to-end retrieval-augmented generation on reference tables from three EU Water Framework Directive guidance documents, scoring answers on 103 and 154 question sets split into text and table question types via LLM-as-a-judge correctness, ROUGE-1, and retrieval accuracy across seven indexing and retrieval pipelines.","document_count":1,"document_ids":["claire_radar:https://github.com/james-a-watson/visual-rag-benchmark"],"document_share":0.0002723311546840959,"domain":"Finance","name":"Visual RAG Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3601,"released":"2026-09-10","source":"claire_radar","url":"https://github.com/james-a-watson/visual-rag-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2606-07872","caveat":"VisualFLIP evaluates multimodal LLMs on visual reasoning with 1,374 images in paired perturbation tasks. Each pair has a fixed question but minimally changed visual evidence so the answer flips. Scoring uses pair accuracy and Collapse Rate to test evidence dependence in capabilities like cardinality, attribute, spatial, and logic.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07872"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VisualFLIP","organization_count":1,"organizations":["claire_radar"],"rank":3602,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07872"},{"aliases":[],"benchmark_id":"claire-radar-2606-07595","caveat":"VisualLeakBench is a 500-image benchmark for evaluating action-boundary propagation failures in vision-language agents, with stratified subsets and oracle diagnostics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07595"],"document_share":0.0002723311546840959,"domain":"Agents","name":"VisualLeakBench","organization_count":1,"organizations":["claire_radar"],"rank":3603,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.07595"},{"aliases":[],"benchmark_id":"opencompass-1748-visualpuzzles","caveat":"VisualPuzzles is a benchmark that targets visual reasoning while deliberately minimizing reliance on specialized knowledge. VisualPuzzles consists of 1168 diverse questions spanning five categories: algorithmic, analogical, deductive, inductive, and spatial reasoning. LLM 能考公务员吗？我们做了个测试…\n\n近年来，大模型（LLM）的能力突飞猛进，似乎“越来越聪明”了。但有一个关键问题仍然摆在眼前：\n🤔 它们真的会“推理”吗？\n\n🚀 我们设计了一个名为 VisualPuzzles 🧩 的全新数据集，专门用来回答这个问题：\n脱离专业知识的支持，大模型能靠逻辑本身解题吗？\n我们从多个来源精心挑选或改编了 1168 道图文逻辑题，其中一个重要来源便是中国国家公务员考试行测中的逻辑推理题（没错，真·考公难度）🎯","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VisualPuzzles"],"document_share":0.0002723311546840959,"domain":"强推理","name":"VisualPuzzles","organization_count":1,"organizations":["opencompass_hub"],"rank":3604,"released":"2025-04-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VisualPuzzles"},{"aliases":[],"benchmark_id":"opencompass-1634-visualsimpleqa","caveat":"VisualSimpleQA is a multimodal fact-seeking benchmark with two key features. VisualSimpleQA 是一个多模态事实寻求基准，具有两个关键特性。首先，它使视觉和语言模态中 LVLMs 的评估更加简化和解耦。其次，它纳入了明确的难度标准，以指导人工标注并促进提取具有挑战性的子集，即 VisualSimpleQA-hard。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VisualSimpleQA"],"document_share":0.0002723311546840959,"domain":"多模态","name":"VisualSimpleQA","organization_count":1,"organizations":["opencompass_hub"],"rank":3605,"released":"2025-03-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VisualSimpleQA"},{"aliases":[],"benchmark_id":"llm-stats-visualwebbench","caveat":"A multimodal benchmark designed to assess the capabilities of multimodal large language models (MLLMs) across web page understanding and grounding tasks. Comprises 7 tasks (captioning, webpage QA, heading OCR, element OCR, element grounding, action prediction, and action grounding) with 1.5K human-curated instances from 139 real websites across 87 sub-domains.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/visualwebbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VisualWebBench","organization_count":1,"organizations":["llm_stats"],"rank":3606,"released":"2024-04-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/visualwebbench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-visulogic","caveat":"VisuLogic evaluates logical reasoning capabilities in visual contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/visulogic?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VisuLogic","organization_count":1,"organizations":["llm_stats"],"rank":3607,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/visulogic?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vita-bench","caveat":"VITA-Bench evaluates AI agents on real-world virtual task automation, measuring their ability to complete complex multi-step tasks in simulated environments.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vita-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"VITA-Bench","organization_count":1,"organizations":["llm_stats"],"rank":3608,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vita-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-27141","caveat":"VitaBench 2.0 evaluates personalized and proactive agent behavior in long-term, multi-session user interactions across food delivery, in-store consumption, and online travel domains. Tasks are per-user sequences of subtasks requiring agents to infer, utilize, and update user preferences from fragmented interaction history, with an extensible memory interface for controlled comparison across memory architectures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.27141"],"document_share":0.0002723311546840959,"domain":"Agents","name":"VitaBench","organization_count":1,"organizations":["claire_radar"],"rank":3609,"released":"2026-05-26","source":"claire_radar","url":"https://arxiv.org/abs/2605.27141"},{"aliases":[],"benchmark_id":"opencompass-2388-vknowu","caveat":"尽管多模态大型语言模型（MLLMs）在识别物体方面已相当熟练，但它们往往缺乏对世界潜在物理和社会原则的直觉的、类似人类的理解。这种高级的、基于视觉的语义，我们称之为视觉知识，其构成了感知与推理之间的桥梁。为了系统地评估这种能力，我们提出了 VKnowU，一个包含 1,249 个视频、1,680 个问题的综合基准，涵盖了 8 种核心类型的视觉知识，既包括以世界为中心的（例如直觉物理），也包括以人类为中心的任务（例如主观意图）。 尽管多模态大型语言模型（MLLMs）在识别物体方面已相当熟练，但它们往往缺乏对世界潜在物理和社会原则的直觉的、类似人类的理解。这种高级的、基于视觉的语义，我们称之为视觉知识，其构成了感知与推理之间的桥梁。为了系统地评估这种能力，我们提出了 VKnowU，一个包含 1,249 个视频、1,680 个问题的综合基准，涵盖了 8 种核心类型的视觉知识，既包括以世界为中心的（例如直觉物理），也包括以人类为中心的任务（例如主观意图）。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VKnowU"],"document_share":0.0002723311546840959,"domain":"多模态","name":"VKnowU","organization_count":1,"organizations":["opencompass_hub"],"rank":3610,"released":"2025-11-25","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VKnowU"},{"aliases":[],"benchmark_id":"llm-stats-vladbench","caveat":"VLADBench is a vision-language autonomous-driving benchmark evaluating understanding of dynamic traffic scenes and participants.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vladbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VLADBench","organization_count":1,"organizations":["llm_stats"],"rank":3611,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vladbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-25376","caveat":"Evaluates post-training quantization of vision-language-action models by rolling out quantized policies on simulation benchmarks and recording per-episode success and diagnostic metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.25376"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"VLAQuantBench","organization_count":1,"organizations":["claire_radar"],"rank":3612,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.25376"},{"aliases":[],"benchmark_id":"opencompass-1541-vlm2-bench","caveat":"VLM²-Bench is the first comprehensive benchmark that evaluates vision-language models' (VLMs) ability to visually link matching cues across multi-image sequences and videos. The benchmark consists of 9 subtasks with over 3,000 test cases. VLM²-Bench 是第一个全面评估视觉语言模型（VLMs）在多图像序列和视频中视觉链接匹配线索能力的基准。该基准包括 9 个子任务，超过 3000 个测试案例，旨在评估人类日常使用的根本视觉链接能力。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VLM2-Bench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"VLM2-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":3613,"released":"2025-02-17","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VLM2-Bench"},{"aliases":[],"benchmark_id":"llm-stats-vlmsarebiased","caveat":"VLMsAreBiased evaluates whether vision-language models rely on visual evidence or fall back on language priors when answering.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vlmsarebiased?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VLMsAreBiased","organization_count":1,"organizations":["llm_stats"],"rank":3614,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsarebiased?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vlmsareblind","caveat":"A vision-language benchmark that probes blind spots and brittle reasoning in multimodal models.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vlmsareblind?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VLMsAreBlind","organization_count":1,"organizations":["llm_stats"],"rank":3615,"released":"2024-07-09","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vlmsareblind?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-02502","caveat":"Evaluates text-to-image models on their ability to generate images that convey visual metaphors, using a set of metaphor prompts and comparative scoring.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.02502"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VMetaphor-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3616,"released":"2026-09-02","source":"claire_radar","url":"https://arxiv.org/abs/2609.02502"},{"aliases":[],"benchmark_id":"claire-radar-github-vnish-global-vnish-global-operator-routing-benchmark","caveat":"Tests multilingual AI assistant routing across 80 cases in ten locales, requiring correct owned-source routing, hard stops on missing evidence, and citation precision.","document_count":1,"document_ids":["claire_radar:https://github.com/vnish-global/vnish-global-operator-routing-benchmark"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VNISH GLOBAL Multilingual Operator Routing Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3617,"released":"2026-08-22","source":"claire_radar","url":"https://github.com/vnish-global/vnish-global-operator-routing-benchmark"},{"aliases":[],"benchmark_id":"llm-stats-vocalsound","caveat":"A dataset for improving human vocal sounds recognition, containing over 21,000 crowdsourced recordings of laughter, sighs, coughs, throat clearing, sneezes, and sniffs from 3,365 unique subjects. Used for audio event classification and recognition of human non-speech vocalizations.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vocalsound?top_n=500"],"document_share":0.0002723311546840959,"domain":"audio","name":"VocalSound","organization_count":1,"organizations":["llm_stats"],"rank":3618,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vocalsound?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-github-kadirnar-voice-agent-next","caveat":"A framework for building real-time speech-to-speech voice agents with a built-in benchmark suite measuring latency.","document_count":1,"document_ids":["claire_radar:https://github.com/kadirnar/voice-agent-next"],"document_share":0.0002723311546840959,"domain":"Agents","name":"voice-agent-next","organization_count":1,"organizations":["claire_radar"],"rank":3619,"released":"2026-09-24","source":"claire_radar","url":"https://github.com/kadirnar/voice-agent-next"},{"aliases":[],"benchmark_id":"llm-stats-voicebench-avg","caveat":"VoiceBench is the first benchmark designed to provide a multi-faceted evaluation of LLM-based voice assistants, evaluating capabilities including general knowledge, instruction-following, reasoning, and safety using both synthetic and real spoken instruction data with diverse speaker characteristics and environmental conditions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/voicebench-avg?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"VoiceBench Avg","organization_count":1,"organizations":["llm_stats"],"rank":3620,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/voicebench-avg?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-10359","caveat":"VoxSumm evaluates joint summarization and translation of long-form spoken news. It comprises 10,045 BBC article-summary pairs across 24 languages and about 703 hours of speech, with scoring based on summarization and translation quality.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10359"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VoxSumm","organization_count":1,"organizations":["claire_radar"],"rank":3621,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10359"},{"aliases":["VPRef: A Cross-Domain Benchmark for Referring Remote Sensing Image Segmentation"],"benchmark_id":"claire-radar-2609-16486","caveat":"Evaluates referring remote sensing image segmentation across visual and textual domain shifts using the Vaihingen-Potsdam Referring dataset with 46,972 triplets and a three-tier linguistic hierarchy.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.16486"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VPRef","organization_count":1,"organizations":["claire_radar"],"rank":3622,"released":"2026-09-15","source":"claire_radar","url":"https://arxiv.org/abs/2609.16486"},{"aliases":[],"benchmark_id":"llm-stats-vqa-rad","caveat":"VQA-RAD (Visual Question Answering in Radiology) is the first manually constructed dataset of medical visual question answering containing 3,515 clinically generated visual questions and answers about radiology images. The dataset includes questions created by clinical trainees on 315 radiology images from MedPix covering head, chest, and abdominal scans, designed to support AI development for medical image analysis and improve patient care.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vqa-rad?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VQA-Rad","organization_count":1,"organizations":["llm_stats"],"rank":3623,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vqa-rad?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-07861","caveat":"Evaluates 12 image preprocessing techniques for cloud VLM-based visual question answering across 3 VQA datasets and 4 commercial models, measuring accuracy, cost, and latency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07861"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VQABench","organization_count":1,"organizations":["claire_radar"],"rank":3624,"released":"2026-08-08","source":"claire_radar","url":"https://arxiv.org/abs/2608.07861"},{"aliases":[],"benchmark_id":"llm-stats-vqav2","caveat":"VQAv2 is a balanced Visual Question Answering dataset that addresses language bias by providing complementary images for each question, forcing models to rely on visual understanding rather than language priors. It contains approximately twice the number of image-question pairs compared to the original VQA dataset.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vqav2?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VQAv2","organization_count":1,"organizations":["llm_stats"],"rank":3625,"released":"2016-12-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vqav2-test","caveat":"VQA v2.0 (Visual Question Answering v2.0) is a balanced dataset designed to counter language priors in visual question answering. It consists of complementary image pairs where the same question yields different answers, forcing models to rely on visual understanding rather than language bias. The dataset contains 1,105,904 questions across 204,721 COCO images, requiring understanding of vision, language, and commonsense knowledge.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28test%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VQAv2 (test)","organization_count":1,"organizations":["llm_stats"],"rank":3626,"released":"2016-12-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28test%29?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-vqav2-val","caveat":"VQAv2 is a balanced Visual Question Answering dataset containing open-ended questions about images that require understanding of vision, language, and commonsense knowledge to answer. VQAv2 addresses bias issues from the original VQA dataset by collecting complementary images such that every question is associated with similar images that result in different answers, forcing models to actually understand visual content rather than relying on language priors.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28val%29?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"VQAv2 (val)","organization_count":1,"organizations":["llm_stats"],"rank":3627,"released":"2016-12-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/vqav2-%28val%29?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2383-vrbench","caveat":"VRBench is the first evaluation benchmark specifically designed to assess multi-step reasoning capabilities over long videos. The benchmark comprises 1,010 long videos with an average duration of 1.6 hours, covering 8 languages and 7 video types. It includes annotations for 9,468 reasoning steps. VRBench是首个专门针对长视频多步推理能力设计的评测基准VRBench。该基准包含 1010 个平均时长1.6小时的长视频，覆盖 8 种语言和 7 种视频类型，标注了 9468 个多步推理问答对和超过 3 万个详细推理步骤。本评测集能够同时对模型的推理过程和推理结果进行评分，是首个同时具备多步推理链标注和评测能力的视频评测集。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/VRBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"VRBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3628,"released":"2025-06-12","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/VRBench"},{"aliases":[],"benchmark_id":"claire-radar-2609-12404","caveat":"Evaluates agents on computer control tasks under finite trial budgets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12404"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VRL-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3629,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12404"},{"aliases":[],"benchmark_id":"claire-radar-2607-08112","caveat":"A large-scale dataset for visual speech recognition in Romanian, with 200 hours of video and annotations. It studies supervision quality, robustness under domain shift, and multimodal fusion.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08112"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VSRo-200","organization_count":1,"organizations":["claire_radar"],"rank":3630,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08112"},{"aliases":[],"benchmark_id":"claire-radar-2608-24228","caveat":"VTC-Bench evaluates multiple LLM generations across five domains using Validated Task Coverage (VTC), which measures the number of distinct useful results obtained within k attempts. Tasks are selected from real data where output quality and task-relevant distinctness can be checked automatically without model-based judges.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.24228"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VTC-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3631,"released":"2026-08-25","source":"claire_radar","url":"https://arxiv.org/abs/2608.24228"},{"aliases":[],"benchmark_id":"claire-radar-2608-00692","caveat":"Vul4Py evaluates automated vulnerability repair in Python across 100 real vulnerabilities from 60 open-source projects, with paired exploit and functional oracles.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00692"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"Vul4Py","organization_count":1,"organizations":["claire_radar"],"rank":3632,"released":"2026-08-01","source":"claire_radar","url":"https://arxiv.org/abs/2608.00692"},{"aliases":[],"benchmark_id":"claire-radar-2607-00107","caveat":"VULBENCH-CPP evaluates the safety of AI-generated C++ code using multi-tier verification including functional testing, static analysis, dynamic analysis, and bounded model checking. It includes 8,918 programs from three LLMs and human authors.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00107"],"document_share":0.0002723311546840959,"domain":"Code generation","name":"VULBENCH-CPP","organization_count":1,"organizations":["claire_radar"],"rank":3633,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2607.00107"},{"aliases":[],"benchmark_id":"claire-radar-2609-15939","caveat":"A benchmark of 500 real-world vulnerabilities requiring models to locate security-relevant files in repository snapshots, also testing appropriate abstention on patched versions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.15939"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"Vulnerability Localization Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3634,"released":"2026-09-14","source":"claire_radar","url":"https://arxiv.org/abs/2609.15939"},{"aliases":[],"benchmark_id":"claire-radar-2606-25592","caveat":"A safety benchmark for image-to-video systems exposed to categorized attacks encoded through visual prompts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.25592"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VVA-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3635,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.25592"},{"aliases":[],"benchmark_id":"claire-radar-2607-10308","caveat":"VVM-Bench evaluates Large Multimodal Models on semantic perception and modality understanding across six real and synthetic modalities, using multiple-choice questions and generation tasks to assess zero-shot generalization to unseen visual modalities.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.10308"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VVM-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3636,"released":"2026-07-11","source":"claire_radar","url":"https://arxiv.org/abs/2607.10308"},{"aliases":[],"benchmark_id":"claire-radar-2609-35641","caveat":"Evaluates image generation models on 10,000 synthetic scene tasks with 32 verifiable constraint types measuring precise visual instruction following.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35641"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VVRBench","organization_count":1,"organizations":["claire_radar"],"rank":3637,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35641"},{"aliases":[],"benchmark_id":"claire-radar-2609-11242","caveat":"VWG-Bench evaluates image-to-video generation models on rule-following, physical and commonsense constraints, and goal realization over time, using 380 annotated initial images across 9 reasoning dimensions and 38 ten-sample task groups, scored with bundled 1–5 metrics and an evaluation toolkit.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.11242"],"document_share":0.0002723311546840959,"domain":"General AI","name":"VWG-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3638,"released":"2026-09-10","source":"claire_radar","url":"https://arxiv.org/abs/2609.11242"},{"aliases":[],"benchmark_id":"claire-radar-2608-22950","caveat":"A reasoning-annotated benchmark for floating waste detection and grounding, with 2,167 images, 13,608 bounding boxes, and ten categories, each with recognition rules.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22950v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WADE","organization_count":1,"organizations":["claire_radar"],"rank":3639,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.22950v1"},{"aliases":[],"benchmark_id":"claire-radar-2607-08729","caveat":"A benchmark for long-term multi-object tracking of Trichogramma wasps in controlled ecological experiments, with 10 sequences of ~12,000 frames each and dense annotations. It evaluates identity preservation over extended durations.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.08729"],"document_share":0.0002723311546840959,"domain":"Software & AI Compute","name":"WaspMOT","organization_count":1,"organizations":["claire_radar"],"rank":3640,"released":"2026-07-09","source":"claire_radar","url":"https://arxiv.org/abs/2607.08729"},{"aliases":[],"benchmark_id":"claire-radar-2606-26443","caveat":"WatchAct evaluates robot manipulation from observed human behavior. Each instance pairs a human-action video and a language instruction with an aligned simulator scene and an executable LIBERO task. It covers 3,000 long-horizon instances across 14 tasks in four capability domains: Event Grounding, Procedural Reasoning, Implicit Intent Inference, and Episodic Reasoning. The evaluation protocol separately measures video-to-plan reasoning, policy execution under oracle plans, and full task completion, in simulation and on a Franka Research 3 robot.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.26443"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"WatchAct","organization_count":1,"organizations":["claire_radar"],"rank":3641,"released":"2026-06-24","source":"claire_radar","url":"https://arxiv.org/abs/2606.26443"},{"aliases":[],"benchmark_id":"claire-radar-2607-20638","caveat":"WaveformQA is a QA benchmark for LLM temporal reasoning over digital waveforms, comprising 360 questions with programmatically generated ground truths across eight categories, including multi-signal correlation and event ordering, with waveforms generated from open-source designs.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.20638"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"WaveformQA","organization_count":1,"organizations":["claire_radar"],"rank":3642,"released":"2026-07-22","source":"claire_radar","url":"https://arxiv.org/abs/2607.20638"},{"aliases":[],"benchmark_id":"claire-radar-2605-25874","caveat":"WBench evaluates interactive video world models across five dimensions: video quality, setting adherence, interaction adherence, consistency, and physics compliance. It includes 289 test cases and 1,058 multi-turn interaction sequences, covering diverse scenes and control types, with 22 automatic sub-metrics validated against human judgment.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.25874"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WBench","organization_count":1,"organizations":["claire_radar"],"rank":3643,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.25874"},{"aliases":["FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches"],"benchmark_id":"claire-radar-2607-17765","caveat":"WC2026-Agents evaluates autonomous LLM forecasting on 104 FIFA World Cup matches using probability forecasts, virtual wagers, and post-match reflections.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.17765"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"WC2026-Agents","organization_count":1,"organizations":["claire_radar"],"rank":3644,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.17765"},{"aliases":[],"benchmark_id":"llm-stats-we-math","caveat":"We-Math evaluates multimodal models on visual mathematical reasoning, requiring models to understand and solve math problems presented with visual elements such as diagrams, charts, and geometric figures.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/we-math?top_n=500"],"document_share":0.0002723311546840959,"domain":"math","name":"We-Math","organization_count":1,"organizations":["llm_stats"],"rank":3645,"released":"2024-07-01","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/we-math?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-05405","caveat":"WearableQA evaluates AI reasoning over longitudinal wearable sensor data. It comprises 4,084 multiple-choice questions derived from 200 real users' wearable time series, blood biomarkers, and demographics. Questions are organized into 16 types along data versus health reasoning and single- versus cross-signal axes. Scoring uses multiple-choice accuracy with a known chance baseline.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05405"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WearableQA","organization_count":1,"organizations":["claire_radar"],"rank":3646,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05405"},{"aliases":[],"benchmark_id":"claire-radar-2606-09426","caveat":"WeaveBench evaluates computer-use agents on 114 long-horizon tasks across 8 real-world work domains. Each task interleaves GUI interaction with command-line and code operations in a single trajectory, with scoring based on a trajectory-aware judge that detects fabricated evidence.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.09426"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WeaveBench","organization_count":1,"organizations":["claire_radar"],"rank":3647,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.09426"},{"aliases":[],"benchmark_id":"llm-stats-web-bench","caveat":"Web Bench evaluates agents on realistic web-development engineering tasks, measuring end-to-end implementation in browser-based workflows.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/web-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"Web Bench","organization_count":1,"organizations":["llm_stats"],"rank":3648,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/web-bench?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-webarena-verified","caveat":"WebArena-Verified evaluates browser agents on realistic web tasks using a verified task set and execution-based grading.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/webarena-verified?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"WebArena-Verified","organization_count":1,"organizations":["llm_stats"],"rank":3649,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/webarena-verified?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-webdev-arena","caveat":"WebDev Arena is a leaderboard for evaluating AI models on web development tasks, including zero-shot generation, complex prompts, and interactive web UI creation. Models are ranked using Elo ratings based on their performance in coding and web development challenges.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/webdev-arena?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"WebDev Arena","organization_count":1,"organizations":["llm_stats"],"rank":3650,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/webdev-arena?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-23067","caveat":"A benchmark for evaluating agent skills in web development, comparing matched conditions with length-matched controls and ablation studies on 31 skills across 50 projects.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.23067v1"],"document_share":0.0002723311546840959,"domain":"Agents","name":"WebDev-Skills-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3651,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.23067v1"},{"aliases":["WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing"],"benchmark_id":"claire-radar-2609-05171","caveat":"Evaluates multimodal agentic image generation and editing, focusing on knowledge-intensive generation and multi-image editing across bilingual prompts.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.05171"],"document_share":0.0002723311546840959,"domain":"Agents","name":"WeBench-MMGenEdit","organization_count":1,"organizations":["claire_radar"],"rank":3652,"released":"2026-09-04","source":"claire_radar","url":"https://arxiv.org/abs/2609.05171"},{"aliases":[],"benchmark_id":"claire-radar-2606-00154","caveat":"WebIGBench evaluates code generation for interactive webpages with 103 complex examples and 871 distinct actions, proposing an automated evaluation pipeline for interactive consistency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.00154"],"document_share":0.0002723311546840959,"domain":"Code","name":"WebIGBench","organization_count":1,"organizations":["claire_radar"],"rank":3653,"released":"2026-05-29","source":"claire_radar","url":"https://arxiv.org/abs/2606.00154"},{"aliases":["Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents"],"benchmark_id":"claire-radar-2609-27353","caveat":"Offline benchmark with 541 tasks and 5,293 steps derived from successful WebArena trajectories, with audited test labels and a deterministic protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27353"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"WebMRE","organization_count":1,"organizations":["claire_radar"],"rank":3654,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27353"},{"aliases":["WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents"],"benchmark_id":"claire-radar-2609-35026","caveat":"An open framework for evaluating web agents, with 152 tasks across six instrumented mock sites, using event-log verification and controlled UI variants.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.35026"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WebPageBench","organization_count":1,"organizations":["claire_radar"],"rank":3655,"released":"2026-09-28","source":"claire_radar","url":"https://arxiv.org/abs/2609.35026"},{"aliases":[],"benchmark_id":"claire-radar-2607-06118","caveat":"Introduces a benchmark with 800 websites and 1,550 tasks for web agent evaluation, plus the NavEval LLM-as-Judge framework and three evaluation protocols.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.06118"],"document_share":0.0002723311546840959,"domain":"Agents","name":"WebRetriever","organization_count":1,"organizations":["claire_radar"],"rank":3656,"released":"2026-07-07","source":"claire_radar","url":"https://arxiv.org/abs/2607.06118"},{"aliases":[],"benchmark_id":"opencompass-1982-webui-bench","caveat":"WebUIBench is a comprehensive benchmark designed to evaluate Multimodal Large Language Models (MLLMs) in Web UI-to-code generation tasks across four key capabilities: UI perception, HTML programming, UI-code understanding, and end-to-end transformation. WebUIBench 是一个面向多模态大语言模型（MLLMs）的综合性评测基准，旨在系统评估模型在 Web UI 到代码生成任务中的四个关键能力：界面感知、HTML 编程、界面-代码理解以及整体转换能力。该基准包含来自 700 多个真实网站的 21,000 个高质量问答对，支持对模型在各阶段的细粒度能力分析。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WebUI-Bench"],"document_share":0.0002723311546840959,"domain":"代码","name":"WebUI-Bench","organization_count":1,"organizations":["opencompass_hub"],"rank":3657,"released":"2025-06-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WebUI-Bench"},{"aliases":[],"benchmark_id":"claire-radar-2610-02617","caveat":"Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that mis…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.02617"],"document_share":0.0002723311546840959,"domain":"Agents","name":"WebUIProof","organization_count":1,"organizations":["claire_radar"],"rank":3658,"released":"2026-10-02","source":"claire_radar","url":"https://arxiv.org/abs/2610.02617"},{"aliases":[],"benchmark_id":"llm-stats-webvoyager","caveat":"WebVoyager evaluates an agent's ability to navigate and complete tasks on real websites by perceiving page screenshots and executing browser actions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/webvoyager?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"WebVoyager","organization_count":1,"organizations":["llm_stats"],"rank":3659,"released":"2024-01-25","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/webvoyager?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-03499","caveat":"WeClawArena is a benchmark and runtime sandbox for multi-party owned-agent collaboration over personal workspaces. It contains 124 base tasks across six domains, expanded into 620 scenario variants with benign and attack-vector conditions, and reports task success and attack success separately.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.03499"],"document_share":0.0002723311546840959,"domain":"Agents","name":"WeClawArena","organization_count":1,"organizations":["claire_radar"],"rank":3660,"released":"2026-08-04","source":"claire_radar","url":"https://arxiv.org/abs/2608.03499"},{"aliases":[],"benchmark_id":"claire-radar-2606-20100","caveat":"WeGenBench evaluates text-to-image generation across 4,000 bilingual prompts, with multi-dimensional tags and novel VLM-based metrics. It assesses generation quality on scene classification and specific sub-categories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.20100"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WeGenBench","organization_count":1,"organizations":["claire_radar"],"rank":3661,"released":"2026-06-18","source":"claire_radar","url":"https://arxiv.org/abs/2606.20100"},{"aliases":[],"benchmark_id":"claire-radar-2608-15092","caveat":"WeSCE is a benchmark for quantifying security drift in LLM-driven code editing. It consists of 400 executable programs derived from real-world code, covering feature addition, removal, bug fixing, and refactoring. The benchmark proposes a continuous risk representation and drift measures that capture changes in overall risk, worst-case severity, and vulnerability distribution.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15092"],"document_share":0.0002723311546840959,"domain":"Cybersecurity","name":"WeSCE","organization_count":1,"organizations":["claire_radar"],"rank":3662,"released":"2026-08-15","source":"claire_radar","url":"https://arxiv.org/abs/2608.15092"},{"aliases":["The Illusion of $\\textit{What If}$: Evaluating the Breakdown of Counterfactual Reasoning in LLMs"],"benchmark_id":"claire-radar-2608-27953","caveat":"WhatIfBench is an open-domain, open-form, long-horizon counterfactual causal reasoning benchmark with 220 what-if questions across STEM, HSS, and Hybrid scenarios, evaluated using the PRISM framework.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.27953"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WhatIfBench","organization_count":1,"organizations":["claire_radar"],"rank":3663,"released":"2026-08-28","source":"claire_radar","url":"https://arxiv.org/abs/2608.27953"},{"aliases":[],"benchmark_id":"claire-radar-github-ethanning-whatworkedbench","caveat":"Evaluates AI agents on predicting component-change effects in 36 executable workflow tasks across 8 families using 30 data sources and 1,248 native outcomes.","document_count":1,"document_ids":["claire_radar:https://github.com/EthanNing/WhatWorkedBench"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"WhatWorkedBench","organization_count":1,"organizations":["claire_radar"],"rank":3664,"released":"2026-09-22","source":"claire_radar","url":"https://github.com/EthanNing/WhatWorkedBench"},{"aliases":[],"benchmark_id":"claire-radar-2607-05189","caveat":"WhisperBench evaluates stealth memory injection attacks on persistent personal agents through a 108-case benchmark spanning five risk categories with fact and preference poisoning, using an IMAP/SMTP workflow and an email agent skill.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.05189"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WhisperBench","organization_count":1,"organizations":["claire_radar"],"rank":3665,"released":"2026-07-06","source":"claire_radar","url":"https://arxiv.org/abs/2607.05189"},{"aliases":[],"benchmark_id":"claire-radar-2607-09996","caveat":"Who&When Pro is a benchmark for automated failure attribution in agentic systems, containing 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks. It uses a controlled pipeline that injects failures after replaying successful prefixes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09996"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Who&When Pro","organization_count":1,"organizations":["claire_radar"],"rank":3666,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.09996"},{"aliases":[],"benchmark_id":"opencompass-1279-whodunitbench","caveat":"WhodunitBench is used to evaluate large multimodal agent under complex tasks and dynamic scenarios. WhodunitBench用于评估大型多模式代理在复杂任务场景下的动态评估。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WhodunitBench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"WhodunitBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3667,"released":"2024-09-26","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WhodunitBench"},{"aliases":[],"benchmark_id":"opencompass-504-wic","caveat":"WiC is a benchmark for the evaluation of context-sensitive word embeddings. WiC is framed as a binary classification task. Each instance in WiC has a target word w, either a verb or a noun, for which two contexts are provided. Each of these contexts triggers a specific meaning of w. The task is to identify if the occurrences of w in the two contexts correspond to the same meaning or not. In fact, the dataset can also be viewed as an application of Word Sense Disambiguation in practise. Word-in-Context是一个词义消歧任务，被视为句子对的二元分类。给定两个文本片段和一个在两个句子中都出现的多义词，任务是确定该词在两个句子中是否具有相同的含义。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WiC"],"document_share":0.0002723311546840959,"domain":"语言","name":"WiC","organization_count":1,"organizations":["opencompass_hub"],"rank":3668,"released":"2018-08-28","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WiC"},{"aliases":[],"benchmark_id":"llm-stats-widesearch","caveat":"WideSearch is an agentic search benchmark that evaluates models' ability to perform broad, parallel search operations across multiple sources. It tests wide-coverage information retrieval and synthesis capabilities.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/widesearch?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"WideSearch","organization_count":1,"organizations":["llm_stats"],"rank":3669,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/widesearch?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2609-24094","caveat":"Evaluates agentic visual analytics tasks involving single- and multi-widget interactions, with benchmark instances released as data archives and a widget-centered runtime for comparisons.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24094"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WidgetVABench","organization_count":1,"organizations":["claire_radar"],"rank":3670,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24094"},{"aliases":[],"benchmark_id":"opencompass-1318-wikicontradict","caveat":"WikiContradict is a benchmark consisting of 253 high-quality, human-annotated instances designed to assess LLM performance when augmented with retrieved passages containing real-world knowledge conflicts. WikiContradict旨在评估LLM遇到包含真实世界知识冲突的段落检索增强时的性能，由253个高质量的人工注释实例组成。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WikiContradict"],"document_share":0.0002723311546840959,"domain":"推理","name":"WikiContradict","organization_count":1,"organizations":["opencompass_hub"],"rank":3671,"released":"2024-06-19","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WikiContradict"},{"aliases":[],"benchmark_id":"opencompass-1129-wikisql","caveat":"WikiSQL is a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia that is an order of magnitude larger than comparable datasets. WikiSQL 是一个包含 80,654 个手动标注示例的问题和 SQL 查询的数据集，分布在来自维基百科的 24,241 个表格中。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WikiSQL"],"document_share":0.0002723311546840959,"domain":"其他","name":"WikiSQL","organization_count":1,"organizations":["opencompass_hub"],"rank":3672,"released":"2017-11-09","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WikiSQL"},{"aliases":[],"benchmark_id":"llm-stats-wild-bench","caveat":"WildBench is an automated evaluation framework that benchmarks large language models using 1,024 challenging, real-world tasks selected from over one million human-chatbot conversation logs. It introduces two evaluation metrics (WB-Reward and WB-Score) that achieve high correlation with human preferences and uses task-specific checklists for systematic evaluation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wild-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Wild Bench","organization_count":1,"organizations":["llm_stats"],"rank":3673,"released":"2024-06-07","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/wild-bench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1555-wildbench","caveat":"Weintroduce WildBench, an automated evaluation framework designed to bench-mark large language models (LLMs) using challenging, real-world user queries, which consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. WildBench推出自动评估框架和数据集，基于真实用户难题评测大语言模型，包含从逾百万人机对话日志中精选的1,024个任务样本。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WildBench"],"document_share":0.0002723311546840959,"domain":"理解","name":"WildBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3674,"released":"2024-06-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WildBench"},{"aliases":["WildBox: A Dataset and Benchmark for Aerial Monocular 3D Detection of African Savanna Wildlife"],"benchmark_id":"claire-radar-2606-21309","caveat":"WildBox evaluates monocular 3D wildlife detection from drone video across six benchmark classes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.21309"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WildBox","organization_count":1,"organizations":["claire_radar"],"rank":3675,"released":"2026-06-19","source":"claire_radar","url":"https://arxiv.org/abs/2606.21309"},{"aliases":[],"benchmark_id":"llm-stats-wildclawbench","caveat":"WildClawBench is an agentic coding benchmark from InternLM/Claw-Eval that reports overall model performance on real-world tool-using development tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wildclawbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"WildClawBench","organization_count":1,"organizations":["llm_stats"],"rank":3676,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/wildclawbench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-2445-wildclawbench","caveat":"WildClawBench evaluates AI agents across six task categories: productivity flow, code intelligence, social interaction, search & retrieval, creative synthesis, and safety alignment. It comprises 60 hand-authored tasks inside a live environment with real tools — browser, bash, file system, etc. WildClawBench 从六个任务维度评测 AI 智能体：生产力流程、代码智能、社交互动、搜索与检索、创意合成和安全对齐。基准包含 60 个人工设计的任务，运行在配备真实工具（浏览器、bash、文件系统、电子邮件等）的真实环境中。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WildClawBench"],"document_share":0.0002723311546840959,"domain":"智能体","name":"WildClawBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3677,"released":"2026-04-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WildClawBench"},{"aliases":[],"benchmark_id":"claire-radar-2608-22959","caveat":"A benchmark containing 500 handwritten documents across three structures, four languages, and nine real-world scenarios, with a Prior-Driven Error metric to quantify reliance on language priors.","document_count":1,"document_ids":["claire_radar:http://arxiv.org/abs/2608.22959v1"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WildHandBench","organization_count":1,"organizations":["claire_radar"],"rank":3678,"released":"2026-08-24","source":"claire_radar","url":"http://arxiv.org/abs/2608.22959v1"},{"aliases":[],"benchmark_id":"claire-radar-2607-09328","caveat":"WildTrace evaluates long-context reasoning over naturally occurring long-form sources, focusing on integrating evidence dispersed across distant passages. It includes 481 tasks over 214 sources with seven source-internal evidence geometries, with multi-stage validation ensuring answer groundedness and clue necessity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.09328"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WildTrace","organization_count":1,"organizations":["claire_radar"],"rank":3679,"released":"2026-07-10","source":"claire_radar","url":"https://arxiv.org/abs/2607.09328"},{"aliases":[],"benchmark_id":"llm-stats-winogrande","caveat":"WinoGrande: An Adversarial Winograd Schema Challenge at Scale. A large-scale dataset of 44,000 pronoun resolution problems designed to test machine commonsense reasoning. Uses adversarial filtering to reduce spurious biases and provides a more robust evaluation of whether AI systems truly understand commonsense or exploit statistical shortcuts. Current best AI methods achieve 59.4-79.1% accuracy, significantly below human performance of 94.0%.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/winogrande?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Winogrande","organization_count":1,"organizations":["llm_stats"],"rank":3680,"released":"2019-11-21","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/winogrande?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1109-winogrande","caveat":"WINOGRANDE is a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. WINOGRANDE 包含 44,000 个问题，受到 WSC 设计的启发，但进行了调整，以提高数据集的规模和难度。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WinoGrande"],"document_share":0.0002723311546840959,"domain":"推理","name":"WinoGrande","organization_count":1,"organizations":["opencompass_hub"],"rank":3681,"released":"2019-11-21","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WinoGrande"},{"aliases":[],"benchmark_id":"claire-radar-2606-18097","caveat":"WireCraft is a simulation benchmark for industrial deformable linear object manipulation, featuring three task families (connector insertion, clip routing, channel seating) with configurable difficulty, two DLO physics models, and shared evaluation metrics for RL, IL, and VLA policies across simulation and physical UR5 trajectories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18097"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"WireCraft","organization_count":1,"organizations":["claire_radar"],"rank":3682,"released":"2026-06-16","source":"claire_radar","url":"https://arxiv.org/abs/2606.18097"},{"aliases":[],"benchmark_id":"llm-stats-wmdp","caveat":"Weapons of Mass Destruction (WMDP) is a multiple-choice benchmark on dual-use biology, chemistry, and cyber knowledge. It measures a model's capacity to enable malicious actors to design, synthesize, acquire, or use chemical, biological, radiological, or nuclear (CBRN) weapons.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wmdp?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"WMDP","organization_count":1,"organizations":["llm_stats"],"rank":3683,"released":"2024-03-05","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/wmdp?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-wmt23","caveat":"The Eighth Conference on Machine Translation (WMT23) benchmark evaluating machine translation systems across 8 language pairs (14 translation directions) including general, biomedical, literary, and low-resource language translation tasks. Features specialized shared tasks for quality estimation, metrics evaluation, sign language translation, and discourse-level literary translation with professional human assessment.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wmt23?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"WMT23","organization_count":1,"organizations":["llm_stats"],"rank":3684,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt23?top_n=500"},{"aliases":[],"benchmark_id":"llm-stats-wmt24","caveat":"WMT24++ is a comprehensive multilingual machine translation benchmark that expands the WMT24 dataset to cover 55 languages and dialects. It includes human-written references and post-edits across four domains (literary, news, social, and speech) to evaluate machine translation systems and large language models across diverse linguistic contexts.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/wmt24%2B%2B?top_n=500"],"document_share":0.0002723311546840959,"domain":"language","name":"WMT24++","organization_count":1,"organizations":["llm_stats"],"rank":3685,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/wmt24%2B%2B?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-13715","caveat":"WorkBench evaluates workplace agents by task completion and unintended harmful actions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.13715"],"document_share":0.0002723311546840959,"domain":"Agents","name":"WorkBench","organization_count":1,"organizations":["claire_radar"],"rank":3686,"released":"2026-06-10","source":"claire_radar","url":"https://arxiv.org/abs/2606.13715"},{"aliases":[],"benchmark_id":"claire-radar-2610-11561","caveat":"LLM-based agents now interact with their environments continuously, shaped by such organizational channels as user instructions, peer messages, and long-term memory.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.11561"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Workerville","organization_count":1,"organizations":["claire_radar"],"rank":3687,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.11561"},{"aliases":[],"benchmark_id":"claire-radar-2606-11042","caveat":"Workflow-GYM evaluates AI agents on long-horizon GUI tasks in professional domains, using specialized software environments and economically valuable workflows. Tasks require end-to-end operation of graphical user interfaces, with success rates measured by task completion.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11042"],"document_share":0.0002723311546840959,"domain":"General AI","name":"Workflow-GYM","organization_count":1,"organizations":["claire_radar"],"rank":3688,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11042"},{"aliases":[],"benchmark_id":"llm-stats-workspace-bench","caveat":"Workspace Bench evaluates AI agents on high-economic-value workplace tasks that span multi-step planning, file processing, and tool use across realistic office and productivity workflows.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/workspace-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"Workspace Bench","organization_count":1,"organizations":["llm_stats"],"rank":3689,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/workspace-bench?top_n=500"},{"aliases":["Workspace-Bench","Workspace Bench","Workspace-Bench 1.0"],"benchmark_id":"workspacebench","caveat":"Large-scale file-workspace tasks; the environment and file dependencies are part of the measurement.","document_count":1,"document_ids":["model_reports:tencent_hy4_preview"],"document_share":0.0002723311546840959,"domain":"professional","name":"WorkspaceBench","organization_count":1,"organizations":["Tencent"],"rank":3690,"released":"2026-05-05","source":"model_reports","url":"https://arxiv.org/abs/2605.03596"},{"aliases":[],"benchmark_id":"claire-radar-2607-25765","caveat":"WorkSurface-Bench evaluates enterprise agents on knowledge routing across documents, tables, and graphs. It includes 1,151 atomic tasks with auditable reference answers and scoring for route, evidence, answer, and efficiency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.25765"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"WorkSurface-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3691,"released":"2026-07-28","source":"claire_radar","url":"https://arxiv.org/abs/2607.25765"},{"aliases":["World Models' Last Exam in Physics","Physics-Last-Exam","phys-last-exam"],"benchmark_id":"world_models_last_exam_physics","caveat":"Forty controlled image-to-video tasks across nine physical families. Overall is a video-weighted composite score, S = 0.15C + 0.85P × I(C ≥ 80), on 0–100; it is not a success rate. Four samples are generated per model-task pair, with different resolutions and frame counts. Seedance uses sample labels rather than transmitted random seeds. Visibility and measurement robustness limit interpretation; empirical difficulty groups are derived from the evaluated models.","document_count":1,"document_ids":["model_reports:world_models_last_exam_physics_v1_results_table"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"World Models’ Last Exam in Physics","organization_count":1,"organizations":["Navers Lab / Einsia.AI"],"rank":3692,"released":"2026-10-06","source":"model_reports","url":"https://lab.einsia.ai/phys-last-exam/"},{"aliases":[],"benchmark_id":"claire-radar-2609-40325","caveat":"As interactive 3D worlds are increasingly used to study intelligent behavior, it becomes important to develop efficient pipelines for identifying anomalies in these simulated environments, such as floating objects, traversable walls, or ob…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.40325"],"document_share":0.0002723311546840959,"domain":"Agents & Tool Use","name":"WorldAuditBench","organization_count":1,"organizations":["claire_radar"],"rank":3693,"released":"2026-09-30","source":"claire_radar","url":"https://arxiv.org/abs/2609.40325"},{"aliases":[],"benchmark_id":"claire-radar-2606-06538","caveat":"WorldBench evaluates multimodal large language models on visually diverse reasoning questions, with accuracy as the metric on a curated dataset.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.06538"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WorldBench","organization_count":1,"organizations":["claire_radar"],"rank":3694,"released":"2026-06-04","source":"claire_radar","url":"https://arxiv.org/abs/2606.06538"},{"aliases":[],"benchmark_id":"claire-radar-2609-01056","caveat":"A multilingual benchmark of persona-grounded everyday workflows for agents, evaluated via sandbox actions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.01056"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WorldBench","organization_count":1,"organizations":["claire_radar"],"rank":3695,"released":"2026-09-01","source":"claire_radar","url":"https://arxiv.org/abs/2609.01056"},{"aliases":[],"benchmark_id":"llm-stats-worldbench","caveat":"WorldBench evaluates real-world visual knowledge and understanding across diverse everyday scenes.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/worldbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"WorldBench","organization_count":1,"organizations":["llm_stats"],"rank":3696,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/worldbench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2606-01869","caveat":"Evaluates autonomous physically grounded 3D world synthesis from natural language, with 2,026 expert-curated tasks across Simulation, Rendering, and Application scenarios, using execution-based verification via StateProbe to check hidden behavioral contracts over runtime states.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.01869"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WorldCoder-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3697,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.01869"},{"aliases":[],"benchmark_id":"claire-radar-2607-18084","caveat":"WorldCupArena evaluates language models and deep-research agents on football forecasting across multiple tasks including result, score, player events, statistics, and tournament outcomes. It uses a composite score and supports adding new schedules for future events.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.18084"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WorldCupArena","organization_count":1,"organizations":["claire_radar"],"rank":3698,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.18084"},{"aliases":[],"benchmark_id":"claire-radar-2608-02603","caveat":"WorldExam evaluates controllable video generation models as world models across four levels: Visual Quality, Control Adherence, Spatial Consistency, and World Reactivity, with 1,474 cases across eight tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.02603"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WorldExam","organization_count":1,"organizations":["claire_radar"],"rank":3699,"released":"2026-08-03","source":"claire_radar","url":"https://arxiv.org/abs/2608.02603"},{"aliases":[],"benchmark_id":"opencompass-1938-worldgenbench","caveat":null,"document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WorldGenBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"WorldGenBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3700,"released":"2025-06-16","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WorldGenBench"},{"aliases":[],"benchmark_id":"claire-radar-2606-18847","caveat":"WorldLines evaluates long-horizon stateful embodied agents in household environments through Memory QA and Embodied Task Planning, using temporally extended household traces with dialogues and state changes.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.18847"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"WorldLines","organization_count":1,"organizations":["claire_radar"],"rank":3701,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.18847"},{"aliases":[],"benchmark_id":"claire-radar-2606-11129","caveat":"WorldOlympiad is a triathlon-style benchmark for video-based world models, evaluating physical faithfulness, geometric consistency, and interaction fidelity across gaming, robotics, and general real-world scenarios. It includes 1,000 long videos and interpretable automatic metrics.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.11129"],"document_share":0.0002723311546840959,"domain":"Robotics & Embodied AI","name":"WorldOlympiad","organization_count":1,"organizations":["claire_radar"],"rank":3702,"released":"2026-06-09","source":"claire_radar","url":"https://arxiv.org/abs/2606.11129"},{"aliases":[],"benchmark_id":"claire-radar-2606-31672","caveat":"WorldRoamBench evaluates interactive world models on long-horizon stability across action, vision, physics, and memory dimensions. It includes 600+ test cases in various scenes and views with continuous interaction.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.31672"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WorldRoamBench","organization_count":1,"organizations":["claire_radar"],"rank":3703,"released":"2026-06-30","source":"claire_radar","url":"https://arxiv.org/abs/2606.31672"},{"aliases":[],"benchmark_id":"opencompass-1735-worldscore","caveat":"WorldScore benchmark is the first unified benchmark for world generation. WorldScore基准测试，这是首个用于世界生成的统一基准测试。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WorldScore"],"document_share":0.0002723311546840959,"domain":"创作","name":"WorldScore","organization_count":1,"organizations":["opencompass_hub"],"rank":3704,"released":"2025-04-01","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WorldScore"},{"aliases":[],"benchmark_id":"claire-radar-2610-08720","caveat":"LLM-based agents are increasingly advancing scientific and engineering problem solving, with physics simulation emerging as a challenging yet practical testbed for reproducing complex physical phenomena with application in embodied AI, gam…","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.08720"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WorldSolver","organization_count":1,"organizations":["claire_radar"],"rank":3705,"released":"2026-10-06","source":"claire_radar","url":"https://arxiv.org/abs/2610.08720"},{"aliases":[],"benchmark_id":"llm-stats-worldvqa","caveat":"WorldVQA is a benchmark designed to evaluate atomic vision-centric world knowledge. It assesses models' ability to understand and reason about visual elements representing real-world knowledge.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/worldvqa?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"WorldVQA","organization_count":1,"organizations":["llm_stats"],"rank":3706,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/worldvqa?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2610-12417","caveat":"Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2610.12417"],"document_share":0.0002723311546840959,"domain":"Multimodal","name":"WOVEN","organization_count":1,"organizations":["claire_radar"],"rank":3707,"released":"2026-10-08","source":"claire_radar","url":"https://arxiv.org/abs/2610.12417"},{"aliases":[],"benchmark_id":"claire-radar-2609-24444","caveat":"Wind power forecasting benchmark covering tasks and dataset scope detailed in the paper.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.24444"],"document_share":0.0002723311546840959,"domain":"Energy & Grid","name":"WPBench","organization_count":1,"organizations":["claire_radar"],"rank":3708,"released":"2026-09-21","source":"claire_radar","url":"https://arxiv.org/abs/2609.24444"},{"aliases":["wps-ai-benchmark-v6"],"benchmark_id":"claire-radar-huggingface-oursecurefuture-wps-ai-benchmark-v6","caveat":"Evaluates AI systems on applying Women, Peace and Security principles when answering operational advice prompts via 30 scenarios and an 11-criterion rubric with LLM judges.","document_count":1,"document_ids":["claire_radar:https://huggingface.co/datasets/OurSecureFuture/wps-ai-benchmark-v6"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WPS AI Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3709,"released":"2026-09-15","source":"claire_radar","url":"https://huggingface.co/datasets/OurSecureFuture/wps-ai-benchmark-v6"},{"aliases":[],"benchmark_id":"llm-stats-writingbench","caveat":"A comprehensive benchmark for evaluating large language models' generative writing capabilities across 6 core writing domains (Academic & Engineering, Finance & Business, Politics & Law, Literature & Art, Education, Advertising & Marketing) and 100 subdomains. Contains 1,239 queries with a query-dependent evaluation framework that dynamically generates 5 instance-specific assessment criteria for each writing task, using a fine-tuned critic model to score responses on style, format, and length dimensions.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/writingbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"legal","name":"WritingBench","organization_count":1,"organizations":["llm_stats"],"rank":3710,"released":"2025-03-07","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/writingbench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1701-writingbench","caveat":"WritingBench: A Comprehensive Benchmark for Generative Writing WritingBench: A Comprehensive Benchmark for Generative Writing","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WritingBench"],"document_share":0.0002723311546840959,"domain":"长文本","name":"WritingBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3711,"released":"2025-03-07","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WritingBench"},{"aliases":[],"benchmark_id":"claire-radar-2605-26068","caveat":"WSADBench is a benchmark for weakly supervised anomaly detection (WSAD) that unifies evaluation across incomplete, inexact, and inaccurate supervision scenarios. It evaluates 36 algorithms across 4 modalities (tabular, video, image features, text embeddings) by systematically varying label quantity, granularity, and quality, with protocols and code provided.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.26068"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WSADBench","organization_count":1,"organizations":["claire_radar"],"rank":3712,"released":"2026-05-25","source":"claire_radar","url":"https://arxiv.org/abs/2605.26068"},{"aliases":[],"benchmark_id":"opencompass-507-wsc","caveat":"WSC is a pronoun disambiguation task, which requires to determine which noun the pronoun refers to according to the context. WSC是一个代词消歧任务，要求根据上下文判断代词指代的是哪个名词。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/WSC"],"document_share":0.0002723311546840959,"domain":"语言","name":"WSC","organization_count":1,"organizations":["opencompass_hub"],"rank":3713,"released":null,"source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/WSC"},{"aliases":[],"benchmark_id":"claire-radar-2608-15654","caveat":"WSE-bench evaluates LLM storytelling in open-ended world simulations, assessing three capacities: Generation Coverage (proportion of planned narrative steps), Consistency (canon coherence), and Richness (meaningful development). Uses a process benchmark to compare agent architectures.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.15654"],"document_share":0.0002723311546840959,"domain":"Agents","name":"WSE-bench","organization_count":1,"organizations":["claire_radar"],"rank":3714,"released":"2026-08-16","source":"claire_radar","url":"https://arxiv.org/abs/2608.15654"},{"aliases":[],"benchmark_id":"claire-radar-2607-01728","caveat":"WUICC-bench evaluates image change captioning for web UI visual regression testing, using natural language descriptions of UI changes. Scoring includes caption quality metrics and assesses suppression of non-meaningful noise.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.01728"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WUICC-bench","organization_count":1,"organizations":["claire_radar"],"rank":3715,"released":"2026-07-02","source":"claire_radar","url":"https://arxiv.org/abs/2607.01728"},{"aliases":[],"benchmark_id":"claire-radar-2607-17745","caveat":"WuYu-EnvLE-Bench evaluates LLMs on environmental law enforcement with 2,521 instances across 14 tasks and 12 pollution-medium subdomains, covering pre-, in-, and post-enforcement workflows. It uses Absolute Environmental Enforcement Score (AES) and Intelligent Enforcement Index (IEI) for evaluation.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.17745"],"document_share":0.0002723311546840959,"domain":"General AI","name":"WuYu-EnvLE-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3716,"released":"2026-07-20","source":"claire_radar","url":"https://arxiv.org/abs/2607.17745"},{"aliases":[],"benchmark_id":"claire-radar-2608-07529","caveat":"A multi-level benchmark evaluates LLMs in solid waste management across foundational knowledge, domain reasoning, and expert decision-making using closed-ended and open-ended questions.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.07529"],"document_share":0.0002723311546840959,"domain":"Factuality","name":"WuYuEval","organization_count":1,"organizations":["claire_radar"],"rank":3717,"released":"2026-07-24","source":"claire_radar","url":"https://arxiv.org/abs/2608.07529"},{"aliases":[],"benchmark_id":"claire-radar-2606-19256","caveat":"X+Slides evaluates audience-conditioned slide generation from source documents, using a dynamic framework of 8,133 source-grounded probes across 113 topics and seven presentation scenes, reporting metrics like Audience Coverage and Efficiency.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.19256"],"document_share":0.0002723311546840959,"domain":"General AI","name":"X+Slides","organization_count":1,"organizations":["claire_radar"],"rank":3718,"released":"2026-06-17","source":"claire_radar","url":"https://arxiv.org/abs/2606.19256"},{"aliases":[],"benchmark_id":"claire-radar-2606-02482","caveat":"Evaluates multimodal large language models on multi-stream streaming understanding, with 4,220 QA pairs across 932 videos covering 11 subtasks in multi-window, multi-view, and multi-device scenarios, using a dual-verification construction pipeline and online inference under a fixed average video-token rate.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.02482"],"document_share":0.0002723311546840959,"domain":"Autonomous Driving","name":"X-Stream","organization_count":1,"organizations":["claire_radar"],"rank":3719,"released":"2026-06-01","source":"claire_radar","url":"https://arxiv.org/abs/2606.02482"},{"aliases":[],"benchmark_id":"claire-radar-2609-27449","caveat":"Simulation benchmark for real-world robot policy evaluation; no artifacts available.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.27449"],"document_share":0.0002723311546840959,"domain":"Robot manipulation","name":"X2Real","organization_count":1,"organizations":["claire_radar"],"rank":3720,"released":"2026-09-23","source":"claire_radar","url":"https://arxiv.org/abs/2609.27449"},{"aliases":[],"benchmark_id":"opencompass-1072-xcodeeval","caveat":"xCodeEval is the largest executable multilingual multitask benchmark to date consisting of 25M document-level coding examples (16.5B tokens) from about 7.5K unique problems covering up to 11 programming languages with execution-level parallelism. xCodeEval 是迄今为止最大的可执行多语言多任务基准，包含 2500 万个文档级编码示例（165 亿个标记），来自约 7500 个独特问题，涵盖多达 11 种编程语言。它包括 7 个任务，涉及代码理解、生成、翻译和检索。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/xCodeEval"],"document_share":0.0002723311546840959,"domain":"代码","name":"xCodeEval","organization_count":1,"organizations":["opencompass_hub"],"rank":3721,"released":"2023-11-06","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/xCodeEval"},{"aliases":[],"benchmark_id":"claire-radar-2609-07784","caveat":"Evaluates LLMs on 248 tasks across 51 scenarios in personal life, white-collar work, learning and research, and cross-domain activities using binary rubrics for explicit and implicit requirements.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.07784"],"document_share":0.0002723311546840959,"domain":"General AI","name":"xDailyBench","organization_count":1,"organizations":["claire_radar"],"rank":3722,"released":"2026-09-07","source":"claire_radar","url":"https://arxiv.org/abs/2609.07784"},{"aliases":[],"benchmark_id":"llm-stats-xdailybench","caveat":"xDailyBench evaluates AI agents on white-collar office work, covering everyday professional tasks such as document handling, consultation, and multi-step productivity workflows.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/xdailybench?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"xDailyBench","organization_count":1,"organizations":["llm_stats"],"rank":3723,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/xdailybench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2608-05799","caveat":"XEWorld is a testbed for evaluating cross-embodiment generalization of action-conditioned world models, but it is primarily a research probe without a defined public benchmark protocol.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.05799"],"document_share":0.0002723311546840959,"domain":"Robotics","name":"XEWorld","organization_count":1,"organizations":["claire_radar"],"rank":3724,"released":"2026-08-06","source":"claire_radar","url":"https://arxiv.org/abs/2608.05799"},{"aliases":[],"benchmark_id":"claire-radar-2607-23545","caveat":"Evaluates instruction hierarchy compliance in multilingual LLMs using same-language and cross-language conflicts across six languages, four domains (rule-following, safety, task-execution, persona), and three hierarchy types (system-user, system-tool, user-tool).","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.23545"],"document_share":0.0002723311546840959,"domain":"General AI","name":"XIH-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3725,"released":"2026-07-26","source":"claire_radar","url":"https://arxiv.org/abs/2607.23545"},{"aliases":[],"benchmark_id":"claire-radar-2608-00036","caveat":"XL-DocBench evaluates evidence-grounded long-document understanding with 1,519 human-verified questions from six professional domains, contexts up to 2,303 pages, multi-page evidence, and typed reasoning rules.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.00036"],"document_share":0.0002723311546840959,"domain":"Biology & Drug Discovery","name":"XL-DocBench","organization_count":1,"organizations":["claire_radar"],"rank":3726,"released":"2026-07-21","source":"claire_radar","url":"https://arxiv.org/abs/2608.00036"},{"aliases":[],"benchmark_id":"llm-stats-xlsum-english","caveat":"Large-scale multilingual abstractive summarization dataset comprising 1 million professionally annotated article-summary pairs from BBC, covering 44 languages. XL-Sum is highly abstractive, concise, and of high quality, designed to encourage research on multilingual abstractive summarization tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/xlsum-english?top_n=500"],"document_share":0.0002723311546840959,"domain":"summarization","name":"XLSum English","organization_count":1,"organizations":["llm_stats"],"rank":3727,"released":"2021-06-25","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/xlsum-english?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2607-03562","caveat":"XPlainVerse evaluates deepfake detection and explanation quality, pairing real images with forgeries from twelve models and providing technical and simplified explanations, with metrics EntityScore and EvidenceScore for reasoning fidelity.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.03562"],"document_share":0.0002723311546840959,"domain":"General AI","name":"XPlainVerse","organization_count":1,"organizations":["claire_radar"],"rank":3728,"released":"2026-07-03","source":"claire_radar","url":"https://arxiv.org/abs/2607.03562"},{"aliases":[],"benchmark_id":"claire-radar-2608-25939","caveat":"Evaluates multilingual repository-level unit test generation across Rust, Go, Julia, PHP, and Ruby using containerized execution and context augmentation strategies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.25939"],"document_share":0.0002723311546840959,"domain":"Code & Software","name":"XREPOTEST","organization_count":1,"organizations":["claire_radar"],"rank":3729,"released":"2026-08-26","source":"claire_radar","url":"https://arxiv.org/abs/2608.25939"},{"aliases":[],"benchmark_id":"llm-stats-xstest","caveat":"XSTest is a test suite designed to identify exaggerated safety behaviours in large language models. It comprises 450 prompts: 250 safe prompts across ten prompt types that well-calibrated models should not refuse to comply with, and 200 unsafe prompts as contrasts that models should refuse. The benchmark systematically evaluates whether models refuse to respond to clearly safe prompts due to overly cautious safety mechanisms.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/xstest?top_n=500"],"document_share":0.0002723311546840959,"domain":"safety","name":"XSTest","organization_count":1,"organizations":["llm_stats"],"rank":3730,"released":"2023-08-02","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/xstest?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-521-xsum","caveat":"XSum is a single-document summarization task which does not favor extractive strategies and calls for an abstractive modeling approach. The idea is to create a short, one-sentence news summary answering the question “What is the article about?”. The dataset collects real-world, large scale data by harvesting online articles from the British Broadcasting Corporation (BBC). XSum是一个单文档摘要任务，不支持抽取式策略，需要采用抽象建模方法。其思想是创建一个简短的一句话新闻摘要，回答“这篇文章是关于什么的？”的问题。该数据集通过从英国广播公司（BBC）收集在线文章，得到了大量的现实数据。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/XSum"],"document_share":0.0002723311546840959,"domain":"理解","name":"XSum","organization_count":1,"organizations":["opencompass_hub"],"rank":3731,"released":"2018-08-27","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/XSum"},{"aliases":[],"benchmark_id":"opencompass-1784-xverify","caveat":"xVerify, an efficient answer verifier for reasoning model evaluations. xVerify demonstrates strong capability in equivalence judgment, enabling it to effectively determine whether the answers produced by reasoning models are equivalent to reference answers across various types of objective questions xVerify，这是一种用于推理模型评估的高效答案验证器。xVerify 在等价判断方面表现出强大的能力，使其能够有效地确定推理模型生成的答案是否等同于各种类型客观问题的参考答案。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/xVerify"],"document_share":0.0002723311546840959,"domain":"推理","name":"xVerify","organization_count":1,"organizations":["opencompass_hub"],"rank":3732,"released":"2025-04-14","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/xVerify"},{"aliases":[],"benchmark_id":"claire-radar-2609-10153","caveat":"A benchmark for Arabic morphological generation across verbs, nouns, adjectives, cliticized forms, and invalid configurations, with over 600K entries.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10153"],"document_share":0.0002723311546840959,"domain":"General AI","name":"YallaMorph","organization_count":1,"organizations":["claire_radar"],"rank":3733,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10153"},{"aliases":[],"benchmark_id":"llm-stats-yc-bench","caveat":"YC-Bench evaluates agents on long-horizon, open-ended business and investment decision-making. The reported metric is the final assets (fund value, in US dollars) accumulated by the agent over the course of the simulation.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/yc-bench?top_n=500"],"document_share":0.0002723311546840959,"domain":"finance","name":"YC-Bench","organization_count":1,"organizations":["llm_stats"],"rank":3734,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/yc-bench?top_n=500"},{"aliases":[],"benchmark_id":"claire-radar-2605-30346","caveat":"YoCausal is a two-level benchmark that evaluates video diffusion models' understanding of temporal causality using the Violation of Expectation paradigm. It temporally reverses real-world videos as counterfactual samples and introduces two metrics: the Reverse Surprise Index (RSI) for arrow-of-time perception and the Causality Cognition Index (CCI) for disentangling genuine causal reasoning from temporal bias.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2605.30346"],"document_share":0.0002723311546840959,"domain":"General AI","name":"YoCausal","organization_count":1,"organizations":["claire_radar"],"rank":3735,"released":"2026-05-28","source":"claire_radar","url":"https://arxiv.org/abs/2605.30346"},{"aliases":[],"benchmark_id":"claire-radar-2607-00664","caveat":"YOMI-Bench evaluates kanji reading and phonological understanding in LLMs for Japanese through four tasks. It is used in a study assessing multilingual and Japanese-specific models.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2607.00664"],"document_share":0.0002723311546840959,"domain":"General AI","name":"YOMI-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3736,"released":"2026-07-01","source":"claire_radar","url":"https://arxiv.org/abs/2607.00664"},{"aliases":[],"benchmark_id":"opencompass-1017-yue-benchmark","caveat":"The benchmarks introduced for evaluating large language models (LLMs) on Cantonese include Yue-TruthfulQA, Yue-GSM8K, Yue-ARC-C, Yue-MMLU, and Yue-TRANS. Each of these benchmarks focuses on different aspects of language understanding and generation in Cantonese, offering a comprehensive means of assessing the capabilities of LLMs in handling this language. Yue_Benchmark用于评估粤语大型语言模型（LLMs）。该评测集包含：Yue TruthtyQA、Yue-GSM8K、Yue-ARC-C、Yue MMLU和Yue TRANS，侧重于粤语语言理解和生成的不同方面，为评估LLM粤语能力提供了一种全面的方法。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/Yue_Benchmark"],"document_share":0.0002723311546840959,"domain":"语言","name":"Yue_Benchmark","organization_count":1,"organizations":["opencompass_hub"],"rank":3737,"released":"2024-08-31","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/Yue_Benchmark"},{"aliases":[],"benchmark_id":"llm-stats-zclawbench","caveat":"ZClawBench evaluates Claw-style agent task execution quality, measuring a model's ability to autonomously complete complex multi-step coding tasks in real-world environments.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/zclawbench?top_n=500"],"document_share":0.0002723311546840959,"domain":"agents","name":"ZClawBench","organization_count":1,"organizations":["llm_stats"],"rank":3738,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/zclawbench?top_n=500"},{"aliases":["Disparity Has a Sign: Stereo Matching Beyond the Zero-Disparity Plane"],"benchmark_id":"claire-radar-2609-06809","caveat":"ZDPShift evaluates stereo matching on 21,495 pairs with dense signed ground truth from seven open movies.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.06809"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ZDPShift","organization_count":1,"organizations":["claire_radar"],"rank":3739,"released":"2026-09-06","source":"claire_radar","url":"https://arxiv.org/abs/2609.06809"},{"aliases":[],"benchmark_id":"llm-stats-zebralogic","caveat":"ZebraLogic is an evaluation framework for assessing large language models' logical reasoning capabilities through logic grid puzzles derived from constraint satisfaction problems (CSPs). The benchmark consists of 1,000 programmatically generated puzzles with controllable and quantifiable complexity, revealing a 'curse of complexity' where model accuracy declines significantly as problem complexity grows.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/zebralogic?top_n=500"],"document_share":0.0002723311546840959,"domain":"reasoning","name":"ZebraLogic","organization_count":1,"organizations":["llm_stats"],"rank":3740,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/zebralogic?top_n=500"},{"aliases":["zero-trust-agent-benchmark"],"benchmark_id":"claire-radar-github-zero-trust-agent-benchmark-zero-trust-agent-benchmark","caveat":"A reproducible benchmark of synthetic JSONL agent tool-call traces in which a defense returns allow or deny per step; the evaluator scores attack blocking, benign false positives, and canary leaks over fixed dev and test splits.","document_count":1,"document_ids":["claire_radar:https://github.com/zero-trust-agent-benchmark/zero-trust-agent-benchmark"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Zero Trust Agent Benchmark","organization_count":1,"organizations":["claire_radar"],"rank":3741,"released":"2026-09-26","source":"claire_radar","url":"https://github.com/zero-trust-agent-benchmark/zero-trust-agent-benchmark"},{"aliases":[],"benchmark_id":"claire-radar-2609-38269","caveat":"Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.38269"],"document_share":0.0002723311546840959,"domain":"Agents","name":"Zero2Repo","organization_count":1,"organizations":["claire_radar"],"rank":3742,"released":"2026-09-29","source":"claire_radar","url":"https://arxiv.org/abs/2609.38269"},{"aliases":[],"benchmark_id":"llm-stats-zerobench","caveat":"ZEROBench is a challenging vision benchmark designed to test models on zero-shot visual understanding tasks.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/zerobench?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ZEROBench","organization_count":1,"organizations":["llm_stats"],"rank":3743,"released":"2025-02-13","source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench?top_n=500"},{"aliases":[],"benchmark_id":"opencompass-1532-zerobench","caveat":"ZeroBench is a challenging visual reasoning benchmark for LMMs. It consists of a main set of 100 high-quality, manually curated questions covering numerous domains, reasoning types and image type. Questions have been designed and calibrated to be beyond the capabilities of current frontier models. ZeroBench 是针对多模态模型（LMMs）的具有挑战性的视觉推理基准。它由一组主要的 100 个高质量人工问题组成，涵盖多个领域、推理类型和图像类型。ZeroBench 中的问题经过设计和校准，已经超出了当前前沿模型的能力范围。","document_count":1,"document_ids":["opencompass_hub:https://hub.opencompass.org.cn/dataset-detail/ZeroBench"],"document_share":0.0002723311546840959,"domain":"多模态","name":"ZeroBench","organization_count":1,"organizations":["opencompass_hub"],"rank":3744,"released":"2025-02-13","source":"opencompass_hub","url":"https://hub.opencompass.org.cn/dataset-detail/ZeroBench"},{"aliases":[],"benchmark_id":"llm-stats-zerobench-sub","caveat":"ZEROBench-Sub is a subset of the ZEROBench benchmark.","document_count":1,"document_ids":["llm_stats:https://api.zeroeval.com/leaderboard/benchmarks/zerobench-sub?top_n=500"],"document_share":0.0002723311546840959,"domain":"multimodal","name":"ZEROBench-Sub","organization_count":1,"organizations":["llm_stats"],"rank":3745,"released":null,"source":"llm_stats","url":"https://api.zeroeval.com/leaderboard/benchmarks/zerobench-sub?top_n=500"},{"aliases":["Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasets"],"benchmark_id":"claire-radar-2606-07032","caveat":"A zero-shot composed-image-retrieval benchmark built from temporally controlled video frames, with evaluation based on ranking positive and negative targets.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.07032"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ZeroSight","organization_count":1,"organizations":["claire_radar"],"rank":3746,"released":"2026-06-05","source":"claire_radar","url":"https://arxiv.org/abs/2606.07032"},{"aliases":[],"benchmark_id":"claire-radar-2606-08841","caveat":"ZIPBench is a zero-shot personalization benchmark with 1.5K users, graph-mined personas, and 40K generated images for evaluating text-to-image personalization.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.08841"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ZIPBench","organization_count":1,"organizations":["claire_radar"],"rank":3747,"released":"2026-06-07","source":"claire_radar","url":"https://arxiv.org/abs/2606.08841"},{"aliases":[],"benchmark_id":"claire-radar-2609-12475","caveat":"A collection of compressed benchmark versions for text, multimodal, and agent tasks, with tools to evaluate models or create new compact benchmarks.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.12475"],"document_share":0.0002723311546840959,"domain":"General AI","name":"ZipBench","organization_count":1,"organizations":["claire_radar"],"rank":3748,"released":"2026-09-11","source":"claire_radar","url":"https://arxiv.org/abs/2609.12475"},{"aliases":["$\\pi$-SUB: A Physics-Informed Synthetic Underwater Benchmark Dataset for Underwater Image Enhancement"],"benchmark_id":"claire-radar-2608-10589","caveat":"Evaluates underwater image enhancement models using a synthetic paired dataset generated by a physics-informed framework with depth-dependent irradiance and Jerlov water types.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2608.10589"],"document_share":0.0002723311546840959,"domain":"General AI","name":"π-SUB","organization_count":1,"organizations":["claire_radar"],"rank":3749,"released":"2026-08-11","source":"claire_radar","url":"https://arxiv.org/abs/2608.10589"},{"aliases":["$\\tau$-Rec: A Verifiable Benchmark for Agentic Recommender Systems"],"benchmark_id":"claire-radar-2606-10156","caveat":"A benchmark for multi-turn agentic recommender systems that exposes dialogue constraints through reveal-tagged elicitation and verifies outcomes against structured catalog predicates.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2606.10156"],"document_share":0.0002723311546840959,"domain":"General AI","name":"τ-Rec","organization_count":1,"organizations":["claire_radar"],"rank":3750,"released":"2026-06-08","source":"claire_radar","url":"https://arxiv.org/abs/2606.10156"},{"aliases":[],"benchmark_id":"artificial-analysis-tau2-bench-telecom","caveat":"Agentic tool use","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/tau2-bench"],"document_share":0.0002723311546840959,"domain":"agentic","name":"τ²-Bench Telecom","organization_count":1,"organizations":["artificial_analysis"],"rank":3751,"released":"2025-06-12","source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/tau2-bench"},{"aliases":[],"benchmark_id":"artificial-analysis-tau3-banking","caveat":"Agentic tool use","document_count":1,"document_ids":["artificial_analysis:https://artificialanalysis.ai/evaluations/tau3-banking"],"document_share":0.0002723311546840959,"domain":"intelligence-index","name":"τ³-Banking","organization_count":1,"organizations":["artificial_analysis"],"rank":3752,"released":null,"source":"artificial_analysis","url":"https://artificialanalysis.ai/evaluations/tau3-banking"},{"aliases":["$\\Phi$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?"],"benchmark_id":"claire-radar-2609-10226","caveat":"Evaluates LLMs on tasks across the LLM infrastructure stack, from kernel completion to end-to-end system optimization using real-world code repositories.","document_count":1,"document_ids":["claire_radar:https://arxiv.org/abs/2609.10226"],"document_share":0.0002723311546840959,"domain":"Language & Knowledge","name":"Φ-Bench","organization_count":1,"organizations":["claire_radar"],"rank":3753,"released":"2026-09-09","source":"claire_radar","url":"https://arxiv.org/abs/2609.10226"},{"aliases":["AbductionRules","Abduction Rules"],"benchmark_id":"abductionrules","caveat":"Four synthetic abductive-reasoning datasets (Animal and Person domains, each Simple and Confounded): the model must supply the missing fact that explains an observation. The paper itself reports that fine-tuned models partly learned to exploit the structure of the data, so a high fine-tuned score need not mean general abduction, and a number without the subset named is not comparable across reports.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"reasoning","name":"AbductionRules","organization_count":0,"organizations":[],"rank":3754,"released":"2022-03-23","source":"model_reports","url":"https://aclanthology.org/2022.findings-acl.19/"},{"aliases":["Agent Cyber Range","ACR"],"benchmark_id":"agentcyberrange","caveat":"Verified success depends on the WEB or POST track, information level and retry policy. WEB vulnerabilities and POST ranges have different scoring denominators; Pass@3 average and Pass@3 maximum are distinct statistics. Read the track and configuration alongside any percentage.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"security","name":"AgentCyberRange","organization_count":0,"organizations":[],"rank":3755,"released":null,"source":"model_reports","url":"https://agentcyberrange.io/"},{"aliases":["Evaluation Agent","Evaluation-Agent"],"benchmark_id":"evaluation_agent","caveat":"A promptable evaluation framework rather than a fixed test set: it plans its own queries per model, so two runs need not ask the same questions. It does report numbers -- an average score per round and the per-prompt scores behind it -- but each is measured over the handful of prompts that run chose for that model, so they describe the run rather than a scale models can be ranked on.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"multimodal","name":"Evaluation Agent","organization_count":0,"organizations":[],"rank":3756,"released":"2024-12-10","source":"model_reports","url":"https://github.com/Vchitect/Evaluation-Agent"},{"aliases":["Gsci-Fact","GsciFact","Gscifact"],"benchmark_id":"gsci_fact","caveat":"About 32.2K general-science claims, balanced across Supported, Refuted and Not-Enough-Info, generated automatically from SciQ crowd-sourced exam questions by the Multi2Claim pipeline (EACL 2023). Annotators judged a quarter of claims answerable without the evidence, so part of a score can come from world knowledge rather than verification. Only 50 examples are public; the full set is on request.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"science","name":"Gsci-Fact","organization_count":0,"organizations":[],"rank":3757,"released":null,"source":"model_reports","url":"https://aclanthology.org/2023.eacl-main.194/"},{"aliases":["Large Language Models Are Not Strong Abstract Reasoners","LLMs Are Not Strong Abstract Reasoners"],"benchmark_id":"llm_abstract_reasoning","caveat":"A suite of abstract-reasoning tasks (IJCAI 2024), several of them text encodings of visual puzzles such as ARC, ACRE and RAVEN, plus PVR and tasks drawn from OpenAI Evals. A text-encoded ARC score is not comparable with a visual ARC score, and the source puzzles are public, so contamination cannot be ruled out.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"reasoning","name":"LLM Abstract Reasoning Benchmark","organization_count":0,"organizations":[],"rank":3758,"released":"2023-05-31","source":"model_reports","url":"https://arxiv.org/abs/2305.19555"},{"aliases":["LogiQA-plus","LogiQA-Plus","LogiQA plus"],"benchmark_id":"logiqa_plus","caveat":"A robustness variant of LogiQA with the same three subsets as ReClor-plus (shuffled options, correct choice replaced by \"none of the other options is correct\", and both). A score is only meaningful next to the original LogiQA score and with the subset named, and the source questions are public.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"reasoning","name":"LogiQA-plus","organization_count":0,"organizations":[],"rank":3759,"released":"2023-10-13","source":"model_reports","url":"https://arxiv.org/abs/2310.09430"},{"aliases":["LogiQAv2-plus","LogiQAv2-Plus","LogiQA 2.0-plus"],"benchmark_id":"logiqav2_plus","caveat":"A robustness variant of LogiQA 2.0 with the same three subsets as ReClor-plus. As with its siblings, the drop from the original LogiQA 2.0 score is the measurement, a single blended number hides which variation caused it, and the source questions are public.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"reasoning","name":"LogiQAv2-plus","organization_count":0,"organizations":[],"rank":3760,"released":"2023-10-13","source":"model_reports","url":"https://arxiv.org/abs/2310.09430"},{"aliases":["Med-Fact","MedFact","Medfact"],"benchmark_id":"med_fact","caveat":"150K medical claims (50K each Supported, Refuted, Not-Enough-Info) generated automatically from MedMCQA exam questions by the Multi2Claim pipeline (EACL 2023), not written by fact-checkers. Labels come from the generator, with sampled annotator agreement rather than full review. Only 50 examples are public; the full set is available from the authors on request.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"health","name":"Med-Fact","organization_count":0,"organizations":[],"rank":3761,"released":null,"source":"model_reports","url":"https://aclanthology.org/2023.eacl-main.194/"},{"aliases":["MER-FG","MER2026 Track 2","MER2026 MER-FG","MER 2026 fine-grained emotion recognition"],"benchmark_id":"mer2026_mer_fg","caveat":"Track 2 of the MER2026 challenge: open-vocabulary, fine-grained emotion recognition from video, audio and subtitles, ranked on a 1,000-sample test set with emotion-wheel-based metrics that group synonyms (happy and joyful count as one label). Because the label space is open, how a system normalizes its vocabulary onto the wheel moves the score independently of recognition. Reference labels were built by merging top MER2025 team predictions and then manually validated.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"multimodal","name":"MER2026 MER-FG","organization_count":0,"organizations":[],"rank":3762,"released":"2026-04-21","source":"model_reports","url":"https://arxiv.org/abs/2604.19417"},{"aliases":["NAFBench","NAF-Bench","NAF Bench"],"benchmark_id":"nafbench","caveat":"Solver-certified normal logic programs verbalized into natural language, each labelled under four negation semantics (SLDNF, well-founded, and credulous and skeptical stable-model reasoning) that can disagree on the same instance. A score without the semantics named is not comparable. Models are also sensitive to rule order. The leaderboard's hidden test split (Hugging Face qbao775/naf-bench, as described by the repository README at revision 83ede91) has 495 rows drawn from only 77 programs, so the effective sample is the program, not the row. The arXiv v1 appendix describes an earlier 385-prompt evaluation set over 77 programs, so a paper number and a leaderboard number are not on the same split.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"reasoning","name":"NAFBench","organization_count":0,"organizations":[],"rank":3763,"released":"2026-09-23","source":"model_reports","url":"https://arxiv.org/abs/2609.27517"},{"aliases":["PARARULE-Plus","PARARULE Plus","ParaRule-Plus"],"benchmark_id":"pararule_plus","caveat":"Synthetic multi-step deductive reasoning over natural-language facts and rules under a closed-world assumption, with about 100,000 examples at each reasoning depth from 2 to 5 (nearly 400,000 in total) over animal and people entities. An aggregate number hides the per-depth breakdown the dataset was built to expose, and a model fine-tuned on the templated training split is not comparable with one prompted zero-shot.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"reasoning","name":"PARARULE-Plus","organization_count":0,"organizations":[],"rank":3764,"released":"2022-07-28","source":"model_reports","url":"https://arxiv.org/abs/2207.14000"},{"aliases":["ReClor-plus","ReClor-Plus","ReClor plus"],"benchmark_id":"reclor_plus","caveat":"A robustness variant of ReClor in three subsets: shuffled options, the correct choice replaced by \"none of the other options is correct\", and both combined. Read it against the same model's score on the original ReClor, because the gap is the signal; one blended number hides which variation caused the drop, and the source questions are public.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"reasoning","name":"ReClor-plus","organization_count":0,"organizations":[],"rank":3765,"released":"2023-10-13","source":"model_reports","url":"https://arxiv.org/abs/2310.09430"},{"aliases":["RSI-Bench","RSI Bench","RSIBench","Recursive Self-Improvement Benchmark"],"benchmark_id":"rsi_bench","caveat":"Announced by Scale AI on 2026-08-07 and still in preview: it is soliciting task contributions rather than reporting results, so no model card in this registry reports it and its adoption count is zero. That zero is a real reading, not a gap in the crawl. Recorded now so the benchmark is addressable by name before any score exists, and so the first card to report it has an id to resolve against. Scores are per-task against a baseline under a fixed compute budget, graded on reliability, efficiency, generality and idea quality, so a single headline number should not be expected. Not to be confused with the llm-stats \"RSI Index\" record, which is a different instrument.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"agent","name":"RSI-Bench","organization_count":0,"organizations":[],"rank":3766,"released":"2026-08-07","source":"model_reports","url":"https://www.rsi-benchmark.com/"},{"aliases":["SciClaimEval","SciClaim Eval","NTCIR-19 SciClaimEval"],"benchmark_id":"sciclaimeval","caveat":"1,664 claims from 180 papers in machine learning, NLP and medicine, verified against tables and figures. Refuted cases keep the authentic claim and edit the evidence instead, so a refuted item can differ from a supported one only in a small change to a table or figure. The authors report that figure-based verification remains particularly challenging for all models, so a blended score depends on the table/figure mix, and table and figure numbers are not interchangeable.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"science","name":"SciClaimEval","organization_count":0,"organizations":[],"rank":3767,"released":"2026-02-07","source":"model_reports","url":"https://arxiv.org/abs/2602.07621"},{"aliases":["Terminal-Bench 4.0","Terminal Bench 4","TB4"],"benchmark_id":"terminal_bench_4","caveat":"Resolution rate on the v4.0.0 task collection. Agent framework, resource limits and task version are part of the result. GPU and multi-container tasks require appropriate infrastructure; scores from earlier versions are not interchangeable with version 4.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"agent","name":"Terminal-Bench 4","organization_count":0,"organizations":[],"rank":3768,"released":null,"source":"model_reports","url":"https://www.tbench.ai/"},{"aliases":["Uni-MMMU","UniMMMU"],"benchmark_id":"uni_mmmu","caveat":"Eight tasks that couple understanding and generation, each scored on a text channel and an image channel. One headline number hides which channel earned it, and a model that only understands or only generates is structurally capped on tasks that require both.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"multimodal","name":"Uni-MMMU","organization_count":0,"organizations":[],"rank":3769,"released":"2025-10-15","source":"model_reports","url":"https://github.com/Vchitect/Uni-MMMU"},{"aliases":["VBench"],"benchmark_id":"vbench","caveat":"Sixteen dimensions folded into a Quality Score and a Semantic Score and then into one Total Score. Every dimension is min-max normalized across the models on the leaderboard at the time, so a total states a model's standing within that pool rather than an absolute rate, and two totals read from different leaderboard revisions are not the same measurement.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"multimodal","name":"VBench","organization_count":0,"organizations":[],"rank":3770,"released":"2023-11-29","source":"model_reports","url":"https://github.com/Vchitect/VBench"},{"aliases":["VBench++","VBench-plus-plus"],"benchmark_id":"vbench_plus_plus","caveat":"Extends VBench to image-to-video generation and adds trustworthiness dimensions, so its totals are computed over a different dimension set. A VBench++ number is not a VBench number for the same model.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"multimodal","name":"VBench++","organization_count":0,"organizations":[],"rank":3771,"released":"2024-11-20","source":"model_reports","url":"https://arxiv.org/abs/2411.13503"},{"aliases":["VBench-2.0","VBench 2.0","VBench2.0"],"benchmark_id":"vbench_2_0","caveat":"Targets intrinsic faithfulness -- physics, commonsense and anatomy -- rather than the surface quality VBench 1.x measures, and is built to stay unsaturated where that suite tops out. Low scores here do not contradict high VBench scores; the two suites ask different questions.","document_count":0,"document_ids":[],"document_share":0.0,"domain":"multimodal","name":"VBench-2.0","organization_count":0,"organizations":[],"rank":3772,"released":"2025-03-27","source":"model_reports","url":"https://arxiv.org/abs/2503.21755"}],"measures":"Counts distinct source documents that record each benchmark. Model reports and registry pages use the same rule. Each document counts once per benchmark record. This measures documentation coverage, not benchmark quality.","organization_count":29,"organizations":{"Ai2":1,"Anthropic":4,"Apexin AI":1,"ApodexAI":1,"BlenderWorld authors":1,"ByteDance":1,"DeepSeek":6,"Google":4,"Meta":2,"Microsoft":1,"Mistral":3,"Moonshot AI":1,"NJU-LINK (Nanjing University)":1,"Navers Lab / Einsia":2,"Navers Lab / Einsia AI":1,"Navers Lab / Einsia.AI":3,"OpenAI":5,"OpenMOSS (Fudan University)":1,"Qwen":4,"Renmin University of China":1,"Stanford Neural Prosthetics Translational Laboratory":1,"Tencent":1,"UC Davis Neuroprosthetics Lab":1,"Z.ai":5,"artificial_analysis":23,"claire_radar":2447,"llm_stats":687,"opencompass_hub":461,"xAI":2},"schema_version":1},"library_taxonomy":{"categories":[{"description":"Competition, olympiad and research-level mathematics, including formal proofs.","group":"reasoning-knowledge","id":"math","name":"Math Reasoning"},{"description":"Broad academic and expert knowledge questions, from exam-style multiple choice to frontier expert questions.","group":"reasoning-knowledge","id":"knowledge-expert-qa","name":"Knowledge & Expert QA"},{"description":"Novel-problem, puzzle, logical, causal and planning reasoning that does not rely on domain knowledge.","group":"reasoning-knowledge","id":"abstract-reasoning","name":"Abstract Reasoning & Planning"},{"description":"Whether models state facts correctly, ground answers in sources and abstain when they do not know.","group":"reasoning-knowledge","id":"factuality","name":"Factuality & Hallucination"},{"description":"Predicting future events and time-series values, including judgmental forecasting and anomaly detection in sequences.","group":"reasoning-knowledge","id":"forecasting","name":"Forecasting & Time Series"},{"description":"Agents that resolve repository issues, work in terminals and complete multi-step software engineering tasks.","group":"coding","id":"agentic-coding","name":"Agentic Coding & SWE"},{"description":"Function- and program-level code writing, competitive programming and code understanding without a long agent loop.","group":"coding","id":"code-generation","name":"Code Generation"},{"description":"Agents doing machine-learning research and engineering: training, post-training, Kaggle-style tasks, paper replication and kernel or systems optimisation.","group":"coding","id":"ai-rnd","name":"AI R&D & ML Engineering"},{"description":"Choosing and calling tools, APIs and MCP servers correctly over single or multi-turn interactions, including customer-service style simulations.","group":"agents","id":"tool-use","name":"Tool Use & Function Calling"},{"description":"Operating desktops, phones and browsers through the screen: OS tasks, web navigation, mobile apps and GUI element grounding.","group":"agents","id":"computer-use","name":"Computer Use & GUI Agents"},{"description":"Finding, retrieving and synthesising information across the web or large corpora, including retrieval-augmented generation.","group":"agents","id":"deep-research","name":"Deep Research & Search"},{"description":"Economically valuable knowledge work: office documents, spreadsheets, enterprise workflows and occupational tasks across finance, consulting, law and similar jobs.","group":"agents","id":"professional-work","name":"Professional & Enterprise Work"},{"description":"Remembering across sessions, updating stored knowledge and adapting to a specific user over time.","group":"agents","id":"agent-memory","name":"Agent Memory & Personalization"},{"description":"Agents that acquire reusable skills, build their own harnesses or improve over repeated episodes.","group":"agents","id":"agent-skills-evolution","name":"Agent Skills & Self-Improvement"},{"description":"Collaboration, negotiation and competition among agents, and interactive game environments.","group":"agents","id":"multi-agent-games","name":"Multi-Agent & Games"},{"description":"Retrieving and reasoning over very long inputs such as books, codebases or long conversations.","group":"language","id":"long-context","name":"Long Context"},{"description":"Following explicit constraints, formats and multi-turn instructions.","group":"language","id":"instruction-following","name":"Instruction Following"},{"description":"Open-ended chat quality judged by humans or models, creative writing and emotional intelligence.","group":"language","id":"chat-writing","name":"Chat, Writing & Preference"},{"description":"Performance across languages and machine translation.","group":"language","id":"multilingual","name":"Multilingual & Translation"},{"description":"Answering and reasoning about images, charts, diagrams and visual math.","group":"multimodal","id":"visual-reasoning","name":"Image & Visual Reasoning"},{"description":"Reading text, tables, layouts and forms in scanned or digital documents.","group":"multimodal","id":"document-ocr","name":"Document Understanding & OCR"},{"description":"Understanding and reasoning over short and long videos, including egocentric and streaming video.","group":"multimodal","id":"video-understanding","name":"Video Understanding"},{"description":"Spatial relations, 3D scenes, counting and geometric perception from images, video or point clouds.","group":"multimodal","id":"spatial-3d","name":"Spatial & 3D Reasoning"},{"description":"Speech recognition and generation, audio understanding, music and real-time voice interaction.","group":"multimodal","id":"speech-audio","name":"Speech & Audio"},{"description":"Generating and editing images, video and 3D assets from text or other inputs.","group":"multimodal","id":"image-video-generation","name":"Image & Video Generation"},{"description":"Robot manipulation, vision-language-action policies, embodied planning, navigation and autonomous driving.","group":"embodied","id":"embodied-robotics","name":"Robotics & Embodied AI"},{"description":"Predicting how environments evolve, interactive world simulation and physical commonsense.","group":"embodied","id":"world-models","name":"World Models & Physical Understanding"},{"description":"Harmful-content refusal, jailbreak robustness, honesty, sycophancy, bias and dangerous-capability (CBRN) evaluations.","group":"safety","id":"safety-alignment","name":"Safety & Alignment"},{"description":"Attacks on and failures of agents: prompt injection, unsafe tool actions, permissions and privacy leakage.","group":"safety","id":"agent-security","name":"Agent Security & Prompt Injection"},{"description":"Offensive and defensive security tasks: capture-the-flag, vulnerability discovery and exploitation.","group":"safety","id":"cybersecurity","name":"Cybersecurity & CTF"},{"description":"Detecting AI-generated or manipulated text, code, images, audio and video, and attributing authorship.","group":"safety","id":"content-detection","name":"AI-Content Detection & Forensics"},{"description":"Scientific research tasks and knowledge in physics, chemistry, biology and earth science, including scientific agents and scientific code.","group":"domains","id":"science","name":"Science"},{"description":"Clinical, medical and health-related tasks for patients and practitioners.","group":"domains","id":"health-medicine","name":"Health & Medicine"},{"description":"Financial analysis and legal reasoning for professional use.","group":"domains","id":"finance-legal","name":"Finance & Law"},{"description":"Engineering tasks such as CAD modelling, chip and circuit design (EDA), power systems and industrial operations.","group":"domains","id":"engineering-design","name":"Engineering & Hardware Design"},{"description":"Aggregate scores that combine many benchmarks into one capability index, such as the Artificial Analysis Intelligence Index or Epoch Capabilities Index.","group":"meta","id":"composite-index","name":"Composite Indexes"}],"groups":[{"id":"reasoning-knowledge","name":"Reasoning & Knowledge"},{"id":"coding","name":"Coding & AI R&D"},{"id":"agents","name":"Agents"},{"id":"language","name":"Language & Interaction"},{"id":"multimodal","name":"Multimodal"},{"id":"embodied","name":"Embodied & Physical"},{"id":"safety","name":"Safety & Security"},{"id":"domains","name":"Application domains"},{"id":"meta","name":"Indexes & Suites"}],"source":{"commit":"522482b200274cf0b6d656df9fc4cd061f5fac23","library_index_sha256":"0d8340cc09406ca5d9708f56f7f4db9e83d91d42c71b41a4fea50ae6d1288c82","repository":"https://github.com/Claire1217/benchmark-radar"},"version":"taxonomy-v5"},"schema_version":1,"topic_definitions":[{"group":"capability","id":"coding","label":{"en":"Coding","zh":"编程"}},{"group":"capability","id":"computer-use","label":{"en":"Computer Use","zh":"电脑操作"}},{"group":"capability","id":"agents","label":{"en":"Tools & Agents","zh":"工具与智能体"}},{"group":"capability","id":"multimodal","label":{"en":"Multimodal","zh":"多模态"}},{"group":"capability","id":"reasoning-math","label":{"en":"Reasoning & Math","zh":"推理与数学"}},{"group":"capability","id":"long-context","label":{"en":"Long Context","zh":"长上下文"}},{"group":"capability","id":"language","label":{"en":"Language","zh":"语言"}},{"group":"capability","id":"safety","label":{"en":"Safety","zh":"安全与可靠性"}},{"group":"domain","id":"cybersecurity","label":{"en":"Cybersecurity","zh":"网络安全"}},{"group":"domain","id":"finance","label":{"en":"Finance","zh":"金融"}},{"group":"domain","id":"medical","label":{"en":"Medical","zh":"医疗"}},{"group":"domain","id":"science","label":{"en":"Science","zh":"科学"}}]}